Metadata-Version: 2.4
Name: gdelt-tui
Version: 0.1.0
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Rust
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Typing :: Typed
Requires-Dist: polars>=1.0
Requires-Dist: pytest>=8 ; extra == 'dev'
Requires-Dist: maturin>=1.7,<2 ; extra == 'dev'
Provides-Extra: dev
License-File: LICENSE
Summary: Download and convert the GDELT Global Database of Events from a terminal UI
Keywords: gdelt,events,parquet,tui,news,geopolitics
Author: ryomenhaider
Requires-Python: >=3.9
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://github.com/ryomenhaider/gdelt
Project-URL: Issues, https://github.com/ryomenhaider/gdelt/issues
Project-URL: Repository, https://github.com/ryomenhaider/gdelt

# gdelt-tui

Download the [GDELT Global Database of Events](https://www.gdeltproject.org/)
and store it as partitioned Parquet, from a terminal UI.

```sh
pip install gdelt-tui
gdelt-tui
```

The TUI asks which table you want, which date range, and where to save, then
shows per-file progress while it downloads and converts. `q` aborts after the
file in flight.

## The three tables

GDELT publishes the event table in three shapes. All three are supported.

| Key | Table | Range | Columns | Notes |
| --- | --- | --- | --- | --- |
| `events` | 1.0 events, daily | 2013-04-01 → today | 58 | ~7MB per day, no header |
| `history` | events, monthly/yearly | 1979 → 2013-03 | 57 | same as `events` minus `SOURCEURL` |
| `master` | MASTERREDUCED V2 | 1979 → 2013 | 17 | aggregated, 1.1GB, has a header |

## Non-interactive use

Every flag works without the TUI, which is what you want in a script or cron job.
Progress falls back to plain lines when stdout is not a terminal.

```sh
# last 7 days of events, the default window
gdelt-tui --dataset events

# an explicit range
gdelt-tui --dataset events --start 2024-01-01 --end 2024-01-31

# one month of history
gdelt-tui --dataset history --start 1979-01-01 --end 1979-01-31
```

| Flag | Meaning |
| --- | --- |
| `-d`, `--dataset` | `events`, `history` or `master` |
| `--start`, `--end` | `YYYY-MM-DD`; a file is included if its period overlaps |
| `-o`, `--output` | root directory, default `./gdelt-data` |
| `--batch-size` | rows per Parquet part, default 250000 |
| `--no-tui` | force plain output |
| `--reconvert` | re-convert files that are already downloaded |

## Output layout

```
gdelt-data/
  raw/                                  zips, kept so you can reconvert
    events/2024/20240115.export.CSV.zip
  parquet/
    year=2024/date=2024-01-15/part-00000.parquet
    year=1979/part-00000.parquet
```

Hive-style partitioning, so Polars, pandas and DuckDB can scan a range without
loading it all:

```python
import polars as pl

events = pl.scan_parquet("gdelt-data/parquet", hive_partitioning=True).filter(
    pl.col("date") >= pl.date(2024, 1, 1)
)
```

```sql
-- DuckDB
SELECT Actor1CountryCode, count(*)
FROM read_parquet('gdelt-data/parquet/**/*.parquet', hive_partitioning=true)
WHERE year = 2024 AND QuadClass = 1
GROUP BY 1 ORDER BY 2 DESC LIMIT 10;
```

## Library use

```python
from datetime import date
from pathlib import Path

from gdelt_tui import BY_KEY, convert, download, fetch_index, select

events = BY_KEY["events"]
raw, out = Path("raw"), Path("parquet")

for entry in select(fetch_index(), events, date(2024, 3, 1), date(2024, 3, 2)):
    archive = raw / entry.name
    download(entry.url, archive, expected=entry.md5)
    convert(archive, events, out, period=entry.period)
```

## Integrity

Every file carries an MD5 on the GDELT index page, so downloads are verified
before the `.part` file is renamed into place. A file that fails its checksum
is retried, then reported and skipped rather than aborting the run.

## A note on the compiled wheel

The TUI is a Rust extension ([ratatui](https://ratatui.rs) via
[PyO3](https://pyo3.rs)), which is what keeps the installed package to one
dependency, `polars`, with no Python TUI libraries.

The trade-off is that this is a platform wheel, not a pure-Python package.
`pip install` from a wheel is instant. Installing from the sdist — on musl
Linux, on a platform without a published wheel, or from a source checkout —
compiles the extension and therefore needs a
[Rust toolchain](https://rustup.rs) present. Building takes a couple of minutes.

## License

MIT

