Metadata-Version: 2.4
Name: internacia-builder
Version: 1.8.0
Summary: Build, validation, and enrichment tooling for internacia-db
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: typer==0.26.8
Requires-Dist: pyyaml==6.0.3
Requires-Dist: tqdm==4.68.3
Requires-Dist: duckdb==1.5.4
Requires-Dist: pandas==3.0.3
Requires-Dist: pyarrow==24.0.0
Requires-Dist: zstandard==0.25.0
Requires-Dist: requests==2.34.2
Requires-Dist: jsonschema==4.26.0
Dynamic: license-file

# Internacia Datasets

[![Validate datasets](https://github.com/commondataio/internacia-db/actions/workflows/validate.yml/badge.svg)](https://github.com/commondataio/internacia-db/actions/workflows/validate.yml)

Comprehensive reference datasets of countries, intergovernmental organizations, and country groups. Source YAML files in `data/countries/`, `data/intblocks/`, and `data/blocktypes/` are validated, enriched, and exported to multiple formats in `data/datasets/`. The project serves as a data source for the **Dateno** search engine.

## Features

- **Multi-format export**: JSONL, YAML, Parquet, and DuckDB (Zstandard compression, level 22)
- **Countries quality pipeline**: schema validation, completeness gates, entity status policy, and field-level provenance
- **Intblocks quality pipeline**: schema validation, blocktype taxonomy checks, duplicate detection, and completeness gates
- **Profile enrichment**: population, area, gini, timezones, and native names from World Bank, Wikidata, and IANA tzdata
- **Data-quality analyzer**: 50+ rules (referential integrity, temporal consistency, geographic plausibility, provenance depth, naming) reported under `dataquality/`; runs in CI and fails on CRITICAL/IMPORTANT findings
- **Build metadata**: `countries.manifest.json`, `intblocks.manifest.json`, and `blocktypes.manifest.json` with version, commit, row count, and schema hash — all sharing a single frozen build identity
- **Artifact consistency guard**: `check_generated_artifacts.py` verifies committed exports agree across formats and match YAML sources (runs in CI and release)
- **CI validation**: pull-request checks, tests, and lint via `.github/workflows/validate.yml`; weekly link validation; tagged releases with dataset assets
- **CLI tools**: Typer-based scripts with tqdm progress bars; console entry points `internacia-build`, `internacia-analyze-quality`, `internacia-validate-countries`, `internacia-validate-intblocks`

## AI agents and LLMs

- [AGENTS.md](AGENTS.md) — root routing hub (all platforms)
- [AGENTS.zh.md](AGENTS.zh.md) · [llms.zh.txt](llms.zh.txt) — 中文指南（Kimi K3、GLM-5.2、通义灵码）
- [llms.txt](llms.txt) — compact index (datasets, join keys, gotchas)
- [llms-full.txt](llms-full.txt) — extended index for crawlers
- [docs/agents/query.md](docs/agents/query.md) — query and join workflow
- [docs/agents/zh/query.md](docs/agents/zh/query.md) — 中文查询工作流
- [docs/agents/contribute.md](docs/agents/contribute.md) — YAML editing workflow
- [docs/query-examples.zh.md](docs/query-examples.zh.md) — 已验证中文 DuckDB 示例
- [docs/ai-consumers.md](docs/ai-consumers.md) — consumption contract, scope boundaries
- [docs/query-examples.md](docs/query-examples.md) — verified DuckDB and Pandas query cookbook
- [CLAUDE.md](CLAUDE.md) / [.github/copilot-instructions.md](.github/copilot-instructions.md) — Claude / Copilot shims
- [.kimi/AGENTS.md](.kimi/AGENTS.md) — Kimi Code
- [.lingma/rules/](.lingma/rules/) — 通义灵码 Project Rules
- [.cursor/skills/](.cursor/skills/) — thin Cursor wrappers pointing to `docs/agents/`

## Installation

Requires Python 3.11+ (CI runs 3.11). Dependencies are pinned for reproducible builds.

```bash
pip install -r requirements.txt        # runtime
pip install -r requirements-dev.txt    # development (adds pytest, ruff, pre-commit)
```

## Quick start

```bash
# Inspect data sources
python3 scripts/builder.py info

# Validate country YAML (no build)
python3 scripts/validate_countries.py
# or: internacia-validate-countries   (after pip install -e .)

# Validate intblock YAML (no build)
python3 scripts/validate_intblocks.py
# or: internacia-validate-intblocks

# Build all datasets
python3 scripts/builder.py build

# Build specific formats only
python3 scripts/builder.py build --formats parquet,duckdb
```

## Output files

Each build writes to `data/datasets/`:

| File | Description |
|------|-------------|
| `countries.jsonl.zst` | Countries (JSONL, zstd) |
| `countries.yaml.zst` | Countries (YAML, zstd) |
| `countries.parquet` | Countries (Parquet, zstd) |
| `countries.manifest.json` | Build metadata (version, commit, row count, schema hash, data license) |
| `countries.meta.json` | Version metadata sidecar for Parquet consumers |
| `intblocks.manifest.json` | Build metadata (version, commit, row count, schema hash, data license) |
| `intblocks.meta.json` | Version metadata sidecar for Parquet consumers |
| `intblocks.jsonl.zst` | International blocks (JSONL, zstd) |
| `intblocks.yaml.zst` | International blocks (YAML, zstd) |
| `intblocks.parquet` | International blocks (Parquet, zstd) |
| `intblocks_aliases.json` | Retired/renamed intblock id → current id map |
| `intblocks_aliases.parquet` | Alias map (Parquet, zstd) |
| `blocktypes.manifest.json` | Build metadata (version, commit, row count, schema hash, data license) |
| `blocktypes.yaml` | Block types (plain YAML copy of source, regenerated on build) |
| `blocktypes.jsonl.zst` | Block types (JSONL, zstd) |
| `blocktypes.yaml.zst` | Block types (YAML, zstd) |
| `blocktypes.parquet` | Block types (Parquet, zstd) |
| `blocktypes.meta.json` | Version metadata sidecar for Parquet consumers |
| `internacia.duckdb` | DuckDB database (`countries`, `intblocks`, `blocktypes`, and `_meta` tables) |

Current row counts: **256** countries, **1078** intblocks, **86** blocktypes.

## Validation and quality

The builder runs both `validate_countries.py` and `validate_intblocks.py` before export. Validation covers:

- JSON Schema conformance (`data/schemas/countries.schema.json`, `data/schemas/intblocks.schema.json`)
- ISO identifier formats and duplicate detection (country codes and intblock ids)
- Completeness thresholds (`data/schemas/countries_completeness.yaml`, `data/schemas/intblocks_completeness.yaml`)
- Entity status policy (`entity_type`, `code_status`)
- Blocktype taxonomy and `partof` reference checks for intblocks
- Intblock cross-references (country `includes` resolve to country sources; `includes[].status` values come from `data/schemas/includes_status.yaml`)
- Referential integrity (borders, `predecessor`/`successor`/`suborganizations`, headquarters countries, duplicate `wikidata_id`) plus temporal, geographic-plausibility, provenance depth, and naming rules — shared between the validators and the quality analyzer (`internacia_builder/validate/*_rules.py`)

```bash
# Full validation with JSON reports
python3 scripts/validate_countries.py --report completeness-report.json
python3 scripts/validate_intblocks.py --report intblocks-report.json

# Enrich profile fields from external sources
python3 scripts/enrich_countries.py
python3 scripts/enrich_countries.py backfill-provenance

# Enrich intblocks from Wikidata (wikidata_id, descriptions, multilingual names)
python3 scripts/enrich_intblocks.py --dry-run
python3 scripts/enrich_intblocks.py
python3 scripts/enrich_intblocks.py backfill-structural   # headquarters + founded

# Generate the data-quality report (dataquality/ — by rule, priority, country)
python3 scripts/builder.py analyze-quality

# Apply entity status annotations
python3 scripts/annotate_entity_status.py

# Compare manifests to main branch baseline
python3 scripts/diff_countries_baseline.py

# Verify committed exports match sources and each other; check doc links
python3 scripts/check_generated_artifacts.py
python3 scripts/check_markdown_links.py

# Run tests and lint
pytest tests/
ruff check internacia_builder/ scripts/ tests/
```

Country code policy (ISO vs user-assigned, filtering examples): [docs/country-code-policy.md](docs/country-code-policy.md)

## Consumer migration

Breaking and semantic changes in the latest countries schema (see [CHANGELOG.md](CHANGELOG.md)):

- **Population / area / gini**: structured as `{value, year, source, source_id}` — use `.value` for the numeric field. `year` is **null** when the source year is unknown (never `0`).
- **Borders**: land neighbors as ISO **alpha-3** codes (e.g. `CAN`, `MEX`), not alpha-2.
- **Entity filter**: `code_status == 'official_iso3166_1'` returns **249** current ISO-style records.
- **Build metadata**: compare `countries.manifest.json` `schema_hash` when upgrading downstream pipelines.

**Pandas example** (structured population):

```python
import pandas as pd

df = pd.read_parquet("data/datasets/countries.parquet")
pop = df["population"].struct.field("value")
```

**DuckDB example** (nested intblock multilingual names):

```python
import duckdb

con = duckdb.connect("data/datasets/internacia.duckdb")
con.execute("""
    SELECT id, name, t.name AS english_name
    FROM intblocks, UNNEST(other_names) AS t
    WHERE t.id = 'en'
    LIMIT 5
""").fetchall()
```

## Versioning and identifier stability

Datasets are **self-describing**. The DuckDB file carries a `_meta` table and each Parquet file has a
`<dataset>.meta.json` sidecar, both mirroring the manifest fields (`version`, `build_date`,
`git_commit`, `row_count`, `schema_hash`, `data_license`):

```python
import duckdb

con = duckdb.connect("data/datasets/internacia.duckdb")
con.execute("SELECT dataset, version, schema_hash FROM _meta").fetchall()
```

**Identifier stability.** Country `code` and intblock `id` are stable join keys. When an intblock id is
renamed, merged, or its acronym is reassigned to a different entity, the old id is recorded in
`intblocks_aliases.json` (and `.parquet`) so downstream joins can remap:

```python
import json

aliases = {a["alias"]: a["target"] for a in json.load(open("data/datasets/intblocks_aliases.json"))}
current_id = aliases.get("ASF", "ASF")  # -> "FSA"
```

A `reason` of `disambiguated` means the old id still exists but now refers to a **different** entity
(e.g. `ASF` is now the African Standby Force; the African Solidarity Fund moved to `FSA`).

## Countries schema

256 country and territory records. Key fields:

| Field | Type | Description |
|-------|------|-------------|
| `code` | String | ISO 3166-1 alpha-2 code (e.g. `US`) |
| `entity_type` | String | `sovereign_state`, `dependent_territory`, `historical_entity`, etc. |
| `code_status` | String | `official_iso3166_1`, `user_assigned`, `obsolete` |
| `recognition_status` | Struct | Optional recognition/dispute metadata |
| `name` | String | Common name |
| `iso3code` | String | ISO 3166-1 alpha-3 code |
| `capital_city` | Struct | `{name, lng, lat}` |
| `region` | Struct | World Bank region `{id, value}` |
| `adminregion` | Struct | World Bank admin region `{id, value}` |
| `incomeLevel` | Struct | World Bank income level `{id, value}` |
| `lendingType` | Struct | World Bank lending type `{id, value}` |
| `numeric_code` | String | ISO 3166-1 numeric code |
| `wikidata_id` | String | Wikidata item ID |
| `official_name` | String | Official full name |
| `languages` | List[Struct] | `{code, name, official}` |
| `currencies` | List[Struct] | `{code, name, symbol}` |
| `un_member` | Boolean | UN member |
| `independent` | Boolean | Independent state |
| `subregion` | String | UN subregion |
| `continents` | List[String] | Continents |
| `borders` | List[String] | Land borders as ISO **alpha-3** codes |
| `landlocked` | Boolean | Landlocked |
| `tld` | String | Top-level domain |
| `calling_codes` | List[String] | Telephone codes |
| `flag_emoji` | String | Flag emoji |
| `car_side` | String | Driving side |
| `start_of_week` | String | Start of week |
| `demonyms` | Struct | `{female, male}` |
| `m49_code` | String | UN M49 code |
| `population` | Struct | `{value, year, source, source_id}` |
| `area` | Struct | Land area sq km `{value, year, source, source_id}` |
| `gini` | Struct | Gini index `{value, year, source, source_id}` |
| `timezones` | List[String] | IANA timezone identifiers |
| `timezone_status` | String | `not_applicable` when no zones apply |
| `native_names` | Map | Lang code → `{official, common}` |
| `other_names` | List[Struct] | Translations `{id, name}` |
| `common_names` | List[String] | Aliases and common names |
| `provenance` | List[Struct] | Field sourcing `{field, source, retrieved_at, url, license}` |

Non-standard codes retained with explicit status: `AN` (obsolete), `JG` (user-assigned grouping), `KV` (user-assigned, disputed).

## International blocks schema

| Field | Type | Description |
|-------|------|-------------|
| `id` | String | Unique identifier |
| `blocktype` | List[String] | Block types |
| `status` | String | Current status |
| `name` | String | Name |
| `languages` | List[String] | Official languages |
| `links` | List[Struct] | `{url, type}` |
| `other_names` | List[Struct] | `{id, name}` translations |
| `founded` | String | Foundation date |
| `geographic_scope` | String | Scope |
| `regions` | List[String] | Regions covered |
| `includes` | List[Struct] | Members `{id, name, type, status, joined, role, note}` — **`id` is authoritative**; `name` is a source label |
| `membership_count` | Integer | Member count |
| `wikidata_id` | String | Wikidata item ID |
| `legal_status` | String | Legal status |
| `description` | String | Description |
| `tags` | List[String] | Tags |
| `topics` | List[Struct] | `{key, name}` |
| `headquarters` | Struct | `{city, country, coordinates}` |
| `acronyms` | List[Struct] | `{lang, value}` |
| `partof` | List[String] | Parent organizations |
| `dissolved` | String | Dissolution date |
| `predecessor` | String | Predecessor |
| `successor` | String | Successor |

The table above lists exported columns. Source YAML may carry additional curation fields that are
validated but not exported (`membership_applicability`, `founding_members`, `suborganizations`,
`secretariat`, `last_verified`, `previous_names`, `official_documents`, `social_media`,
`recognition_status`); see `data/schemas/intblocks.schema.json` for the full source contract.
Valid `includes[].status` values are cataloged in `data/schemas/includes_status.yaml`.

## Data sources

**YAML sources**

- `data/countries/*.yaml` — 256 country/territory records
- `data/intblocks/<category>/*.yaml` — 1078 international block records across 63 domain categories (`intorg`, `aviation`, `agriculture`, `health`, `climate`, etc.)

**External enrichment**

- [World Bank](https://data.worldbank.org/) — population, area, gini, income classifications
- [Wikidata](https://www.wikidata.org/) — entity linking, native names, fallbacks
- [IANA tzdata](https://data.iana.org/time-zones/) — timezone mapping (`scripts/data/zone1970.tab`)

## Scripts

| Script | Purpose |
|--------|---------|
| `internacia_builder/` | Installable package (`pip install -e .`): build/export, quality analyzer, validation rule modules, shared paths/HTTP helpers |
| `scripts/builder.py` | Shim → `internacia_builder.build` / `internacia_builder.quality` (`build`, `info`, `analyze-quality`) |
| `scripts/validate_countries.py` | Shim → `internacia_builder.validate.countries` |
| `scripts/validate_intblocks.py` | Shim → `internacia_builder.validate.intblocks` |
| `scripts/validate_links.py` | Intblock URL and Wikidata validation (run weekly in CI) |
| `scripts/enrich_countries.py` | Enrich country profiles; `backfill-provenance` subcommand |
| `scripts/enrich_intblocks.py` | Enrich intblocks from Wikidata (wikidata_id, descriptions, multilingual names); `backfill-structural` fills headquarters and founded dates |
| `scripts/annotate_entity_status.py` | Set `entity_type` and `code_status` |
| `scripts/diff_countries_baseline.py` | Manifest diff vs git baseline (countries, intblocks, blocktypes) |
| `scripts/check_generated_artifacts.py` | Cross-format primary-key parity, source/export parity, build-identity guard |
| `scripts/check_markdown_links.py` | Internal Markdown link checker |

One-off migration scripts live in `dev/scripts/` and are not part of the maintained pipeline.

## Releases

Tagged releases (`vX.Y.Z`) automatically rebuild all formats and attach them as GitHub Release assets (`.github/workflows/release.yml`). Consumers can either clone the repository (datasets are committed under `data/datasets/`) or download versioned assets from the Releases page.

## Notes

- All text files use UTF-8 encoding; generated outputs overwrite existing files.
- Decompress zstd files: `zstd -d data/datasets/countries.jsonl.zst`
- Gap analysis research: `dev/research/countries_gaps_manus_20260528.md`
- `data/_legacy/` contains pre-1.0 Airtable JSON exports kept for reference only; nothing in the pipeline consumes them.

## License

- **Code** (everything under `scripts/`, `tests/`, and build tooling) — MIT, see [LICENSE](LICENSE).
- **Data** (curated sources under `data/` and generated artifacts in `data/datasets/`) —
  Creative Commons Attribution 4.0 (CC BY 4.0), see [DATA_LICENSE](DATA_LICENSE).

Upstream sources (World Bank, Wikidata, IANA tzdata) and citation guidance are documented in
[ATTRIBUTION.md](ATTRIBUTION.md). The data license SPDX identifier is recorded in each build manifest
and in the `_meta`/`*.meta.json` metadata.

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for the YAML authoring guide, validation workflow, and PR checklist.

## Related projects

- [internacia-api](../internacia-api) — REST API
- [internacia-python](../internacia-python) — Python SDK

## Roadmap

- [x] Python SDK — [internacia-python](../internacia-python)
- [x] REST API — [internacia-api](../internacia-api)
