Metadata-Version: 2.4
Name: dofjson
Version: 0.3.0
Summary: Prototype client for SIDOF's undocumented JSON open data service for Mexico's official gazette (DOF).
License: Apache-2.0
Project-URL: Homepage, https://github.com/INGEOTEC/LegalIA/tree/main/packages/dofjson
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: requests>=2.31
Requires-Dist: pypdf>=3.0
Provides-Extra: test
Requires-Dist: pytest>=7.0; extra == "test"

# dofjson

Prototype client for the JSON open data service exposed by
[sidof.segob.gob.mx](https://sidof.segob.gob.mx/datos_abiertos), the
Secretaría de Gobernación's system for Mexico's official gazette (DOF,
Diario Oficial de la Federación).

The service's public docs only show sample responses, but its real,
unauthenticated endpoints were found under `https://sidof.segob.gob.mx/dof/sidof/`:

| Endpoint | Description |
|---|---|
| `GET /diarios/porFecha/DD-MM-YYYY` | Edition metadata for a date (Matutina/Vespertina/Extraordinaria) |
| `GET /diarios/{YYYY}` | A whole year's `FechasSinPublicacion` — the dates it claims had no gazette |
| `GET /notas/DD-MM-YYYY` | Notes/documents published on a date |
| `GET /notas/nota/{codNota}` | Full detail of a single note, including its HTML content |
| `GET /indicadores/DD-MM-YYYY` | Economic indicators (exchange rate, TIIE, UDIS) |

Note that this service reports a missing day as `200 OK` with empty note
lists, not as an error, and that some of the dates it lists as unpublished
were in fact published — see
[the days SIDOF loses](#the-days-sidof-loses-and-where-they-are-recovered-from---respaldo).

This is an experimental package for evaluating whether this service is a
viable alternative (or complement) to `dof2md`'s PDF download + Markdown
conversion pipeline — notes already come with structured HTML content,
which may be easier to work with than OCR'd PDFs.

On top of the raw endpoints, the client offers note-scoped downloads that
resolve a note's page span (`infer_paginas`) and fetch it in whichever form
you want:

- `download_nota_imagenes(codNota)` — the note's scanned page image(s).
- `download_nota_pdf(codNota)` — the note as its own PDF: the whole edition
  PDF (there is no per-note PDF endpoint) sliced to just the note's pages,
  using `pypdf`.

## Usage

```bash
pip install -e "packages/dofjson[test]"
dofjson 2026-07-16 --endpoint notas --outdir output
```

## Building a local archive of daily indexes (`--archivo`)

`dofjson --archivo` downloads the **daily notes index** incrementally, day by
day, over a whole date range (by default from January 2, 1917 to today). For
each date it does exactly what `dofjson YYYY-MM-DD --endpoint notas` does —
`get_notas(date)` filtered with `quita_notas_sin_titulo` — and saves one JSON
per day. It does **not** download each note's content or scanned images: only
the index.

```bash
dofjson --archivo                                      # 1917-01-02 -> today
dofjson --archivo --desde 1980-01-01 --hasta 1980-12-31
dofjson --archivo --pausa 1.0                          # slower (kinder to the server)
```

Output goes to `notas-archivo/` (configurable with `--outdir`), a **local,
never-committed** directory (it is in the repo's `.gitignore`), with the same
per-day filenames the plain command produces:

```
notas-archivo/
  .completados                 # registry of finished days (for resuming)
  2026/
    15072026-notas.json        # index for 2026-07-15 (get_notas, filtered)
    16072026-notas.json
  1980/
    02011980-notas.json
```

The mode is resumable and idempotent: the `.completados` registry records the
finished days, so each run only fetches what is missing. Days that fail with
network errors are *not* marked and get retried on the next run; days with no
edition (holidays, weekends) *are* marked so they are not retried forever.
"Today" is never marked, so late additions are picked up by a later run. You
can interrupt with Ctrl-C and resume at any time.

> The full range is ~40,000 days: a long download, meant to be run in parts.
> Start with a bounded range via `--desde/--hasta` if you only need an era.

### The days SIDOF loses, and where they are recovered from (`--respaldo`)

SIDOF does not report a day it is missing as an error. It answers **200 OK
with every note list empty** — which is also how it reports a Sunday — and
lists the date under `FechasSinPublicacion` in `GET /diarios/{year}`. Most of
those dates are genuine: weekends and holidays. Some are not.

On 8 March 1999 the DOF published the decree amending articles 16, 19, 22 and
123 of the Constitution. SIDOF has no trace of it: the day is empty, the
note's `codNota` returns `{"Nota": []}`, and its `codDiario` 404s. Sampling
four years (1999, 2006, 2010, 2020) turned up **eight** such days — dates
SIDOF calls unpublished that were published.

`www.dof.gob.mx`, the DOF's own website, is a separate system on a separate
database, and it has them. So an empty answer is no longer taken at face
value: on a weekday, the day is put to the website before being written off.

```bash
dofjson --archivo                        # habiles (default): re-check Mon-Fri
dofjson --archivo --respaldo todos       # also weekends (~10,000 more requests)
dofjson --archivo --respaldo nunca       # trust SIDOF alone
```

```
[1999-03-08] SIDOF no la tiene; recuperada de dof.gob.mx
```

The same applies to a single date, so a lost day is reachable directly:

```bash
dofjson 1999-03-08 --endpoint notas      # -> "fuente": "dof.gob.mx", 22 notas
```

**The text of those notes is recoverable too**, not just their titles.
`dofweb.get_nota(codNota)` reads a note's page on the website and returns it in
the shape `client.get_nota()` uses, with the note's HTML in `cadenaContenido` —
the same string SIDOF would have served, so `nota2md` converts it to Markdown
by the ordinary HTML path:

```python
from dofjson import dofweb

dofweb.get_nota(4997808)["Nota"]["cadenaContenido"]   # DOF 03-03-1999
```

On a note both sources have, the recovered HTML differs from SIDOF's only in
escaping its accents as entities, and the Markdown built from either is
identical. The page carries no `codDiario`, `codEdicion` or `pagina`, so the
image and PDF paths stay SIDOF-only; a note the site has no HTML for comes back
with `existeHtml` `"N"`, and an unknown `codNota` with `"Nota": []`, as SIDOF
answers.

**Which source a day came from is recorded, never inferred.** Every saved day
carries a `fuente` key (`"sidof"` or `"dof.gob.mx"`), and the registry stores
it next to the date, so provenance can be audited after the fact:

```
1999-03-06	sin-edicion
1999-03-08	dof.gob.mx
1999-03-09	sidof
```

In the compact `--titulos` dataset the marker rides along on the notes it
applies to: a note carries `fuente` only when its day did *not* come from
SIDOF, since repeating `"sidof"` on all ~1.2 million rows would cost more
than it says.

#### What the fallback carries, and what it does not

The website's daily index lists the substantive gazette — `PE`, `PJ`, `PL`,
`OA`, `OD` — and leaves out the three bulk-announcement groups, which are
reachable on the site only through its POST search form:

| | |
|---|---|
| `CV` | convocatorias for public-sector procurement |
| `VG` | convocatorias for civil-service vacancies |
| `AV` | avisos judiciales y generales |

On days both sources have, the recovered set of `codNota` matches SIDOF's
**exactly** once those three are excluded (checked on days sampled from 1999
through 2026). A recovered day is therefore complete with respect to what the
gazette *enacted* and short of what it *announced*, and says so in
`notasIncompletas` rather than passing for a whole day.

The website's per-note index **starts in January 1999**; before that it holds
only scanned images, so an older day returns an edition with no index. Those
come back in `edicionesSinIndice` as `{"codEdicion", "codDiario"}`: no titles
to list, but proof the gazette was published, which is what keeps the day off
the empty pile. Every day confirmed lost from SIDOF so far is 1999 or later,
inside the range where titles can actually be recovered.

> **On a page served for the wrong date.** The index prints the date it is
> actually serving, and it has been seen answering with a *different* day's
> page. Since the parser stamps each note with the date that was **asked
> for**, taking such a page at face value would file real notes under the
> wrong day. Every page carrying content is therefore checked against what it
> claims to be, and a mismatch raises `dofweb.PaginaDeOtroDia`. `--archivo`
> treats that like a network error and leaves the day to retry — believing it
> would corrupt the day, and calling the day empty would bury it for good.
> Editions the gazette never ran carry no date and no content, which is not a
> mix-up and is not treated as one.

> **On TLS.** `www.dof.gob.mx` serves its leaf certificate without the
> intermediate that signs it, so verification fails with "unable to get local
> issuer certificate" on any client that does not chase the issuer itself. The
> missing GoDaddy intermediate — and its root — ship in
> `dofjson/certs/dof-gob-mx-chain.pem`. The system trust store is tried first
> and the bundled chain only on failure. Certificate verification is never
> disabled.

## Building a compact titulo dataset from the release (`--titulos`)

`dofjson --titulos` builds a small `codNota` + `titulo` + `fecha` dataset out
of every note ever published, sourced from the [`notas-archivo` GitHub
release](https://github.com/INGEOTEC/LegalIA/releases/tag/notas-archivo)
(one `notas-YYYY.tgz` per year, 1917 to last year, plus one
`notas-YYYY-MM.tgz` per month of the current year). Each asset is downloaded
straight into memory, its daily JSON indexes are read without ever writing
them to disk, and only `codNota`/`titulo`/`fecha` are kept from every note
(`titulo` is Spanish for "title", `fecha` for "date") — `codNota` to fetch
that note's full content later, `titulo` for exploratory analysis of the
titles themselves, `fecha` to place each title in time. The
result is a single gzip-compressed JSONL file (~1.2 million notes fit in a
few tens of MB): small enough to move to a Colab GPU runtime for
experiments.

```bash
dofjson --titulos                    # -> titulos/titulos.jsonl.gz
dofjson --titulos --outdir /content  # e.g. from a Colab notebook
```

```python
import gzip, json
with gzip.open("titulos/titulos.jsonl.gz", "rt", encoding="utf-8") as f:
    notas = [json.loads(line) for line in f]
# notas[0] == {"codNota": 4434476, "titulo": "CIRCULAR nº. 164, ...", "fecha": "23-03-1917"}
```

Or use the function directly:

```python
from pathlib import Path
from dofjson.titulos import download_titulos

download_titulos(Path("titulos.jsonl.gz"))
```

## Development

```bash
pytest packages/dofjson
```
