Metadata-Version: 2.4
Name: dhis2-fetch
Version: 0.1.0
Summary: Read-only Python client for the DHIS2 Web API, returning pandas DataFrames
Author-email: Valérian Turbé <valerian.turbe@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/Valerian8/dhis2-fetch
Project-URL: Repository, https://github.com/Valerian8/dhis2-fetch
Project-URL: Issues, https://github.com/Valerian8/dhis2-fetch/issues
Keywords: dhis2,health,hmis,pandas,api,data
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Healthcare Industry
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests
Requires-Dist: pandas
Requires-Dist: tqdm
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: responses>=0.23; extra == "dev"
Dynamic: license-file

# dhis2-fetch

A **read-only** Python client for the DHIS2 Web API that returns tidy `pandas` DataFrames, built for analysts working in notebooks and scripts.

It authenticates with **your own DHIS2 account** and can only access data that account is already permitted to see in the DHIS2 web interface. It issues only `GET` requests (there is a single HTTP call site in the entire codebase) and cannot create, modify, or delete items on a DHIS2 instance.

To help users struggling with intermittent internet connection and/or unstable DHIS2 instances, long downloads are resumable: progress is written to a local SQLite file as it goes, so an interrupted job continues where it stopped instead of starting over.

**Who this is for:** Bulk analysis of DHIS2 data is easiest against the database directly, but most analysts have no SQL access to it. The database sits behind the instance and making SQL queries is usually restricted to admins. This package uses the Web API with the account you already have and is designed to be practical for data analysis: it handles the batching, paging, retries and reshaping, and hands back a clean DataFrame you can work with, without needing access to database credentials.

---

## Install

```bash
pip install dhis2-fetch
```

Requires Python 3.9+, and installs `requests`, `pandas` and `tqdm`.

## Quick start

```python
import dhis2_fetch as dfetch

dfetch.connect('https://your-dhis2-instance.org')      # prompts for username and password

df = dfetch.download_data(
    uids=['Uvn6LCg7dVU'],
    start_period='202401',
    end_period='202412',
    period_unit='Month',
    download_level=4,
)
```

`df` is a wide DataFrame: one row per period × org unit, one column per data item, with the full org-unit
hierarchy and org-unit attributes already joined on.

`connect()` prompts for credentials so they never sit in your script. You can pass them explicitly
(`dfetch.connect(url, username, password)`) for unattended runs, and calling `connect()` with no arguments
prompts for the URL too. Calling anything before connecting raises `NotConnectedError`.

On success it confirms which instance and account you are on.

```
Connected to https://your-dhis2-instance.org as jdoe
```

To check later which connection is in use:

```python
dfetch.active().url          # 'https://your-dhis2-instance.org'
dfetch.active().username     # 'jdoe'
```

`active()` raises `NotConnectedError` if you have not connected yet. There is one connection per Python
process: calling `connect()` again replaces it rather than adding a second. `connect()` also returns the
connection, so `client = dfetch.connect(...)` is equivalent, and if you need two instances live at
once, build `Dhis2Client` objects directly instead.

## Useful functions

| Function | What it gives you |
|---|---|
| `connect(url, username=None, pwd=None)` | Authenticates once; everything after uses that session |
| `active()` | The connection `connect()` made — check which instance and account you are on |
| `fetch_data_items(verbose=False)` | Every dataElement / indicator / programIndicator / dataSet on the instance — this is how you find UIDs before downloading data |
| `fetch_orgunit_levels()` | `{1: 'National', 2: 'Regional', ...}` — tells you what `download_level` to ask for |
| `download_data(...)` | The main event: UIDs + period range + level → a wide DataFrame |
| `fetch_shapefiles(boundary_ou_uid=None, levels=None)` | `{level_name: GeoJSON FeatureCollection}` for mapping |
| `save_shapefiles(shapes, output_dir='./outputs')` | Writes those out as one `.geojson` per level; returns the paths |
| `fetch_visualizations()` | List all saved visualizations on the instance |
| `validate_visualization_uid(uid, viz_list=None)` | Resolves a visualization UID to `(name, object_type)` |
| `download_visualization(uid, object_type)` | The data behind a saved visualization, as a DataFrame |
| `fetch_programs()` | Tracker programs, their stages, and each stage's data elements (one row per data element) |
| `download_event_data(...)` | A program's events as a wide DataFrame, one row per event |
| `download_events(...)` | The lower-level primitive behind it: raw events into the job database |

Lower-level building blocks (`download_data_analytics`, `select_dataset_elements`, `consolidate_data`,
`pivot_data`, `generate_periods`, …) are available in their modules in case you need to drive a step yourself: `dhis2_fetch.analytics`,
`.orgunits`, `.periods`, `.visualizations`, `.events`.

## Events

`download_event_data` is to events what `download_data` is to aggregate data — one call, one DataFrame:

```python
programs = dfetch.fetch_programs()        # find the program and stage UIDs

df = dfetch.download_event_data(
    program_uid='lxAQ7Zs9VYR',
    start_date='2026-01-01',
    end_date='2026-12-31',
    download_level=4,                  # the level events are captured at
    stage_uid=None,                    # optional: restrict to one stage
)
```

One row per event, with the org unit hierarchy and attribute columns joined on, the event date split into
`Year`/`Month`, and each data element as a named column. Resumable in the same way, with the same
`df.attrs` (`events_available`, `db_path`, `job_name`, `created_date`).

Two things differ from aggregate downloads:

- **`download_level` is the level events were captured at**, usually the facility level. Events are read
  from those exact org units, not from their descendants, so asking at district level returns only events
  registered against the districts themselves.
- **It makes one request per org unit**, so a whole country's facilities means thousands of requests. Pass
  `filter_ou_uids` to scope it to the units you need; you will get a warning above 200.

## Finding the UIDs you want

```python
items = dfetch.fetch_data_items()

malaria = items[items['dataname'].str.contains('paludisme', case=False)]
uids = malaria['dataid'].tolist()
```

`items` has columns `datatype` (`DE`, `I`, `PI`, `DS`), `dataid` and `dataname`. Pass `verbose=True` to
also get `valueType`/`aggregationType` for data elements and `numerator`/`denominator` for indicators.
Feed the `dataid` values straight into `download_data(uids=...)`.

## Choosing the org unit level

```python
dfetch.fetch_orgunit_levels()
# {1: 'National', 2: 'Regional', 3: 'Province', 4: 'District', 5: 'Commune', 6: 'Facility'}
```

`download_level=4` downloads at district level. Data stored below that is aggregated up by DHIS2; data
stored only at facility level will come back empty if you ask for a level it isn't reported at, which is
the most common reason for an unexpectedly empty result.

To restrict to part of the hierarchy, pass `filter_ou_uids=['uid1', 'uid2']` — these may be at the
download level itself (downloads exactly those units) or above it (downloads everything beneath them).
That second form is how you pull a sub-hierarchy: passing hospital UIDs with `download_level` set to the
level below returns every department of those hospitals.

## Org unit attribute columns

Every result also carries the org unit's **group set** memberships as columns, one per group set defined on the instance, describing the units at `download_level`. Pass `ou_attributes=False` to omit them.

These are the columns that tell you *what kind of thing* each row is, which matters because a level often mixes different kinds of unit. For example: on one instance, level N could hold 50 health districts **and** 700+ hospital departments, because hospitals are registered a level up, beside provinces, rather than inside districts. The group sets separate them cleanly:

| Column | Health districts | Hospital departments |
|---|---|---|
| `District_attribute` | `HD` | blank |
| `Type` | blank | `Hospital department` |

```python
df = dfetch.download_data(uids=[...], download_level=4, ...)

geographic = df[df['District_attribute'] == 'HD']        # the 50 districts
hospital   = df[df['Type'] == 'Hospital department']          # the 700+ hospital departments
```

Two things worth knowing before you filter:

- **The two sets are disjoint and both real.** Check whether they sum exactly to the national total,
  so filtering to one would genuinely drops the other's data rather than removing duplication. Hospital
  departments can be a small share of a routine indicator but a large share of specialist activity.
- **Mixing is not universal.** On the same instance, levels 5 and 6 are entirely geographic — the
  hospital branch has no children — so nothing needs filtering there. Check your own instance before
  assuming either way.

Group set names and values are defined per instance, so yours will differ; `District` and `Type` above are
that instance's own. The column is suffixed `_attribute` only when a group set shares its name with an org
unit level, as `District` does here.

## Periods

`period_unit` must be one of:

| Unit | Example | Unit | Example |
|---|---|---|---|
| `Year` | `2024` | `Weekly` | `2024W1` |
| `Month` | `202401` | `BiWeekly` | `2024BiW1` |
| `Quarter` | `2024Q1` | `Daily` | `20240101` |
| `BiMonth` | `2024B1` | `FinancialApril` | `2024April` |
| `SixMonth` | `2024S1` | `FinancialJuly` | `2024July` |
| `SixMonthApril` | `2024AprilS1` | `FinancialOct` / `FinancialNov` | `2024Oct` |

`start_period` and `end_period` use the same format, and every period in between is generated for you.
Periods with no data are still present as rows, with `NA` values.

## Datasets and reporting metrics

If any UID is a **dataSet**, you are asked what to pull from it — its data elements, its reporting
metrics, or both. There are five reporting metrics: `REPORTING_RATE`, `REPORTING_RATE_ON_TIME`,
`ACTUAL_REPORTS`, `ACTUAL_REPORTS_ON_TIME` and `EXPECTED_REPORTS`. To skip the prompt and run
unattended, answer in advance:

```python
df = dfetch.download_data(
    uids=['Uvn6LCg7dVU', 'vc6nF5yZsPR'],
    start_period='202401', end_period='202412', period_unit='Month',
    download_level=4,
    dataset_choices={'vc6nF5yZsPR': {'elements': 'all', 'metrics': True}},
)
```

`elements` accepts `'all'`, an explicit list of data element UIDs, or `None` for metrics only. Reporting
metric columns are placed after the raw data columns.

## Resuming an interrupted download

Each job writes to `./outputs/<job_name>.db`. Re-running the same call picks up where it left off and
tells you how many requests were already done. You will be asked to confirm; pass `force=True` to skip
that prompt.

If the job's defining parameters changed (instance URL, org units, period unit, disaggregation), it
stops rather than mixing incompatible data — delete the `.db` file to start fresh. The database also
means you can inspect exactly what was retrieved, and it is safe to delete once you have your DataFrame.

Useful extras on the result:

```python
df.attrs['data_available']   # {'Yes': [...], 'No': [...], 'Failed': [...], 'Skipped': [...]}
df.attrs['db_path']          # where the job database lives
df.attrs['job_name']
df.attrs['created_date']
```

### Naming a job yourself

The job name is built from the country, org units, level and period unit, so two downloads that differ
only in their UIDs or period range want the same file. The second one stops rather than mixing data.
Pass `job_name` to keep them apart:

```python
df_2023 = dfetch.download_data(uids=[...], start_period='202301', end_period='202312',
                            job_name='malaria_2023', ...)
df_2024 = dfetch.download_data(uids=[...], start_period='202401', end_period='202412',
                            job_name='malaria_2024', ...)
```

Each gets its own `.db` and resumes independently. The name is cleaned up for use as a filename, so
spaces and slashes are fine. Reusing a name with different settings still stops, which is the point —
`job_name` separates jobs, it does not switch the check off.

Other options: `disaggregation=True` for category option combos, `ou_attributes=False` to skip the
org-unit group-set columns, and `output_dir` to move the job database.

## How the requests are split up

A download of *n* data items over *m* periods can be sliced into requests in several ways, and
`download_data` picks one automatically from how many of each you asked for. You can override it:

```python
df = dfetch.download_data(..., download_strategy='by_period')
```

| Strategy | One request per … |
|---|---|
| `by_uid` | data item, covering all periods |
| `by_period` | period, covering all data items |
| `batched_uids` | period × batch of 20 data items |
| `batched_periods` | data item × batch of 20 periods |
| `auto` (default) | chosen from the size of the job |

**Worth trying if a download misbehaves.** On some instances a query that comes back empty, times out, or
errors will succeed when sliced a different way — typically a smaller strategy such as `by_uid` or
`batched_periods`. The causes sit on the server: request size and timeout
limits, memory pressure while a large analytics query is assembled, proxies cutting off slow responses,
and the state of the analytics tables all play a part, and they differ between instances. There is no
reliable way to predict which shape an instance will accept, so if results look wrong or a job keeps
failing, changing `download_strategy` is a quick and easy thing to try before assuming the data isn't there.

The log line at the start of a download tells you what was chosen and how many requests it implies:

```
Download strategy: batched_uids (auto, score 40×24=960) — 24 periods × batches of 20 UIDs (~48 requests)
Download strategy: by_uid (auto, score 2×12=24) — 2 requests (all 12 periods per UID)
```

## Saving your results

The package returns DataFrames but does not write a data file for you, so saving is one line of pandas:

```python
df = dfetch.download_data(...)

df.to_csv('outputs/malaria_2024.csv', index=False)
df.to_excel('outputs/malaria_2024.xlsx', index=False)
df.to_parquet('outputs/malaria_2024.parquet')           
```

One caveat: the run metadata on `df.attrs` (`data_available`, `db_path`, `job_name`, `created_date`) is
**lost by `to_csv` and `to_excel`**, which store only the table itself. Read what you need from `.attrs`
before saving, or use `df.to_pickle(...)`, which preserves it:

```python
failed = df.attrs['data_available']['Failed']   # check this before you save
df.to_pickle('outputs/malaria_2024.pkl')        # keeps .attrs intact
```

## Boundaries and maps

```python
shapes = dfetch.fetch_shapefiles()          # {level_name: GeoJSON FeatureCollection}
shapes
# ShapefileSet(District: 13 features, Chiefdom: 150 features)
```

The result is an ordinary dict — `shapes['District']` is a GeoJSON `FeatureCollection` you can hand
straight to geopandas, folium or `json.dump`. It only prints as a summary, because the real output can be
hundreds of thousands of coordinates.

To write one file per level:

```python
dfetch.save_shapefiles(shapes, output_dir='./outputs')
# {'District': './outputs/District.geojson', 'Chiefdom': './outputs/Chiefdom.geojson'}
```

It creates `output_dir` if needed, makes level names safe to use as filenames, and returns the paths.
This is the one place the package writes data files, and only because you asked it to — everything else
returns a value and leaves saving to you.

Levels without geometry are skipped with a warning.

## Errors

All exceptions inherit from `Dhis2Error`, so `except Dhis2Error` catches anything the package raises:
`AuthenticationError`, `Dhis2ConnectionError`, `Dhis2TimeoutError`, `Dhis2RequestError`,
`JobConflictError`, `NotConnectedError`. Bad arguments raise plain `ValueError`.

Individual failed requests inside a long download do not abort the job — they are recorded and reported
in `df.attrs['data_available']['Failed']`, and retried when you re-run.

## Windows sleep prevention

On Windows, downloads keep the machine awake for their duration and release it when finished, so a long
job doesn't stall when the laptop suspends. This is automatic and needs nothing from you. The display is
unaffected, and it is a no-op on other platforms. To wrap your own long-running cell the same way:

```python
from dhis2_fetch import keep_awake

with keep_awake():
    ...
```

## Development

```bash
pip install -e ".[dev]"
pytest
```

The test suite is offline — no DHIS2 server needed. CI runs it on Python 3.9 through 3.13.

`examples/smoke_test.py` is a separate, manual end-to-end check against a live server. It defaults to the
public DHIS2 demo (`admin`/`district`) and is configurable by environment variable:

```bash
python examples/smoke_test.py

DHIS2_URL=https://your-instance.org DHIS2_USERNAME=you DHIS2_PASSWORD=... DHIS2_INDICATOR_UID=... DHIS2_DATASET_UID=... DHIS2_LEVEL=4 python examples/smoke_test.py
```

The public demo servers have no analytics tables built, so the indicator and reporting-metric sections
report SKIPPED there; point it at an instance with analytics to exercise those paths.

## Requirements

Python 3.9+ - `requests` - `pandas` - `tqdm`

Tested on Python 3.9 through 3.13, on Linux and Windows. The public functions carry type hints.

## API stability

The package works and is tested, but the version is still `0.x`: the functions and their arguments may
still change between minor versions while the interface settles. Pin a version if you are building
something that has to keep working unattended. Changes are listed in [CHANGELOG.md](https://github.com/Valerian8/dhis2-fetch/blob/main/CHANGELOG.md).

## Licence

MIT — see [LICENSE](https://github.com/Valerian8/dhis2-fetch/blob/main/LICENSE).

## Disclaimer

An independent tool, not affiliated with or endorsed by the DHIS2 project, the University of Oslo, or any
ministry of health. DHIS2 is a trademark of the University of Oslo.
