Metadata-Version: 2.4
Name: pymetadataeditor
Version: 0.5.0
Summary: Metadata Editor client for python
License: MIT
License-File: LICENSE
Author: Gordon Blackadder
Author-email: gblackadder@worldbank.org
Requires-Python: >=3.11,<4.0
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Provides-Extra: llm
Requires-Dist: markitdown (>=0.0.1a3,<0.0.2) ; extra == "llm"
Requires-Dist: metadataschemas (>=0.1.3,<0.2.0)
Requires-Dist: openai (>=2.8.0,<3.0.0) ; extra == "llm"
Requires-Dist: pandas (>=2.2.2,<3.0.0)
Requires-Dist: pydantic (>=2.7.4,<3.0.0)
Requires-Dist: requests (>=2.32.3,<3.0.0)
Requires-Dist: tiktoken (>=0.8.0) ; extra == "llm"
Requires-Dist: tqdm ; extra == "llm"
Requires-Dist: urllib3 (>=2.2.2,<3.0.0)
Description-Content-Type: text/markdown

# pyMetadataEditor

A Python client for [Metadata Editor](https://github.com/worldbank/metadata-editor): create, edit, and manage metadata for microdata, indicators, geospatial data, documents, scripts, images, and videos.

## Installation

This project uses [Poetry](https://python-poetry.org/docs/basic-usage/):

```bash
pip install poetry
cd pymetadataeditor
poetry install
```

The LLM-assisted metadata features (OpenAI, `tiktoken`, `markitdown`) are optional: `poetry install --extras llm`, or `pip install "pymetadataeditor[llm]"`.

For development (pytest, pre-commit):

```bash
poetry install --with dev
poetry run pre-commit install
```

## Quick start

```python
from pymetadataeditor import MetadataEditor
import os

me = MetadataEditor(
    api_url=os.getenv("METADATA_API_URL"),   # e.g. https://host/index.php/api
    api_key=os.getenv("METADATA_API_KEY"),
    verify_ssl=False,  # set True when SSL is verified on your server
)
```

### List projects

`list` returns one page (summary fields, no metadata) plus the server `total`:

```python
response = me.projects.list(limit=10, metadata_type="indicator", keywords="literacy rate")
print(response["total"])
for project in response["projects"]:
    print(project["id"], project["type"], project["title"])
```

Filters: `keywords`, `metadata_type`, `collection` (+ `exclude_collections=True`), `tag`, `ownership`, `users`,
`created_by`, `changed_by`, `date_start` / `date_end` (last changed), `sort_by`. List filters accept Python lists.
`me.projects.facets()` returns the available types, collections and their counts.

To go through every matching project, iterate instead of loading everything at once:

```python
for project in me.projects.iter_all(metadata_type="indicator", page_size=500):
    ...

me.projects.count(collection=12)                # just the number
me.projects.list_all(metadata_type="table")     # everything in memory: {"projects": [...], "total": n}
```

`iter_all` pages in id order, so projects edited while you iterate are neither skipped nor repeated. Other large
listings have the same pattern: `me.collections.iter_projects`, `me.jobs.iter_all`, `me.codelists.iter_all`,
`me.codelists.iter_codes`, `me.data_structures.iter_all`, `me.indicators.iter_data_rows`.

### Create a project (template-aware)

```python
demo_name = "my_indicator_demo"
metadata = {
    "idno": demo_name,
    "series_description": {
        "idno": demo_name,
        "name": "Version 1",
    },
}

project_id = me.metadata.create(
    metadata,
    metadata_type_or_template_uid="indicator",
)
```

### Read metadata back

```python
model = me.metadata.get(project_id, output_mode="pydantic")
model.pretty_print()
```

## API overview (v2)

Thin clients return **raw API JSON** (`dict`). Workflow helpers on `me.metadata` handle validation and Pydantic/Excel conversion.

| Client | Role |
|--------|------|
| `me.projects` | List, get, create, update, patch, delete projects |
| `me.metadata` | Template-aware create, update, get (dict / pydantic / excel) |
| `me.collections` | Collections and membership |
| `me.templates` | Template CRUD |
| `me.resources` | External resources (reports, questionnaires, files) |
| `me.jobs` | Background jobs (publish to NADA, indicator data import, poll, list) |
| `me.publish` | Publish metadata and indicator DSD/data to NADA (synchronous) |
| `me.uploads` | Resumable (chunked) file uploads |
| `me.data_structures` | Data structure (DSD) registry: structures, components, CSV/JSON/SDMX import, export |
| `me.codelists` | Codelist registry: codelists, codes, translations, import/export |
| `me.indicators` | Indicator projects: bind a DSD, load/export data, chart data, `notes`, publish |
| `me.api` | Generic `get` / `post` / `put` / `delete` for any path |

API errors raise `ApiError` (`status_code`, `message`, `errors`), a subclass of `requests` `HTTPError`.

Methods still on `me` directly include `make_metadata_outline`, `read_metadata_from_excel`, `draft_metadata_from_files`, and related helpers.

**Breaking change:** v1 flat methods (`list_projects`, `create_project_log`, …) were removed. See [legacy-migration.MD](legacy-migration.MD).

### Direct HTTP (`me.api`)

For endpoints without a typed wrapper yet:

```python
me.api.get("/editor", params={"limit": 5})
me.api.post("/collections/add_projects", json={"collections": [1], "id_format": "id", "projects": [101]})
```

POST body styles:

- `json={...}` — JSON API payloads
- `data={...}` — form fields
- `data={...}` + `files={...}` — multipart uploads

Many deletes use **POST** in restricted networks (e.g. `me.projects.delete`, `me.collections.delete`), not HTTP DELETE.

### Publish to NADA

`me.publish.*` publishes now and returns NADA's answers; `me.jobs.queue_*` methods queue a job on the
editor's worker and return its `uuid` (pass `wait=True` to block until it finishes). Both take the same steps:

```python
options = {"overwrite": "yes", "published": 1, "access_policy": "licensed", "repositoryid": "MY_COLLECTION"}

# now: {"results": {...}, "errors": {...}} per step
me.publish.project(123, catalog_connection_id=1, publish_thumbnail=False, options=options)

# background job
job = me.jobs.queue_publish_to_nada(123, catalog_connection_id=1, publish_thumbnail=False, options=options)
result = me.jobs.wait(job["uuid"], timeout_seconds=3600)
```

Single steps: `me.publish.metadata`, `indicator_data`, `thumbnail`, `resources`, `resource`, `delete_nada_resources`.

Step-by-step notebook: [`demo/publish_to_nada_jobs.ipynb`](demo/publish_to_nada_jobs.ipynb).

## Guides

- [Indicators](docs/indicators.md) — data structures, loading data, notes, publishing to NADA, pandas helpers
- [Metadata workflows](docs/metadata-workflows.md) — outlines, [Pydantic](docs/metadata-workflows.md#pydantic), [Excel](docs/metadata-workflows.md#excel), [retrieving metadata](docs/metadata-workflows.md#retrieving-existing-metadata)
- [LLM-assisted metadata](docs/llm-assisted-metadata.md) — draft and augment from documents and URLs
- [API examples](docs/examples.md) — collections, copying projects, and common tasks
- [Migrating from v1](legacy-migration.MD) — old → new method mapping
- [API reference](docs/API_Reference.md) — regenerate with `python make_docs.py`

## Contributing

### Poetry troubleshooting (Windows)

If NumPy or path-length errors occur:

1. Enable [long paths in Windows](https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=powershell#enable-long-paths-in-windows-10-version-1607-and-later), or
2. Run `poetry config virtualenvs.in-project true` then `poetry install`

### Tests

```bash
poetry run pytest tests/test_interface.py -q
```

### API documentation

```bash
python make_docs.py             # demo notebooks -> docs/*.md, and docs/API_Reference.md
python make_docs.py --api-only  # only docs/API_Reference.md
```

## Notes

Parts of this codebase were written with the assistance of ChatGPT, in keeping with World Bank Group practice.

