Metadata-Version: 2.4
Name: odmlib
Version: 0.2.1
Summary: Python library for working with CDISC ODM documents and extensions
Author-email: Sam Hume <swhume@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/swhume/odmlib
Project-URL: Documentation, https://swhume.github.io/odmlib
Project-URL: Repository, https://github.com/swhume/odmlib
Project-URL: Issues, https://github.com/swhume/odmlib/issues
Keywords: CDISC,ODM,Define-XML,Dataset-JSON,clinical data
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Healthcare Industry
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE.md
Requires-Dist: xmlschema>=1.10.0
Requires-Dist: validators>=0.18.2
Requires-Dist: Cerberus>=1.3.4
Requires-Dist: pathvalidate>=2.3.1
Provides-Extra: dataframe
Requires-Dist: pandas>=1.5; extra == "dataframe"
Provides-Extra: test
Requires-Dist: pytest>=7.0; extra == "test"
Requires-Dist: pytest-cov>=4.0; extra == "test"
Requires-Dist: hypothesis>=6.0; extra == "test"
Provides-Extra: docs
Requires-Dist: sphinx>=5.0; extra == "docs"
Requires-Dist: sphinx-rtd-theme>=1.0; extra == "docs"
Provides-Extra: dev
Requires-Dist: odmlib[dataframe,docs,test]; extra == "dev"
Requires-Dist: mypy>=1.0; extra == "dev"
Dynamic: license-file

# odmlib

[![CI](https://github.com/swhume/odmlib/actions/workflows/ci.yml/badge.svg)](https://github.com/swhume/odmlib/actions/workflows/ci.yml)
[![PyPI version](https://img.shields.io/pypi/v/odmlib)](https://pypi.org/project/odmlib/)
[![Python versions](https://img.shields.io/pypi/pyversions/odmlib)](https://pypi.org/project/odmlib/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE.md)

A Python library for creating, parsing, and validating CDISC ODM (Operational Data Model)
documents and extensions including Define-XML, Dataset-XML, and CT-XML.

## Supported Standards

| Standard | Package | Status | Bundled XSD |
|----------|---------|--------|-------------|
| ODM 1.3.2 | `odmlib.odm_1_3_2` | Stable | ✅ |
| ODM 2.0 | `odmlib.odm_2_0` | Draft | ✅ |
| Define-XML 2.0 | `odmlib.define_2_0` | Stable | ✅ |
| Define-XML 2.1 | `odmlib.define_2_1` | Stable | ✅ |
| Dataset-XML 1.0.1 | `odmlib.dataset_1_0_1` | Stable | — |
| CT-XML 1.1.1 | `odmlib.ct_1_1_1` | Stable | — |
| ARM 1.0 | `odmlib.arm_1_0` | Stable | ✅ |
| Dataset-JSON v1.1 | `odmlib.dataset_json_1_1` | Stable | — |

## Features

- **Object-oriented interface**: work with ODM elements as Python objects
- **Type-validated attributes**: all assignments validated at assignment time
- **Bidirectional serialization**: convert between XML, JSON, and Python dicts
- **Validation**: OID uniqueness, ref/def integrity, Cerberus conformance, element ordering
- **Bundled CDISC schemas**: XSD validation for ODM, Define-XML, and ARM with no downloads
- **Dynamic OID checking**: automatic ref/def mapping via model introspection
- **Extensible**: create custom extensions by subclassing model classes
- **Builder API**: fluent `ODMBuilder` for programmatic document construction
- **Context managers**: `open_odm()` and `open_define()` for safe file handling
- **Dataset-JSON v1.1**: create, read, write, and convert CDISC Dataset-JSON documents
- **Define-XML roundtrip**: flatten Define-XML to tabular Dataset-JSON and rebuild via `DefineFlattener`/`DefineBuilder`
- **Permissive loading**: load non-conformant files for inspection and repair with graduated validation control
- **Pandas integration**: export metadata/data to DataFrames; import DataFrame rows as ODM objects (optional, `pip install odmlib[dataframe]`)

See ROADMAP.md for the path to v1.0

## Installation

```bash
pip install odmlib
```

For development:

```bash
git clone https://github.com/swhume/odmlib.git
cd odmlib
pip install -e ".[dev]"
```

With optional Pandas support:

```bash
pip install odmlib[dataframe]
```

## Claude Code Skill

odmlib ships a [Claude Code](https://claude.com/claude-code) skill that teaches Claude how to
use this library correctly — the right loader per standard, namespace registration, element
ordering, validation layers, and the serialization pitfalls that are easy to get wrong by hand.

The skill lives in this repository at `.claude/skills/odmlib/`. It is **not** part of the PyPI
package, so `pip install odmlib` does not install it — copy the directory into whichever
`.claude/skills/` directory you want it available from:

```bash
git clone https://github.com/swhume/odmlib.git

# For one project:
mkdir -p /path/to/your-project/.claude/skills
cp -r odmlib/.claude/skills/odmlib /path/to/your-project/.claude/skills/

# Or for every project on this machine:
mkdir -p ~/.claude/skills
cp -r odmlib/.claude/skills/odmlib ~/.claude/skills/
```

Claude picks the skill up automatically when a task involves ODM, Define-XML, Dataset-JSON,
or ARM content — you do not need to name it.

**The skill describes odmlib 0.2.1 and later.** Several behaviors it documents (namespace-aware
`to_xml_string()`, opt-in context-manager writing, full error enumeration under
`collect_errors=True`) do not hold on 0.2.0, so upgrade before relying on it.

If Claude generates incorrect odmlib code while using the skill, please
[open a skill feedback issue](https://github.com/swhume/odmlib/issues/new?template=skill-feedback.yml) —
those reports are what the skill is refined against.

## Loading Documents

### Load an ODM-XML file

```python
import odmlib.odm_loader as OL
import odmlib.loader as LD

loader = LD.ODMLoader(OL.XMLODMLoader())
loader.open_odm_document("study.xml")
odm = loader.root()

mdv = loader.MetaDataVersion()
for item_group in mdv.ItemGroupDef:
    print(f"{item_group.OID}: {item_group.Name}")

# Find a specific element
item = mdv.find("ItemDef", "OID", "IT.AGE")
```

### Load an ODM-JSON file

```python
import odmlib.odm_loader as OL
import odmlib.loader as LD

loader = LD.ODMLoader(OL.JSONODMLoader())
loader.open_odm_document("study.json")
mdv = loader.MetaDataVersion()
```

### Load from a string

```python
import odmlib.odm_loader as OL
import odmlib.loader as LD

loader = LD.ODMLoader(OL.XMLODMLoader())
loader.load_odm_string(xml_string)
odm = loader.root()
```

### Load a Define-XML file

```python
import odmlib.define_loader as DL
import odmlib.loader as LD

# Define-XML 2.1 (default)
loader = LD.ODMLoader(DL.XMLDefineLoader(model_package="define_2_1"))
loader.open_odm_document("define.xml")
mdv = loader.MetaDataVersion()

for item_def in mdv.ItemDef:
    print(f"{item_def.OID}: {item_def.Name} ({item_def.DataType})")

# Find all value lists
vl = mdv.find("ValueListDef", "OID", "VL.AEDECOD")
```

### Load an ARM-extended Define-XML (ADaM)

```python
import odmlib.arm_loader as AL
import odmlib.loader as LD

loader = LD.ODMLoader(AL.XMLArmLoader())
loader.open_odm_document("define-adam.xml")
mdv = loader.MetaDataVersion()

# Access analysis result displays
for rd in mdv.AnalysisResultDisplays:
    print(f"{rd.OID}: {rd.Name}")
    for ar in rd.AnalysisResult:
        print(f"  Result: {ar.OID} ({ar.AnalysisPurpose})")

# Schema-validate it against the bundled ARM XSD
from odmlib.odm_parser import ODMSchemaValidator

validator = ODMSchemaValidator(standard="arm", version="1.0-define2.1")
validator.validate_file("define-adam.xml")
```

## Creating Documents

### Create an ODM document programmatically

```python
import odmlib.odm_1_3_2.model as ODM
import odmlib.ns_registry as NS

# Register the namespace (required once before creating XML)
NS.NamespaceRegistry(prefix="odm",
    uri="http://www.cdisc.org/ns/odm/v1.3",
    is_default=True, is_reset=True)

# Build elements bottom-up
tt = ODM.TranslatedText(_content="Subject identifier", lang="en")
desc = ODM.Description(TranslatedText=[tt])
item_def = ODM.ItemDef(
    OID="IT.SUBJID", Name="SUBJID", DataType="text",
    Length=8, Description=desc
)

item_ref = ODM.ItemRef(ItemOID="IT.SUBJID", Mandatory="Yes", OrderNumber=1)
igd = ODM.ItemGroupDef(
    OID="IG.DM", Name="Demographics", Repeating="No",
    ItemRef=[item_ref]
)

mdv = ODM.MetaDataVersion(OID="MDV.001", Name="Version 1")
mdv.ItemGroupDef.append(igd)
mdv.ItemDef.append(item_def)

gv = ODM.GlobalVariables(
    StudyName=ODM.StudyName(_content="My Study"),
    StudyDescription=ODM.StudyDescription(_content="Phase II trial"),
    ProtocolName=ODM.ProtocolName(_content="PROT-001"),
)
study = ODM.Study(OID="S.001", GlobalVariables=gv, MetaDataVersion=[mdv])
odm = ODM.ODM(
    FileOID="F.001", FileType="Snapshot",
    CreationDateTime="2024-01-01T00:00:00", Study=[study]
)

odm.write_xml("study.xml")
odm.write_json("study.json")
```

### Use the fluent builder

The `ODMBuilder` provides a chainable API that tracks the current study,
MetaDataVersion, and ItemGroupDef context automatically.

```python
from odmlib.builder import ODMBuilder

odm = (ODMBuilder("odm_1_3_2")
    .set_file(FileOID="F.001", FileType="Snapshot",
              CreationDateTime="2024-01-01T00:00:00")
    .add_study(OID="S.001",
               study_name="My Study",
               study_description="Phase II trial",
               protocol_name="PROT-001")
    .add_metadata_version(OID="MDV.001", Name="Version 1")
    .add_item_group_def(OID="IG.DM", Name="Demographics", Repeating="No")
    .add_item_ref(ItemOID="IT.SUBJID", Mandatory="Yes", OrderNumber=1)
    .add_item_ref(ItemOID="IT.AGE", Mandatory="No", OrderNumber=2)
    .add_item_def(OID="IT.SUBJID", Name="SUBJID", DataType="text", Length=8)
    .add_item_def(OID="IT.AGE", Name="AGE", DataType="integer")
    .add_code_list(
        OID="CL.SEX", Name="Sex", DataType="text",
        items=[
            {"CodedValue": "M", "Decode": "Male"},
            {"CodedValue": "F", "Decode": "Female"},
        ]
    )
    .build())

odm.write_xml("study.xml")
```

### Context Managers

Context managers load a document on entry, making read-modify-write workflows
concise. **Writing is opt-in** (since 0.2.1): a bare `open_odm(path)` loads
read-only and writes nothing on exit. Ask for a write by passing `output_file=`
to write elsewhere, or `write_on_exit=True` to update the input file in place.

```python
from odmlib.context import open_odm, open_define

# Update an ODM file in place (explicit opt-in via write_on_exit=True)
with open_odm("study.xml", write_on_exit=True) as odm:
    odm.FileOID = "F.002"
    mdv = odm.Study[0].MetaDataVersion[0]
    mdv.ItemGroupDef.append(new_igd)
# study.xml is overwritten on clean exit

# Without write_on_exit= or output_file=, the load is READ-ONLY
with open_odm("study.xml") as odm:
    odm.FileOID = "F.002"       # discarded on exit; study.xml is untouched

# Write to a different output file
with open_odm("study.xml", output_file="study_updated.xml") as odm:
    odm.FileOID = "F.002"

# Read-only inspection — input file is never modified
with open_odm("study.xml", write_on_exit=False) as odm:
    print(odm.FileOID)
    print(len(odm.Study[0].MetaDataVersion[0].ItemDef))

# Define-XML (defaults to define_2_1 model)
# NOTE: in Define-XML, Study and MetaDataVersion are single objects, not lists
with open_define("define.xml") as define:
    mdv = define.Study.MetaDataVersion
    print(len(mdv.ItemDef))

# JSON format is auto-detected from the file extension
with open_odm("study.json", write_on_exit=True) as odm:
    odm.FileOID = "F.002"
```

### Load a non-conformant file

```python
from odmlib import permissive
import odmlib.loader as LD
import odmlib.odm_loader as OL

loader = LD.ODMLoader(OL.XMLODMLoader())

with permissive():
    loader.open_odm_document("broken_define.xml")
    odm = loader.root()

# Fix issues, then validate (max_errors caps a badly broken document)
errors = odm.validate(collect_errors=True, max_errors=100)
```

## Serialization

All odmlib elements support bidirectional conversion:

```python
# To/from XML
xml_string = item_def.to_xml_string()   # self-contained: declares its own xmlns
xml_elem = item_def.to_element()        # namespace-resolved ElementTree Element

# To/from JSON
json_string = mdv.to_json()
python_dict = mdv.to_dict()

# Write to file
odm.write_xml("output.xml")
odm.write_json("output.json")
```

`to_xml_string()` adds the xmlns declarations the tree actually uses, so the result
re-parses and schema-validates on its own. The string and file paths line up exactly:

```python
odm.to_xml_string()                       # the bytes write_xml() writes AFTER <?xml ...?>
odm.to_xml_string(xml_declaration=True)   # exactly what write_xml() writes
```

`to_element()` gives you a real tree — Clark-notation tags
(`{http://www.cdisc.org/ns/def/v2.1}leaf`), so namespace-aware `find()`,
`ET.canonicalize()`, `ET.indent()` pretty-printing, and grafting into a host document all
work. It costs one serialize + reparse (~8 ms for a 166 KB Define-XML document).

`to_xml()` is different: it is odmlib's internal *serialization buffer*, with literal
prefixed tags (`def:leaf`) and **no** `xmlns` at all — the shared tree builder behind
`to_xml_string()` and `write_xml()`. `ET.tostring(obj.to_xml())` is therefore not a
substitute — on Define-XML it fails to parse (`unbound prefix`), and on ODM it parses into
no namespace, which odmlib will re-load with the `FileOID` intact and every `Study`
silently dropped. Use `to_xml_string()` for text and `to_element()` when you want a tree.

## Validation

odmlib provides four independent validation layers.

### OID uniqueness and ref/def integrity

```python
from odmlib.oid_generator import create_oid_checker

checker = create_oid_checker("odm_1_3_2")
odm.verify_oids(checker)               # raises OdmlibOIDError on failure

# Find unreferenced definitions
orphans = odm.unreferenced_oids(checker)
```

### Cerberus conformance validation

```python
from odmlib.odm_1_3_2.rules.metadata_schema import MetadataSchema

validator = MetadataSchema()
odm.verify_conformance(validator)      # raises on failure
```

### XML schema (XSD) validation

odmlib bundles the official CDISC schemas, so there is nothing to download.
Select one by `(standard, version)`:

```python
from odmlib.odm_parser import ODMSchemaValidator

validator = ODMSchemaValidator(standard="odm", version="1.3.2")
validator.validate_file("study.xml")               # raises OdmlibSchemaValidationError on failure
```

| `standard` | `version` | Validates |
|------------|-----------|-----------|
| `"odm"`    | `"1.3.2"` | ODM 1.3.2 |
| `"odm"`    | `"2.0"`   | ODM 2.0 |
| `"define"` | `"2.0"`   | Define-XML 2.0 |
| `"define"` | `"2.1"`   | Define-XML 2.1 |
| `"arm"`    | `"1.0"`   | ARM 1.0 in a Define-XML 2.0 document |
| `"arm"`    | `"1.0-define2.1"` | ARM 1.0 in a Define-XML 2.1 document |

ARM has two entries because it layers onto Define-XML, and the two Define-XML
versions use different `def:` namespace URIs. Pick the one matching your
document — `"1.0-define2.1"` is the pairing `odmlib.arm_1_0` models. The ARM
schemas are supersets of the corresponding Define-XML schema, so they also
validate ARM-free Define-XML documents.

Both `standard` and `version` are required; there is no default. For a custom
or local schema, pass a path instead:

```python
validator = ODMSchemaValidator(xsd_file="/path/to/schema.xsd")
```

### Combined validation with error collection

Collect every error in a single pass instead of stopping at the first failure.
Each layer — element order, OID integrity, and conformance — reports *all* the
problems it finds, so one run gives you the complete picture:

```python
from odmlib.oid_generator import create_oid_checker

checker = create_oid_checker("odm_1_3_2")
errors = odm.validate(collect_errors=True, oid_checker=checker)
for err in errors:
    print(err)
```

Use `max_errors` to cap the list on a badly broken document — the final entry is
then an `OdmlibErrorLimitError` marking that more problems may exist:

```python
errors = odm.validate(collect_errors=True, oid_checker=checker, max_errors=100)
```

An OID checker accumulates every OID it sees, so use one per document or call
`checker.reset()` between runs. The deprecated `rules/oid_ref.py` `OIDRef`
classes do not support error collection and contribute at most one error.

### Element ordering

```python
try:
    odm.verify_order()
except OdmlibElementOrderError:
    odm.reorder_object()    # fix ordering automatically (issues a warning)
```

## OID Index Lookup

Build an index for fast OID lookups across the entire document tree:

```python
idx = odm.build_oid_index()
elements = idx.find_all("IT.AGE")    # returns list of odmlib objects with that OID
```

## Finding Elements

```python
# First match
item = mdv.find("ItemDef", "OID", "IT.AGE")

# All matches
text_items = mdv.find_all("ItemDef", "DataType", "text")

# Multi-attribute match
item = mdv.find_by("ItemDef", DataType="integer", Length=4)
```

## Dataset-JSON

```python
from odmlib.dataset_json_1_1 import DatasetJSON, Column

ds = DatasetJSON(
    datasetJSONVersion="1.1.0",
    fileOID="F.DS.001",
    creationDateTime="2024-01-01T00:00:00",
    datasetJSONCreationDateTime="2024-01-01T00:00:00",
    records=3,
    name="DM",
    label="Demographics",
)
ds.columns = [
    Column(itemOID="IT.DM.SUBJID", name="SUBJID", label="Subject ID",
           dataType="string", targetDataType="string"),
    Column(itemOID="IT.DM.AGE", name="AGE", label="Age",
           dataType="integer", targetDataType="integer"),
]
ds.rows = [["SUBJ-001", 34], ["SUBJ-002", 28], ["SUBJ-003", 45]]

ds.write_json("dm.json")
```

Convert between Dataset-XML and Dataset-JSON:

```python
from odmlib.dataset_json_1_1 import dataset_xml_to_dataset_json, dataset_json_to_dataset_xml

dataset_json = dataset_xml_to_dataset_json("dm.xml", "define.xml")
dataset_json.write_json("dm.json")
```

Flatten Define-XML 2.1 metadata into Dataset-JSON datasets:

```python
from odmlib.dataset_json_1_1 import DefineFlattener

flattener = DefineFlattener("define.xml")
datasets = flattener.flatten()           # dict of dataset name → DatasetJSON
datasets["IG"].write_json("ig.json")
```

## Pandas Integration

Requires `pip install odmlib[dataframe]`.

```python
from odmlib.dataframe import (
    metadata_to_dataframe,
    clinical_data_to_dataframe,
    define_metadata_to_dataframes,
    dataset_json_to_dataframe,
    dataframe_to_items,
)

# Export all ItemDef metadata as a DataFrame
df = metadata_to_dataframe(mdv, "ItemDef")
print(df[["OID", "Name", "DataType", "Length"]].to_string())

# Flatten ClinicalData to a tabular DataFrame
df = clinical_data_to_dataframe(odm)

# Flatten all Define-XML metadata tables at once
dfs = define_metadata_to_dataframes("define.xml")
print(dfs["variables"].head())

# Convert Dataset-JSON to DataFrame
df = dataset_json_to_dataframe(ds)

# Create odmlib elements from a DataFrame
items = dataframe_to_items(df, ODM.ItemDef)
```

## Namespace Management

When creating documents from scratch, register namespaces before writing XML.
The `is_reset=True` flag clears any previously registered namespaces.

```python
import odmlib.ns_registry as NS

NS.NamespaceRegistry(prefix="odm",
    uri="http://www.cdisc.org/ns/odm/v1.3",
    is_default=True, is_reset=True)

# For Define-XML, add additional namespaces
NS.NamespaceRegistry(prefix="def", uri="http://www.cdisc.org/ns/def/v2.1")
NS.NamespaceRegistry(prefix="xs", uri="http://www.w3.org/2001/XMLSchema-instance")
# xlink is required: def:leaf carries xlink:href, and without this the
# xmlns:xlink declaration is omitted and the output cannot be re-parsed
NS.NamespaceRegistry(prefix="xlink", uri="http://www.w3.org/1999/xlink")
```

`is_reset=True` clears every prefix, including the ones each model package registers when
it is imported — so register every prefix the document uses, not only the ones you set
yourself.

## Running Tests

```bash
# Run all tests
python -m pytest tests/ -v

# Run with coverage report
python -m pytest tests/ --cov=odmlib --cov-report=term-missing

# Run a specific test file
python -m pytest tests/test_odm_loader.py -v
```

## Documentation

- [API Reference](https://swhume.github.io/odmlib)
- [Example Programs](https://github.com/swhume/odmlib_examples)
- [CDISC ODM Specification](https://www.cdisc.org/standards/data-exchange/odm)

## Dependencies

- [xmlschema](https://pypi.org/project/xmlschema/) — XML Schema validation
- [validators](https://pypi.org/project/validators/) — email/URL validation
- [Cerberus](https://docs.python-cerberus.org/) — conformance schema validation
- [pathvalidate](https://pypi.org/project/pathvalidate/) — filename validation

**Optional:**

- [pandas](https://pandas.pydata.org/) ≥ 1.5 — DataFrame integration (`pip install odmlib[dataframe]`)

## Known Limitations

- No `ItemData[Type]` support (typed item data elements, deprecated in ODM v2.0)
- No `ds:Signature` support (digital signatures)
- Single `MetaDataVersion` per load by default (use `idx` parameter for others)
- ODM v2.0 implementation is still draft, but every **metadata** element the ODM 2.0
  XSD defines is now modelled, and the model's divergence from the schema is pinned by
  `tests/test_odm_2_0_xsd_alignment.py`. Two areas remain:
  - **ClinicalData / ReferenceData are not modelled.** `ODM` has no `ClinicalData`,
    `ReferenceData` or `Association` child, and `Location` has no `Query`. The 24
    elements of that data layer are scoped to v0.3.0 — see `ROADMAP.md`.
  - **Three deliberate approximations**, each documented in the class docstring, where
    the XSD says something odmlib's descriptor model cannot express. Each is caught by
    `ODMSchemaValidator` rather than at build time:
    - `FormalExpression` — the XSD requires exactly one of `Code` or `ExternalCodeLib`;
      odmlib cannot express an `xs:choice`, so both are optional.
    - `StudyEventGroupDef` — the XSD's repeating `(StudyEventGroupRef?, StudyEventRef?)`
      group is approximated by two parallel lists, so an interleaved ordering cannot be
      reproduced. Both forms are schema-valid; only the ordering is lost.
    - `TranslatedText` — the XSD allows XHTML markup via a mixed-content `xhtml:div`
      child. odmlib models the text-only form; an `xhtml:div` in a source document is
      dropped on load.

## License

[MIT](LICENSE.md)

## Contributing

Issues and pull requests are welcome at
[https://github.com/swhume/odmlib/issues](https://github.com/swhume/odmlib/issues).
