Metadata-Version: 2.4
Name: robokop-orion
Version: 2.0.9
Summary: ORION ingests data from knowledge bases and converts it into interoperable Biolink Model knowledge graphs.
License-Expression: MIT
License-File: LICENSE
Requires-Dist: bmt>=1.4.8
Requires-Dist: jsonlines>=4.0.0
Requires-Dist: orjson>=3.11.7
Requires-Dist: pydantic-settings>=2.13.0
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: requests>=2.33.1
Requires-Dist: requests-toolbelt>=1.0.0
Requires-Dist: robokop-genetics>=0.8.0
Requires-Dist: uuid-utils>=0.14.1
Requires-Dist: xxhash>=3.6.0
Requires-Dist: beautifulsoup4>=4.12.3 ; extra == 'robokop'
Requires-Dist: celery>=5.4.0 ; extra == 'robokop'
Requires-Dist: curies>=0.7.9 ; extra == 'robokop'
Requires-Dist: intermine ; extra == 'robokop'
Requires-Dist: mysql-connector-python>=9.4.0 ; extra == 'robokop'
Requires-Dist: neo4j>=5.28.0 ; extra == 'robokop'
Requires-Dist: openpyxl>=3.1.5 ; extra == 'robokop'
Requires-Dist: pandas>=2.3.3 ; extra == 'robokop'
Requires-Dist: polars>=1.19.0 ; extra == 'robokop'
Requires-Dist: prefixmaps>=0.2.6 ; extra == 'robokop'
Requires-Dist: psycopg2-binary>=2.9.9 ; extra == 'robokop'
Requires-Dist: pyoxigraph>=0.5.9 ; extra == 'robokop'
Requires-Dist: redis>=5.2.1 ; extra == 'robokop'
Requires-Python: >=3.12
Provides-Extra: robokop
Description-Content-Type: text/markdown

# ORION

*Operational Routine for the Ingest and Output of Networks*

ORION ingests data from over 40 knowledge sources and converts them into interoperable [Biolink Model](https://biolink.github.io/biolink-model/) knowledge graphs in [KGX](https://github.com/biolink/kgx) format.

Knowledge graphs can be composed of any number of data sources. ORION operates like a dependency manager; to build a graph ORION resolves data source dependencies as needed, by either downloading preprocessed artifacts from a centralized repository or by running an ingest pipeline to build graphs from scratch, before merging them together into unified graphs. Sub-graphs downloaded or generated by the ingest pipeline are stored in versioned directories with accompanying metadata and can be reused in subsequent graph builds.

ORION tracks versions of data sources and is designed to automatically update source data dependencies as they become available. It also tracks versions of the software, services, and data models used so that it can build new graphs as appropriate when underlying dependencies change.

ORION generates thorough metadata about its data, processes, and pipelines, for troubleshooting and understanding, and for distribution with graph outputs. These files align with FAIR principles aimed to increase transparency, provenance, and (re)usability of knowledge graphs for humans and machines.

### Installation

ORION requires Python 3.12+ and [uv](https://docs.astral.sh/uv/) for dependency management (uv will install a suitable Python for you if needed).

Alternatively, using Docker is the best way to ensure the environment is configured correctly, and is highly recommended for any usage of ORION involving neo4j.

```bash
git clone https://github.com/RobokopU24/ORION.git
cd ORION
uv sync --extra robokop
```

`uv sync` installs the `orion-*` commands (listed under [CLI Commands](#cli-commands)) into the project's `.venv`. Run them with `uv run`, e.g. `uv run orion-build ...`, or activate the environment first with `source .venv/bin/activate` and call them directly.

The core library is also available on PyPI (`pip install robokop-orion`), but the full repository is needed to utilize ingest modules from the [ROBOKOP](https://robokop.renci.org/) project.

### Quickstart

Build a customized graph from any of the available data sources:

```bash
uv run orion-build My_Graph --sources DrugCentral,HGNC
```

Outputs are written to `~/ORION-workspace/graphs/<graph_id>/<version>/` by default. See [Configuration](#configuration) to select a different location.

### CLI Commands

The following commands are available (prefix with `uv run` if not using a uv-managed shell):

| Command               | Description                                                                                                                                                           |
|-----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `orion-build`         | Build complete knowledge graphs from a Graph Spec, by specifying sources on the command line, or by naming a single data source to materialize it directly as a graph |
| `orion-ingest`        | Run the ingest pipeline to download and process individual data sources from scratch                                                                                  |
| `orion-merge`         | Merge multiple pairs of KGX node/edge files into one knowledge graph                                                                                                  |
| `orion-meta-kg`       | Generate MetaKG and test data files                                                                                                                                   |
| `orion-schema`        | Generate a KGX schema describing the nodes, edges and attributes in a pair of KGX files                                                                               |
| `orion-schema-diff`   | Compare two KGX schemas and report what changed between them                                                                                                          |
| `orion-redundant-kg`  | Generate edge files with redundant biolink predicates                                                                                                                 |
| `orion-neo4j-dump`    | Generate Neo4j database dumps                                                                                                                                         |
| `orion-memgraph-dump` | Generate Memgraph database dumps                                                                                                                                      |

### Graph Specs

A Graph Spec yaml file defines which sources to include in a knowledge graph. Every `*.yaml` file at the top level of `graph_specs/` is loaded automatically when you run `orion-build`, so any `graph_id` defined in those files can be built directly by name. Project-specific specs that aren't part of the default build live under `graph_specs/optional/`.

To build a graph outside the default available set, you have several options:

```bash
# 1. Provide any local file or http(s) URL for a Graph Spec outside of ORION.
orion-build My_Graph --graph_spec /path/to/my-graph-spec.yaml
orion-build My_Graph --graph_spec https://example.org/my-graph-spec.yaml

# 2. Build a simple graph from a list of sources without writing a yaml file.
orion-build My_Graph --sources DrugCentral,HGNC
orion-build My_Graph --sources DrugCentral,HGNC --output_format neo4j

# 3. Build one of the optional specs in ORION by providing its file path.
orion-build LitCoin_ORION --graph_spec graph_specs/optional/litcoin-graph-spec.yaml

# 4. Materialize a single data source directly as a graph (using its data source id).
orion-build CTD
```

You can build several graphs (or sources) in one invocation by separating their ids with commas, e.g. `orion-build HGNC,CTD` or `orion-build My_Robokop_Graph,LitCoin_ORION`. Note that a `graph_id` cannot be the same as a data source id — the two share one namespace so every id maps to exactly one thing to build.

Add `--conflation` (`-c`) to any `orion-build` command to force conflation on for every source, overriding the spec.

Here is a simple Graph Spec example:

```yaml
graphs:
  - graph_id: Example_Graph
    graph_name: Example Graph
    graph_description: A free text description of what is in the graph.
    output_format: neo4j
    sources:
      - id: DrugCentral
      - id: HGNC
```

See the full list of data sources and their identifiers in the [data sources file](https://github.com/RobokopU24/ORION/blob/master/orion/data_sources.py).

#### Graph Spec Parameters

Graph Specs allow a number of options for customization. The following parameters can be set per data source:

- **merge_strategy** - alternative merge strategies
- **strict_normalization** - discard nodes that fail to normalize (true/false)
- **conflation** - conflate genes with proteins and chemicals with drugs during normalization (true/false) see [Babel's Conflation docs](https://github.com/NCATSTranslator/Babel/blob/main/docs/Conflation.md).

The following can be set at the graph level:

- **output_format** - which database/file formats to generate. Valid values: `jsonl` (the default KGX node/edge files, always produced), `neo4j`, `memgraph`, `redundant_jsonl`, `redundant_neo4j`, `collapsed_qualifiers_jsonl`, `collapsed_qualifiers_neo4j`. Combine multiple with `+`, e.g. `neo4j+jsonl`. The `neo4j`/`memgraph` formats (and the Reactome ingest) require a Neo4j installation; the provided Docker image includes it. If Neo4j is not installed, the dump is logged as an error and skipped, but the rest of the graph still builds.
- **add_edge_id** - whether to add unique identifiers to edges (true/false)
- **edge_id_type** - if add_edge_id is true, the type of identifier can be specified (uuid or orion)
- **base_release_version** - release version floor for the graph (e.g. `"2.0"` or `"2.1.0"`)

#### Graph Versions

Every built graph has two versions:

- A **build version** — a deterministic hash of the graph's inputs (source versions, normalization scheme, merge strategy, graph dependency versions). Identical inputs always produce the same build version.
- A **release version** — a human-facing semantic version (`MAJOR.MINOR.PATCH`) used in output directory names, URLs, and metadata.

ORION picks the release version automatically. On a new build it checks existing releases of the same graph (from the graph registry, then local storage). If a previous release has the same build version, that release is reused. Otherwise the highest existing release version's `PATCH` component is incremented. To start at or jump to a different `MAJOR`/`MINOR`, set `base_release_version:` in the graph spec — it acts as a floor that the auto-bump never goes below.

See the `graph_specs/` directory for more examples.

### Configuration

ORION can be configured via environment variables, which can be set directly or through an `.env` file. 

All variables are optional; the table below lists some that users typically override.

| Variable | Purpose | Default |
|---|---|---|
| `ORION_STORAGE` | Path to a directory for data ingest pipeline storage | `~/ORION-workspace/storage` |
| `ORION_GRAPHS` | Path to a directory for Knowledge Graph outputs | `~/ORION-workspace/graphs` |
| `ORION_LOGS` | Path to a Log file directory (if unset, logs go to stdout) | `None` |
| `ORION_OUTPUT_URL` | Base URL recorded in graph metadata and distribution links | `https://localhost` |
| `ORION_USE_GRAPH_REGISTRY` | Consult the remote graph registry for prebuilt dependencies and version discovery. Set `false` for fully offline/air-gapped builds. | `true` |
| `ORION_GRAPH_REGISTRY_URL` | Base URL of the graph registry. | `https://robokop-graph-registry.apps.renci.org` |

ORION will create `~/ORION-workspace/storage/` and `~/ORION-workspace/graphs/` on first use if the corresponding env vars aren't set.

When the graph registry is enabled (the default), ORION checks it for prebuilt graph dependencies before building them from scratch. If the registry is unreachable or misconfigured, ORION logs a warning and falls back to local storage and building — it never fails a build. For offline use, set `ORION_USE_GRAPH_REGISTRY=false` to skip the registry entirely.

For more customization and settings, copy or rename `.env.example` to `.env` and uncomment the variables you want to override. 

If you'd prefer repo-adjacent output directories during development, as opposed to your home directory, this optional helper script will create them and set the corresponding environment variables for you:

```bash
source ./set_up_dev_env.sh
```

Configuration is managed by [pydantic-settings](https://docs.pydantic.dev/latest/concepts/pydantic_settings/) — environment variables override `.env` file values, and sensible defaults are provided where possible. See `orion/config.py` for the full list of settings.

### Running with Docker

ORION's Docker compose does require `ORION_STORAGE` and `ORION_GRAPHS` to be set on the host - docker-compose mounts those host paths into the container as volumes, so outputs persist on disc after the container exits. Either export them in your shell or put them in `.env`.

Build the image:

```bash
docker compose build
```

Build a graph using docker compose:

```bash
docker compose run orion orion-build Example_Graph
```

Run the ingest pipeline for a single data source:

```bash
docker compose run orion orion-ingest DrugCentral
```

See available data sources and options:

```bash
docker compose run orion orion-ingest -h
```

### Running with Helm

[Helm charts](https://github.com/RobokopU24/ORION/blob/master/helm/orion/) are available for running ORION on Kubernetes. See `helm/orion/values.yaml` for the full set of values, including `graphSpecUrl`, `graphSpecYaml`, and `optionalGraphSpec` for a variety of options for providing graph specs.

### Development

Install dev dependencies with [uv](https://docs.astral.sh/uv/):

```bash
uv sync --extra robokop --group dev
```

Run tests:

```bash
uv run pytest tests/
```

### Contributing

Contributions are welcome, see the [Contributor README](README-CONTRIBUTER.md).
