Metadata-Version: 2.4
Name: cvcdocdb
Version: 1.5.0
Summary: Graph-based document representation library with Neo4j and NetworkX backends
Home-page: https://github.com/CVC-DAG/cvcdocdb
Author: Oriol Ramos Terrades
Author-email: oriolrt@cvc.uab.cat
License: GPL-3.0-or-later
Project-URL: Documentation, https://cvc-dag.github.io/cvcdocdb/
Project-URL: Source, https://github.com/CVC-DAG/cvcdocdb
Project-URL: Tracker, https://github.com/CVC-DAG/cvcdocdb/issues
Keywords: document representation,knowledge graph,neo4j,memgraph,networkx,sparql,jena,rdf,document analysis
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Description-Content-Type: text/x-rst
License-File: LICENSE
Requires-Dist: neo4j
Requires-Dist: networkx
Requires-Dist: numpy
Requires-Dist: filelock
Requires-Dist: tqdm
Provides-Extra: rdf
Requires-Dist: rdflib; extra == "rdf"
Requires-Dist: pyyaml; extra == "rdf"
Provides-Extra: schema
Requires-Dist: pyyaml; extra == "schema"
Provides-Extra: vector
Requires-Dist: hnswlib; extra == "vector"
Provides-Extra: torch
Requires-Dist: torch; extra == "torch"
Requires-Dist: torch_geometric; extra == "torch"
Provides-Extra: graphrag
Requires-Dist: neo4j-graphrag<2,>=1.21; extra == "graphrag"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: license-file
Dynamic: project-url
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: summary

Document Representation Models
===============================

Archives, libraries, and document collections are rarely flat files: documents have internal structure -- pages, sections, regions, entities -- and rich relationships to other documents, people, places, and events. A **Document Representation Model (DRM)** captures this structure as a graph, where nodes stand for the objects that make up (or are described by) a document, and typed edges capture how those objects relate to, contain, or depend on one another. This graph-first view makes document content queryable, composable, and reusable across archival, historical, and document-analysis workflows.

CVCDocDB
--------

**CVCDocDB** is a Python library developed by the Document Analysis Group (DAG) at the `Centre de Visió per Computador <https://www.cvc.uab.es/>`_ (CVC), within the framework of the **SUKIDI** project, to represent the contents of documents according to a Document Representation Model. It offers a graph-based API with two interchangeable backends -- a persistent Neo4j store and an in-memory NetworkX store for testing and tutorials -- together with semantic entity definitions, WeakNode hierarchies with cascade delete, foreign key validation, vector search, and ready-to-run example datasets for getting started quickly.

Features
--------

* **Two backends**: Full Neo4j integration (``Neo4jGraph``, targeting Neo4j Community Edition; opt-in Enterprise-only features, see *Neo4j editions* below) a Memgraph backend (``MemgraphGraph``, same propagation policy as Neo4j), an Apache Jena SPARQL backend (``JenaGraph``, RDF 1.2 in Fuseki), or in-memory NetworkX (``NetworkXGraph``) for testing and tutorials
* **WeakNode hierarchy**: Child entities with composite primary keys and automatic cascade delete through parent-child edges. Inserting a WeakNode inserts all its ancestors. A chain has at most ``MAX_WEAK_CHAIN_DEPTH`` = 3 nodes (the root plus two levels, e.g. ``Document → Section → Page``). Deeper WeakNodes emit a ``WeakNodeDepthWarning``, and the next major version will give them an automatic surrogate key
* **ON DELETE strategies**: CASCADE, RESTRICT, SET NULL -- choose the deletion semantics that fit your use case
* **Semantic entities** (optional module, see below): Domain-specific node types such as ``IndividuPadro``, ``LlocPadro``, and ``Fotografia``
* **FK validation**: Foreign key constraints on relations prevent dangling references
* **Query and filtering**: Secondary index on scalar properties, multi-filter search with intersection/union, debug snapshots
* **Vector search (NetworkX only)**: HNSW-based ANN indexing on node properties with ``cosine``, ``l2``, and ``ip`` distance spaces
* **RDF/OWL ontology conversion**: Generate Python entity classes from RDF/OWL ontologies (RiC-O, etc.)
* **Backend-to-backend migration**: ``cvcdocdb.migration.migrate()`` copies an entire graph -- nodes, edges, and vector indexes -- between any two ``GraphStore`` backends
* **PyTorch / PyTorch Geometric dataloader**: ``cvcdocdb.torch_dataloader`` streams a graph into PyG-ready tensors for node embedding models (``MetaPath2Vec``) and link prediction

Installation
------------

Install from PyPI::

    pip install cvcdocdb

Optional features are installed as extras, e.g. ``pip install "cvcdocdb[rdf,vector]"``:

* ``rdf`` -- RDF/OWL ontology import (``cvcdocdb.rdf_schema``)
* ``schema`` -- entity-class generation from a YAML schema (``cvcdocdb.schema_gen``)
* ``vector`` -- vector indexes on ``NetworkXGraph``
* ``torch`` -- PyTorch / PyG data loaders (``cvcdocdb.torch_dataloader``)
* ``graphrag`` -- ``Text2Cypher`` on a ``Neo4jGraph`` (Python >= 3.10). Not needed on ``MemgraphGraph`` or ``NetworkXGraph``

Or install from source in development mode (``requirements.txt`` adds the
documentation/notebook tools and some optional dependencies used for development)::

    git clone https://github.com/CVC-DAG/cvcdocdb.git
    cd cvcdocdb
    pip install -e . -r requirements.txt

Register the recommended Jupyter kernel for tutorials::

    python -m ipykernel install --user --name cvcdocdb --display-name "Python (cvcdocdb)"

Quick Start
-----------

.. code:: python

    from cvcdocdb import NetworkXGraph, Node, WeakNode

    # In-memory backend -- no database required
    graph = NetworkXGraph()

    # Create a document hierarchy
    doc = Node(pk={"doc": "DOC-001"}, main_label="Document")
    graph.insertNode(doc)

    section = WeakNode(parent=doc, pk={"section": 1}, main_label="Section")
    graph.insertNode(section, insert_parent=True)

    page = WeakNode(parent=section, pk={"page": 1}, main_label="Page")
    graph.insertNode(page, insert_parent=True)

    # Query the graph
    print("Nodes:", graph.get_node_ids())
    print("Edges:", graph.get_edges())
    graph.close()

Optional modules
----------------

The core (``Node``/``WeakNode``/``Relation``, ``GraphStore``, ``migrate()`` and
the backends) is what ``import cvcdocdb`` loads. These modules build on it and
are **optional**: they aren't imported by ``import cvcdocdb``, and some need an
extra.

* ``cvcdocdb.drm_entities`` (no extra) -- DRM semantic entities: ``IndividuPadro``, ``LlocPadro``, ``Fotografia``...
* ``cvcdocdb.rico_entities`` (no extra) -- RiC-O archival entities generated from the ontology
* ``cvcdocdb.rdf_schema`` (``[rdf]``) -- RDF/OWL ontology to YAML schema to entity classes
* ``cvcdocdb.schema_gen`` (``[schema]``) -- Entity classes from a YAML schema
* ``cvcdocdb.torch_dataloader`` (``[torch]``) -- Stream a graph into PyTorch / PyTorch Geometric
* ``cvcdocdb.text2query`` (as the translator) -- Natural-language questions on any backend (SPARQL on Jena, Cypher elsewhere)
* ``cvcdocdb.text2cypher`` (``[graphrag]``, on Neo4j only) -- Natural-language questions to Cypher
* ``cvcdocdb.text2sparql`` (no extra) -- Natural-language questions to SPARQL (``JenaGraph``)

::

    from cvcdocdb.drm_entities import IndividuPadro   # not: from cvcdocdb import IndividuPadro

``from cvcdocdb import IndividuPadro`` (and the other DRM entities) still works
but emits a ``DeprecationWarning``; it will be removed in the next major version.

Tutorial Notebooks
------------------

Runnable Jupyter notebooks in ``docs/tutorials/notebooks/``. Each notebook installs the package automatically from the latest release when run.

You can also view them rendered in the `hosted documentation <https://cvc-dag.github.io/cvcdocdb/>`_.

General (any backend)
~~~~~~~~~~~~~~~~~~~~~

These use only the common ``GraphStore`` API, so they run unchanged on
``NetworkXGraph``, ``Neo4jGraph`` and ``MemgraphGraph``:

* ``intro_basics`` -- Minimal end-to-end workflow: insert nodes, create WeakNode hierarchies
* ``querying_and_filtering`` -- Query operations: ``get_node()``, ``find_nodes()``, property filtering
* ``weaknodes_interactive`` -- Build hierarchies with an interactive widget panel
* ``delete_strategies`` -- Compare CASCADE, RESTRICT, SET NULL strategies
* Datasets, each loaded into NetworkX and Neo4j: ``karate_club``, ``movies``, ``game_of_thrones``, ``bibliography_openalex``

Backend-specific
~~~~~~~~~~~~~~~~

* **NetworkX**: ``vector_search`` -- HNSW vector indexing and nearest-neighbor search (``NetworkXGraph`` only)
* **Neo4j**: ``propagation_demo`` -- full propagation workflow on a real Neo4j database (also as a script: ``python -m cvcdocdb.exemples.demo_propagation``)
* **Memgraph**: no specific notebook; every general one runs on ``MemgraphGraph`` as is

Optional modules
~~~~~~~~~~~~~~~~

* ``cvcdocdb.rdf_schema`` / ``cvcdocdb.schema_gen``: ``generating_classes_from_owl`` -- Generate Python entity classes from RDF/OWL ontologies
* ``cvcdocdb.rico_entities``: ``ric_o_demo`` (Neo4j) and ``ric_o_networkx_demo`` (NetworkX) -- the RiC-O model on each backend
* ``cvcdocdb.torch_dataloader``: ``torch_dataloader_bibliography`` -- PyTorch/PyG dataloader, MetaPath2Vec training and link prediction

RDF/OWL Ontology Conversion
---------------------------

Generate Python entity classes from RDF/OWL ontologies in one step::

    from cvcdocdb.rdf_schema import download_ontology_and_convert

    # Downloads, converts to YAML, and generates Python classes
    output_path = download_ontology_and_convert(
        "https://raw.githubusercontent.com/ICA-EGAD/RiC-O/master/ontology/current-version/RiC-O_1-1.rdf",
        "rico",
        output_dir="cvcdocdb/"
    )
    # Generates cvcdocdb/rico_entities.py (677 classes from RiC-O)

Step by step::

    from cvcdocdb.rdf_schema import download_ontology, rdf_to_yaml
    from cvcdocdb.schema_gen import generate_classes

    # 1. Download ontology
    ont_path = download_ontology(url, output_dir="ontologies/")

    # 2. Convert RDF to YAML DRM
    yaml_str = rdf_to_yaml(ont_path, "my_db")

    # 3. Generate Python classes
    py_source = generate_classes(yaml_str)

    # 4. Write file
    with open("cvcdocdb/entities_my_db.py", "w") as f:
        f.write(py_source)

The pipeline maps OWL constructs to DRM:

* ``owl:Class`` -- Node label
* ``rdfs:subClassOf`` -- ``WeakNode`` hierarchy (parent)
* ``owl:DatatypeProperty`` -- Node properties
* ``owl:ObjectProperty`` -- Relationships
* ``owl:hasKey`` -- Primary key fields
* ``rdfs:comment`` -- Class docstring

Example Dataset Loaders (cvcdocdb.exemples)
-------------------------------------------

The package includes ready-to-run loaders for common graph domains:

* ``cvcdocdb.exemples.networkx_karate`` -- Karate Club graph (NetworkX classic)
* ``cvcdocdb.exemples.networkx_bibliografia`` -- Bibliographic references from OpenAlex
* ``cvcdocdb.exemples.neo4j_movies`` -- Movie-domain graph
* ``cvcdocdb.exemples.neo4j_got`` -- Game of Thrones character-house graph

Command-line loader
~~~~~~~~~~~~~~~~~~~

.. code:: bash

    python -m cvcdocdb.exemples --dataset karate --backend networkx
    python -m cvcdocdb.exemples --dataset all --backend both --quiet

Programmatic usage
~~~~~~~~~~~~~~~~~~

.. code:: python

    from cvcdocdb import NetworkXGraph
    from cvcdocdb.exemples import load_karate_club, load_bibliografia_openalex

    graph = NetworkXGraph()
    print(load_karate_club(graph))
    print(load_bibliografia_openalex(graph, query="graph database", per_page=15))
    graph.close()

Memgraph backend
----------------

``MemgraphGraph`` stores the graph in `Memgraph <https://memgraph.com/>`_. It
subclasses ``Neo4jGraph`` (Memgraph speaks Bolt and Cypher, and uses the same
``neo4j`` driver), so the API and the **change-propagation policy** are
identical to the Neo4j backend:

* **Insert**: a WeakNode inserts its parent, and the parent→child edge
  carries ``_propagate=TRUE``. A missing parent, child keys that don't match
  the parent's, or a duplicate key are refused. Dependencies become ``Valor``
  nodes.
* **Update**: ``update=True`` merges attributes. ``replace=True`` deletes the
  old node with propagation (its WeakNode descendants too) and recreates it.
* **Delete**: RESTRICT by default, ``propagation=True`` for WeakNode
  descendants, ``detach=True`` (CASCADE), or ``on_delete="set_null"``.
* **Relations**: FK validation of both endpoints, plus ``update``/``replace``.

.. code-block:: python

    from cvcdocdb import MemgraphGraph

    graph = MemgraphGraph("bolt://localhost:7687", "", "")  # no auth by default

``test/test_memgraph_graph.py`` runs the same propagation scenarios on Neo4j
and Memgraph and checks that both leave the same graph and raise the same
errors. The only Memgraph-specific code is pk indexes (``SHOW INDEX INFO``,
``CREATE INDEX ON :Label(props)``) and label/relationship-type listing for
``schema_yaml()``. The Neo4j Enterprise features are not available on
Memgraph (``EnterpriseFeatureError``), and neither is ``drop_constraint()``.
Tested with Memgraph 3.13 (Community).

Apache Jena backend (SPARQL)
----------------------------

``JenaGraph`` stores the graph as RDF 1.2 in `Apache Jena Fuseki <https://jena.apache.org/documentation/fuseki2/>`_
(or another SPARQL 1.2 store), with the same API and behaviour as every other
backend:

.. code-block:: python

    from cvcdocdb import JenaGraph, Node

    graph = JenaGraph("http://localhost:3030/ds")          # a Fuseki dataset
    graph.insertNode(Node(pk={"doc": "D1"}, main_label="Document", title="Padró"))
    graph.query("PREFIX label: <urn:cvcdocdb:label/> PREFIX prop: <urn:cvcdocdb:prop/> "
                "SELECT ?t WHERE { ?d a label:Document ; prop:title ?t }")   # [{'t': 'Padró'}]

* **Same behaviour:** it reuses the ``NetworkXGraph`` logic, which is identical
  to ``Neo4jGraph``'s, on an in-memory copy of the graph. The graph has to fit
  in RAM.
* **Atomic writes:** every mutating call, or a whole ``batch()``, is sent as
  one atomic SPARQL Update with only what changed.
* **Concurrent writes:** a version stamp detects writes from other clients
  and raises ``ConcurrentModificationError``; nothing is applied.
* **Querying:** ``query()`` runs SPARQL strings on the server. SPARQL Update is
  refused, so writes always go through the API. Dict filters and Cypher run
  on the copy.
* **RDF layout:** nodes are ``<urn:cvcdocdb:node/ID>``, with ``a label:…`` and
  ``prop:…`` values. Relationships are ``rel:…`` triples, and their properties
  are RDF 1.2 annotations (``?a rel:X ?b {| prop:p ?v |}``).
* **Isolation:** ``namespace`` and ``graph_iri`` keep it apart from other data
  in the dataset.
* **Natural language:** ``Text2SPARQL``, or ``Text2Query`` for any backend.

Tested with Apache Jena Fuseki 6.2.0. Vector indexes are not supported (as on Neo4j).

Neo4j editions: Community (default) and Enterprise
--------------------------------------------------

CVCDocDB targets **Neo4j Community Edition**. It is the edition the test suite
and CI run on (``neo4j:5-community``), and ``Neo4jGraph`` uses it by default
(``edition="community"``). Everything described above works on both Community
and Enterprise.

A few Neo4j **Enterprise-only** features are available as an opt-in mode
(see ``cvcdocdb.neo4j_enterprise``)::

    graph = Neo4jGraph(url, user, password, edition="enterprise")  # emits a UserWarning
    graph.create_node_key_constraint("Document", ["doc"])           # NODE KEY
    graph.create_property_existence_constraint("Document", "title") # IS NOT NULL
    graph.create_property_type_constraint("Document", "year", "INTEGER")  # Neo4j 5.9+
    graph.create_database("projecte1")                              # multiple databases
    graph.drop_database("projecte1")

In the default Community mode these methods raise ``EnterpriseFeatureError``
without contacting the server. In Enterprise mode, every call also checks that
the server really is Enterprise (``graph.server_edition()``).

.. warning::

   The Enterprise features require a valid **Neo4j Enterprise license** and
   are **not fully tested**. The regular suite only checks the generated
   Cypher and the Community-mode guards. The tests against a real Enterprise
   server only run when ``NEO4J_ENTERPRISE_URL`` is set, and CI does not set
   it. No constraint is created automatically. A ``NODE KEY`` on a label used
   with several pk shapes rejects the nodes of the other shapes.

Configuration
-------------

CVCDocDB uses environment variables for Neo4j connections. Multiple targets are supported via the ``NEO4J_TARGET`` selector::

    # Default target
    export NEO4J_DEV_URL=bolt://dev-host:7687
    export NEO4J_DEV_USER=neo4j
    export NEO4J_DEV_PASSWORD=your_dev_password
    export NEO4J_DEV_DATABASE=neo4j

    # Custom target
    export NEO4J_TARGET=LOCAL
    export NEO4J_LOCAL_URL=bolt://localhost:7687
    export NEO4J_LOCAL_USER=neo4j
    export NEO4J_LOCAL_PASSWORD=your_password
    export NEO4J_LOCAL_DATABASE=neo4j

Running Tests
-------------

.. code:: bash

    pip install -r requirements-test.txt
    python -m pytest test/ -v

Three test levels:

* **Unit** (``-m unit``) -- fast, no graph store
* **Integration** (``-m integration``) -- NetworkXGraph (in-memory)
* **Neo4j** (``-m slow``) -- requires a real Neo4j connection. If none is
  reachable and Docker is available, a disposable ``neo4j:5-community``
  container is started automatically; otherwise these tests auto-skip.
  See ``test/README.md`` for details and the manual ``docker-compose.neo4j.yml`` option.

Skip Neo4j tests: ``pytest test/ -v -m "not slow"``

Documentation
-------------

* **Hosted docs**: https://cvc-dag.github.io/cvcdocdb/
* **Source docs**: ``docs/`` -- Sphinx documentation source

Generate HTML docs with Sphinx::

    cd docs
    sphinx-build -b html . _build/html

Authors and Contributors
------------------------

* Oriol Ramos Terrades
* Jialuo Chen
* Adrià Molina

Acknowledgements
----------------

This work has been partially supported by the Spanish projects PID2021-126808OB-I00 and PID2024-157778OB-I00, Ministerio de Ciencia e Innovación, the Departament de Cultura of the Generalitat de Catalunya, and the CERCA Program / Generalitat de Catalunya. Adrià Molina is funded with the PRE2022-101575 grant provided by MCIN / AEI / 10.13039 / 501100011033 and by the European Social Fund (FSE+).
