Metadata-Version: 2.4
Name: urgap
Version: 3.3.15
Project-URL: Homepage, https://github.com/urgap/urgap
Project-URL: Documentation, https://urgap.github.io/urgap/
Project-URL: Repository, https://github.com/urgap/urgap.git
Project-URL: Issues, https://github.com/urgap/urgap/issues
Project-URL: Changelog, https://github.com/urgap/urgap/blob/main/CHANGELOG.md
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX
Classifier: Operating System :: POSIX :: SunOS/Solaris
Classifier: Operating System :: Unix
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Topic :: Scientific/Engineering :: Chemistry
Requires-Python: >=3.11
Requires-Dist: click<9.0.0,>=8.3.1
Requires-Dist: jsonschema<5.0.0,>=4.26.0
Requires-Dist: networkx<4.0.0,>=3.6.1
Requires-Dist: packaging<26.0.0,>=25.0
Requires-Dist: requests<2.35,>=2.26
Requires-Dist: sqlalchemy<3.0.0,>=2.0.47
Provides-Extra: all
Requires-Dist: apache-beam<3.0.0,>=2.75.0; extra == 'all'
Requires-Dist: azure-identity<2.0.0,>=1.25.2; extra == 'all'
Requires-Dist: azure-keyvault-secrets<5.0.0,>=4.10.0; extra == 'all'
Requires-Dist: azure-monitor-opentelemetry<2.0.0,>=1.7.0; extra == 'all'
Requires-Dist: azure-servicebus<8.0.0,>=7.14.3; extra == 'all'
Requires-Dist: azure-storage-blob<13.0.0,>=12.28.0; extra == 'all'
Requires-Dist: azure-storage-file-datalake<13.0.0,>=12.22.0; extra == 'all'
Requires-Dist: azure-storage-file-share<13.0.0,>=12.24.0; extra == 'all'
Requires-Dist: betterproto==2.0.0b6; extra == 'all'
Requires-Dist: cloud-sql-python-connector~=1.0; extra == 'all'
Requires-Dist: fastapi<1.0.0,>=0.133.0; extra == 'all'
Requires-Dist: fastexcel<1.0.0,>=0.19.0; extra == 'all'
Requires-Dist: fastmcp<4.0.0,>=3.0.2; extra == 'all'
Requires-Dist: flask-wtf<2.0.0,>=1.2.2; extra == 'all'
Requires-Dist: flask<4.0.0,>=3.1.3; extra == 'all'
Requires-Dist: google-apitools<1.0.0,>=0.5.35; extra == 'all'
Requires-Dist: google-cloud-logging<4.0.0,>=3.13.0; extra == 'all'
Requires-Dist: google-cloud-secret-manager<3.0.0,>=2.26.0; extra == 'all'
Requires-Dist: google-cloud-storage<4.0.0,>=3.9.0; extra == 'all'
Requires-Dist: google-crc32c<2.0.0,>=1.8.0; extra == 'all'
Requires-Dist: httpx2<3.0.0,>=2.7.0; extra == 'all'
Requires-Dist: ipython<10.0.0,>=9.15.0; extra == 'all'
Requires-Dist: jinja2<4.0.0,>=3.1.6; extra == 'all'
Requires-Dist: mcp<2.0.0,>=1.26.0; extra == 'all'
Requires-Dist: nbformat<6.0.0,>=4.2.0; extra == 'all'
Requires-Dist: nbsphinx<1.0.0,>=0.9.8; extra == 'all'
Requires-Dist: openpyxl<4.0.0,>=3.1.5; extra == 'all'
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc<2.0.0,>=1.39.0; extra == 'all'
Requires-Dist: opentelemetry-sdk<2.0.0,>=1.39.0; extra == 'all'
Requires-Dist: pandas<4,>=2; extra == 'all'
Requires-Dist: pg8000~=1.29; extra == 'all'
Requires-Dist: plotly<7.0.0,>=6.9.0; extra == 'all'
Requires-Dist: polars<2.0.0,>=1.33.0; extra == 'all'
Requires-Dist: prefect-kubernetes<1.0.0,>=0.7.3; extra == 'all'
Requires-Dist: prefect<4.0.0,>=3.6.13; extra == 'all'
Requires-Dist: psycopg2-binary<3.0.0,>=2.9.12; extra == 'all'
Requires-Dist: pyarrow<25.0.0,>=18.1.0; extra == 'all'
Requires-Dist: pygithub<3.0.0,>=2.8.1; extra == 'all'
Requires-Dist: pymongo<5.0.0,>=3.8.0; extra == 'all'
Requires-Dist: pysmb<2.0.0,>=1.2.13; extra == 'all'
Requires-Dist: pytest-asyncio<2.0.0,>=1.0; extra == 'all'
Requires-Dist: pytest<10.0.0,>=8.4; extra == 'all'
Requires-Dist: pyvis<1.0.0,>=0.3.2; extra == 'all'
Requires-Dist: sphinx-rtd-theme<4.0.0,>=3.1.0; extra == 'all'
Requires-Dist: sphinx<10.0.0,>=9.0.4; extra == 'all'
Requires-Dist: tqdm<5.0.0,>=4.67.3; extra == 'all'
Requires-Dist: uvicorn<1.0.0,>=0.40.0; extra == 'all'
Requires-Dist: xlsxwriter<4.0.0,>=3.2.9; extra == 'all'
Provides-Extra: cloud
Requires-Dist: apache-beam<3.0.0,>=2.75.0; extra == 'cloud'
Requires-Dist: azure-identity<2.0.0,>=1.25.2; extra == 'cloud'
Requires-Dist: azure-keyvault-secrets<5.0.0,>=4.10.0; extra == 'cloud'
Requires-Dist: azure-monitor-opentelemetry<2.0.0,>=1.7.0; extra == 'cloud'
Requires-Dist: azure-servicebus<8.0.0,>=7.14.3; extra == 'cloud'
Requires-Dist: azure-storage-blob<13.0.0,>=12.28.0; extra == 'cloud'
Requires-Dist: azure-storage-file-datalake<13.0.0,>=12.22.0; extra == 'cloud'
Requires-Dist: azure-storage-file-share<13.0.0,>=12.24.0; extra == 'cloud'
Requires-Dist: betterproto==2.0.0b6; extra == 'cloud'
Requires-Dist: cloud-sql-python-connector~=1.0; extra == 'cloud'
Requires-Dist: fastmcp<4.0.0,>=3.0.2; extra == 'cloud'
Requires-Dist: google-apitools<1.0.0,>=0.5.35; extra == 'cloud'
Requires-Dist: google-cloud-logging<4.0.0,>=3.13.0; extra == 'cloud'
Requires-Dist: google-cloud-secret-manager<3.0.0,>=2.26.0; extra == 'cloud'
Requires-Dist: google-cloud-storage<4.0.0,>=3.9.0; extra == 'cloud'
Requires-Dist: google-crc32c<2.0.0,>=1.8.0; extra == 'cloud'
Requires-Dist: mcp<2.0.0,>=1.26.0; extra == 'cloud'
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc<2.0.0,>=1.39.0; extra == 'cloud'
Requires-Dist: opentelemetry-sdk<2.0.0,>=1.39.0; extra == 'cloud'
Requires-Dist: pg8000~=1.29; extra == 'cloud'
Requires-Dist: prefect-kubernetes<1.0.0,>=0.7.3; extra == 'cloud'
Requires-Dist: prefect<4.0.0,>=3.6.13; extra == 'cloud'
Requires-Dist: pysmb<2.0.0,>=1.2.13; extra == 'cloud'
Requires-Dist: pyvis<1.0.0,>=0.3.2; extra == 'cloud'
Provides-Extra: databases
Requires-Dist: psycopg2-binary<3.0.0,>=2.9.12; extra == 'databases'
Requires-Dist: pymongo<5.0.0,>=3.8.0; extra == 'databases'
Provides-Extra: dev
Requires-Dist: apache-beam<3.0.0,>=2.75.0; extra == 'dev'
Requires-Dist: azure-identity<2.0.0,>=1.25.2; extra == 'dev'
Requires-Dist: azure-keyvault-secrets<5.0.0,>=4.10.0; extra == 'dev'
Requires-Dist: azure-monitor-opentelemetry<2.0.0,>=1.7.0; extra == 'dev'
Requires-Dist: azure-servicebus<8.0.0,>=7.14.3; extra == 'dev'
Requires-Dist: azure-storage-blob<13.0.0,>=12.28.0; extra == 'dev'
Requires-Dist: azure-storage-file-datalake<13.0.0,>=12.22.0; extra == 'dev'
Requires-Dist: azure-storage-file-share<13.0.0,>=12.24.0; extra == 'dev'
Requires-Dist: betterproto==2.0.0b6; extra == 'dev'
Requires-Dist: cloud-sql-python-connector~=1.0; extra == 'dev'
Requires-Dist: fastapi<1.0.0,>=0.133.0; extra == 'dev'
Requires-Dist: fastexcel<1.0.0,>=0.19.0; extra == 'dev'
Requires-Dist: fastmcp<4.0.0,>=3.0.2; extra == 'dev'
Requires-Dist: flask-wtf<2.0.0,>=1.2.2; extra == 'dev'
Requires-Dist: flask<4.0.0,>=3.1.3; extra == 'dev'
Requires-Dist: google-apitools<1.0.0,>=0.5.35; extra == 'dev'
Requires-Dist: google-cloud-logging<4.0.0,>=3.13.0; extra == 'dev'
Requires-Dist: google-cloud-secret-manager<3.0.0,>=2.26.0; extra == 'dev'
Requires-Dist: google-cloud-storage<4.0.0,>=3.9.0; extra == 'dev'
Requires-Dist: google-crc32c<2.0.0,>=1.8.0; extra == 'dev'
Requires-Dist: httpx2<3.0.0,>=2.7.0; extra == 'dev'
Requires-Dist: ipython<10.0.0,>=9.15.0; extra == 'dev'
Requires-Dist: jinja2<4.0.0,>=3.1.6; extra == 'dev'
Requires-Dist: mcp<2.0.0,>=1.26.0; extra == 'dev'
Requires-Dist: nbformat<6.0.0,>=4.2.0; extra == 'dev'
Requires-Dist: openpyxl<4.0.0,>=3.1.5; extra == 'dev'
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc<2.0.0,>=1.39.0; extra == 'dev'
Requires-Dist: opentelemetry-sdk<2.0.0,>=1.39.0; extra == 'dev'
Requires-Dist: pandas<4,>=2; extra == 'dev'
Requires-Dist: pg8000~=1.29; extra == 'dev'
Requires-Dist: plotly<7.0.0,>=6.9.0; extra == 'dev'
Requires-Dist: polars<2.0.0,>=1.33.0; extra == 'dev'
Requires-Dist: prefect-kubernetes<1.0.0,>=0.7.3; extra == 'dev'
Requires-Dist: prefect<4.0.0,>=3.6.13; extra == 'dev'
Requires-Dist: psycopg2-binary<3.0.0,>=2.9.12; extra == 'dev'
Requires-Dist: pyarrow<25.0.0,>=18.1.0; extra == 'dev'
Requires-Dist: pygithub<3.0.0,>=2.8.1; extra == 'dev'
Requires-Dist: pymongo<5.0.0,>=3.8.0; extra == 'dev'
Requires-Dist: pysmb<2.0.0,>=1.2.13; extra == 'dev'
Requires-Dist: pytest-asyncio<2.0.0,>=1.0; extra == 'dev'
Requires-Dist: pytest<10.0.0,>=8.4; extra == 'dev'
Requires-Dist: pyvis<1.0.0,>=0.3.2; extra == 'dev'
Requires-Dist: tqdm<5.0.0,>=4.67.3; extra == 'dev'
Requires-Dist: uvicorn<1.0.0,>=0.40.0; extra == 'dev'
Requires-Dist: xlsxwriter<4.0.0,>=3.2.9; extra == 'dev'
Provides-Extra: docs
Requires-Dist: ipython<10.0.0,>=9.15.0; extra == 'docs'
Requires-Dist: nbsphinx<1.0.0,>=0.9.8; extra == 'docs'
Requires-Dist: sphinx-rtd-theme<4.0.0,>=3.1.0; extra == 'docs'
Requires-Dist: sphinx<10.0.0,>=9.0.4; extra == 'docs'
Provides-Extra: github
Requires-Dist: pygithub<3.0.0,>=2.8.1; extra == 'github'
Provides-Extra: server
Requires-Dist: fastapi<1.0.0,>=0.133.0; extra == 'server'
Requires-Dist: fastmcp<4.0.0,>=3.0.2; extra == 'server'
Requires-Dist: flask-wtf<2.0.0,>=1.2.2; extra == 'server'
Requires-Dist: flask<4.0.0,>=3.1.3; extra == 'server'
Requires-Dist: jinja2<4.0.0,>=3.1.6; extra == 'server'
Requires-Dist: mcp<2.0.0,>=1.26.0; extra == 'server'
Requires-Dist: uvicorn<1.0.0,>=0.40.0; extra == 'server'
Provides-Extra: tabular
Requires-Dist: fastexcel<1.0.0,>=0.19.0; extra == 'tabular'
Requires-Dist: openpyxl<4.0.0,>=3.1.5; extra == 'tabular'
Requires-Dist: pandas<4,>=2; extra == 'tabular'
Requires-Dist: polars<2.0.0,>=1.33.0; extra == 'tabular'
Requires-Dist: pyarrow<25.0.0,>=18.1.0; extra == 'tabular'
Requires-Dist: tqdm<5.0.0,>=4.67.3; extra == 'tabular'
Requires-Dist: xlsxwriter<4.0.0,>=3.2.9; extra == 'tabular'
Provides-Extra: viz
Requires-Dist: ipython<10.0.0,>=9.15.0; extra == 'viz'
Requires-Dist: nbformat<6.0.0,>=4.2.0; extra == 'viz'
Requires-Dist: plotly<7.0.0,>=6.9.0; extra == 'viz'
Description-Content-Type: text/x-rst

Urgap - node wrapping framework
===============================

Urgap is a cloud-native framework for file-based data engineering, containing abstraction layers for data and meta data, extensive re-run skipping logic and data versioning. Urgap can be incorporated with any scheduling/pipelining tool making pipeline development independent from business logic and data storage, while offering standardized logging and execution, which makes monitoring and debugging easy.

Urgap gives us the governance constraints required for a decentralized data domain autonomy as Urgap will enforce shared common data IO for storage, a common meta data capturing process in form of an interface thus can be plugged into any existing processes and finally global data lineages. 

|build-status-azure|

.. |build-status-azure| image:: https://dev.azure.com/DevOps-RD/RD-DSO/_apis/build/status%2Fgsk-tech.urgap?repoName=gsk-tech%2Furgap&branchName=refs%2Fpull%2F388%2Fmerge
   :target: https://dev.azure.com/DevOps-RD/RD-DSO/_build?definitionId=13513
   :alt: ADO CI status

Learn More
----------

Watch our introduction talk **urgap - unified resource governance and data provenance** by Christian Fufezan to get a comprehensive overview of urgap's design and capabilities:

.. image:: https://img.youtube.com/vi/63pYK1xZPx8/0.jpg
   :target: https://www.youtube.com/watch?v=63pYK1xZPx8
   :alt: Watch the video
   :width: 560

How to Setup
------------

Prerequisites
~~~~~~~~~~~~~

We recommend using a virtual environment for Python projects. This guide uses `uv` for dependency management.

Installation
~~~~~~~~~~~~

**Basic Installation** (local file system access only):

.. code-block:: bash

    uv sync

**With Cloud Storage Support:**

.. code-block:: bash

    uv sync --extra cloud

**With All Optional Dependencies:**

.. code-block:: bash

    uv sync --all-extras

Available extras include:

* ``server``: FastAPI/Flask/uvicorn + MCP server for ``uctl run``
* ``tabular``: pandas/polars/pyarrow and Excel engines for tabular unodes
* ``viz``: plotly-based reporting and visualizer unodes
* ``github``: GitHub-backed file IO
* ``cloud``: Azure and Google Cloud storage backends
* ``databases``: MongoDB and PostgreSQL backends
* ``docs``: Sphinx documentation build
* ``dev``: test/lint tooling (pytest, black) plus all runtime extras
* ``all``: All optional dependencies

Running Tests
~~~~~~~~~~~~~

Install test dependencies:

.. code-block:: bash

    uv sync --extra dev

Run the test suite:

.. code-block:: bash

    uv run pytest tests

Quickstart: Writing Your First Pipeline
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The best way to learn urgap is through a complete example. Check out the end-to-end filter CSV pipeline:

* **Location:** `tests/integrationtests/end2end/test_filter_csv_pipeline.py`
* **What it demonstrates:** Complete pipeline setup, node configuration, and execution
* **Requirements:** Everything needed to run this example is included in the repository

This example can be run entirely on your local machine without any external dependencies.

To run the example:

.. code-block:: bash

    pytest tests/integrationtests/end2end/test_filter_csv_pipeline.py

Documentation
--------------

Please use sphinx in the docs folder


.. note::

    Currently CI does not include pushing the documentation to readthedocs, therefore please 
    #. checkout the repo
    #. pip install -e .
    #. cd docs
    #. make html
    #. open docs/build/index.html
