Metadata-Version: 2.5
Name: docling-pipelines
Version: 2.0.0
Summary: Docling Pipelines — a composable document processing pipeline framework built on Docling.
Project-URL: Homepage, https://github.com/IBM/docling-pipelines
Project-URL: Documentation, https://github.com/IBM/docling-pipelines/blob/main/docs/README.md
Project-URL: Source, https://github.com/IBM/docling-pipelines
Project-URL: Bug Tracker, https://github.com/IBM/docling-pipelines/issues
Maintainer-email: Jojo Joseph <jo-in@ibm.com>, Aditya Rohan Singh <adityars@ibm.com>, Nipun Bayas <nsbayas@ibm.com>, Paul Baby <paulbaby@ibm.com>, Docling Pipeline Team <docling_pipeline_team-dg@ibm.com>
License-File: LICENSE
Requires-Python: <3.13,>=3.12
Requires-Dist: accelerate==1.13.0
Requires-Dist: aiohttp==3.14.3
Requires-Dist: alembic==1.18.4
Requires-Dist: azure-storage-blob==12.28.0
Requires-Dist: babel==2.18.0
Requires-Dist: boto3==1.42.55
Requires-Dist: botocore==1.42.55
Requires-Dist: box-sdk-gen==1.17.0
Requires-Dist: cryptography==50.0.0
Requires-Dist: data-prep-toolkit-transforms==1.1.8
Requires-Dist: datatrove==0.10.0
Requires-Dist: datefinder==1.0.0
Requires-Dist: defusedxml==0.7.1
Requires-Dist: docling[rapidocr]==2.130.0
Requires-Dist: dropbox==12.2.1
Requires-Dist: duckdb==1.3.2
Requires-Dist: fastapi==0.141.1
Requires-Dist: fasttext-wheel==0.9.2
Requires-Dist: filelock==3.24.2
Requires-Dist: ftfy==6.3.1
Requires-Dist: google-api-python-client==2.190.0
Requires-Dist: google-auth-httplib2==0.3.0
Requires-Dist: google-auth-oauthlib==1.2.4
Requires-Dist: google-cloud-storage==3.9.0
Requires-Dist: google-re2==1.1.20251105
Requires-Dist: griffe==1.15.0
Requires-Dist: httpx==0.28.1
Requires-Dist: ibm-cos-sdk==2.14.3
Requires-Dist: ibm-watsonx-ai==1.5.10
Requires-Dist: langchain-classic==1.0.7
Requires-Dist: langchain-community==0.4.1
Requires-Dist: langchain-core==1.4.6
Requires-Dist: langchain-experimental==0.4.1
Requires-Dist: langchain-text-splitters==1.1.2
Requires-Dist: langchain==1.3.9
Requires-Dist: langdetect==1.0.9
Requires-Dist: langsmith==0.8.18
Requires-Dist: litellm==1.88.6
Requires-Dist: lxml==6.1.0
Requires-Dist: mako==1.4.2
Requires-Dist: mcp<2.0,>=1.0
Requires-Dist: milvus-lite==3.2.1
Requires-Dist: mlx==0.32.2; sys_platform == 'darwin'
Requires-Dist: msal==1.37.0
Requires-Dist: nltk==3.10.3
Requires-Dist: o365==2.1.9
Requires-Dist: ollama==0.6.1
Requires-Dist: openai==2.33.0
Requires-Dist: opensearch-py==3.1.0
Requires-Dist: orjson==3.11.7
Requires-Dist: pandas==2.3.3
Requires-Dist: passlib[bcrypt]==1.7.4
Requires-Dist: pillow==12.3.0
Requires-Dist: polars==1.32.3
Requires-Dist: prefect==3.8.3
Requires-Dist: psycopg2-binary==2.9.12
Requires-Dist: pyarrow==24.0.0
Requires-Dist: pyasn1==0.6.4
Requires-Dist: pydantic-settings==2.14.2
Requires-Dist: pydantic==2.12.5
Requires-Dist: pygments==2.20.0
Requires-Dist: pyiceberg-core==0.7.0
Requires-Dist: pyiceberg[glue]==0.9.1
Requires-Dist: pyjwt==2.15.0
Requires-Dist: pymilvus==2.6.13
Requires-Dist: pyopenssl==26.4.0
Requires-Dist: pypdf==6.19.0
Requires-Dist: pyphen==0.17.2
Requires-Dist: python-dotenv==1.2.2
Requires-Dist: python-jose[cryptography]==3.5.0
Requires-Dist: python-ldap==3.4.5
Requires-Dist: python-multipart==0.0.32
Requires-Dist: pyyaml==6.0.3
Requires-Dist: requests==2.33.0
Requires-Dist: sentence-transformers==5.6.0
Requires-Dist: setuptools==83.0.0
Requires-Dist: spacy>=3.7.4
Requires-Dist: sqlalchemy==2.0.49
Requires-Dist: sqlglot==27.13.2
Requires-Dist: sqlmodel==0.0.38
Requires-Dist: tabulate==0.9.0
Requires-Dist: tenacity==8.3.0
Requires-Dist: toml==0.10.2
Requires-Dist: torch==2.13.0
Requires-Dist: transformers<5.18,>=5.17
Requires-Dist: unicategories==0.1.2
Requires-Dist: unicodedataplus==16.0.0.post1
Requires-Dist: urllib3==2.8.0
Requires-Dist: uvicorn[standard]==0.40.0
Provides-Extra: benchmark
Requires-Dist: matplotlib==3.11.0; extra == 'benchmark'
Requires-Dist: psutil==7.2.2; extra == 'benchmark'
Requires-Dist: scipy<1.19.0,>=1.13.0; extra == 'benchmark'
Provides-Extra: cpu
Requires-Dist: torch; extra == 'cpu'
Requires-Dist: torchvision; extra == 'cpu'
Provides-Extra: dev
Requires-Dist: black==26.3.1; extra == 'dev'
Requires-Dist: detect-secrets==1.5.0; extra == 'dev'
Requires-Dist: faker==40.8.0; extra == 'dev'
Requires-Dist: flake8==7.3.0; extra == 'dev'
Requires-Dist: httpx==0.28.1; extra == 'dev'
Requires-Dist: interrogate==1.7.0; extra == 'dev'
Requires-Dist: mypy==2.1.0; extra == 'dev'
Requires-Dist: openlineage-python==1.52.0; extra == 'dev'
Requires-Dist: pre-commit==4.5.1; extra == 'dev'
Requires-Dist: pytest-asyncio==1.4.0; extra == 'dev'
Requires-Dist: pytest-cov==7.0.0; extra == 'dev'
Requires-Dist: pytest-mock==3.15.1; extra == 'dev'
Requires-Dist: pytest-timeout==2.3.1; extra == 'dev'
Requires-Dist: pytest-xdist==3.6.1; extra == 'dev'
Requires-Dist: pytest==9.0.3; extra == 'dev'
Requires-Dist: ruff==0.15.20; extra == 'dev'
Provides-Extra: easyocr
Requires-Dist: docling[easyocr]==2.130.0; extra == 'easyocr'
Provides-Extra: event-streaming
Requires-Dist: confluent-kafka[schemaregistry]==2.15.1; extra == 'event-streaming'
Provides-Extra: gpu
Requires-Dist: torch; extra == 'gpu'
Requires-Dist: torchvision; extra == 'gpu'
Provides-Extra: lineage
Requires-Dist: openlineage-python==1.52.0; extra == 'lineage'
Provides-Extra: notebooks
Requires-Dist: ipykernel==7.3.0; extra == 'notebooks'
Requires-Dist: jupyter==1.1.1; extra == 'notebooks'
Requires-Dist: jupyterlab==4.6.4; extra == 'notebooks'
Requires-Dist: matplotlib==3.11.0; extra == 'notebooks'
Requires-Dist: nbconvert==7.17.1; extra == 'notebooks'
Requires-Dist: nbformat==5.10.4; extra == 'notebooks'
Requires-Dist: seaborn==0.13.2; extra == 'notebooks'
Provides-Extra: ocrmac
Requires-Dist: docling[ocrmac]==2.130.0; extra == 'ocrmac'
Provides-Extra: rapidocr
Requires-Dist: docling[rapidocr]==2.130.0; extra == 'rapidocr'
Provides-Extra: telemetry
Requires-Dist: opentelemetry-api==1.41.1; extra == 'telemetry'
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc==1.41.1; extra == 'telemetry'
Requires-Dist: opentelemetry-instrumentation-fastapi==0.62b1; extra == 'telemetry'
Requires-Dist: opentelemetry-sdk==1.41.1; extra == 'telemetry'
Provides-Extra: tesserocr
Requires-Dist: docling[tesserocr]==2.130.0; extra == 'tesserocr'
Provides-Extra: vlm-asr
Requires-Dist: docling[asr,vlm]==2.130.0; extra == 'vlm-asr'
Description-Content-Type: text/markdown

# Docling pipelines

[![PyPI version](https://img.shields.io/pypi/v/docling-pipelines)](https://pypi.org/project/docling-pipelines/)
[![Python 3.12](https://img.shields.io/badge/python-3.12-blue.svg)](https://www.python.org/downloads/release/python-3120/)
[![License MIT](https://img.shields.io/github/license/IBM/docling-pipelines)](https://opensource.org/licenses/MIT)
[![CI](https://github.com/IBM/docling-pipelines/actions/workflows/ci.yml/badge.svg)](https://github.com/IBM/docling-pipelines/actions/workflows/ci.yml)
[![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![DCO](https://img.shields.io/badge/DCO-required-brightgreen)](https://developercertificate.org/)
[![OpenSSF Best Practices](https://www.bestpractices.dev/projects/15122/badge)](https://www.bestpractices.dev/projects/15122)

## What is Docling pipelines?

Docling pipelines is an enterprise-grade document curation pipeline for Retrieval Augmented Generation (RAG) applications. It ingests data from unstructured sources, curates documents, and writes entities and vector embeddings to targets — enabling AI-ready pipelines at scale.

It connects to cloud document sources (S3, OneDrive, SharePoint, Google Drive, Box, and more) and extracts content and entities from PDF, DOCX, HTML, images, and other formats using [Docling](https://github.com/docling-project/docling). Extracted content is curated for LLMs, converted into chunks and embeddings, and stored in a vector database such as Milvus or OpenSearch.

## Features

- 📥 **Multi-source ingestion** — local filesystem, Amazon S3, IBM COS, SharePoint, OneDrive, Google Drive, Box, Dropbox, and web pages
- 📄 **Document extraction** — PDF, DOCX, HTML, images, and more via Docling, with optional VLM and ASR pipelines
- 🧠 **Entity extraction** — LLM-based extraction via LiteLLM (100+ providers), IBM watsonx.ai, or Docling templates
- ✂️ **Chunking** — Docling-native and semantic chunking strategies
- 🔢 **Embeddings** — vector embedding generation for any downstream vector store
- 🔍 **Quality operators** — language detection, readability scoring, PII/HAP detection, deduplication, redaction, SQL filtering, document classification, and ML enrichment
- 🗄️ **Vector storage** — write to OpenSearch or Milvus
- 🔀 **DAG-based flows** — define pipelines as JSON with automatic dependency resolution and parallel execution
- 🔌 **Extensible** — load custom operators from Python packages, local paths, or S3 without modifying core code
- 🖥️ **Multiple interfaces** — CLI, Python API (`DocpipeFlowManager`), and REST API (FastAPI)

## Installation

```bash
pip install docling-pipelines
```

Requires Python 3.12. Works on macOS and Linux (x86_64 and arm64).

### OCR Installation

RapidOCR is included in the default PyPI installation and works out of the box. For advanced setups, platform-specific extras, or other OCR backends, see the [Extract Operator Guide](docs/operators/extract/extract_operator_readme.md).

### Lightweight variant

For a lightweight version of docling-pipelines that excludes certain operator dependencies, see [docs/guides/SLIM_VARIANT.md](docs/guides/SLIM_VARIANT.md).

## Quickstart

### 1. Run a flow (CLI)

```bash
docling-pipelines --flow-file path/to/flow.json
```

Validate without executing:

```bash
docling-pipelines --flow-file flow.json --validate
```

List all available operators:

```bash
docling-pipelines --list-operators
```

### 2. Python API

```python
from docpipe import DocpipeFlowManager

manager = DocpipeFlowManager(flow_file="path/to/flow.json")
result = manager.execute()
```

Log verbosity is controlled via `DS_LOG_LEVEL` (`DEBUG`, `INFO`, `WARNING`).

## UI

Docling pipelines includes a React + TypeScript frontend (IBM Carbon Design System) served by the FastAPI backend at `/ui`.

### Development setup

**Prerequisites:** Node.js 22.15.1 (use `nvm use` in the `frontend/` directory), npm.

**1. Start the backend:**
```bash
uvicorn docpipe.api.main:app --reload --host 0.0.0.0 --port 8080
```

**2. Start the frontend dev server (separate terminal):**
```bash
cd frontend
nvm use
npm install
npm run dev
# UI available at http://localhost:3000
```

### Production (bundled with the wheel)

The frontend is automatically built and bundled into the wheel during `uv build`. After installing:
```bash
uvicorn docpipe.api.main:app --host 0.0.0.0 --port 8080
# UI available at http://localhost:8080/ui
```

For full details see [FRONTEND_BUILD_INTEGRATION.md](FRONTEND_BUILD_INTEGRATION.md).

## Documentation

Check out the full [documentation](docs/README.md) for installation, flow authoring, operator reference, and more:

- [Quick Start Guide](QUICKSTART.md) — first pipeline in under 5 minutes
- [Pipeline Setup Guide](USER_GUIDE_PIPELINE_SETUP.md) — complete setup with Ollama, OpenSearch, and flow examples
- [Flow Authoring Format](docs/guides/FLOW_AUTHORING_FORMAT.md) — declarative flow authoring
- [Operator Reference](docs/reference/OPERATORS.md) — full parameter specs for all operators
- [Architecture](ARCHITECTURE.md) — system design and distributed execution patterns
- [Troubleshooting](TROUBLESHOOTING.md) — common issues and solutions

## Available Operators

| Category | Operators |
|---|---|
| **Ingest** | Local Filesystem & Remote Source Ingest (`ingest_source`) — [filesystem, S3, IBM COS, SharePoint, OneDrive, Google Drive, Box, Dropbox, web](docs/operators/ingest/ingest_source_readme.md) |
| **Extract** | Document Extractor (`extract_operator`), ACL Extraction (`acl_operator`) |
| **Functional** | Chunking (`chunker`), Embeddings (`embeddings`), Branching Operator (`branching`), Merge Operator (`merge`), Document ID Hash (`doc_id_hash`), Entity Curation (`entity_curation`), No-op (`noop`) |
| **Quality** | Language Annotator (`lang_detect`), Readability Operator (`readability`), PII and HAP Annotator (`pii_and_hap`), Document Classifier (`document_classifier`), Annotation Filter (`sql_filter`), Redaction (`redaction`), De-duplicator (`ededup`), ML Text Enrichment (`ml_enrichment`), Document Quality (`doc_quality`) |
| **VectorDB** | Vector Database (`vectordb`) — OpenSearch, Milvus |
| **Storage** | Document Set (`document_set`) — DuckDB-backed document collections, Storage Output (`storage_output`) — write pipeline output to filesystem, S3, IBM COS, and more |

For per-operator configuration guides, see [Operator Configuration Guides](docs/reference/OPERATORS.md).

## Examples

Explore [sample flows](examples/) and [DocpipeFlowManager examples](examples/docpipe_flow_manager/) for common pipeline patterns.

For interactive, hands-on tutorials, see the [Jupyter notebook examples](examples/notebooks/README.md).

## Contributing

Please read [Contributing to Docling pipelines](CONTRIBUTING.md) for development setup, code standards, testing requirements, and the pull request process.

## License

The Docling pipelines codebase is under the [MIT License](LICENSE).
