Metadata-Version: 2.4
Name: docext
Version: 0.1.12
Summary: Onprem information extraction from documents
Home-page: https://github.com/nanonets/docext
Author: Souvik Mandal
Author-email: souvik@nanonets.com
Classifier: Programming Language :: Python :: 3.11
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: accelerate
Requires-Dist: gradio==5.23.2
Requires-Dist: json-repair
Requires-Dist: litellm
Requires-Dist: loguru
Requires-Dist: mdpd
Requires-Dist: pandas
Requires-Dist: python-dotenv
Requires-Dist: requests
Requires-Dist: setuptools
Requires-Dist: tabulate
Requires-Dist: types-requests
Requires-Dist: vllm==v0.8.3
Requires-Dist: xgrammar==0.1.17
Provides-Extra: dev
Requires-Dist: pre-commit; extra == "dev"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license-file
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

<h1 align="center">docext</h1>

<p align="center"><em>An on-premises, OCR-free document information extraction tool powered by vision-language models.</em></p>

<p align="center">
  <a href="https://pepy.tech/projects/docext">
    <img src="https://static.pepy.tech/badge/docext/week" alt="PyPI Downloads" />
  </a>
  <a href="https://opensource.org/licenses/Apache-2.0">
    <img src="https://img.shields.io/badge/License-Apache_2.0-blue.svg" alt="License" />
  </a>
  <a href="https://colab.research.google.com/drive/1r1asxGeezfWnJvw8jimfFAB2sGjk1HdM?usp=sharing">
    <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" />
  </a>
  <a href="https://pypi.org/project/docext/">
    <img alt="PyPI - Version" src="https://img.shields.io/pypi/v/docext">
  </a>
</p>

![Demo](https://raw.githubusercontent.com/NanoNets/docext/main/assets/demo.png)

## Overview

docext is an OCR-free tool for extracting structured information from documents such as invoices, passports, and other documents. It leverages vision-language models (VLMs) to accurately identify and extract both field data and tabular information from document images.


## Features

- **Flexible extraction**: Define custom fields or use pre-built templates
- **Table extraction**: Extract structured tabular data from documents
- **Confidence scoring**: Get confidence levels for extracted information
- **On-premises deployment**: Run entirely on your own infrastructure (Linux, MacOS)
- **Multi-page support**: Process documents with multiple pages
- **REST API**: Programmatic access for integration with your applications
- **Pre-built templates**: Ready-to-use templates for common document types:
  - Invoices
  - Passports
  - Add/delete new fields/columns for other templates.

## Table of Contents
- [Getting Started](#Getting-Started)
  - [Quickstart](#Quickstart)
  - [Installation](#Installation)
  - [Web Interface](#Web-Interface)
  - [API access](#API-access)
- [Supported Models & Platforms](#Supported-Models-&-Platforms)
  - [Models with vLLM (Linux)](#Models-with-vLLM-Linux)
  - [Models with Ollama (Linux and MacOS)](#Models-with-Ollama-Linux-and-MacOS)
  - [Supported Vendor-Hosted Models](#Supported-Vendor-Hosted-Models)
- [Docker](#Docker)
- [About](#About)
- [Contributing](#Contributing)
- [Troubleshooting](#Troubleshooting)

## Getting Started

### Quickstart
- [Colab notebook for onprem deployment](https://colab.research.google.com/drive/1r1asxGeezfWnJvw8jimfFAB2sGjk1HdM?usp=sharing)
- [Colab notebook for vendor-hosted models (openai, anthropic, openrouter)](https://colab.research.google.com/drive/1yBnDv_1mZEuNtSMEYc8INGG0Z3UoLakD?usp=sharing)
- [Docker](https://github.com/NanoNets/docext/blob/main/README.md#Docker)

### Installation

```bash
# create a virtual environment
## install uv if not installed
curl -LsSf https://astral.sh/uv/install.sh | sh
## create a virtual environment with python 3.11
uv venv --python=3.11
source .venv/bin/activate

# Install from PyPI
uv pip install docext

# Or install from source
git clone https://github.com/nanonets/docext.git
cd docext
uv pip install -e .
```
Check [Supported Models](#Supported-Models-&-Platforms) section for more options.

### Web Interface

docext includes a Gradio-based web interface for easy document processing:

```bash
# Start the web interface with default configs
python -m docext.app.app

# Start the web interface with custom configs
python -m docext.app.app --model_name "hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct-AWQ" --max_img_size 1024 # `--help` for more options
```

The interface will be available at `http://localhost:7860` with default credentials: (You can change the port by using `--ui_port` flag)

- Username: `admin`
- Password: `admin`

### API access

docext also provides a REST API for programmatic access to the document extraction functionality.
1. start the API server
```bash
# increase the concurrency limit to process more requests in parallel, default is 1
python -m docext.app.app --concurrency_limit 10
```

2. use the API to extract information from a document
```python
import pandas as pd
import concurrent.futures
from gradio_client import Client, handle_file


def dataframe_to_custom_dict(df: pd.DataFrame) -> dict:
    return {
        "headers": df.columns.tolist(),
        "data": df.values.tolist(),
        "metadata": None  # Modify if metadata is needed
    }

def dict_to_dataframe(d: dict) -> pd.DataFrame:
    return pd.DataFrame(d["data"], columns=d["headers"])


def get_extracted_fields_and_tables(
    client_url: str,
    username: str,
    password: str,
    model_name: str,
    fields_and_tables: dict,
    file_inputs: list[dict]
):
    client = Client(client_url, auth=(username, password))
    result = client.predict(
        file_inputs=file_inputs,
        model_name=model_name,
        fields_and_tables=fields_and_tables,
        api_name="/extract_information"
    )
    fields_results, tables_results = result
    fields_df = dict_to_dataframe(fields_results)
    tables_df = dict_to_dataframe(tables_results)
    return fields_df, tables_df


fields_and_tables = dataframe_to_custom_dict(pd.DataFrame([
    {"name": "invoice_number", "type": "field", "description": "Invoice number"},
    {"name": "item_description", "type": "table", "description": "Item/Product description"}
    # add more fields and table columns as needed
]))

file_inputs = [
    {
        # "image": handle_file("https://your_image_url/invoice.jpg") # incase the image is hosted on the internet
        "image": handle_file("assets/invoice_test.jpeg") # incase the image is hosted on the local machine
    }
]

## send single request
### client url can be the local host or the public url like `https://6986bdd23daef6f7eb.gradio.live/`
fields_df, tables_df = get_extracted_fields_and_tables(
    "http://localhost:7860", "admin", "admin", "hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct-AWQ", fields_and_tables, file_inputs
)
print("========Fields:=========")
print(fields_df)
print("========Tables:=========")
print(tables_df)


## send multiple requests in parallel
# Define a wrapper function for parallel execution
def run_request():
    return get_extracted_fields_and_tables(
        "http://localhost:7860", "admin", "admin", "hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct-AWQ", fields_and_tables, file_inputs
    )

# Use ThreadPoolExecutor to send 10 requests in parallel
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
    future_results = [executor.submit(run_request) for _ in range(10)]

    for future in concurrent.futures.as_completed(future_results):
        fields_df, tables_df = future.result()
        print("========Fields:=========")
        print(fields_df)
        print("========Tables:=========")
        print(tables_df)
```

## Requirements

- Python 3.11+
- CUDA-compatible GPU (for optimal performance). Use Google Colab for free GPU access.
- Dependencies listed in requirements.txt

## Supported Models & Platforms
### Models with vLLM (Linux)

docext uses vision-language models for document understanding. By default, it uses: `Qwen/Qwen2.5-VL-7B-Instruct-AWQ` but you can use any other models supported by vLLM.

Recommended models based on GPU memory:
| Model | GPU Memory | `--model_name` |
|-------|------------|--------------|
| Qwen/Qwen2.5-VL-7B-Instruct-AWQ | 16GB | `hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct-AWQ` |
| Qwen/Qwen2.5-VL-7B-Instruct | 24GB | `hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct` |
| Qwen/Qwen2.5-VL-32B-Instruct-AWQ | 48GB | `hosted_vllm/Qwen/Qwen2.5-VL-32B-Instruct-AWQ` |
| Qwen/Qwen2.5-VL-32B-Instruct | 80 GB | `hosted_vllm/Qwen/Qwen2.5-VL-32B-Instruct` |

```bash
# will download the default model (Qwen/Qwen2.5-VL-7B-Instruct-AWQ) and host it on your local machine with vLLM on port 8000
python -m docext.app.app

# will download the model (Qwen/Qwen2.5-VL-32B-Instruct-AWQ) and host it on your local machine with vLLM on port 9000
python -m docext.app.app --model_name hosted_vllm/Qwen/Qwen2.5-VL-32B-Instruct-AWQ --vlm_server_port 9000

# If you already have a vLLM server running on ip <your_ip> and port <your_port>, you can use the following command:
export API_KEY=<your_api_key> # incase you have used a API key to host the model
python -m docext.app.app --model_name hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct-AWQ --vlm_server_host <your_ip> --vlm_server_port <your_port>
```


### Models with Ollama (Linux and MacOS)
> Ollama is supported on Windows. But I have not tested it.
1. Install [ollama](https://ollama.com/download) in your machine.
2. Download the checkpoint `ollama pull llama3.2-vision`.
3. Run the following command to start the ollama server.

```bash
# You can use the ollama server running on your local machine
python -m docext.app.app --model_name ollama/llama3.2-vision --max_img_size 1024

# incase you have a ollama server running on ip <your_ip> and port <your_port>
python -m docext.app.app --model_name ollama/llama3.2-vision --max_img_size 1024 --vlm_server_host <your_ip> --vlm_server_port <your_port>
```
If you have a machine with GPU >= 16GB, change the `--max_img_size` to 2048.

### Supported Vendor-Hosted Models

docext supports integration with various cloud-based vision-language models.

**Important**: Please review each provider's data privacy policy before using their services. We recommend using local models for sensitive data.

| Provider | Model Examples | Environment Variable | Usage Example |
|----------|---------------|---------------------|---------------|
| OpenAI | gpt-4o | `OPENAI_API_KEY` | `--model_name gpt-4o` |
| Anthropic | Claude 3 Sonnet | `ANTHROPIC_API_KEY` | `--model_name claude-3-sonnet-20240229` |
| OpenRouter | Meta Llama models | `OPENROUTER_API_KEY` | `--model_name openrouter/meta-llama/llama-4-maverick:free` |
| Google | gemini-2.0-flash | `GEMINI_API_KEY` | `--model_name gemini/gemini-2.0-flash` |

Example usage:
```bash
export OPENROUTER_API_KEY=sk-...
python -m docext.app.app --model_name "openrouter/meta-llama/llama-4-maverick:free"
```

## Docker
1. Add your [huggingface token](https://huggingface.co/docs/hub/en/security-tokens) to the environment variable. Not needed if you are using the default model.
2. Utilize all available GPUs or specify a particular one as needed (e.g., --gpus '"device=0"'). CPU mode is not supported; for trying out the app, we recommend using Google Colab, which offers free GPU access.
```bash
docker run --rm \
  --env "HUGGING_FACE_HUB_TOKEN=<secret>" \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --network host \
  --shm-size=20.24gb \
  --gpus all \
  nanonetsopensource/docext:v0.1.11 --model_name "hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct-AWQ"
```
3. If you are using vendor-hosted models, you can use the following command:
```bash
docker run --rm \
  --env "OPENROUTER_API_KEY=<secret>" \
  --network host \
  nanonetsopensource/docext:v0.1.11 --model_name "openrouter/meta-llama/llama-4-maverick:free"
```

## About

docext is developed by [Nanonets](https://nanonets.com), a leader in document AI and intelligent document processing solutions. Nanonets is committed to advancing the field of document understanding through open-source contributions and innovative AI technologies. If you are looking for information extraction solutions for your business, please visit [our website](https://nanonets.com) to learn more.

## Contributing

We welcome contributions! Please see [contribution.md](https://github.com/NanoNets/docext/blob/main/contribution.md) for guidelines.
If you have a feature request or need support for a new model, feel free to open an issue—we'd love to discuss it further!

## Troubleshooting

If you encounter any issues while using `docext`, please refer to our [Troubleshooting guide](https://github.com/NanoNets/docext/blob/main/Troubleshooting.md) for common problems and solutions.


## License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
