Metadata-Version: 2.4
Name: ai-parrot-loaders
Version: 1.0.0
Summary: Document loaders for AI-Parrot RAG pipelines
Author-email: Jesus Lara <jesuslara@phenobarbital.info>
License-Expression: MIT
Project-URL: Homepage, https://github.com/phenobarbital/ai-parrot
Project-URL: Repository, https://github.com/phenobarbital/ai-parrot
Project-URL: Documentation, https://github.com/phenobarbital/ai-parrot/
Keywords: ai,rag,document-loaders,pdf,youtube,agents,asyncio
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: POSIX :: Linux
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Framework :: AsyncIO
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: ai-parrot>=1.0.0
Requires-Dist: decorator>=5
Requires-Dist: openpyxl>=3.1
Requires-Dist: tabulate>=0.9
Requires-Dist: semantic-text-splitter<1.0,>=0.30
Provides-Extra: youtube
Requires-Dist: pytube==15.0.0; extra == "youtube"
Requires-Dist: youtube_transcript_api==1.0.3; extra == "youtube"
Requires-Dist: yt-dlp>=2026.02.21; extra == "youtube"
Provides-Extra: audio
Requires-Dist: whisperx==3.8.5; extra == "audio"
Requires-Dist: av==16.1.0; extra == "audio"
Requires-Dist: resemblyzer==0.1.4; extra == "audio"
Requires-Dist: torch-audiomentations==0.12.0; extra == "audio"
Requires-Dist: torch-pitch-shift==1.2.5; extra == "audio"
Requires-Dist: torchmetrics==1.8.2; extra == "audio"
Provides-Extra: pdf
Requires-Dist: weasyprint==69.0; extra == "pdf"
Requires-Dist: fpdf==1.7.2; extra == "pdf"
Requires-Dist: markdownify>=1.2.2; extra == "pdf"
Requires-Dist: markitdown[docx,pdf,pptx,xlsx]>=0.1.2; extra == "pdf"
Requires-Dist: python-docx==1.1.2; extra == "pdf"
Requires-Dist: paddleocr==3.2.0; extra == "pdf"
Requires-Dist: easyocr==1.7.2; extra == "pdf"
Provides-Extra: images
Requires-Dist: easyocr==1.7.2; extra == "images"
Provides-Extra: web
Requires-Dist: beautifulsoup4>=4.12; extra == "web"
Requires-Dist: html2text>=2024.0; extra == "web"
Provides-Extra: ebook
Requires-Dist: ebooklib>=0.19; extra == "ebook"
Requires-Dist: beautifulsoup4>=4.12; extra == "ebook"
Requires-Dist: markdownify>=1.2.2; extra == "ebook"
Requires-Dist: mobi<0.5,>=0.4.1; extra == "ebook"
Provides-Extra: documents
Requires-Dist: ai-parrot-loaders[ebook]; extra == "documents"
Requires-Dist: python-docx==1.1.2; extra == "documents"
Requires-Dist: pymupdf>=1.27; extra == "documents"
Requires-Dist: pymupdf4llm>=0.0.27; extra == "documents"
Provides-Extra: video
Requires-Dist: moviepy==2.2.1; extra == "video"
Requires-Dist: ffmpeg==1.4; extra == "video"
Provides-Extra: document
Requires-Dist: mammoth>=1.11.0; extra == "document"
Provides-Extra: scraping
Requires-Dist: trafilatura>=1.12; extra == "scraping"
Requires-Dist: beautifulsoup4>=4.12; extra == "scraping"
Requires-Dist: markdownify>=1.2.2; extra == "scraping"
Provides-Extra: ml
Requires-Dist: torch<2.9,>=2.8.0; extra == "ml"
Requires-Dist: torchaudio<2.9,>=2.8.0; extra == "ml"
Requires-Dist: torchvision<0.24,>=0.23.0; extra == "ml"
Requires-Dist: pytorch-lightning==2.6.5; extra == "ml"
Requires-Dist: pytorch-metric-learning==2.9.0; extra == "ml"
Requires-Dist: nvidia-cudnn-cu12>=9.10.2.21; extra == "ml"
Provides-Extra: ml-heavy
Requires-Dist: torch<2.9,>=2.8.0; extra == "ml-heavy"
Requires-Dist: torchaudio<2.9,>=2.8.0; extra == "ml-heavy"
Requires-Dist: numpy<3,>=2.2.2; extra == "ml-heavy"
Requires-Dist: accelerate>=1.1.0; extra == "ml-heavy"
Requires-Dist: bitsandbytes==0.49.2; extra == "ml-heavy"
Requires-Dist: datasets>=3.0.2; extra == "ml-heavy"
Requires-Dist: transformers<5.0,>=4.48.0; extra == "ml-heavy"
Requires-Dist: tensorflow>=2.20.0; extra == "ml-heavy"
Requires-Dist: tf-keras>=2.20.0; extra == "ml-heavy"
Requires-Dist: opencv-python==4.10.0.84; extra == "ml-heavy"
Provides-Extra: all
Requires-Dist: ai-parrot-loaders[audio,document,documents,ebook,images,ml,ml-heavy,pdf,scraping,video,web,youtube]; extra == "all"

# AI-Parrot Loaders

**ai-parrot-loaders** provides document loaders for [AI-Parrot](https://pypi.org/project/ai-parrot/) RAG (Retrieval-Augmented Generation) pipelines. Each loader transforms a specific document format into text chunks that can be embedded and searched.

## Installation

```bash
pip install ai-parrot-loaders
```

Install only the extras you need:

```bash
pip install ai-parrot-loaders[pdf]
pip install ai-parrot-loaders[youtube]
pip install ai-parrot-loaders[audio]
pip install ai-parrot-loaders[web]

# Everything
pip install ai-parrot-loaders[all]
```

## Available Extras

| Extra | Description |
|-------|-------------|
| `pdf` | PDF loading with OCR support (PaddleOCR) |
| `youtube` | YouTube transcript and video download |
| `audio` | Audio transcription (WhisperX, pyannote) |
| `web` | HTML/web page loading |
| `ebook` | Structured EPUB and MOBI e-book loading |
| `documents` | PDF, DOCX, EPUB, and MOBI ingestion without OCR dependencies |
| `video` | Video processing (MoviePy, FFmpeg) |

## Supported Formats

`uv pip install 'ai-parrot[wiki]'` automatically installs
`ai-parrot-loaders[documents]` and the parsing dependencies for PDF, DOCX, EPUB,
and MOBI. To use those loaders independently, install
`uv pip install 'ai-parrot-loaders[documents]'`.

The `documents` extra uses the native PDF and DOCX loaders plus the `ebook`
extra. The existing `document` extra is separate and supplies Mammoth for
document conversion; `pdf` includes the broader OCR stack.

| Loader | Format | Description |
|--------|--------|-------------|
| `TextLoader` | `.txt` | Plain text files |
| `CSVLoader` | `.csv` | CSV files |
| `ExcelLoader` | `.xlsx`, `.xls` | Excel spreadsheets |
| `MSWordLoader` | `.docx` | Microsoft Word documents |
| `HTMLLoader` | `.html` | HTML files |
| `MarkdownLoader` | `.md` | Markdown files |
| `PDFLoader` | `.pdf` | PDF documents |
| `PDFMarkdownLoader` | `.pdf` | PDF to Markdown conversion |
| `PDFTablesLoader` | `.pdf` | PDF table extraction |
| `PowerPointLoader` | `.pptx` | PowerPoint presentations |
| `EpubLoader` | `.epub` | EPUB e-books |
| `MobiLoader` | `.mobi` | MOBI/KF8 e-books via KindleUnpack |
| `WebLoader` | URL | Web pages |
| `YoutubeLoader` | URL | YouTube video transcripts |
| `VimeoLoader` | URL | Vimeo video transcripts |
| `AudioLoader` | `.mp3`, `.wav`, etc. | Audio transcription |
| `VideoLoader` | URL | Video download + transcription |
| `VideoLocalLoader` | `.mp4`, etc. | Local video transcription |
| `DocumentConverterLoader` | multiple | Auto-detect format and convert |

## Quick Start

### Structured EPUB and MOBI books

Install the optional ebook dependencies:

```bash
uv pip install 'ai-parrot-loaders[ebook]'
```

This includes `ebooklib`, `beautifulsoup4`, `markdownify`, and `mobi` (KindleUnpack).

```python
from parrot_loaders.epubloader import EpubLoader
from parrot_loaders.mobiloader import MobiLoader
from parrot.loaders.ebook import ebook_sections

documents = await EpubLoader("book.epub").load()
sections = ebook_sections(documents)
await pageindex.insert_ebook("my-book", [section.model_dump() for section in sections])

# Same output contract and loader options for MOBI:
mobi_documents = await MobiLoader("book.mobi").load()
```

Create the PageIndex tree before inserting. Bookstore, GraphIndex and wiki
ingestion select this structural path automatically for EPUB and MOBI documents.

```bash
bookstore add book.mobi --no-llm
```

MOBI support handles unencrypted MOBI7 HTML/NCX and KF8 EPUB output. MOBI7 uses
NCX navigation when present, then explicit nested HTML navigation, then HTML
heading structure. If the source has no hierarchy, the loader cannot recover
one that was never encoded. Invalid/encrypted books and PDF-only Print Replica
output raise `MobiLoaderError`. Decoding runs off the event loop with serialized
backend calls, and temporary files are removed on both success and failure.
Section provenance always refers to the original `.mobi` source.

Each document contains one section's body and an `ebook_section` metadata record:
`section_id`, `parent_id`, `title`, `href` (including fragments), `depth`,
`toc_order`, `reading_order`, `source_uri`, `target_found`, and `origin`.
`per_chapter=False` returns one readable book document with complete records in
`ebook_sections`. `include_toc_document=True` adds a nested navigation document.

Navigation parents and short TOC sections are retained even when they have no
body. Missing targets remain in the index with `target_found=False`. When a
document has no TOC entries, HTML headings supply its fallback structure.
Reading order follows the spine, while TOC order is recorded separately.
Repeated targets retain distinct navigation identities; their body is stored
once, on the last entry for that target, rather than duplicated.

**Compatibility:** `load()` defaults to `split_documents=False` for ebooks.
Explicit chunking remains available for vector-store callers. Section bodies no
longer carry synthetic `Section:`/`Title:` prefixes or duplicate chapter headings;
titles and relationships are metadata. Use `ebook_markdown(sections)` for a
readable Markdown export. The structural PageIndex path preserves all navigation
levels and does not merge short nodes or infer parents from Markdown headings.

### Other documents

```python
from parrot_loaders.factory import get_loader_class

# Auto-detect loader by file extension
LoaderClass = get_loader_class("report.pdf")
loader = LoaderClass(source="report.pdf")
documents = await loader.load()

for doc in documents:
    print(doc.page_content[:200])
```

Or use a specific loader directly:

```python
from parrot_loaders.youtube import YoutubeLoader

loader = YoutubeLoader(source="https://www.youtube.com/watch?v=...")
documents = await loader.load()
```

## Requirements

- Python >= 3.11
- [ai-parrot](https://pypi.org/project/ai-parrot/) >= 0.23.18

## License

MIT
