Metadata-Version: 2.4
Name: scrapy-ingest
Version: 2.2.0
Summary: One pipeline. Full crawl observability in SQL or Elasticsearch.
Home-page: https://github.com/fawadss1/scrapy-ingest
Author: Fawad Ali
Author-email: fawadstar6@gmail.com
Project-URL: Documentation, https://scrapy-ingest.readthedocs.io/
Project-URL: Source, https://github.com/fawadss1/scrapy-ingest
Project-URL: Tracker, https://github.com/fawadss1/scrapy-ingest/issues
Keywords: scrapy,database,postgresql,mysql,web-scraping,data-pipeline
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Framework :: Scrapy
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Database
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: scrapy>=2.18.0
Requires-Dist: psycopg2-binary>=2.9.12
Requires-Dist: itemadapter>=0.13.1
Requires-Dist: pytz>=2026.3
Requires-Dist: w3lib>=2.4.1
Requires-Dist: PyMySQL>=1.2.0
Requires-Dist: opensearch-py>=3.2.0
Provides-Extra: docs
Requires-Dist: sphinx>=5.0.0; extra == "docs"
Requires-Dist: sphinx_rtd_theme>=1.2.0; extra == "docs"
Requires-Dist: myst-parser>=0.18.0; extra == "docs"
Requires-Dist: sphinx-autodoc-typehints>=1.19.0; extra == "docs"
Requires-Dist: sphinx-copybutton>=0.5.0; extra == "docs"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: ruff>=0.8.0; extra == "dev"
Requires-Dist: black>=22.0.0; extra == "dev"
Requires-Dist: flake8>=5.0.0; extra == "dev"
Requires-Dist: mypy>=0.991; extra == "dev"
Requires-Dist: pre-commit>=2.20.0; extra == "dev"
Provides-Extra: test
Requires-Dist: pytest>=7.0.0; extra == "test"
Requires-Dist: pytest-cov>=4.0.0; extra == "test"
Requires-Dist: pytest-mock>=3.8.0; extra == "test"
Provides-Extra: dashboard
Requires-Dist: fastapi>=0.115.0; extra == "dashboard"
Requires-Dist: httpx>=0.27.0; extra == "dashboard"
Requires-Dist: uvicorn[standard]>=0.32.0; extra == "dashboard"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license-file
Dynamic: project-url
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

<p align="center">
  <img src="https://raw.githubusercontent.com/fawadss1/scrapy-ingest/master/static/logo.png" alt="scrapy-ingest" width="640">
</p>

# Scrapy Ingest

[![PyPI version](https://img.shields.io/pypi/v/scrapy-ingest?color=blue)](https://pypi.org/project/scrapy-ingest/)
[![Python versions](https://img.shields.io/pypi/pyversions/scrapy-ingest)](https://pypi.org/project/scrapy-ingest/)
[![Downloads](https://static.pepy.tech/badge/scrapy-ingest)](https://pepy.tech/project/scrapy-ingest)
[![Documentation](https://readthedocs.org/projects/scrapy-ingest/badge/?version=latest)](https://scrapy-ingest.readthedocs.io/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](https://github.com/fawadss1/scrapy-ingest/blob/master/LICENSE)
[![Changelog](https://img.shields.io/badge/changelog-releases-informational)](https://github.com/fawadss1/scrapy-ingest/releases)

**One pipeline. Full crawl observability in SQL or Elasticsearch — plus an optional read-only web dashboard.**

Set `DB_URL`, `SEARCH_URL`, or both — or use **no DB settings** and ingest writes to **`ingest.db`** (SQLite) in your project folder. Items, requests, logs, stats, parent URLs, and errors are captured automatically. Browse the same data in the browser with **`scrapy-ingest dashboard`** (2.0+).

## Install

**Pipeline only** (Postgres, MySQL, SQLite, Elasticsearch/OpenSearch):

```bash
pip install scrapy-ingest
```

**Pipeline + read-only web dashboard** (adds FastAPI, Uvicorn, httpx):

```bash
pip install "scrapy-ingest[dashboard]"
```

Upgrade an existing install to include the dashboard:

```bash
pip install -U "scrapy-ingest[dashboard]"
```

The base install includes `psycopg2-binary`, `PyMySQL`, and `opensearch-py`. The `[dashboard]` extra is optional and only needed for `scrapy-ingest dashboard`.

## Minimal setup (settings.py)

Only the item pipeline is required — requests, logs, stats, parent_url, and error logging are enabled automatically.

**Local dev (SQLite, no server):**

```python
ITEM_PIPELINES = {
    "scrapy_ingest.pipelines.IngestPipeline": 300,
}
CREATE_TABLES = True  # creates ingest.db in the project directory
```

**Postgres / MySQL:**

```python
ITEM_PIPELINES = {
    "scrapy_ingest.pipelines.IngestPipeline": 300,
}

DB_URL = "postgresql://user:password@localhost:5432/database"
```

Run your spider:

```bash
scrapy crawl your_spider
```

Log level follows Scrapy `LOG_LEVEL`.

## Choose your destination

Set **`DB_URL`** (or `DB_*` fields) for Postgres/MySQL/SQLite, **`SEARCH_URL`** for Elasticsearch/OpenSearch, or both. If nothing is set, **`ingest.db`** (SQLite) is used unless `INGEST_SQLITE_ENABLED = False`.

| Mode                                | Settings                | Result                                   |
|-------------------------------------|-------------------------|------------------------------------------|
| **Default (dev)**                   | *(none)*                | SQLite (`ingest.db`)                     |
| **Database only**                   | `DB_URL` or `DB_*`      | SQL only                                 |
| **Elasticsearch / OpenSearch only** | `SEARCH_URL`            | Indexes only                             |
| **Both**                            | `DB_URL` + `SEARCH_URL` | SQL first, then Elasticsearch/OpenSearch |

```python
# Database only
DB_URL = "postgresql://user:password@localhost:5432/database"

# Elasticsearch / OpenSearch only
SEARCH_URL = "http://localhost:9200"

# Both
DB_URL = "postgresql://user:password@localhost:5432/database"
SEARCH_URL = "http://localhost:9200"
```

When both are configured, SQL is written first, then Elasticsearch/OpenSearch indexes. An indexing failure after a successful SQL write is logged but does not roll back database rows.

## Database configuration

PostgreSQL or MySQL — pick **one** connection style.

**Single URL:**

```python
DB_URL = "postgresql://user:password@localhost:5432/database"
# DB_URL = "mysql://user:password@localhost:3306/database"
```

**Discrete fields** (no URL encoding for special characters in passwords):

```python
DB_TYPE = "postgres"  # or "mysql" / "mariadb"
DB_HOST = "localhost"
DB_PORT = 5432  # MySQL: 3306
DB_USER = "user"
DB_PASSWORD = "password"
DB_NAME = "database"
```

Tables, indexes, and named constraints are created automatically when `CREATE_TABLES = True` (default). See [SQL schema](https://scrapy-ingest.readthedocs.io/en/latest/configuration.html#sql-schema-create-tables) in the configuration docs for full DDL details.

**PostgreSQL:** optional `DB_SCHEMA = "public"` (default). Ingest sets `search_path` before creating tables (fixes empty `search_path` on some servers).

**Constraint / index names:** `uq_<table>_<column>`, `fk_<table>_<column>`, `ix_<table>_<column>` (e.g. `fk_ingest_items_job_id`, `ix_ingest_jobs_spider_id`).

## Elasticsearch / OpenSearch configuration

Elasticsearch and OpenSearch use the same REST bulk API. The package connects via **`opensearch-py`**, which works with both.

**Minimum:**

```python
SEARCH_URL = "http://localhost:9200"
```

**With authentication (optional):**

```python
SEARCH_URL = "https://search.example.com:9200"
SEARCH_USER = "elastic"
SEARCH_PASSWORD = "secret"
SEARCH_SSL_VERIFY = True  # set False only for self-signed certs in dev
```

**Index names** match SQL table names by default:

| Table / index         | Contents                                              |
|-----------------------|-------------------------------------------------------|
| `ingest_spiders`      | Stable spider id, name, class path                    |
| `ingest_jobs`         | Job metadata: status, counts, elapsed time            |
| `ingest_job_meta`     | Spider args, ingest settings snapshot, Scrapy stats   |
| `ingest_items`        | Scraped items as JSON documents                       |
| `ingest_requests`     | Requests: url, fingerprint, parent_url, status, error |
| `ingest_logs`         | Log lines: level, logger, message, exception          |

Override table/index names with `SPIDERS_TABLE`, `JOBS_TABLE`, `JOB_META_TABLE`, `ITEMS_TABLE`, `REQUESTS_TABLE`, and `LOGS_TABLE` (same setting for SQL and Elasticsearch/OpenSearch).

Indexes are created on first write if they do not exist.

## What is stored

### Relational database (Postgres / MySQL)

| Table             | Contents                                                                                                                                                                                                                                                                             |
|-------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `ingest_spiders`  | One row per spider name: integer `id`, `class_path`, timestamps                                                                                                                                                                                                                      |
| `ingest_jobs`     | One row per crawl: `id`, unique `job_id` string (e.g. `3/328`), `spider_id` → spiders, status, start/finish, counts, items/min. `errors_count` = failed requests + standalone ERROR/CRITICAL logs (not double-counted from `scrapy.core.scraper` when the request is already failed) |
| `ingest_job_meta` | Per job: `spider_args`, ingest-related settings snapshot, Scrapy `stats` at finish (`job_id` → `ingest_jobs.id`, CASCADE)                                                                                                                                                            |
| `ingest_items`    | JSON items (`crawled_at` added). `job_id` → `ingest_jobs.id` (`fk_ingest_items_job_id`, ON DELETE CASCADE)                                                                                                                                                                           |
| `ingest_requests` | url, parent_url, parent_id, status, response_time_secs, fingerprint, error, success (download errors, HTTP 4xx/5xx, spider callback tracebacks). FKs: `job_id` → jobs, `parent_id` → self                                                                                            |
| `ingest_logs`     | time, logger, level, message, exception; `job_id` → jobs (CASCADE)                                                                                                                                                                                                                   |

Indexes on `job_id`, filters (`spider_id`, `status`, `level`, …), and lookup columns (`fingerprint`, `parent_id`, `created_at`) are created automatically. `ingest_jobs.job_id` has a named unique constraint (`uq_ingest_jobs_job_id`).

Request `parent_url` is the page that scheduled the request (e.g. sitemap → product). Start URLs are `null`. The request `fingerprint` is a SHA1 hash of method + canonical URL for parent lookup.

### Elasticsearch / OpenSearch

Same tables as indexes, with matching columns per document. Child documents reference `ingest_jobs.id` (integer), not the external `job_id` string. The jobs index uses the external `job_id` (e.g. `3/328`) as the document `_id` for lookups. `spider_args`, crawl settings, and stats live in `ingest_job_meta`. Search mode does not link `parent_id`; `parent_url` is stored on each request document.

## Ingest flow and connections

**One connection per destination per crawl** — not connect → insert → disconnect for every product. Items, requests, and logs are buffered in memory and written in batches using the same Postgres/MySQL connection and/or Elasticsearch/OpenSearch client for the whole spider run.

Full diagrams: [docs/ingest-flow.rst](docs/ingest-flow.rst) (Read the Docs: [Ingest flow](https://scrapy-ingest.readthedocs.io/en/latest/ingest-flow.html)).

### Crawl lifecycle

```mermaid
flowchart TB
    START([scrapy crawl starts]) --> OPEN[IngestPipeline open_spider]
    OPEN --> FLSTART[IngestFlusher.start]

    FLSTART --> DB{DB configured?}
    DB -->|yes| DBCONN[Open one Postgres/MySQL connection<br/>application_name = Ingest]
    DB -->|no| SKIPDB[Skip SQL]

    FLSTART --> SRCH{SEARCH_URL set?}
    SRCH -->|yes| ESCONN[Open one SearchClient<br/>X-Opaque-Id: Ingest]
    SRCH -->|no| SKIPES[Skip search]

    DBCONN --> JOB[Create job row / index doc]
    ESCONN --> JOB
    JOB --> TIMER[Start 10s periodic flush timer]
    TIMER --> CRAWL([Spider crawls])

    CRAWL --> END([Engine stops])
    END --> FINAL[Final flush + finish_job]
    FINAL --> SUMMARY[Print crawl summary]
    SUMMARY --> CLOSE[Close DB + search clients]
```

### When data is flushed

```mermaid
flowchart TB
    subgraph inputs [Buffered in memory]
        I1[Items]
        I2[Requests]
        I3[Logs]
    end

    inputs --> BUF[(Collector buffer)]

    BUF --> T1{Size ≥ INGEST_BATCH_SIZE?<br/>default 50}
    BUF --> T2{Timer every 10s?}

    T1 -->|yes| FLUSH[flush batch]
    T2 -->|yes| FLUSH

    FLUSH --> SQL[SQL: INSERT + COMMIT<br/>same connection]
    FLUSH --> ES[Search: bulk index<br/>same HTTP client]
```

| Setting                 | Default | Effect                                          |
|-------------------------|---------|-------------------------------------------------|
| `INGEST_BATCH_SIZE`     | `50`    | Flush when buffer reaches this many rows        |
| `INGEST_FLUSH_INTERVAL` | `10`    | Also flush every N seconds (set `0` to disable) |

**Example:** 100,000 products → ~2,000 batch flushes (not 100,000 connections).

When both SQL and search are configured, SQL commits first; a search failure after SQL success is logged but does not roll back database rows.

When the spider closes, a crawl summary is printed to stderr (visible even when `LOG_LEVEL` is `ERROR`) with job id, spider, database URL, Elasticsearch/OpenSearch URL, table/index names, counts, and elapsed time. Set `INGEST_SHOW_SUMMARY = False` to hide it.

## CLI

Run from your Scrapy project directory (loads `settings.py` via Scrapy).

### `check-config`

Validate settings and ping configured destinations before running a spider:

```bash
scrapy-ingest check-config
scrapy-ingest check-config --db-url "postgresql://user:pass@localhost:5432/db"
scrapy-ingest check-config --search-url "http://localhost:9200"
```

Shows a spinner per step, prints a summary table to stderr, and exits `0` when all configured destinations are reachable.

### `jobs show`

List recent jobs or print one job summary from SQL or Elasticsearch/OpenSearch (requires `DB_URL` and/or `SEARCH_URL`). When both are configured, queries use SQL; crawls still dual-write to both.

```bash
scrapy-ingest jobs show                                    # list recent jobs
scrapy-ingest jobs show --limit 20                         # list up to 20 jobs
scrapy-ingest jobs show --status running                   # filter by status (list mode)
scrapy-ingest jobs show --spider my_spider                 # filter by spider (list mode)
scrapy-ingest jobs show 3/328                              # one job (Field | Value table)
scrapy-ingest jobs show --search-url "http://localhost:9200"
```

**List mode** (no job id): table of recent jobs — job id, spider, status, timestamps, counts. **Detail mode** (with job id): Field | Value table from the `jobs` row. `--status` and `--spider` apply in list mode only.

The job id is the string shown in the end-of-crawl summary (or your `JOB_ID` setting). Exit `0` on success (including an empty list); exit `1` if a specific job id is missing or the destination is unreachable.

### `dashboard`

Read-only web UI for jobs, items, requests, and logs — same destination rules as `jobs show` (SQL when `DB_URL` / default SQLite is used; search when only `SEARCH_URL` is set; **SQL wins** when both are configured).

Install (once):

```bash
pip install "scrapy-ingest[dashboard]"
```

Run from your Scrapy project directory:

```bash
scrapy-ingest dashboard
scrapy-ingest dashboard --host 0.0.0.0 --port 8765
scrapy-ingest dashboard --no-browser
python -m ingest_dashboard --db-url "sqlite:///./ingest.db"
```

Open `http://127.0.0.1:8765` (browser opens automatically unless `--no-browser`). **Home:** summary cards, jobs table with elapsed time, spider/status filters, 5s auto-refresh. **Job detail:** overview, stats, paginated items (JSON modal), requests, logs — hash routes like `#/job/<job_id>?tab=items`. Dark/light theme and a connection popover show where data is read from.

Typical workflow:

```bash
pip install "scrapy-ingest[dashboard]"   # optional UI
scrapy-ingest check-config
scrapy crawl your_spider
scrapy-ingest jobs show
scrapy-ingest dashboard
scrapy-ingest jobs show --status finished --spider your_spider
scrapy-ingest jobs show <job_id>
```

See [CLI docs](https://scrapy-ingest.readthedocs.io/en/latest/cli.html) for details.

## Useful settings

| Setting                                                                                        | Default        | Description                                                                      |
|------------------------------------------------------------------------------------------------|----------------|----------------------------------------------------------------------------------|
| `DB_URL` / `DB_*`                                                                              | —              | Database connection (enables SQL ingest)                                         |
| `SEARCH_URL`                                                                                   | —              | Elasticsearch or OpenSearch URL (enables indexing)                               |
| `SEARCH_USER` / `SEARCH_PASSWORD`                                                              | —              | HTTP basic auth for Elasticsearch/OpenSearch (optional)                          |
| `SEARCH_SSL_VERIFY`                                                                            | `True`         | Verify HTTPS certificates                                                        |
| `CREATE_TABLES`                                                                                | `True`         | Auto-create SQL tables, indexes, and named FKs on startup                        |
| `DB_SCHEMA`                                                                                    | `public`       | PostgreSQL schema (`search_path`) for ingest tables                              |
| `INGEST_BATCH_SIZE`                                                                            | `50`           | Flush when this many rows are buffered                                           |
| `INGEST_FLUSH_INTERVAL`                                                                        | `10`           | Periodic flush in seconds                                                        |
| `INGEST_SHOW_SUMMARY`                                                                          | `True`         | Print crawl summary when the spider closes                                       |
| `DB_TYPE`                                                                                      | `postgres`     | `postgres` / `mysql` / `mariadb` for discrete `DB_*` fields                      |
| `SPIDERS_TABLE`, `JOB_META_TABLE`, `ITEMS_TABLE`, `REQUESTS_TABLE`, `LOGS_TABLE`, `JOBS_TABLE` | see defaults   | Override SQL table and index names (same value for both)                         |
| `TIMEZONE`                                                                                     | `Asia/Karachi` | Timezone for `created_at`                                                        |
| `JOB_ID`                                                                                       | auto           | Omit for `spider_id/run` (e.g. `3/328`). Set `JOB_SESSION_ID` to fix the prefix. |
| `JOB_NEW_SESSION_PER_CRAWL`                                                                    | `False`        | Use a new session prefix on every crawl instead of spider id                     |

## Troubleshooting

- **Password has `@` or `$` in `DB_URL`?** Encode them: `@` → `%40`, `$` → `%24`. Or use discrete `DB_*` fields.
- **Yield items** from callbacks (not only `return` inside a generator).
- **Elasticsearch / OpenSearch connection refused?** Check `SEARCH_URL`, firewall, and that the cluster is running. Test with `curl http://localhost:9200`.
- **HTTPS / self-signed cert?** Set `SEARCH_SSL_VERIFY = False` in development only.

## Standalone components

If you only want part of the collection:

```python
# Items only
ITEM_PIPELINES = {"scrapy_ingest.pipelines.ItemsPipeline": 300}

# Requests only (parent_url + errors)
ITEM_PIPELINES = {"scrapy_ingest.pipelines.RequestsPipeline": 300}

# Logs only
EXTENSIONS = {"scrapy_ingest.extensions.LoggingExtension": 500}
```

## Benchmarks and comparisons

How **scrapy-ingest** compares to other Scrapy storage packages — feature matrices, architecture notes, and a reproducible write-path benchmark.

| Package                  |     SQL     | ES/OS | Mongo | Kafka | Requests / logs | Dashboard | Batched  |
|--------------------------|:-----------:|:-----:|:-----:|:-----:|:---------------:|:---------:|:--------:|
| **scrapy-ingest**        | ✓ PG/MySQL |  ✓   |   —   |   —   |      auto       |    ✓     | ✓ (50)  |
| scrapy-elasticsearch     |      —      |  ✓   |   —   |   —   |        —        |     —     | ✓ (500) |
| scrapy-mongodb           |      —      |   —   |  ✓   |   —   |        —        |     —     | per-item |
| scrapy-sql               |  ✓ FEEDS   |   —   |   —   |   —   |        —        |     —     |   bulk   |
| os-scrapy-kafka-pipeline |      —      |   —   |   —   |  ✓   |        —        |     —     | per-item |
| Manual psycopg2 pipeline |     DIY     |   —   |   —   |   —   |       DIY       |     —     | often ✗ |
| Scrapy `FEEDS`           |    files    |   —   |   —   |   —   |        —        |     —     |  stream  |

Also in docs: **scrapy-pipelines** (Scrapedia/Mongo), **Scrapy MongoPipeline** (official example).

Synthetic benchmark (SQLite, 100k rows, batch 50): **~3.3×** faster write path vs per-row `COMMIT` on a sample run — real crawls are usually network-bound. Run yourself:

```bash
python scripts/benchmark_ingest.py --rows 100000 --batch-size 50
```

Full comparison, methodology, and production tuning: [docs/benchmarks.rst](docs/benchmarks.rst) · [Read the Docs](https://scrapy-ingest.readthedocs.io/en/latest/benchmarks.html).

## Links

- Docs: https://scrapy-ingest.readthedocs.io/
- Changelog: [docs/development/changelog.rst](docs/development/changelog.rst) — **[2.2.0](docs/development/changelog.rst)** search index mappings, `check-config` layout checks, and `spider_id/run` doc examples; **[2.1.0](docs/development/changelog.rst)** spider registry and SQL-aligned search docs; **[2.0.0](docs/development/changelog.rst)** web dashboard and `[dashboard]` extra
- Issues: https://github.com/fawadss1/scrapy-ingest/issues

## License

MIT License. See [LICENSE](LICENSE).
