Metadata-Version: 2.4
Name: crawlingo
Version: 1.0.0a2
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Summary: Crawlingo - validated datasets from web pages, powered by Rust
Home-Page: https://crawlingo-docs.vercel.app
License: MIT
Requires-Python: >=3.8, <3.14
Description-Content-Type: text/markdown
Project-URL: Documentation, https://crawlingo-docs.vercel.app
Project-URL: Funding, https://github.com/sponsors/Vamshavardhan50
Project-URL: Repository, https://github.com/Vamshavardhan50/crawlingo

# Crawlingo for Python

Turn web pages into validated datasets using a Rust engine and a small Python API.

Supply a URL file and explicit field selectors. DatasetJob validates records,
removes exact duplicates, writes JSONL/CSV shards and resumes committed work.
Records stay in the Rust pipeline; Python receives a summary report.

```python
from crawlingo import DatasetJob

job = DatasetJob({
    "urls_file": "urls.txt",
    "output_dir": "datasets/pages",
    "fields": [{"name": "title", "selector": "h1", "required": True}],
})
report = job.run()
print(report)
```

Create urls.txt with one HTTP(S) URL per line before running. Check pages_failed
and records_rejected even when complete is true. Repeat unchanged input and
configuration to resume; use a new directory for changed inputs.

This is an alpha API. This release targets CPython 3.8–3.13; Python 3.14 is outside
its release matrix and package metadata. Available wheels depend on the release
platform matrix. Source builds require Rust, C/C++ tools, CMake and libclang.
Linux wheels target glibc 2.28 or newer on x64 and ARM64; musl wheels are not built.

[Installation and full Python guide](https://github.com/Vamshavardhan50/crawlingo/blob/main/sdk/python/README.md) ·
[API reference](https://crawlingo-docs.vercel.app/reference/dataset-job) ·
[Release status](https://github.com/Vamshavardhan50/crawlingo/blob/main/guides/RELEASE_STATUS.md) ·
[Measured scale](https://github.com/Vamshavardhan50/crawlingo/blob/main/SCALE_RESULTS.md)

MIT license. No hosted API, semantic deduplication or checkpointed Parquet jobs
are included in this dataset workflow. Performance depends on the source and workload.

