Metadata-Version: 2.5
Name: crawlerflow
Version: 0.1.0
Summary: Declarative YAML workflow engine for browser automation and web scraping
Project-URL: Homepage, https://github.com/mehmetemineker/crawlerflow
Project-URL: Repository, https://github.com/mehmetemineker/crawlerflow
Project-URL: Issues, https://github.com/mehmetemineker/crawlerflow/issues
Project-URL: Documentation, https://github.com/mehmetemineker/crawlerflow/tree/main/docs
Author: Mehmet Emin Eker
License-Expression: MIT
License-File: LICENSE
Keywords: automation,browser,crawler,scraping,workflow,yaml
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Browsers
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic>=2.8
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.7
Requires-Dist: typer>=0.12
Provides-Extra: browser
Requires-Dist: pydoll-python<3,>=2.23; extra == 'browser'
Provides-Extra: dev
Requires-Dist: pydoll-python<3,>=2.23; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.2; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Description-Content-Type: text/markdown

# CrawlerFlow

CrawlerFlow is a declarative, YAML-based workflow engine for browser automation and web
scraping. Workflows describe what should happen; adapters and steps decide how it happens.

## Current foundation

- Versioned YAML workflow loading and validation
- Browser-independent adapter contract and a lazy-starting Pydoll implementation
- Extensible step registry
- Isolated plugin API with typed YAML settings, lifecycle hooks, steps, filters, and subscribers
- Async workflow executor and event bus
- Variable interpolation and a built-in expression engine
- Per-run `today` and `now` date variables
- Nested `foreach`, `foreach_date`, `foreach_select`, and declarative `if` control flow
- Reusable parameterized workflow macros
- Per-step retry and continue/fail error policies
- JSON Lines workflow, step, retry, and request event logging
- Built-in navigation, interaction, cookies, downloads, screenshots, and selective HTML output
- `run`, `validate`, `list-steps`, `list-plugins`, and `doctor` CLI commands

## Development

```bash
python -m pip install -e ".[dev]"
pytest
crawlerflow validate examples/basic.yaml
crawlerflow run examples/basic.yaml
```

Run multiple workflows sequentially by supplying more paths. Execution stops at the first failed
workflow:

```bash
crawlerflow run examples/first-site.yaml examples/second-site.yaml
```

A directory argument discovers its directly contained `.yaml` and `.yml` files and runs them in
alphabetical order:

```bash
crawlerflow run examples
```

Use asynchronous mode to run every supplied workflow in parallel. All workflows are allowed to
finish; the command exits with code `1` if any workflow fails:

```bash
crawlerflow run --mode async examples/first-site.yaml examples/second-site.yaml
```

Directory discovery can also be combined with parallel execution:

```bash
crawlerflow run --mode async examples
```

Add `--progress` to display a live progress bar based on the total workflow count. The bar advances
as each workflow succeeds or fails in both sequential and asynchronous modes:

```bash
crawlerflow run --mode async --progress examples
```

Use `--concurrency` (or `-c`) to limit how many workflows run at the same time in asynchronous
mode. This avoids starting every HTTP client or browser session simultaneously:

```bash
crawlerflow run --mode async --concurrency 8 --progress examples
```

Omitting the option preserves unlimited parallel execution. `--concurrency` accepts positive
integers and can only be used with `--mode async`.

Install browser support and select Pydoll in a workflow:

```bash
python -m pip install -e ".[browser,dev]"
```

```yaml
browser:
  engine: pydoll
  headless: true
```

Browser-free workflows omit the `browser` section. Applications can also inject another
`BrowserAdapter` into `WorkflowRunner`. See `docs/http-requests.md` for direct HTTP requests and
shortened map URL coordinate resolution.

External extensions can register entry points under `crawlerflow.plugins`; workflows activate only
the plugins they list. See `docs/plugins.md` and the installable `examples/plugins/example` package
for the plugin contract, discovery command, and packaging example.
