Metadata-Version: 2.4
Name: scrape-applied-links
Version: 0.1.0
Summary: Bulk-run render-and-parse over the urls in a links_applied_by_date.json file, recording the resulting output paths back into it.
Author-email: Umer Khalid <umerkha2007@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/umerkhalid/scrape-applied-links
Keywords: scraping,render-url,cli,job-applications
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: render-url

# WebScraper-Caller

Small CLI that reads a JSON file of objects, calls `render-and-parse` on
each object's `url`, and records the resulting output file paths back into
the JSON for later use. The only field it cares about is `url` — everything
else is optional and passed through untouched.

## Installation

```bash
pip install scrape-applied-links
playwright install chromium
```

This installs the `scrape-applied-links` command along with its
`render-url` dependency (which provides `render-and-parse`).

Installing from source instead:

```bash
pip install -e .
playwright install chromium
```

## Input format

The tool only requires a `url` field on each object — everything else is
optional and simply passed through untouched. It accepts either of two
shapes for the input file:

**A flat list of objects**, each needing nothing but a `url`:

```json
[
  { "url": "https://example.com/careers/123" },
  { "url": "https://example.com/careers/456" }
]
```

**An object mapping arbitrary labels (e.g. dates) to lists of objects** —
useful if you want to group entries, such as by the date they were scraped:

```json
{
  "2026-09-16": [
    { "url": "https://example.com/careers/123" },
    { "url": "https://example.com/careers/456" }
  ]
}
```

Objects can carry any extra fields you like (`company`, `title`,
`description`, etc.) — they're ignored by this tool and left as-is. If a
`company` field is present it's shown in log output while processing, but
it isn't required.

| Field | Type | Required | Notes |
|---|---|---|---|
| `url` | string | **yes** | The page passed to `render-and-parse`. Objects without a `url` are skipped entirely. |
| anything else | any | no | Passed through untouched. |

After an object is processed, this tool adds two fields to it:

| Field | Type | Description |
|---|---|---|
| `rendered_page` | string | Path to the `rendered_page_N.json` file written by `render-and-parse` for this url. |
| `parsed_page` | string | Path to the `parsed_page_N.json` file written by `render-and-parse` for this url. |

## Usage

```bash
scrape-applied-links [json_file] [options]
```

- `json_file` (optional, positional) — path to the input JSON file (either
  input shape described above). Defaults to `links_applied_by_date.json`
  in the current directory.

### Options

| Flag | Description |
|---|---|
| `--dry-run` | List the URLs that would be processed without actually calling `render-and-parse`. |
| `--force` | Re-scrape jobs that already have a `rendered_page`/`parsed_page` recorded. Without this flag, jobs already processed are skipped. |
| `--output-dir <dir>` | Directory where `render-and-parse` writes its `rendered_page_N.json` / `parsed_page_N.json` files. Defaults to the current directory. |

### Examples

Process the default file:

```bash
scrape-applied-links
```

Preview which URLs would be scraped, without calling anything:

```bash
scrape-applied-links --dry-run
```

Re-run and re-scrape every job, even ones already processed:

```bash
scrape-applied-links --force
```

Use a different input file and write render output to a subfolder:

```bash
scrape-applied-links my_links.json --output-dir out/
```

## What it does

1. Loads the JSON file (a flat list or a label-keyed object of lists) and
   collects every object that has a `url`.
2. Skips objects that already have a `rendered_page` value, unless
   `--force` is passed.
3. For each remaining object, runs `render-and-parse <url>` (in
   `--output-dir` if given).
4. Detects the newly created `rendered_page_N.json` and `parsed_page_N.json`
   files and adds their paths to that job's entry as `rendered_page` and
   `parsed_page`.
5. Saves the JSON file after each successfully processed job, so progress
   isn't lost if a later URL fails.
6. If any URLs failed, prints a summary of them at the end and exits with a
   non-zero status.
