Metadata-Version: 2.5
Name: rer.linkchecker
Version: 1.0.0a6
Summary: An addon that check all links in site contents and generate a report with broken ones
Project-URL: Homepage, https://github.com/RegioneER/rer-linkchecker
Project-URL: PyPI, https://pypi.org/project/rer.linkchecker
Project-URL: Source, https://github.com/RegioneER/rer-linkchecker
Project-URL: Tracker, https://github.com/RegioneER/rer-linkchecker/issues
Author-email: RedTurtle Technology <sviluppo@redturtle.it>
License: GPL-2.0-only
License-File: LICENSE.GPL
License-File: LICENSE.md
Keywords: CMS,Plone,Python
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Web Environment
Classifier: Framework :: Plone
Classifier: Framework :: Plone :: 6.1
Classifier: Framework :: Plone :: 6.2
Classifier: Framework :: Plone :: Addon
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: <3.14,>=3.11
Requires-Dist: plone-api
Requires-Dist: plone-restapi
Requires-Dist: plone-volto
Requires-Dist: products-cmfplone
Requires-Dist: requests
Provides-Extra: test
Requires-Dist: horse-with-no-namespace; extra == 'test'
Requires-Dist: plone-app-testing; extra == 'test'
Requires-Dist: plone-restapi[test]; extra == 'test'
Requires-Dist: pytest; extra == 'test'
Requires-Dist: pytest-cov; extra == 'test'
Requires-Dist: pytest-plone>=1.0.0a2; extra == 'test'
Description-Content-Type: text/markdown

# rer.linkchecker

A Plone addon that checks every internal and external link found in site contents (fields and Volto blocks) and generates a report of the broken ones.

## Features

- Adds a `portal_linkchecker` tool that crawls the whole site, collecting every link found in content fields and in Volto blocks (`resolveuid/...` links included).
- Checks internal links by resolving them against the catalog, and external links concurrently over HTTP (configurable timeout, thread pool size, and per-host throttling).
- Caches external link statuses for a configurable TTL, so repeated runs only re-check links that are due.
- Distinguishes real broken links from bot-protection responses (`403`, `429`, LinkedIn's `999`) and from `http://` links that only work over `https://` (reported so they can be fixed in place, not counted as broken).
- Reports the conditions that are not a plain http status with their own negative status, so they can be told apart in the CSV: `-1` timeout, `-2` works only over `https` (update the link), `-3` connection error.
- Does not verify TLS certificates: only reachability matters here, and many otherwise working servers omit their intermediate certificate (browsers fetch it themselves, `requests` does not), which would be reported as a broken link.
- Exposes the results as a CSV report via `tool.get_rows()`, with the same columns and labels as the report panel: site content, link to check, link type, outcome (the status code), outcome description and the action to take (to fix, to update, to check), translated in the language of the request.
- Ships a `check_broken_links` console script to run a check from the command line or from cron, without going through the web.

## Installation

Install `rer.linkchecker` with uv:

```shell
uv add rer.linkchecker
```

or add it to a zc.buildout-based project as a develop egg / source, alongside `plone.volto` (required: it provides the `blocks`/`blocks_layout` fields the linkchecker scans, and `plone.distribution` used by `addPloneSite`).

Then add `rer.linkchecker` to the `eggs` of your Plone instance and install the add-on from the Plone control panel (or via a GenericSetup profile) as usual.

## Usage

### The `portal_linkchecker` tool

```python
from plone import api

tool = api.portal.get_tool("portal_linkchecker")
tool.check_site()  # crawl the whole site and check every link

for uid, broken_links in tool.get_page_with_broken_links():
    ...  # [(link, status), ...] per content UID

for item in tool.get_broken_links():
    ...  # flat view: one dict per broken link, with the page it sits on

import csv

with open("broken_links.csv", "w", newline="") as fh:
    writer = csv.writer(fh, quoting=csv.QUOTE_ALL)
    for row in tool.get_rows():
        writer.writerow(row)
```

`get_broken_links()` is the flat source every report is built from, and `get_rows()` is its csv
rendering. Both include the bot-protected links (each with its own description), while
`get_page_with_broken_links()` leaves them out: it answers "how many real problems are there".
Filter either with `tool.filter_links(items, status=[404], link_type="EXTERNAL")`.

`check_site(ttl=3600 * 6, timeout=15, max_workers=10)` accepts:

- `ttl`: seconds a cached external link status stays valid (`0` forces a full recheck).
- `timeout`: per-request timeout, in seconds, for external links.
- `max_workers`: number of concurrent threads checking external links.

### The `check_broken_links` console script

Installed as a standard `console_scripts` entry point (`[project.scripts]` in `pyproject.toml`), so it lands in `bin/` both in a uv-managed virtualenv and in a zc.buildout instance. Since it runs Zope/ZODB code, it must be launched through `zconsole`/`instance run`, not called directly:

```shell
# uv-managed instance (Makefile: make check-broken-links)
./bin/zconsole run instance/etc/zope.conf ./.venv/bin/check_broken_links

# zc.buildout instance
./bin/instance run bin/check_broken_links
```

Options:

- `--ttl`, `--workers`, `--timeout`: same meaning as on `check_site()`.
- `--output-dir`: also dump the run as a csv in this directory (default: no csv).
- `--site-id`: id of the Plone site to check (default: the `PLONE_SITE_ID` env var, or `Plone`).
- `--url <url>`: verify a single url and log its status, without touching the site.
- `--content <path-or-UID>`: verify a single content's links and log them, without touching the site.

The script stores the report in the site, which is where the REST API endpoints below read it
from: that is the normal way to get at the result. With `--output-dir` it also writes
`<output-dir>/<siteid>_broken_links_<YYYYMMDD-HHMMSS>.csv`, byte for byte the csv
`@linkchecker-csv` serves — worth it only to keep an archive of past runs, since the site holds
the last one alone.

### The REST API endpoints

Two read-only endpoints on the site root expose the stored report, both guarded by the
`rer.linkchecker.ViewReport` permission (granted to `Manager`, `Site Administrator` and
`Editor`).

They **never run a check**: a full check takes minutes on a medium site and would time the
request out, so they only serve what the last run stored. Refresh the data out of band, with
the `check_broken_links` script from cron. Because the data is therefore asynchronous, both
responses carry the timestamp it dates from.

```shell
# the report as json, batched (plone.restapi conventions: b_start, b_size)
curl -u user:pass "$SITE/++api++/@linkchecker" -H 'Accept: application/json'

# the same report as a csv download
curl -u user:pass -OJ "$SITE/++api++/@linkchecker-csv"
```

Both accept the same filters:

- `status`: repeatable, e.g. `?status=404&status=-2` (negative values are the `STATUS_*`
  constants: `-1` timeout, `-2` https-only, `-3` connection error).
- `type`: `INTERNAL` or `EXTERNAL`.

An invalid filter value answers `400` rather than silently returning an empty report.

`@linkchecker` returns `last_update` and `duration` for the whole run, `items_total`, the
batched `items`, and a `summary` listing `{status, status_description, count}` sorted by count.
The summary is computed over the **unfiltered** report on purpose, so the counts a UI shows in
its filter chips do not move as filters are applied. When no check has ever run, `last_update`
is `null` and `items`/`summary` are empty, which a UI can tell apart from "nothing is broken".

Each item describes the page it sits on with the usual plone.restapi field names, so a client
can treat it as any other content reference, and names the link's own fields apart from those:

```json
{
  "@id": "http://site/bandi-e-avvisi",
  "@type": "Document",
  "title": "Bandi e avvisi",
  "UID": "1a568f09734340dfba2c1a53730b9cf6",
  "link": "https://unimc.it/careerday",
  "link_type": "EXTERNAL",
  "status": 404,
  "status_description": "Not Found",
  "last_update": "2026-07-30T03:00:12"
}
```

`@linkchecker-csv` returns the exact same csv the console script writes (same columns, same
quoting), names the file after the date of the *data* rather than of the download, and repeats
that date in the `X-Linkchecker-Last-Update` response header.

## Development

### Prerequisites

- An [operating system](https://6.docs.plone.org/install/create-project-cookieplone.html#prerequisites-for-installation) that runs all the requirements mentioned.
- [uv](https://6.docs.plone.org/install/create-project-cookieplone.html#uv)
- [Make](https://6.docs.plone.org/install/create-project-cookieplone.html#make)
- [Git](https://6.docs.plone.org/install/create-project-cookieplone.html#git)
- [Docker](https://docs.docker.com/get-started/get-docker/) (optional)

### Setup

```shell
git clone git@github.com:RegioneER/rer-linkchecker.git
cd rer-linkchecker/backend
make install
```

### Common tasks

```shell
make start                  # start a Plone instance on localhost:8080
make create-site            # create a new site from scratch
make check-broken-links     # run the linkchecker and write a csv report
make test                   # run the test suite
```

## Contribute

- [Issue tracker](https://github.com/RegioneER/rer-linkchecker/issues)
- [Source code](https://github.com/RegioneER/rer-linkchecker/)

## License

The project is licensed under GPLv2.

## Credits

Developed with the support of [Regione Emilia Romagna](http://www.regione.emilia-romagna.it/).

Regione Emilia Romagna supports the [PloneGov initiative](http://www.plonegov.it/).

## Authors

This product was developed by **RedTurtle Technology** team.

[![RedTurtle Technology](https://avatars1.githubusercontent.com/u/1087171?s=100&v=4)](http://www.redturtle.it/)
