Metadata-Version: 2.5
Name: refaudit
Version: 0.4.7
Summary: Verify bibliography entries against Crossref, DataCite, arXiv and OpenAlex before you submit.
Project-URL: Homepage, https://github.com/uw-share-lab/refaudit
Project-URL: Issues, https://github.com/uw-share-lab/refaudit/issues
License: MIT
License-File: LICENSE
Keywords: arxiv,bibtex,citations,crossref,openalex,research-integrity
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Text Processing :: Markup :: LaTeX
Classifier: Typing :: Typed
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: mypy>=1.5; extra == 'dev'
Requires-Dist: pdoc>=14; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Requires-Dist: types-defusedxml>=0.7; extra == 'dev'
Provides-Extra: xml
Requires-Dist: defusedxml>=0.7; extra == 'xml'
Description-Content-Type: text/markdown

# refaudit

Verify the entries in a `.bib` against Crossref, arXiv and OpenAlex, and report
the ones a human needs to look at.

Written for venues that run automated checks for hallucinated or malformed
references. The failure it is built to catch is not a missing DOI — it is a DOI
that resolves to **a different paper than the entry claims**. That one is
invisible when you read the reference list, because the title and authors look
fine and only the identifier is wrong.

## Install

```bash
pip install refaudit
```

Python 3.10+. No runtime dependencies.

Latest development version:

```bash
pip install git+https://github.com/uw-share-lab/refaudit.git
```

## Quick start

It reads the whole `.bib` and checks **every** entry in it — you point it at the
file and it works through the lot, one entry at a time.

```bash
# every entry in the file
refaudit refs.bib --email you@uwaterloo.ca

# only the entries actually cited in the paper, which is usually what you want
refaudit refs.bib --email you@uwaterloo.ca --tex paper/sections --only-cited
```

`--email` is required. Crossref and OpenAlex give identified callers a separate,
more reliable request pool, and it is the courtesy their documentation asks for.
Set it once instead of typing it each time:

```bash
export REFAUDIT_EMAIL=you@uwaterloo.ca
```

Results are written to `refaudit-out/` as `reference_check.txt` (readable) and
`reference_check.csv` (sortable), and printed to stdout.

### Working from Overleaf

Download the `.bib` (Menu → Download → Source, or just the file), then point
`--tex` at the unzipped `sections/` directory so `--only-cited` can tell which
keys actually reach the PDF:

```bash
refaudit sample-base.bib --email you@uwaterloo.ca --tex sections/ --only-cited
```

## What it reports

| Verdict | Meaning |
|---|---|
| `TITLE_MISMATCH` | an identifier resolved to a different paper — check first |
| `DEAD_DOI` | the DOI is registered with no agency — confirmed against `doi.org` |
| `AUTHOR_MISMATCH` | titles agree, first author does not |
| `YEAR_MISMATCH` | titles agree, year is off by more than one |
| `NOT_FOUND` | no identifier, and no title match anywhere |

An arXiv identifier counts wherever the entry keeps it: `eprint`, or free text
such as `journal={arXiv preprint arXiv:2506.08872}`, which is what Google
Scholar exports look like.
| `UNVERIFIED` | **not checked — a source was unreachable, or the DOI is real but unindexed. Says nothing about the entry** |
| `SKIPPED` | `@misc`/`@online`/`@phdthesis` and similar with no identifier; nothing to check against |
| `OK` | resolved and consistent |

Exit status is `1` if there is at least one finding, `0` if not, `2` on a usage
error — so it drops into CI or a pre-submission script.

## Options

| Flag | Effect |
|---|---|
| `--email` | contact address sent to the APIs (or `REFAUDIT_EMAIL`). Required. |
| `--tex PATH` | LaTeX file or directory, used to work out which keys are cited |
| `--only-cited` | check only cited keys; requires `--tex` |
| `--resolvers` | comma-separated subset of `crossref:doi`, `datacite:doi`, `doi:content`, `arxiv:id`, `dblp`, `openalex`, `crossref:title`, `openlibrary` |
| `--out DIR` | output directory (default `refaudit-out`) |
| `--cache PATH` / `--no-cache` | cache location, or disable it |
| `--ttl-days N` | how long cached results stay valid (default 90) |
| `--timeout N` | per-request timeout in seconds (default 20) |
| `--title-match N` | similarity at or above which two titles are the same work (default 0.75) |
| `--workers` | entries checked in parallel (default 4). Each service keeps its own rate limit regardless |
| `--no-duplicates` | skip the offline duplicate-entry pass |
| `--no-shared-pacing` | do not share rate limits with other refaudit runs on this machine |
| `--quiet` | suppress per-entry progress, print only the summary |
| `-v`, `--verbose` | log every request, retry and rate-limit change to stderr |

A run over a few hundred references takes minutes, because it is deliberately
paced. Successful lookups are cached, so it is safe to interrupt with Ctrl-C and
re-run — it picks up where it stopped.

## The one design rule

**"We could not check" and "this is wrong" are different answers and never
collapse into each other.**

This sounds obvious and is easy to get wrong. arXiv rate-limits whole networks;
when that happens, a naive checker either silently passes the entry (false
comfort, the worse failure) or falls back to a title search, finds something
loosely related, and reports a mismatch (false alarm, which trains you to ignore
it). Both are worse than saying "I could not check this one."

So every resolver returns exactly one of `Found`, `NotFound`, or `Unavailable`,
and only `Found` can produce a negative verdict. `UNVERIFIED` results are listed
separately from findings and are never cached, so a transient outage does not
get baked into later runs.

Relatedly, evidence is weighted by strength: a title search returning something
different is only a finding when there was no identifier to go on. Otherwise
every arXiv-only workshop paper that Crossref does not index would be flagged.

The same care applies to DOIs, and for a while it did not. No registration
agency speaks for the whole DOI system: Crossref registers most published
literature, DataCite registers preprints and deposits — including every arXiv
DOI (`10.48550/*`), Zenodo and figshare. Reading Crossref's 404 as "this DOI
does not exist" reported 22 live preprints in a real bibliography as dead
references. `DEAD_DOI` is now the one verdict confirmed against a second source:
every agency has to disown the DOI *and* `doi.org` — which answers for all of
them — has to report it unregistered. If the DOI resolves but nothing indexes
it, that is `UNVERIFIED`; if `doi.org` cannot be reached, that is also
`UNVERIFIED`. A reference is never called dead on one agency's silence.

## Sources, and why there are several

No source is load-bearing. Every one of them can be missing, metered or
unreachable without the run producing a false finding — that is the point of
having several, not redundancy for its own sake.

| Source | Used for | Why it is in the list |
|---|---|---|
| Crossref | DOI + title | registers most published literature; publishes the rate-limit headers we pace ourselves by |
| DataCite | DOI | registers preprints and deposits — every `10.48550/*` arXiv DOI, Zenodo, figshare |
| `doi:content` | DOI | content negotiation via `doi.org`, which answers for **any** agency, including the ones we do not query directly (mEDRA, JaLC, KISTI, OP) |
| arXiv | arXiv ID | authoritative for preprints, and the only source with the versioned record |
| DBLP | title | hand-curated for computer science, free and unmetered; usually returns the DOI too, turning "not found" into a correction |
| OpenAlex | title | broad but noisier, and **now meters usage** — a fallback, no longer the backbone |
| Open Library | title | monographs, which no article index will ever hold |

Sources are tried in order of how much their answer is worth: identifier lookups
before title searches, curated indexes before harvested ones. An identifier that
resolves to a different paper ends the search — that disagreement *is* the
finding. A weak title hit does not, because otherwise the first index to return
anything at all would mask a better answer from the next.

**If a source is down or metered, drop it** and the rest still work:

```bash
refaudit refs.bib --email you@uwaterloo.ca \
  --resolvers crossref:doi,datacite:doi,doi:content,arxiv:id,dblp,crossref:title
```

## Duplicates

The same work cited under two keys is invisible to the per-entry check: both
copies resolve, both are correct, both report `OK`. Only comparing entries with
each other finds it, so that runs as its own pass — offline, on entry order, and
matching on DOI, arXiv ID (including the arXiv DOI form of the same ID) and
near-identical titles. It costs nothing and works when every network source is
refusing us.

## Rate limiting

Each source declares the limit its own documentation specifies, next to the code
that calls it:

| Source | Rate used | Why |
|---|---|---|
| Crossref | 2/s, then whatever the response headers say | Crossref publishes `X-Rate-Limit-Limit` / `-Interval`; the client reads and obeys them |
| arXiv | 1 per 3s | [arXiv's terms of use](https://info.arxiv.org/help/api/tou.html) specify exactly this |
| DBLP | 1/s | a small academic service that asks callers not to hammer it |
| Open Library | 1/s | donation-funded; asks for a descriptive User-Agent and modest rates |
| DataCite | 2/s | [asks for reasonable use](https://support.datacite.org/docs/api) and throttles heavy callers; matched to our Crossref pace |
| doi.org | 2/s | the proxy redirects to the owning agency, so each call costs a third party a real request |
| OpenAlex | 3/s | metered against a [daily budget](https://openalex.org/pricing) that resets at midnight UTC, not a request ceiling; a free caller exhausts it quickly, so treat it as a fallback |

Requests retry up to four times with exponential backoff and **full jitter**, so
a transient failure does not become a false `UNVERIFIED` and simultaneous
retries do not synchronise into a thundering herd. A definitive `4xx` is never
retried — it is an answer, not a failure. Every request sent is paced, retries
and redirect hops included: a redirect is a real request to a real server, so it
costs the same as any other.

Limits are held **per host, not per resolver**: two resolvers calling
`api.crossref.org` are still one caller as far as Crossref is concerned, so they
share a single bucket and the most cautious rate either declares. The same
applies to the circuit breaker — when a host starts refusing us, every resolver
that calls it backs off together. Raising `--workers` does not raise any of
this: threads queue on the same buckets.

A `429` is treated as instruction rather than noise: `Retry-After` is honoured,
the token bucket is halved, and after repeated refusals a circuit breaker stops
asking that host so the rest of the run still finishes. Retries use exponential
backoff with full jitter.

If a `429` carries a `Retry-After` longer than a minute, that is an answer
rather than a delay: the request fails immediately and the host is stood down
for the period it asked for, so the rest of the run skips it instead of
rediscovering the same refusal entry by entry. Anything that source would have
answered comes back `UNVERIFIED` — no verdict, never a finding.

The halving is a penalty the run can work off. Each success afterwards edges the
rate back up by a twentieth of the host's ceiling, and a penalty never takes it
below a sixteenth of that ceiling. Decrease is multiplicative and recovery is
additive — quick to yield, slow to re-probe — so a burst of refusals early on
costs you seconds rather than leaving the remaining few hundred entries crawling
for the rest of the run. Rates a *service* publishes are different: those are a
ceiling, and recovery never climbs past one.

`--email` is required because Crossref and OpenAlex give identified callers a
separate, more reliable pool, and it is the courtesy their docs ask for.

### Several people running it at once

Everyone runs under their own `--email`, so each of you is a separate identified
caller with your own allowance, your own backoff and your own circuit breaker.
One person hitting a 429 does not slow anybody else down, and no amount of
parallel use by a lab gets an individual blocked. There is nothing to configure.

### Several runs by one person at once

Two terminals, a shell loop or a cluster job array are a different matter: those
are one caller as far as Crossref is concerned, and a per-process limit would
send it a multiple of the rate we promised.

So the token bucket for each host is shared by every refaudit *you* are running,
through a small state file under your own cache directory (`~/.cache/refaudit/`,
or `%LOCALAPPDATA%` on Windows). Four runs at once share one allowance rather
than taking four. A `429` in any of them slows all of them, and the recovery
afterwards is shared too.

It is per user, not per machine, which is deliberate: two people on a shared
server are still two identified callers and should not be throttling each other.

Everything about it degrades. If the state file cannot be created, read, written
or locked, each run paces itself exactly as it did before — correct on its own,
uncoordinated with the others — so the worst case is the behaviour that came
first. `--no-shared-pacing`, or `REFAUDIT_NO_SHARED_PACING=1`, turns it off
entirely and keeps refaudit from writing anything outside `--out`.

Two runs *in the same directory* also share the cache file, which is safe.
Writes are atomic, each flush merges what is already on disk rather than
overwriting it, and the read-merge-write happens under a lock, so neither run
loses the other's entries. Where `flock` is not honoured, as on some network
mounts, the lock falls back to an atomic lock *directory* with two rules that
keep it from being worse than no lock at all: one left behind by a process that
died is taken over after 30 seconds, and waiting for it is bounded, after which
the run carries on unlocked rather than not at all. If you would rather keep the
runs entirely separate, give each one its own `--out`.

## Security

- **HTTPS only.** Plain-`http` URLs are refused, not silently upgraded.
- **Bounded redirects**, kept on https, so a redirect cannot downgrade transport
  or forward the `mailto` identifier somewhere unexpected.
- **Response size cap and per-request timeouts** on every call.
- **XML is parsed with entity declarations refused**, blocking billion-laughs and
  XXE. Uses `defusedxml` when installed, otherwise a hardened stdlib path; both
  raise the same exception type so callers cannot miss one.
- **DOIs and arXiv ids are validated against a pattern before being interpolated
  into a request path**, so a malformed field cannot steer the URL.
- **No credentials in code.** Optional API keys come from the environment and are
  sent as headers, never query parameters, so they stay out of logs.
- **No runtime dependencies.** This gets installed in a hurry near a deadline,
  often on a machine someone else administers; that is the wrong moment to widen
  the supply chain.

## Troubleshooting

**`UNVERIFIED` on an entry whose DOI you know is fine.** The DOI resolves, but
none of the indexes we can read hold metadata for it, so the reference is
unchecked rather than wrong. This is the intended outcome, not a failure.

**Lots of `UNVERIFIED` results.** A source refused your network. arXiv in
particular rate-limits by IP and will 429 an entire institution or VPN
regardless of your own pace. These are not findings — the entries were simply
not checked. Try again later, from a different network, or lean on OpenAlex,
which indexes arXiv identifiers too:

```bash
refaudit refs.bib --email you@uwaterloo.ca --resolvers crossref:doi,openalex
```

**A correct entry is flagged `NOT_FOUND`.** Workshop papers, theses and
tech reports are often in no citation index. If the entry has no DOI and no
arXiv id there is nothing to verify it against; confirm it by hand and move on.

**A correct entry is flagged `TITLE_MISMATCH`.** This one is worth taking
seriously: it means the DOI or arXiv id in your `.bib` resolves to a different
paper. Usually the identifier was copied from the wrong row, or generated rather
than looked up. Check the `found` line in the report against what you meant to
cite.

**You want to see what it is actually doing.** Add `-v`. Every request, retry,
rate-limit change and circuit-breaker trip goes to stderr, which is the fastest
way to tell a slow network apart from a service that is refusing you:

```bash
refaudit refs.bib --email you@uwaterloo.ca -v
```

Diagnostics go to stderr and the report to stdout, so you can keep them apart:

```bash
refaudit refs.bib --email you@uwaterloo.ca -v 2> refaudit.log
```

**Everything is `SKIPPED`.** `@misc` and `@online` entries with no identifier
cannot be checked. That is expected for datasets, blog posts and software.

## API reference

Generated from the docstrings and published on every push to `main`:
**https://uw-share-lab.github.io/refaudit/refaudit.html**

Build it locally with `pdoc` (included in the `dev` extra):

```bash
pdoc refaudit refaudit.checker refaudit.models refaudit.resolvers -o site --docformat google
```

## Library use

```python
from refaudit import Checker, default_resolvers, parse_file

entries = parse_file("refs.bib")
checker = Checker(default_resolvers("you@university.edu"))

for result in checker.check_all(entries):
    if result.verdict.is_finding:
        print(result.key, result.verdict.value, result.note)
```

`Resolver` is a `Protocol`: implement `name`, `rate`, `can_handle` and `resolve`
to add a source, and pass it to `Checker` alongside the built-ins.

Importing refaudit configures no logging and prints nothing — it attaches only a
`NullHandler`, so level and destination stay your application's decision. To see
what the network is doing, handle the `refaudit` logger yourself:

```python
import logging
logging.basicConfig(level=logging.INFO)
logging.getLogger("refaudit").setLevel(logging.DEBUG)
```

`WARNING` carries the things worth acting on — rate-limit penalties, retries and
circuit-breaker trips. `DEBUG` adds one line per request.

## Development

```bash
pip install -e ".[dev]"
pytest          # offline: the suite uses fake resolvers and never hits a network
ruff check .
mypy src
```

The suite is hermetic so CI cannot be failed by a busy upstream. That has a
cost worth being honest about: a green run proves the code behaves correctly
against fixtures, and says nothing about whether it still parses what Crossref
actually sends. One test does check that, and it is opt-in:

```bash
REFAUDIT_LIVE_EMAIL=you@uni.edu pytest tests/test_live.py -v -m live
```

It checks seven references against the real services: one correct DOI, one
arXiv id, one DataCite DOI, a real DOI deliberately paired with the wrong
title, a DOI that does not exist, and an entry with no identifier at all. What
it asserts is the thing that must never break, which is that a correct
reference is never reported as a finding. A source being unreachable is allowed
to turn any of them `UNVERIFIED`, because that is a fact about the network
rather than the entry.

It also runs weekly from [`.github/workflows/live.yml`](.github/workflows/live.yml),
away from pull requests, so an upstream having a bad day costs a notification
rather than a blocked merge. Worth running by hand before cutting a release:
0.4.1 exists because it caught something 127 hermetic tests could not.

`main` is protected: changes go through a pull request with one approving
review, and CI must pass on Python 3.10-3.13. See
[CONTRIBUTING.md](CONTRIBUTING.md).

Repository settings that releases depend on are kept as code in
[`.github/scripts/configure-repo.sh`](.github/scripts/configure-repo.sh) and are
safe to re-run.

## Licence

MIT.
