Metadata-Version: 2.5
Name: invenio-pidbox
Version: 0.2.12
Summary: InvenioRDM PIDBox integration for people, organizations, and works.
Project-URL: Homepage, https://codeberg.org/front-matter/invenio-pidbox
Project-URL: Repository, https://codeberg.org/front-matter/invenio-pidbox
Project-URL: Bug Tracker, https://codeberg.org/front-matter/invenio-pidbox/issues
License: MIT
License-File: LICENSE
Keywords: author,citation,invenio,organization,pidbox
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Internet :: WWW/HTTP :: Dynamic Content
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.14
Requires-Dist: commonmeta-py[backend]>=0.272
Requires-Dist: commonmeta-rs>=0.9.80
Requires-Dist: invenio-app-rdm>=14.0.0
Provides-Extra: tests
Requires-Dist: invenio-app>=1.4.0; extra == 'tests'
Requires-Dist: invenio-rdm-records>=14.0.0; extra == 'tests'
Requires-Dist: invenio-search[opensearch2]>=3.0.0; extra == 'tests'
Requires-Dist: pytest-black>=0.3.0; extra == 'tests'
Requires-Dist: pytest-isort>=3.1.0; extra == 'tests'
Requires-Dist: pytest<9,>=7.4.0; extra == 'tests'
Description-Content-Type: text/markdown

# invenio-pidbox

InvenioRDM integration that serves people, organizations and citations from a
commonmeta SQLite store, read through commonmeta-py and commonmeta-rs.

The store is the one a commonmeta import already maintains — ORCID persons, ROR
organizations, works and the relations between them. This package reads it, and
adds no tables of its own: no application-specific people or organizations
tables, no citation store, no schema. Everything it exposes is a view onto that
one file.

## Features

**Profiles** — people and organization profile pages for InvenioRDM, with list
and search pages, template context injected on the profile routes, and a
stylesheet and search-result components shipped as webpack bundles.

**Deposit autocomplete** — creator, affiliation and funder suggestions served
from commonmeta-rs Tantivy search, with the deposit form's overridable
components registered to route lookups to pidbox when configured.

**Record relations** — affiliations and funders resolved from the store as well
as the vocabularies, so a record may name any organization the store knows.
Opt-in, because it changes which records an instance accepts.

**Facet labels** — search facets fall back to the store for ids the vocabulary
cannot label, so an organization accepted by the relations above does not appear
in the sidebar as a bare ROR id.

**Citations** — citation counts and citing works, read from the store's
`pid_relations` table and served as commonmeta works, plus a
`pidbox:citations` custom field for the works a record itself cites.

**Bounded store access** — every read runs with a deadline behind a circuit
breaker, so a store that has gone slow degrades the feature that reads it
instead of holding request threads.

## Requirements

- Python 3.14+
- A commonmeta SQLite database readable through commonmeta-rs

Internally, commonmeta-rs reads `pid_annotations` records with source_id 4 for
people (ORCID) and source_id 3 for organizations (ROR), works from
`pid_records`, and citations from `pid_relations`.

## Installation

```bash
uv add invenio-pidbox
```

This package depends on commonmeta-py, commonmeta-rs, and invenio-app-rdm at
runtime, so installing invenio-pidbox pulls in the profile backend
automatically.

## Configuration

### Required

- `PIDBOX_SQLITE_PATH` — path to the commonmeta SQLite store, used by every
  feature below. Defaults to `/var/lib/commonmeta/commonmeta.sqlite3`.

### Deposit autocomplete

- `APP_RDM_DEPOSIT_FORM_AUTOCOMPLETE_NAMES`
- `APP_RDM_DEPOSIT_FORM_AUTOCOMPLETE_NAMES_PROVIDER`
- `PIDBOX_AUTOCOMPLETE_API`

To enable pidbox autocomplete in the deposit form, set:

```python
APP_RDM_DEPOSIT_FORM_AUTOCOMPLETE_NAMES = "search"
APP_RDM_DEPOSIT_FORM_AUTOCOMPLETE_NAMES_PROVIDER = "pidbox"
```

The endpoint URLs can be overridden with:

```python
PIDBOX_AUTOCOMPLETE_API = {
    "names": "/api/pidbox/names",
    "affiliations": "/api/pidbox/affiliations",
    "funders": "/api/pidbox/funders",
}
```

The funders endpoint returns only organizations with `"funder"` in their `types`
list, so the deposit form offers exactly the set a commit will accept.

### Record relations

```python
PIDBOX_RECORD_RELATIONS = True
```

Off by default. InvenioRDM validates every affiliation and funder id on each
record commit against the `affiliations` and `funders` vocabularies, so a record
naming an organization that was never imported into them is rejected with
`InvalidRelationValue`. With this on, the store is consulted first and the
vocabulary second, making the accepted set the union of the two rather than a
swap — an unreadable store degrades to exactly the stock behaviour.

Turning this on changes which records an instance will accept, which is why
installing the package does not.

### Facet labels

Facets label their buckets from the vocabularies, so an organization accepted by
the relations above but absent from the vocabulary shows as a raw ROR id. The
store-backed labels fill only those gaps, asking the vocabulary first so a
curated, localized title keeps winning:

```python
from invenio_pidbox.facets import StoreAffiliationsLabels, StoreFundersLabels
```

Use them where the facet is declared, in place of the stock
`AffiliationsLabels` / `FundersLabels` — for example
`value_labels=StoreFundersLabels("funders")`.

### Store access

- `PIDBOX_STORE_TIMEOUT` — seconds a single store read may take before the
  caller gives up (default `2.0`; `0` disables the deadline)
- `PIDBOX_STORE_FAILURE_THRESHOLD` — consecutive timeouts before the store is
  treated as unavailable (default `3`)
- `PIDBOX_STORE_RECOVERY_TIME` — seconds to leave an unavailable store alone
  before trying again (default `60.0`)

These exist because a store shared with a writer can get very slow: reads that
normally take a millisecond have blocked for minutes while an import ran against
the same file, and since a relation is resolved on every record commit, that was
enough to exhaust the request threads and have gunicorn kill the worker. A slow
store is therefore treated as an unavailable one.

## Routes

| Route | Purpose |
|---|---|
| `/people/`, `/organizations/` | profile list and search pages |
| `/people/<orcid>` | person profile page |
| `/organizations/<ror>` | organization profile page |
| `/api/pidbox/names` | creator autocomplete |
| `/api/pidbox/affiliations` | affiliation autocomplete |
| `/api/pidbox/funders` | funder autocomplete |
| `/api/pidbox/citations/<doi>` | works citing a DOI |

Each API route is also registered without the `/api` prefix, because the same
extension is registered on the UI app and the API app, and Invenio mounts the
API app *at* `/api` and strips the prefix.

Autocomplete requests carrying `?suggest=` are capped at 50 results; the profile
search pages are fixed to 10 results per page and 100 in total.

### Citations

```http
GET /api/pidbox/citations/<doi>
```

`<doi>` is a bare DOI or a DOI URL. The response uses the usual hits envelope,
and each hit is a commonmeta work, unchanged:

```json
{"hits": {"hits": [{"id": "https://doi.org/10.1234/a", "title": "Work a"}], "total": 3}}
```

Which relations count as a citation (`References`, `Cites`, `IsSupplementedBy` —
DataCite's definition) is decided by commonmeta-rs, not by this package.

`total` is the citation count and can exceed the number of hits: it is read from
the relation index, while hits are hydrated works, so a citing DOI whose work
has not been imported yet is counted but cannot be listed.

Concept DOIs are also left out of the hits. A versioned publisher — Rogue
Scholar, Zenodo — registers a version DOI and a parent DOI standing over it, and
both cite, so the same work would otherwise appear twice under two identifiers
with one citation text between them. The parent is the one carrying a
`HasVersion` relation, and it is dropped in favour of the specific version that
did the citing. The count knows nothing about versions, so such a citation
counts two and lists one.

Listing hydrates every citing work, so `size=0` returns just the count, which is
answered from an index and stays cheap on a heavily cited DOI. Otherwise `size`
(default 10, max 100) and `page` paginate the response.

### The `pidbox:citations` custom field

Records can also carry the works they cite, as a list of
`{identifier, scheme, reference}` entries indexed with the record:

```json
{
  "identifier": "10.59350/4q8j1-1ap35",
  "scheme": "doi",
  "reference": "Willighagen, E. (2007, May 25). Numbers are copyrighted?. <i>Chem-bla-ics</i>."
}
```

`reference` is the citing work rendered as a formatted citation, so a landing
page can show a reference list rather than a column of DOIs. It uses the
instance's own `RDM_CITATION_STYLES_DEFAULT` (`apa` unless changed), so a stored
reference reads the way the citation box on the same page does. RDM has no
config for the locale — it renders in the requesting user's language, which a
background sweep does not have — so `PIDBOX_CITATIONS_LOCALE` supplies it,
defaulting to the `en-US` RDM itself falls back to.

This is the record-side view of a citation, and it is independent of the
store-side one above — the field says what a record cites, the store says what
cites a DOI. They point in opposite directions, so their numbers are not
expected to match.

The field is not registered automatically, because a custom field has to exist
in the search mapping before a record can use it. To enable it:

```python
from invenio_pidbox.custom_fields import (
    CITATIONS_FACET,
    CITATIONS_QUERY_FIELD,
    CitationsCF,
    PIDBOX_NAMESPACE,
)

RDM_NAMESPACES = {**RDM_NAMESPACES, **PIDBOX_NAMESPACE}
RDM_CUSTOM_FIELDS = [*RDM_CUSTOM_FIELDS, CitationsCF(name="pidbox:citations")]
RDM_FACETS = {**RDM_FACETS, "citations": CITATIONS_FACET}
```

Then create the mapping:

```bash
invenio rdm-records custom-fields init -f pidbox:citations
```

`CITATIONS_QUERY_FIELD` is a `QueryParser` mapping entry, so adding
`"citations": CITATIONS_QUERY_FIELD` to `RDM_SEARCH`'s parser mapping makes
`citations:10.5555/12345678` searchable. Declaring `CITATIONS_FACET` does not
display it — the facet also has to be named in `RDM_SEARCH`'s `facets` list.

commonmeta-rs fills the field when it pushes a record: citing works become
`IsReferencedBy` relations, the InvenioRDM serializer maps those to
`custom_fields["pidbox:citations"]`, and `commonmeta push`/`put --to inveniordm`
sends them to the REST API. Reading records back reverses it. So the store stays
the source of truth and the field is a projection of it onto each record.

That name is hardcoded in commonmeta-rs, which is why this package requires
commonmeta-rs >= 0.9.80. Earlier versions write `rs:citations`, and against them
the field is defined, indexed, and never populated.

This field comes from Rogue Scholar, where it was `rs:citations`. It is
unchanged apart from the namespace, and the two names are different fields: an
instance holding records under the old one has to migrate and reindex them, and
until it does, those records read as having no citations.

### Keeping the field current

A push writes the citations a record had *at that moment*. What makes it stale
afterwards is other records arriving, which is not an event on this record, so
nothing in InvenioRDM would ever notice.

The package therefore ships a sweep, as a job in the **Jobs** administration
dashboard:

```text
Refresh citations   (pidbox_refresh_citations)
```

Installing the package makes the job available there; it does not schedule it.
Whether it runs at all, how often, and with what arguments is set in the
dashboard — this one edits records, which is not something installing a package
should start doing on a timer. Running it needs the Celery worker a standard
deployment already has.

For every record with a DOI the sweep compares `count_sqlite_citations` — an
index lookup — against the number of citations the record already stores, and
skips the record without hydrating anything when they agree. Only a mismatch
pays for the full read, and only a genuine difference in identifiers is written
back, as a metadata edit through the records service (same version, same DOI,
new revision).

For a first run, set the dashboard's custom arguments to bound it and check the
counts it reports:

```json
{"max_records": 100}
```

It returns `{"scanned", "hydrated", "updated", "failed"}`. The same task can be
called directly, for example from `invenio shell`:

```python
from invenio_pidbox.tasks import refresh_citations

refresh_citations(max_records=100)
```

Rendering a citation costs about 25 ms, so a reference already stored for a DOI
is kept rather than rendered again: a run only formats citations the record did
not already have. Records written by `commonmeta push` carry identifiers with no
reference text, and the sweep backfills those.

Four behaviours worth knowing:

- The dashboard's **`since` is ignored**. A record's citations change when other
  records are imported, which does not modify the record, so no timestamp on it
  reflects the change — filtering by one would skip exactly the records that
  need updating. Being cheap on unchanged records is what replaces it.
- A record cited by a work the store has not imported has a count higher than
  anything that can be listed, so it takes the slow path on every run and writes
  nothing. Correct, but not free. A citing work with a concept DOI (below) does
  the same.
- The sweep **never clears** a record's citations. Every store read degrades to
  "none" when the store is unavailable, which is indistinguishable from a DOI
  nothing cites, so a record that holds citations keeps them rather than risk
  emptying every record in the instance.

## How It Works

When a request path matches `/people/<orcid>` or `/organizations/<ror>`, the
extension loads the corresponding commonmeta record from SQLite and exposes
template-friendly objects such as person, organization, employment, identifiers,
relations, location, orcid, ror, and search_config. It also registers the
profile routes and a `country_name` template filter, so it runs against a
standard invenio-app-rdm installation.

For deposit autocomplete it patches invenio-app-rdm's form config at runtime to
inject the endpoint URLs, and loads overridable components that switch the
person, organization and funder remote selects to those endpoints. Awards lookup
stays on the existing awards API.

Reads are optional throughout, and each has a defined fallback: a profile falls
back to a record search, an autocomplete to no suggestions, a relation to the
vocabulary, a facet label to the bare id, a citation listing to none. An absent,
unreadable or slow store therefore costs a feature, not the page.

## Development

```bash
uv sync --extra tests
uv run pytest
```

## Entry points

| Group | Name | Value |
|---|---|---|
| `invenio_base.apps` | `invenio_pidbox` | `invenio_pidbox.ext:InvenioPidbox` |
| `invenio_base.api_apps` | `invenio_pidbox` | `invenio_pidbox.ext:InvenioPidbox` |
| `invenio_assets.webpack` | `invenio_pidbox` | `invenio_pidbox.webpack:profiles` |

## License

MIT
