Metadata-Version: 2.4
Name: splent_feature_elasticsearch
Version: 0.3.1
License-Expression: LGPL-3.0-or-later
Requires-Python: >=3.13
Description-Content-Type: text/markdown
License-File: COPYING
License-File: COPYING.LESSER
Requires-Dist: splent_framework>=1.30.0
Requires-Dist: elasticsearch<9,>=8
Dynamic: license-file

# splent_feature_elasticsearch

Reusable Elasticsearch infrastructure for SPLENT products (service archetype).
Ships a single-node Elasticsearch 8.x container and a generic, domain-agnostic
`ElasticsearchService` that any feature can consume through the service locator.

## What it provides

- `ElasticsearchService` (registered in the service locator):
  `available()`, `index_name(name)`,
  `ensure_index(name, mappings=None, settings=None)`,
  `index_document(index, id, body)`, `bulk_index(index, documents)`,
  `delete_document(index, id)`, `delete_by_query(index, query)`,
  `refresh_index(name)`, `search(index, query_body)`, `delete_index(name)`,
  `alias_targets(alias)`, `point_alias(alias, index)`, `health()`.
- Docker: `docker/docker-compose.dev.yml` (host port published, security
  off) and `docker/docker-compose.prod.yml` (no port published, security
  on), both with a healthcheck and a named `elasticsearch_data` volume.
- CLI: `splent feature:elasticsearch status`.

## Security

The production node runs with security on and publishes no port: the app
reaches it by service name on the product's network, and nothing outside the
host reaches it at all. The node takes `ELASTICSEARCH_PASSWORD` as the
password of its `elastic` superuser, and the app signs in as
`ELASTICSEARCH_USERNAME` with that same password. `docker/.env.prod.example`
ships it as `<SET>`, so `product:deploy` generates one and
`docker compose up` refuses to start until it is set. HTTP stays plain inside
the product's network, which is the only place it travels.

The node reads the password each time its container is created, so changing
it in `.env` and running `docker compose up -d` rotates it. That stops
holding once someone sets the `elastic` password through the security API,
which then wins over the variable. The healthcheck signs in with the same
password, which curl takes from its environment, so it never shows on a
command line.

Still open: the app signs in as `elastic`, the node's superuser, and the
product's web container gets the whole `.env`, that password included. A
leak from that container gives control of the whole node (security,
snapshots, settings), not only of the product's `<prefix>_*` indices. The
fix is a role limited to those indices and a user or API key of the app's
own, with the `elastic` password left to the operator; the feature does not
create them yet. A key made by hand goes in `ELASTICSEARCH_API_KEY`, and it
needs the `monitor` cluster privilege besides the index ones, because
`available()` and `status` ask the node who it is.

The development node runs without security, like the rest of the development
stack, and `ELASTICSEARCH_PASSWORD` stays empty there. The CI test job runs
the development setup too.

`ELASTICSEARCH_API_KEY`, when set, wins over the user and the password. That
is how a product talks to a cluster it does not run: set it together with
`ELASTICSEARCH_URL`. Credentials never appear in a message the service writes
or in what `status` prints, even when they are written into the URL.

`splent feature:elasticsearch status` tells a node that does not answer
(`unreachable`) from one that answers and refuses (`refused`), which after
turning security on almost always means the credentials:

```
  URL:     http://splent_feature_elasticsearch:9200
  Auth:    user elastic
  Enabled: yes
  Indices: uvlhub_splent_*
  Status:  refused
  Reason:  Elasticsearch at http://splent_feature_elasticsearch:9200 rejected the request: AuthenticationException(401, ...)
```

## One node, many products

Each product runs its own node, with its own data volume, on its own network.
Index names are still a flat, node-wide namespace, though, and a product may
be pointed at a node somebody else runs: two products that both index `pages`
there would overwrite each other's documents and answer searches with each
other's material.

Every index name handed to the service is prefixed with
`ELASTICSEARCH_INDEX_PREFIX`, whose shipped value `__PRODUCT__` is resolved to
the product name by `product:env --merge` and `product:build`. A feature keeps
using its own short name and never has to know.

```python
es.ensure_index("pages")          # creates innosoft_app_pages
es.index_name("pages")            # "innosoft_app_pages"
```

`splent feature:elasticsearch status` prints the namespace in use, which is the
quickest way to check that a product is not sharing indices with its neighbour.

An index is a cache and never the authority on visibility. This service stores
and returns whatever documents it is given; whether a particular reader may see
a particular hit is decided at request time by the feature that owns the
material.

## Analyzers and mappings

`ensure_index` takes the index settings as well as the mappings. A mapping
that names a custom analyzer needs it in the settings at creation, because an
open index takes no new analyzers:

```python
es.ensure_index(
    "pages_v1",
    settings={"analysis": {"analyzer": {"folded": {
        "type": "custom", "tokenizer": "standard",
        "filter": ["lowercase", "asciifolding"],
    }}}},
    mappings={"properties": {"title": {"type": "text", "analyzer": "folded"}}},
)
```

An index that already exists is left as it is, whatever either argument says.

## Changing a mapping without emptying search

Deleting an index to build it again with a new mapping leaves search empty
until the reindex ends. Instead, build each version under a name of its own
and reach it through an alias, which `point_alias` moves in one atomic step:

```python
es.ensure_index("pages_v2", mappings=MAPPINGS, settings=SETTINGS)
es.bulk_index("pages_v2", documents)
es.refresh_index("pages_v2")
for old in es.point_alias("pages", "pages_v2"):
    es.delete_index(old)
```

Readers and writers keep using `pages` throughout. `point_alias` returns the
indices the alias pointed at before, and `alias_targets("pages")` says where it
points now.

An index created before versioning holds the alias's own name, and an alias
cannot share it. `point_alias("pages", "pages_v1")` refuses in that case;
`replace_index=True` deletes that index in the same atomic step, once the new
one is built.

## Graceful degradation and typed errors

Elasticsearch is a soft dependency. `available()` never raises, and every other
operation raises `ElasticsearchError`, never a raw client exception. Two
subclasses say what happened and what to do about it.

`available()` is False both when the node does not answer and when it
answers and refuses, and a consumer falls back either way. The second is
almost always wrong credentials, so it is logged as a warning, at most one
every five minutes per node, while a node that is down is only logged at
debug.

| Exception | Meaning | Usual response |
| --- | --- | --- |
| `ElasticsearchUnavailable` | Disabled, or the node could not be reached. | Fall back to a slower search. |
| `ElasticsearchRequestError` | The node answered and refused the request (malformed query, rejected mapping, missing permission or credentials). | Let it surface. Retrying will not help. |

```python
from splent_framework.services.service_locator import service_proxy
from splent_io.splent_feature_elasticsearch.services import ElasticsearchUnavailable

es = service_proxy("ElasticsearchService")

try:
    results = es.search("posts", {"query": {"match": {"title": query}}})
except ElasticsearchUnavailable:
    results = fallback_sql_search(query)
```

## Bulk indexing

A full reindex is thousands of documents, and one HTTP round trip each turns a
maintenance command into something nobody runs. `bulk_index` consumes any
iterable lazily, sends batches through the `_bulk` API and reports what
happened, so a single bad document does not cost you the other 9999.

```python
report = es.bulk_index(
    "pages",
    ({"id": page.id, "body": {"title": page.title, "body": page.text}} for page in pages),
)
es.refresh_index("pages")
# {"indexed": 9999, "failed": 1, "errors": [{"id": 42, "error": "..."}]}
```

Only a failure of the request itself raises. Per-document failures come back in
the report, with the error list capped so a broken mapping cannot flood a log.

## Deleting by query

`delete_by_query(index, query)` deletes every document the query clause
matches and returns how many went:

```python
es.delete_by_query("pages", {"term": {"dataset_id": 7}})   # 12
```

`query` is the clause itself, not a whole search body. A document changed
while the deletion runs is skipped rather than failing the rest, the deletion
is visible to the next search unless `refresh=False`, and a missing index
deletes nothing without raising.

## Configuration

| Env var | Default | Purpose |
| --- | --- | --- |
| `ELASTICSEARCH_URL` | `http://localhost:9200` | Node URL. The Docker `.env` sets `http://splent_feature_elasticsearch:9200` for container-to-container access. |
| `ELASTICSEARCH_ENABLED` | `true` | Feature kill switch. When false all operations degrade immediately. |
| `ELASTICSEARCH_INDEX_PREFIX` | the product's name | Namespace for this product's indices. Declared as `__PRODUCT__`, resolved to the product name; with neither a prefix nor `SPLENT_APP` the service refuses to name an index. |
| `ELASTICSEARCH_USERNAME` | `elastic` | User the app signs in as when a password is set. |
| `ELASTICSEARCH_PASSWORD` | empty | Password of that user, and of the production node's `elastic` user. Empty in development; `<SET>` in `docker/.env.prod.example`. |
| `ELASTICSEARCH_API_KEY` | empty | API key for a cluster this product does not run. Wins over the user and the password. |
| `ELASTICSEARCH_HOST_PORT` | `9201` | Host port published in development only. |
| `ELASTICSEARCH_JAVA_OPTS` | `-Xms512m -Xmx512m` | Heap for the production node (`docker/.env.prod.example`). |

`ELASTICSEARCH_ENABLED`, `ELASTICSEARCH_INDEX_PREFIX` and
`ELASTICSEARCH_USERNAME` are declared in `[tool.splent.config]`. The URL and
the host port, which name the feature's container, live in
`docker/.env.example`; production reads `docker/.env.prod.example` instead,
which asks for the password and carries the heap.

## Upgrading from 0.1

- A production deployment needs `ELASTICSEARCH_PASSWORD`. `product:deploy`
  asks for it and offers a random one; on a host without the CLI, set it in
  `deploy/.env` before `docker compose up`. An existing data volume keeps its
  indices: the node takes the password on the next start.
- `health()` reports `auth`, `status` and `error` besides what it did.
- `available()` calls the client's `info()` instead of `ping()`. A test that
  mocks the client and sets `ping()` has to set `info()` now.
- `_client()` and `_translated_errors()` stay as they were, but they were
  never public: `ensure_index(settings=...)`, `delete_by_query` and the
  aliases cover what a consumer reached them for.

## Tests

```bash
splent feature:test splent_feature_elasticsearch
```

Unit tests mock the client boundary, so no network access is required.
