Metadata-Version: 2.4
Name: genorbex-dynamic-search
Version: 0.1.0
Summary: Tenant-isolated dynamic search service for Genorbex applications
Author: Genorbex
License: MIT
Keywords: genorbex,multi-tenant,opensearch,postgresql,search
Requires-Python: >=3.12
Requires-Dist: asyncpg>=0.29
Requires-Dist: fastapi>=0.110
Requires-Dist: pydantic-settings>=2.2
Requires-Dist: pydantic>=2.7
Requires-Dist: sqlalchemy[asyncio]>=2.0
Provides-Extra: dev
Requires-Dist: alembic>=1.13; extra == 'dev'
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: opensearch-py[async]>=2.5; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: migrations
Requires-Dist: alembic>=1.13; extra == 'migrations'
Provides-Extra: opensearch
Requires-Dist: opensearch-py[async]>=2.5; extra == 'opensearch'
Provides-Extra: test
Requires-Dist: aiosqlite>=0.20; extra == 'test'
Requires-Dist: httpx>=0.27; extra == 'test'
Requires-Dist: pytest-asyncio>=0.23; extra == 'test'
Requires-Dist: pytest>=8; extra == 'test'
Description-Content-Type: text/markdown

# genorbex-dynamic-search

An async, multi-tenant search service package for Genorbex resources. PostgreSQL full-text
search is the default provider; OpenSearch is an optional adapter with the same response model.
The package is intentionally an installable service library, not a new web deployment or
database infrastructure definition.

## Install and configure

Python 3.12 or newer is required.

```sh
python -m pip install -e '.[migrations,test]'
```

PostgreSQL is the default. Use the application database or a dedicated PostgreSQL database and
run the package migration before mounting the API:

```env
SEARCH_PROVIDER=postgres
SEARCH_DATABASE_URL=postgresql://user:password@host:5432/genorbex
SEARCH_ENVIRONMENT=dev
SEARCH_QUERY_RATE_LIMIT=60
SEARCH_QUERY_RATE_WINDOW_SECONDS=60
```

```sh
alembic -c packages/genorbex-dynamic-search/alembic.ini upgrade head
```

The migration is package-owned Alembic SQL. This repository's main web application currently
uses Prisma migrations and has no Python FastAPI host. Keep migration ownership explicit: either
run this Alembic migration as part of the Python-service release or port the SQL into the host's
Prisma migration workflow before sharing deployment ownership of the table.

## Use the service

Application code depends on `DynamicSearchService`, not provider classes:

```python
from datetime import datetime, timezone

from genorbex_dynamic_search import ResourceType, SearchDocument, SearchRequest
from genorbex_dynamic_search.config import SearchSettings
from genorbex_dynamic_search.service import create_search_service

service = create_search_service(SearchSettings())
await service.initialize()
await service.index(SearchDocument(
    id="workflow:flow-123",
    tenant_id="org-123",  # trusted organization ID; use user ID for personal tenants
    resource_type=ResourceType.WORKFLOW,
    title="Customer onboarding",
    content="Verify the account and provision the customer workspace.",
    source_id="flow-123",
    created_at=datetime.now(timezone.utc),
    updated_at=datetime.now(timezone.utc),
))
response = await service.search(SearchRequest(tenant_id="org-123", query="onboarding"))
await service.aclose()
```

`SearchRequest` supports resource type, project, tag, creator, date, sort, cursor, and page-size
filters. Results include ranked hits, PostgreSQL/OpenSearch highlights, type facets, total count,
latency, and an opaque `next_cursor`. Cursors are tied to their sort order; pass the same sort
on the next request. Treat highlight strings as untrusted display text and escape them in HTML.

## FastAPI integration and authentication

```python
from fastapi import FastAPI
from genorbex_dynamic_search.api.router import create_search_router
from genorbex_dynamic_search.service import create_search_service

app = FastAPI()
search_service = create_search_service()
app.include_router(create_search_router(search_service, identity_dependency=genorbex_identity))
```

`genorbex_identity` must adapt the authenticated principal established by the host's middleware
to `SearchIdentity(user_id=..., tenant_id=..., role=..., permissions=...)`. The default adapter
reads `request.state.user` or `request.state.current_user`; it fails closed if no trusted identity
is present and uses `organizationId` as the tenant, falling back to the authenticated user ID for
personal workspaces. The package never reads a tenant from query parameters or request headers.
Manual index/delete endpoints require an admin/owner role or an explicit search-admin permission.
The GET endpoint applies a configurable per-tenant rate limit and max query length. Its built-in
limiter is process-local; inject a shared limiter implementation for multi-worker deployments.

The current Genorbex web application is Next.js/Prisma, not FastAPI, so its authentication
middleware cannot be imported directly into a Python process. Mount the router only in a host
that supplies that trusted identity adapter.

## Resource indexing and queue lifecycle

Indexers are provided for workflows, executions, agents, generated apps, connectors, documents,
knowledge chunks, and audit events. Call `enqueue_upsert` after create/update and `enqueue_delete`
after delete, passing the host's durable `SearchJobQueue` adapter. That adapter's `enqueue` method
must persist the serialized job before returning. A worker deserializes `SearchIndexJob` and calls
`process_search_job(service, job)`; failures use bounded exponential retries and are logged with
structured context. Upserts and deletes are idempotent.

The repository has a Prisma-backed message queue in the TypeScript application, but no Python
consumer/adapter contract exists yet. The package therefore defines the queue protocol and worker
handler without creating another queue, Redis service, or cloud resource. Connect the protocol to
the existing queue consumer at the application boundary.

Indexers deliberately avoid workflow definitions, prompts, execution input/output blobs, connector
settings, and credential fields. SearchDocument additionally redacts common credential assignments
and JWT-shaped strings, and metadata is filtered through an explicit allowlist. Review extracted
document text before indexing if it may contain regulated or confidential data.

## Backfill

The CLI requires exactly one scope. `--all-tenants` is an explicit opt-in and never the default:

```sh
GENORBEX_SEARCH_BACKFILL_SOURCE=my_app.search_backfill:create_source \
  python -m genorbex_dynamic_search.jobs.backfill --tenant-id org-123

GENORBEX_SEARCH_BACKFILL_SOURCE=my_app.search_backfill:create_source \
  python -m genorbex_dynamic_search.jobs.backfill --all-tenants
```

The source factory returns an adapter implementing `authorize_backfill(tenant_id, all_tenants)`,
`list_tenant_ids()`, and async `iter_documents(tenant_id)`. The authorization method must reject
operators who are not allowed to backfill the requested scope; it runs before any tenant records
are fetched. Every yielded document is checked against the tenant being backfilled. This adapter is
where host-specific Prisma/ORM reads and the package indexers belong.

## OpenSearch later

OpenSearch is optional and is never started or provisioned by this package:

```sh
python -m pip install -e '.[opensearch]'
```

```env
SEARCH_PROVIDER=opensearch
SEARCH_OPENSEARCH_URL=https://search.example.internal
SEARCH_OPENSEARCH_USERNAME=...
SEARCH_OPENSEARCH_PASSWORD=...
SEARCH_ENVIRONMENT=prod
# Defaults to genorbex-search-prod; override with SEARCH_OPENSEARCH_INDEX if needed.
```

The provider creates an index with text, keyword, timestamp, tenant, and metadata mappings when
`await service.initialize()` runs. Every query includes a mandatory tenant filter. No embedding
service or vector index is required; vector/semantic search can be added later without changing
the `SearchResponse` contract.

## Migrations, tests, and observability

```sh
alembic -c packages/genorbex-dynamic-search/alembic.ini upgrade head
python -m unittest discover -s packages/genorbex-dynamic-search/tests
```

Set `SEARCH_TEST_DATABASE_URL` to a disposable PostgreSQL database with the migration applied to
enable integration tests. Unit tests mock OpenSearch. The service emits structured query latency,
result-count, index failure, retry, and provider-health logs. Optionally inject `SearchMetrics` to
export query latency/result counts, index failures, and provider health; inject `SearchJobMetrics`
into the worker to export job retries and failures.
