Metadata-Version: 2.5
Name: autoinference-utils
Version: 0.2.19
Summary: Shared endpoint abstractions for autoinference deployments
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# autoinference-utils

Shared endpoint abstractions (`SGLangEndpoint`, `VLLMEndpoint`) for autoinference deployments.

## Publishing

Bump `version` in `pyproject.toml` and merge to `main` — CI publishes automatically via trusted publishing.

To publish manually: `cd autoinference_utils && uv build && uv publish`

## SGLang defaults

`SGLangEndpoint` applies defaults that any deployment can override:

- Server args in `DEFAULT_OPERATIONAL_ARGS` (metrics, cache report,
  `--dist-timeout 3600`). Any `extra_server_args` entry for the same flag
  replaces the default.
- `--tokenizer-backend fastokens` when the image has `fastokens` installed. Pass
  `--tokenizer-backend huggingface` to opt out. Under fastokens the server also
  gets `SGLANG_PARALLEL_PROMPT_ENCODE=0`.
- Environment in `DEFAULT_ENV` (keep-alives, the metrics device timer). The
  image env and `extra_env` override it.

## SGLang lifecycle

- `start()` launches SGLang in its own session, so Modal's SIGTERM+SIGINT to the
  container's process group does not reach it. `stop()` sends SIGTERM, which
  SGLang drains on, waits up to `drain_timeout` (default 150 s, keep it below the
  app's `exit_grace_period`), then kills the server's whole process group.
- In a Modal container, `start()` then watches the server: if it exits or
  `/health` keeps failing for 60 s, it ends the container so Modal replaces it.
  Watching stops once `stop()` begins, so stop the server with `stop()` rather
  than signalling its process directly. Pass `watch=False` to opt out.
