Metadata-Version: 2.4
Name: vllm-hust-operator-optimizations
Version: 0.1.1
Summary: Extensible operator optimization bundle for vLLM-HUST
License-Expression: Apache-2.0
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Operating System :: POSIX :: Linux
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# vLLM-HUST Operator Optimizations

This package is an updateable vLLM-HUST Extension Bundle for independently
gated operator optimizations. Version `0.1.1` contains one optimization:

- `persistent-matmul-empty`: replaces the redundant `torch.zeros` allocation
  in Ascend batch-invariant Triton `linear_persistent` with `torch.empty`.

The optimization is limited to the batch-invariant Triton path. It does not
change ordinary ACLNN MatMul or other linear implementations. The persistent
kernel writes every valid output element, including tail tiles. Existing NPU
evidence covers FP16/BF16/FP32, divisible and non-divisible M/N/K shapes, and
matched operator/end-to-end measurements. Unknown source structures are
rejected rather than patched.

## Architecture

The distribution exposes two entry points:

- `vllm_hust.extension_bundles` lets `vllm-hust-ext` discover, check and enable
  the `0.2-experimental` bundle.
- `vllm.general_plugins` installs a lazy import hook in each vLLM process. The
  hook changes only enabled target modules and has no device or network side
  effects at import time.

Optimizations are selected with `VLLM_HUST_OPERATOR_OPTIMIZATIONS`. The bundle
manager sets it to `persistent-matmul-empty` when this extension is enabled.

## Install and use

The supported release line is intentionally narrow: Ascend 910B2, CANN
9.1.0-beta.3, ATB 9.1.T6.B010, Python 3.11, Torch 2.10.0,
torch_npu 2.10.0.post4 and Triton-Ascend 3.2.2. Exact source commits are in
`compatibility/verified-line.json` inside the package. Other combinations are
unsupported until separately validated.

```bash
export VLLM_SRC=/path/to/vllm
export VLLM_ASCEND_SRC=/path/to/vllm-ascend-hust
pip install vllm-hust-operator-optimizations==0.1.1
vllm-hust-operator-check \
  --vllm-src "$VLLM_SRC" \
  --vllm-ascend-src "$VLLM_ASCEND_SRC"
vllm-hust-ext extension check org.vllm-hust.operator-optimizations
vllm-hust-ext extension enable org.vllm-hust.operator-optimizations
VLLM_BATCH_INVARIANT=1 vllm-hust-ext run -- vllm serve /path/to/model \
  --block-size 128
```

`TRITON_ASCEND_SRC` may additionally point to the locked Triton checkout. If
it is absent, the checker verifies the installed `triton-ascend==3.2.2`
wheel; this is the mode used by the validated server environment.

Repository maintainers can build with `uv build --no-sources --out-dir dist`
and use `scripts/install_verified_line.sh` to gate and install a local wheel.

If `VLLM_PLUGINS` is manually set, it must include both the Ascend platform
plugin name and `operator_optimizations`; otherwise vLLM intentionally filters
out this runtime entry point.

## Add another optimization

1. Add an `Optimization` implementation under `optimizations/` with a stable
   ID, target module and fail-closed `apply()` method.
2. Register it in `registry.register_builtins()`.
3. Add its ID to the manifest environment value if it should be enabled by
   this bundle version, or publish a separately selectable bundle profile.
4. Add unit correctness tests and NPU shape/tail, operator-share and matched
   end-to-end evidence.
5. Increment `_version.py` and the manifest `extension_version` together.

The framework deliberately keeps every optimization independent so a future
release can add, disable or reject one optimization without changing the
others.

## Validation

```bash
uv run --with pytest pytest
uv run --with ruff ruff check src tests
uv build --no-sources --out-dir dist
```

For `0.1.1`, reuse the existing Qwen2.5-7B Graph correctness and performance
evidence for the exact locked line. Re-run NPU validation only when that line
or the optimization changes. Release checks remain `extension list`, `inspect`,
`check`, and `run --dry-run`; the completed real service run is not repeated
merely for packaging changes.
