Metadata-Version: 2.4
Name: vllm-hust-opset
Version: 0.3.0
Summary: OPset: updateable Ascend operator optimizations and Graph fusions for vLLM-HUST
License-Expression: Apache-2.0
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Operating System :: POSIX :: Linux
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# OPset for vLLM-HUST

OPset is an updateable vLLM-HUST Extension Bundle for independently gated
Ascend operator optimizations and Graph fusions. Version `0.3.0` contains:

- `persistent-matmul-empty`: replaces the redundant `torch.zeros` allocation
  in Ascend batch-invariant Triton `linear_persistent` with `torch.empty`.
- `linear-swiglu-graph`: fuses adjacent unquantized gate/up GEMM and SwiGLU in
  the Graph path. The validated decode/small-batch range (`M <= 128`) uses the
  fused Triton kernel; larger dynamic prefill shapes fail closed to the native
  GEMM plus `npu_swiglu` path.

Both optimizations are limited to the batch-invariant Graph/Triton path and do
not change ordinary ACLNN MatMul or unrelated linear implementations. Existing
NPU evidence covers supported dtypes, divisible and non-divisible shapes, and
matched operator/end-to-end measurements. Unknown source structures are
rejected rather than patched.

## Architecture

The distribution exposes two entry points:

- `vllm_hust.extension_bundles` lets `vllm-hust-ext` discover, check and enable
  the `0.2-experimental` bundle.
- `vllm.general_plugins` installs a lazy import hook in each vLLM process. The
  hook changes only enabled target modules and has no device or network side
  effects at import time.

Optimizations are selected with `VLLM_HUST_OPERATOR_OPTIMIZATIONS`. The bundle
manager enables both shipped optimizations; either ID can also be selected
independently.

## Install and use

This distribution succeeds `vllm-hust-operator-optimizations`. Existing
Python imports, extension IDs, environment variables, and the
`vllm-hust-operator-check` command remain available. Upgrade an existing
installation with:

```bash
pip uninstall -y vllm-hust-operator-optimizations
pip install vllm-hust-opset==0.3.0
```

The package records the fixed experiment line in
`compatibility/verified-line.json`. This keeps current results reproducible
without repeatedly rebuilding the environment; the line will be refreshed and
revalidated periodically as the underlying stack advances.

```bash
export VLLM_SRC=/path/to/vllm
export VLLM_ASCEND_SRC=/path/to/vllm-ascend-hust
pip install vllm-hust-opset==0.3.0
vllm-hust-opset \
  --vllm-src "$VLLM_SRC" \
  --vllm-ascend-src "$VLLM_ASCEND_SRC"
vllm-hust-ext extension check org.vllm-hust.operator-optimizations
vllm-hust-ext extension enable org.vllm-hust.operator-optimizations
VLLM_BATCH_INVARIANT=1 vllm-hust-ext run -- vllm serve /path/to/model \
  --block-size 128
```

`TRITON_ASCEND_SRC` may additionally point to the locked Triton checkout. If
it is absent, the checker verifies the installed `triton-ascend==3.2.2`
wheel; this is the mode used by the validated server environment.

Repository maintainers can build with `uv build --no-sources --out-dir dist`
and use `scripts/install_verified_line.sh` to gate and install a local wheel.

If `VLLM_PLUGINS` is manually set, it must include both the Ascend platform
plugin name and `operator_optimizations`; otherwise vLLM intentionally filters
out this runtime entry point.

## Add another optimization

1. Add an `Optimization` implementation under `optimizations/` with a stable
   ID, target module and fail-closed `apply()` method.
2. Register it in `registry.register_builtins()`.
3. Add its ID to the manifest environment value if it should be enabled by
   this bundle version, or publish a separately selectable bundle profile.
4. Add unit correctness tests and NPU shape/tail, operator-share and matched
   end-to-end evidence.
5. Increment `_version.py` and the manifest `extension_version` together.

The framework deliberately keeps every optimization independent so a future
release can add, disable or reject one optimization without changing the
others.

## Validation

```bash
uv run --with pytest pytest
uv run --with ruff ruff check src tests
uv build --no-sources --out-dir dist
```

For `0.3.0`, the linear-SwiGLU Graph path was validated with Qwen2.5-7B,
ShareGPT, three matched baseline/candidate rounds and operator correctness
coverage. Steady-state TPOT decreased by 3.82% and output-token throughput
increased by 5.24%. Release checks remain `extension list`, `inspect`, `check`,
and `run --dry-run`; full NPU validation is refreshed periodically or when the
optimization changes.
