Metadata-Version: 2.4
Name: vortexrt
Version: 0.3.1
Summary: Unified AI model inference — auto-detect, auto-route, auto-serve with multi-engine priority routing.
Author: VortexRT Contributors
License: MIT
Project-URL: Homepage, https://github.com/vortexrt/vortexrt
Project-URL: Repository, https://github.com/vortexrt/vortexrt
Project-URL: Issues, https://github.com/vortexrt/vortexrt/issues
Project-URL: Documentation, https://github.com/vortexrt/vortexrt#readme
Keywords: ai,inference,transformers,diffusers,onnx,vllm,serving,llm,tts,gguf,flux
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: huggingface-hub>=0.26.0
Requires-Dist: fastapi>=0.104.0
Requires-Dist: uvicorn[standard]>=0.24.0
Requires-Dist: psutil>=5.9.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: torch>=2.1.0
Requires-Dist: transformers>=4.46.0
Requires-Dist: diffusers>=0.31.0
Requires-Dist: accelerate>=0.30.0
Requires-Dist: onnxruntime>=1.18.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: safetensors>=0.4.0
Requires-Dist: Pillow>=10.0
Requires-Dist: scipy>=1.10.0
Requires-Dist: sentencepiece>=0.2.0
Requires-Dist: tokenizers>=0.19.0
Provides-Extra: vllm
Requires-Dist: vllm>=0.7.0; extra == "vllm"
Provides-Extra: vllm-omni
Requires-Dist: vllm-omni>=0.1.0; extra == "vllm-omni"
Provides-Extra: gguf
Requires-Dist: llama-cpp-python>=0.3.0; extra == "gguf"
Provides-Extra: diffusion-gguf
Requires-Dist: stable-diffusion-cpp-python>=0.4.0; extra == "diffusion-gguf"
Provides-Extra: gpu
Requires-Dist: pynvml>=11.5.0; extra == "gpu"
Provides-Extra: all
Requires-Dist: vortexrt[diffusion-gguf,gguf,gpu,vllm,vllm-omni]; extra == "all"
Requires-Dist: bitsandbytes>=0.41.0; extra == "all"
Requires-Dist: flash-attn>=2.5.0; extra == "all"
Requires-Dist: soundfile>=0.12.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: httpx>=0.25.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Dynamic: license-file

<div align="center">

# VortexRT

### Unified AI Model Inference — One Library, Every Model

[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
[![License: MIT](https://img.shields.io/badge/License-MIT-green)](LICENSE)

</div>

---

VortexRT is a Python library for AI model inference. Give it a model
name, it detects the task, picks the best engine for your hardware,
installs missing dependencies, and runs inference.  

```python
from vortexrt import VortexRT
vrt = VortexRT()
llm = vrt.load("Qwen/Qwen2.5-1.5B-Instruct")
result = vrt.predict(llm.id, {"prompt": "Explain quantum computing"})
```

---

## Engines

VortexRT is built on 7 open-source inference engines. Each engine is
independently licensed and can be used in any other project.

| Engine | License | Specialisation | % of Tasks |
|---|---|---|---|
| [Transformers](https://github.com/huggingface/transformers) | Apache 2.0 | Text, classification, QA, NER, embeddings, TTS, ASR | 45% |
| [Diffusers](https://github.com/huggingface/diffusers) | Apache 2.0 | Text-to-image, image-to-image | 10% |
| [ONNX Runtime](https://github.com/microsoft/onnxruntime) | MIT | Custom ONNX, audio, TTS, ASR, DirectML | 15% |
| [vLLM](https://github.com/vllm-project/vllm) | Apache 2.0 | High-throughput LLM (PagedAttention) | 10% |
| [vLLM-Omni](https://github.com/vllm-project/vllm-omni) | Apache 2.0 | Accelerated image/video (DiT, Cache-DiT) | 10% |
| [llama.cpp](https://github.com/ggerganov/llama.cpp) | MIT | GGUF LLMs on CPU | 5% |
| [Stable Diffusion.cpp](https://github.com/leejet/stable-diffusion.cpp) | MIT | GGUF diffusion on CPU/GPU | 5% |

All engines are open source and permissible for commercial use.
Our library is MIT licensed — compatible with all of them.

### Hardware Support

| | NVIDIA CUDA | AMD ROCm | Intel XPU/Arc | CPU Only | Apple MPS |
|---|---|---|---|---|---|
| **Transformers** | ✓ | ✓ | ✓ | ✓ | ✓ |
| **Diffusers** | ✓ | — | — | ✓ | ✓ |
| **ONNX Runtime** | ✓ CUDA EP | ✓ ROCm EP | ✓ OpenVINO EP | ✓ | ✓ CoreML |
| **vLLM** | ✓ | ✓ | ✓ | ✓ (slow) | — |
| **vLLM-Omni** | ✓ | ✓ | ✓ | — | — |
| **llama.cpp** | ✓ Vulkan | ✓ Vulkan | ✓ Vulkan | ✓ | ✓ Metal |
| **diffusion-gguf** | ✓ Vulkan | ✓ Vulkan | ✓ Vulkan | ✓ | — |

**Yes, it works on AMD, Intel, and CPU.** The library automatically
detects your hardware and picks the best available engine. If vLLM
isn't available (no NVIDIA GPU), it falls back to Transformers for
text and Diffusers for images.

### Video Generation

Video generation is registered as a task (`text-to-video`) but
requires a video-capable model and engine. vLLM-Omni supports DiT-based
video models (Sora-style, CogVideo, etc.) when installed with a
compatible GPU.

---

## Installation

```bash
pip install vortexrt
```

Installs the core engines (Transformers, Diffusers, ONNX Runtime) and
all dependencies automatically. For niche engines:

```bash
pip install "vortexrt[all]"
```

---

## Quick Start

### Text Generation

```python
from vortexrt import VortexRT

vrt = VortexRT()
llm = vrt.load("Qwen/Qwen2.5-1.5B-Instruct")

result = vrt.predict(llm.id, {"prompt": "Explain AI in one sentence."})
print(result.data["generated_text"])
# → "AI is the simulation of human intelligence by machines..."

print(f"Engine: {result.engine}  Time: {result.timing.total_ms:.0f}ms")
print(f"Tokens: {result.usage}")
```

### Chat (Multi-turn)

```python
chat = vrt.chat(llm.id, system="You are a Python expert. Keep replies short.")

reply = chat.send("How do I sort a dict by value?")
print(reply.data["generated_text"])

# Stream token by token
for token in chat.stream("Show me a one-line example"):
    print(token, end="", flush=True)

chat.reset()  # clear history
```

### Image Generation

```python
img = vrt.load("runwayml/stable-diffusion-v1-5")

result = vrt.predict(img.id, {
    "prompt": "A mountain lake at sunrise",
    "num_inference_steps": 20,
    "save_to": True,     # auto-saves to Desktop
    "show_progress": True,  # live progress bar
})
print(result.data["saved_path"])
```

### Multiple Models on One Port

```python
vrt.load("Qwen/Qwen2.5-1.5B-Instruct")
vrt.load("runwayml/stable-diffusion-v1-5")
vrt.serve()  # → http://localhost:8000
```

### Engine Control

```python
# Let the system choose (recommended)
vrt.load("model")

# Force a specific engine
vrt.load("model", engine="transformers")

# With optimizations
vrt.load("model", quantize="int8", torch_dtype="bfloat16")

# See what engine will be chosen
vrt.routing_table()
```

### Authentication

Gated models (Llama, FLUX, Gemma) need a HuggingFace token:

```python
from vortexrt import set_token
set_token("hf_...")
```

---

## REST API

```python
vrt.serve()  # http://localhost:8000
```

| Method | Endpoint | Description |
|---|---|---|
| `GET` | `/v1/health` | Health + hardware + available engines |
| `GET` | `/v1/engines` | Engine routing table |
| `GET` | `/v1/models` | All loaded models |
| `POST` | `/v1/models/load` | Load a model |
| `POST` | `/v1/models/{id}/predict` | Inference (returns timing + usage) |
| `POST` | `/v1/models/{id}/stream` | SSE token stream |
| `POST` | `/v1/models/{id}/chat` | Multi-turn chat |
| `DELETE` | `/v1/models/{id}` | Unload |

---

## License

MIT — see [LICENSE](LICENSE).

Built on Apache 2.0 and MIT licensed engines.
All third-party engines remain under their original licenses.
