Metadata-Version: 2.4
Name: distributed-llm
Version: 0.6.2.1
Summary: Distributed LLM inference — pool GPUs across multiple devices to run models no single machine can handle
Author: Distributed LLM Contributors
License: Apache-2.0
Project-URL: Homepage, https://github.com/distributed-llm/distributed-llm
Project-URL: Documentation, https://github.com/distributed-llm/distributed-llm#readme
Project-URL: Repository, https://github.com/distributed-llm/distributed-llm
Project-URL: Issues, https://github.com/distributed-llm/distributed-llm/issues
Keywords: llm,inference,distributed,pipeline-parallelism,gpu-sharing,peer-to-peer,open-source
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Distributed Computing
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fastapi>=0.109.0
Requires-Dist: python-multipart>=0.0.6
Requires-Dist: uvicorn>=0.27.0
Requires-Dist: pydantic>=2.5.0
Requires-Dist: pydantic-settings>=2.0.0
Requires-Dist: httpx>=0.27
Requires-Dist: pyyaml>=6.0
Requires-Dist: loguru>=0.7.0
Requires-Dist: pyjwt>=2.0.0
Requires-Dist: numpy>=1.26.0
Requires-Dist: psutil>=5.9.0
Requires-Dist: grpcio>=1.60.0
Requires-Dist: protobuf>=4.25.0
Requires-Dist: setuptools>=84.1.0
Provides-Extra: observability
Requires-Dist: prometheus-client>=0.20; extra == "observability"
Requires-Dist: opentelemetry-api>=1.21.0; extra == "observability"
Requires-Dist: opentelemetry-sdk>=1.21.0; extra == "observability"
Requires-Dist: opentelemetry-exporter-prometheus>=0.42b0; extra == "observability"
Provides-Extra: self-hosted
Requires-Dist: torch>=2.5.0; extra == "self-hosted"
Requires-Dist: transformers>=4.48.0; extra == "self-hosted"
Requires-Dist: accelerate>=1.0.0; extra == "self-hosted"
Requires-Dist: safetensors>=0.4.0; extra == "self-hosted"
Requires-Dist: huggingface-hub>=0.20.0; extra == "self-hosted"
Requires-Dist: sentencepiece>=0.1.99; extra == "self-hosted"
Requires-Dist: nvidia-ml-py>=12.0; extra == "self-hosted"
Provides-Extra: ray
Requires-Dist: ray[default]>=2.30.0; extra == "ray"
Provides-Extra: backends
Requires-Dist: vllm<0.9.0,>=0.7.0; extra == "backends"
Requires-Dist: sglang<0.5.0,>=0.4.0; extra == "backends"
Requires-Dist: llama-cpp-python>=0.3.0; extra == "backends"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
Requires-Dist: hypothesis>=6.90; extra == "dev"
Requires-Dist: pytest-benchmark>=4.0; extra == "dev"
Requires-Dist: black==26.1.0; extra == "dev"
Requires-Dist: ruff==0.16.7; extra == "dev"
Requires-Dist: mypy>=1.8; extra == "dev"
Requires-Dist: opentelemetry-test-utils>=0.42b0; extra == "dev"
Requires-Dist: bandit[toml]>=1.7; extra == "dev"
Requires-Dist: detect-secrets>=1.5; extra == "dev"
Requires-Dist: pip-audit>=2.7; extra == "dev"
Provides-Extra: sdk
Requires-Dist: httpx>=0.27.0; extra == "sdk"
Provides-Extra: cli
Requires-Dist: typer[all]>=0.9.0; extra == "cli"
Requires-Dist: rich>=13.7.0; extra == "cli"
Requires-Dist: prompt_toolkit>=3.0.0; extra == "cli"
Provides-Extra: ui
Requires-Dist: jinja2>=3.1.0; extra == "ui"
Requires-Dist: python-multipart>=0.0.6; extra == "ui"
Requires-Dist: qrcode[pil]>=7.4; extra == "ui"
Provides-Extra: tokenize
Requires-Dist: tiktoken>=0.7.0; extra == "tokenize"
Provides-Extra: examples
Requires-Dist: langchain-openai>=0.1.0; extra == "examples"
Requires-Dist: langchain-core>=0.1.0; extra == "examples"
Requires-Dist: llama-index-llms-openai>=0.1.0; extra == "examples"
Requires-Dist: llama-index-core>=0.10.0; extra == "examples"
Requires-Dist: crewai>=0.30.0; extra == "examples"
Requires-Dist: haystack-ai>=2.0.0; extra == "examples"
Requires-Dist: agency-swarm>=0.2.0; extra == "examples"
Requires-Dist: openai>=1.0.0; extra == "examples"
Provides-Extra: security
Requires-Dist: cryptography>=41.0; extra == "security"
Requires-Dist: pynacl>=1.5.0; extra == "security"
Provides-Extra: moderation
Requires-Dist: transformers>=4.36.0; extra == "moderation"
Requires-Dist: onnxruntime>=1.16.0; extra == "moderation"
Provides-Extra: mlflow
Requires-Dist: mlflow>=2.0.0; extra == "mlflow"
Provides-Extra: monitoring
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc>=1.21.0; extra == "monitoring"
Requires-Dist: opentelemetry-proto>=1.21.0; extra == "monitoring"
Requires-Dist: nvidia-ml-py>=12.0; extra == "monitoring"
Requires-Dist: psutil>=5.9.0; extra == "monitoring"
Provides-Extra: model-extras
Requires-Dist: bitsandbytes>=0.41.0; extra == "model-extras"
Requires-Dist: peft>=0.14.0; extra == "model-extras"
Provides-Extra: wan
Requires-Dist: aioquic>=1.0.0; extra == "wan"
Provides-Extra: wandb
Requires-Dist: wandb>=0.18; extra == "wandb"
Requires-Dist: nvidia-ml-py>=12.0; extra == "wandb"
Provides-Extra: spark
Requires-Dist: pyspark>=3.4.0; extra == "spark"
Provides-Extra: spark-mlflow
Requires-Dist: distributed-llm[mlflow,spark]; extra == "spark-mlflow"
Provides-Extra: openai-agents
Requires-Dist: openai-agents>=0.1.0; extra == "openai-agents"
Requires-Dist: openai>=1.0.0; extra == "openai-agents"
Provides-Extra: agno
Requires-Dist: agno>=1.0.0; extra == "agno"
Provides-Extra: autogpt
Requires-Dist: autogpt>=0.4.0; extra == "autogpt"
Provides-Extra: vector-db-pinecone
Requires-Dist: pinecone>=3.0.0; extra == "vector-db-pinecone"
Provides-Extra: vector-db-qdrant
Requires-Dist: qdrant-client>=1.9.0; extra == "vector-db-qdrant"
Provides-Extra: vector-db-weaviate
Requires-Dist: weaviate-client>=4.0.0; extra == "vector-db-weaviate"
Provides-Extra: vector-db-milvus
Requires-Dist: pymilvus>=2.3.0; extra == "vector-db-milvus"
Provides-Extra: vector-db-all
Requires-Dist: distributed-llm[vector-db-milvus,vector-db-pinecone,vector-db-qdrant,vector-db-weaviate]; extra == "vector-db-all"
Provides-Extra: spot
Requires-Dist: httpx>=0.27; extra == "spot"
Provides-Extra: all
Requires-Dist: distributed-llm[agno,autogpt,backends,cli,examples,mlflow,observability,openai-agents,ray,sdk,security,self-hosted,spark,spot,ui,wan]; extra == "all"
Requires-Dist: distributed-llm[model-extras]; extra == "all"
Provides-Extra: all-dev
Requires-Dist: distributed-llm[all,dev]; extra == "all-dev"
Requires-Dist: locust>=2.20; extra == "all-dev"
Requires-Dist: safety>=3.0; extra == "all-dev"
Provides-Extra: testing
Requires-Dist: pytest>=7.0; extra == "testing"
Requires-Dist: pytest-asyncio>=0.23; extra == "testing"
Requires-Dist: pytest-cov>=4.0; extra == "testing"
Requires-Dist: pytest-timeout>=2.2; extra == "testing"
Requires-Dist: pytest-mock>=3.12; extra == "testing"
Requires-Dist: hypothesis>=6.90; extra == "testing"
Requires-Dist: pytest-benchmark>=4.0; extra == "testing"
Requires-Dist: httpx>=0.27; extra == "testing"
Requires-Dist: torch>=2.5.0; extra == "testing"
Requires-Dist: transformers>=4.48.0; extra == "testing"
Requires-Dist: accelerate>=1.0.0; extra == "testing"
Requires-Dist: safetensors>=0.4.0; extra == "testing"
Requires-Dist: huggingface-hub>=0.20.0; extra == "testing"
Requires-Dist: sentencepiece>=0.1.99; extra == "testing"
Requires-Dist: nvidia-ml-py>=12.0; extra == "testing"
Requires-Dist: cryptography>=41.0; extra == "testing"
Requires-Dist: pynacl>=1.5.0; extra == "testing"
Requires-Dist: zstandard>=0.22; extra == "testing"
Requires-Dist: zeroconf>=0.131; extra == "testing"
Dynamic: license-file

# DistLLM — Distributed Inference Across All Your Devices

**Pool GPUs from every device you own to run models no single machine can handle.**

[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)

You have a gaming PC with an RTX 4090. Your laptop has an RTX 4060. Your friend has a desktop with an RTX 3080. None of you can run Llama 3.1 70B alone. **Together, you can.**

DistLLM splits large language models across all your devices using pipeline parallelism. Each device runs a fraction of the model layers. Automatic discovery. Auto-partitioning. Works over LAN, WiFi, or internet.

```
Your Laptop (RTX 4060) ────┐
                            │
Your Gaming PC (RTX 4090) ──┼──► DistLLM Cluster ──► Run 70B models
                            │
Friend's PC (RTX 3080) ────┘
```

## Why DistLLM?

| Problem | Solution |
|---------|----------|
| One GPU can't run today's best models | Split across all your devices |
| Cloud inference costs thousands/month | Use the GPUs you already own |
| Data privacy concerns with cloud APIs | Your data stays on your devices |
| Slow single-device inference | Pipeline parallelism = faster generation |
| Setting up distributed systems is hard | One command to start, one to join |

## Quick Start

```bash
pip install distributed-llm[cli,self-hosted]

# On your main machine — start a cluster
distllm cluster start --model meta-llama/Llama-3.2-7B

# On every other machine — join the cluster
distllm cluster join
```

## How It Works

DistLLM uses **pipeline parallelism** — the model is split across devices by layers:

```
Device 1 (Laptop): Layers 0-5 ──→ Device 2 (Desktop): Layers 6-11 ──→ Device 3 (Friend's PC): Layers 12-17

Each device runs ~6 layers → fits in 6-8GB VRAM
Combined pool → runs models up to 70B parameters
```

Key capabilities:
- **Auto-discovery**: devices find each other on the same network automatically
- **Auto-partitioning**: automatically assigns layers based on each device's GPU
- **Node recovery**: if a device disconnects, remaining nodes take over
- **Straggler detection**: slow nodes are detected and worked around
- **WAN optimization**: token accumulation for low-latency cross-internet inference
- **Privacy-first**: keep sensitive layers on your own devices

## Installation

```bash
# Core package + CLI
pip install distributed-llm[cli]

# With PyTorch/transformers for self-hosted GPU inference (recommended)
pip install "distributed-llm[cli,self-hosted]"

# With vLLM backend (recommended for NVIDIA GPUs)
pip install "distributed-llm[self-hosted,backends]"

# Full install (all backends, integrations, SDKs)
pip install "distributed-llm[all]"

# Development
pip install -e ".[dev]"
```

## Key Features

- **Pipeline parallelism** — split any HuggingFace model across N devices
- **Auto-discovery** — mDNS/zeroconf device finding on LAN
- **12 backends** — vLLM, llama.cpp, TensorRT-LLM, ExLlamaV2, ONNX, PyTorch, Triton, NIM, MLX, WebGPU, Ollama, TGI
- **Auth plugin** — JWT authentication + RBAC role-based access control
- **Health watchdog** — continuous node health monitoring with auto circuit-breaking
- **Semantic caching** — deduplicate repeated prompts with embedding similarity
- **Token streaming** — `generate_stream()` for real-time token-by-token responses
- **Config validation** — cross-field validation catches invalid combinations at load time
- **Circuit breaker** — graduated backpressure for load shedding
- **Auto-partitioning** — hardware-aware DP solver for optimal layer assignment
- **Node recovery** — checkpoint-based recovery when nodes disconnect
- **Straggler detection** — statistical outlier detection for slow nodes
- **Dynamic rebalancing** — redistribute layers when nodes join/leave
- **P2P KV cache gossip** — CRDT-based cache sharing between nodes
- **Wide-area support** — token accumulation for internet-scale inference
- **Quantization** — 4-bit/8-bit to fit larger models on consumer GPUs
- **OpenAI-compatible API** — use any OpenAI client to send requests
- **Full observability** — Prometheus metrics, OTel tracing, structured logging
- **Interactive chat** — `distllm chat` for CLI-based interaction
- **Health endpoints** — `/healthz` (liveness) and `/readyz` (readiness) for Kubernetes probes

## CLI Commands

```bash
# Start the REST API server (local mode)
distllm system run --model meta-llama/Llama-3.2-1B --local --port 8000

# Start the API server on a cluster node
distllm-api --model meta-llama/Llama-3.2-1B --local

# Start a coordinator node with workers (modern CLI)
distllm cluster start --model meta-llama/Llama-3.2-7B

# Join a cluster as a worker node
distllm cluster join --coordinator 192.168.1.100

# System diagnostics (local environment check)
distllm cli-doctor

# Cluster diagnostics (check a running coordinator)
distllm system doctor

# Quantize / optimize model (adaptive precision optimizer)
distllm tune quantize --model meta-llama/Llama-3.2-7B

# Profile optimal batch size
distllm tune batch --model meta-llama/Llama-3.2-7B --gpus 1

# Size KV cache for expected concurrency
distllm tune cache --model meta-llama/Llama-3.2-7B

# Interactive chat
distllm chat
```

## Documentation

- [Architecture](docs/ARCHITECTURE.md) — Pipeline parallelism, node topology, KV cache management
- [Deployment](DEPLOYMENT.md) — Local, Docker, and multi-machine deployment
- [API Reference](docs/api.md) — OpenAI-compatible API docs

## Project Structure

```
src/distllm/
├── core/              # Coordinator, engine strategies, schedulers, HA
├── dist/              # Distributed inference engine
│   ├── pipeline/      # PipelineOrchestrator, transport, 1F1B scheduler
│   ├── partition/     # Hardware-aware auto-partitioner, quantization
│   ├── p2p/           # P2P gossip protocol, Kademlia DHT, discovery
│   ├── backends/      # Node-level backend adapters
│   ├── worker.py      # WorkerNode — per-device model subset (entry point)
│   ├── recovery.py    # Node failure recovery + checkpoint replay
│   ├── straggler.py   # Straggler detection
│   ├── rebalancer.py  # Dynamic pipeline rebalancing
│   └── wide_area.py   # WAN-optimized inference
├── api/               # OpenAI-compatible REST API (FastAPI)
├── backends/          # Inference backends: vLLM, llama.cpp, TensorRT, ONNX, MLX, Triton
├── models/            # Model loading, partitioning, hub
├── cli/               # Typer CLI (`distllm`, `distllm cluster`, `distllm tune`)
├── config/            # Pydantic settings, YAML/env loader
├── sdk/               # Python client library
├── observability/     # Prometheus metrics, OTel tracing, structured logging
└── plugins/           # Auth, caching, watchdog, and other built-in plugins
```

## Supported Model Architectures

GPT-2, GPT-Neo, Llama 2/3, Mistral, Mixtral, Qwen2.5, Phi, DeepSeek, StableLM, Pythia, Baichuan, ChatGLM, InternLM, and more via HuggingFace AutoModel.

## Roadmap

- **v0.4.x** (current): Pipeline parallelism across LAN devices, auto-partitioning via hardware-aware DP solver, health watchdog, semantic cache, RBAC auth, 12 backends, speculative decoding, mDNS auto-discovery
- **Planned**:
  - **GPU marketplace reputation system**
  - **Privacy-preserving split inference with differential privacy**
  - **Federated clusters, cross-internet NAT traversal, incentive systems** (2027)

> Full canonical project map: [`docs/_map/UNIFIED.md`](docs/_map/UNIFIED.md) — single entry point for architecture, modules, API, ops, security, and the hygiene log.

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md).

## License

Apache 2.0. See [LICENSE](LICENSE).
