Metadata-Version: 2.5
Name: maque
Version: 0.4.0
Summary: Python toolkit for ML, CV, NLP and multimodal AI development
Project-URL: homepage, https://github.com/beidongjiedeguang/maque
Project-URL: repository, https://github.com/beidongjiedeguang/maque
Project-URL: documentation, https://github.com/beidongjiedeguang/maque#readme
Project-URL: Issues, https://github.com/beidongjiedeguang/maque/issues
Project-URL: Source, https://github.com/beidongjiedeguang/maque
Author-email: kunyuan <beidongjiedeguang@gmail.com>
License-File: LICENSE
Keywords: Machine Learning,cli,cv,nlp
Classifier: Development Status :: 5 - Production/Stable
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Build Tools
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Requires-Dist: aiohttp
Requires-Dist: argcomplete
Requires-Dist: attrs>=22.2.0
Requires-Dist: chevron
Requires-Dist: colour
Requires-Dist: deprecated
Requires-Dist: diff-match-patch
Requires-Dist: fire
Requires-Dist: flexllm>=0.2.2
Requires-Dist: json5
Requires-Dist: loguru>=0.6.0
Requires-Dist: lxml
Requires-Dist: more-itertools
Requires-Dist: orjson
Requires-Dist: pillow
Requires-Dist: pretty-errors>=1.2.25
Requires-Dist: psutil
Requires-Dist: pyahocorasick
Requires-Dist: pyyaml
Requires-Dist: requests
Requires-Dist: rich
Requires-Dist: tabulate
Provides-Extra: cli
Requires-Dist: asciinema; extra == 'cli'
Requires-Dist: docker; extra == 'cli'
Requires-Dist: gitpython; extra == 'cli'
Requires-Dist: httpie; extra == 'cli'
Requires-Dist: icrawler; extra == 'cli'
Requires-Dist: objprint; extra == 'cli'
Requires-Dist: orjsonl; extra == 'cli'
Requires-Dist: paramiko; extra == 'cli'
Requires-Dist: prompt-toolkit>=3.0.0; extra == 'cli'
Requires-Dist: schedule; extra == 'cli'
Requires-Dist: twine; extra == 'cli'
Requires-Dist: typer; extra == 'cli'
Requires-Dist: viztracer; extra == 'cli'
Provides-Extra: clustering
Requires-Dist: hdbscan>=0.8.0; extra == 'clustering'
Requires-Dist: matplotlib; extra == 'clustering'
Requires-Dist: scikit-learn>=1.0.0; extra == 'clustering'
Requires-Dist: umap-learn>=0.5.0; extra == 'clustering'
Provides-Extra: crawl
Requires-Dist: crawl4ai; extra == 'crawl'
Requires-Dist: icrawler; extra == 'crawl'
Provides-Extra: dev
Requires-Dist: asciinema; extra == 'dev'
Requires-Dist: black; extra == 'dev'
Requires-Dist: concurrent-log-handler; extra == 'dev'
Requires-Dist: fastapi>=0.80.0; extra == 'dev'
Requires-Dist: gpustat>=1.0.0; extra == 'dev'
Requires-Dist: icrawler; extra == 'dev'
Requires-Dist: ordered-set; extra == 'dev'
Requires-Dist: orjson; extra == 'dev'
Requires-Dist: pandas; extra == 'dev'
Requires-Dist: pendulum>=2.1.2; extra == 'dev'
Requires-Dist: pillow; extra == 'dev'
Requires-Dist: pre-commit>=2.8; extra == 'dev'
Requires-Dist: psutil>=5.9.2; extra == 'dev'
Requires-Dist: pyinstrument; extra == 'dev'
Requires-Dist: pysnooper; extra == 'dev'
Requires-Dist: scalene; extra == 'dev'
Requires-Dist: twine; extra == 'dev'
Requires-Dist: uvicorn>=0.16.0; extra == 'dev'
Provides-Extra: embedding
Requires-Dist: fastapi>=0.80.0; extra == 'embedding'
Requires-Dist: numpy; extra == 'embedding'
Requires-Dist: sentence-transformers>=2.2.0; extra == 'embedding'
Requires-Dist: uvicorn>=0.16.0; extra == 'embedding'
Provides-Extra: lancedb
Requires-Dist: lancedb>=0.20.0; extra == 'lancedb'
Requires-Dist: pylance>=0.20.0; extra == 'lancedb'
Provides-Extra: latex
Requires-Dist: opencv-python-headless<4.3; extra == 'latex'
Requires-Dist: pix2tex[gui]; extra == 'latex'
Provides-Extra: llm
Requires-Dist: flaxkv2>=0.1.0; extra == 'llm'
Requires-Dist: tiktoken>=0.5.0; extra == 'llm'
Provides-Extra: ml
Requires-Dist: fastapi>=0.80.0; extra == 'ml'
Requires-Dist: marisa-trie>=0.7.8; extra == 'ml'
Requires-Dist: orjson; extra == 'ml'
Requires-Dist: pysnooper; extra == 'ml'
Requires-Dist: ray; extra == 'ml'
Requires-Dist: uvicorn>=0.16.0; extra == 'ml'
Provides-Extra: nlp
Requires-Dist: jionlp; extra == 'nlp'
Requires-Dist: levenshtein; extra == 'nlp'
Requires-Dist: nltk; extra == 'nlp'
Requires-Dist: rouge-chinese; extra == 'nlp'
Provides-Extra: npu
Requires-Dist: torch-npu>=2.1.0; extra == 'npu'
Provides-Extra: other
Requires-Dist: aiortc; extra == 'other'
Requires-Dist: arrayfire; extra == 'other'
Requires-Dist: awkward; extra == 'other'
Requires-Dist: cn2an; extra == 'other'
Requires-Dist: gradio; extra == 'other'
Requires-Dist: grpcio-reflection~=1.46.3; extra == 'other'
Requires-Dist: grpcio-tools~=1.46.3; extra == 'other'
Requires-Dist: grpcio~=1.46.3; extra == 'other'
Requires-Dist: keyboard; extra == 'other'
Requires-Dist: memray; extra == 'other'
Requires-Dist: protobuf~=3.19.1; extra == 'other'
Requires-Dist: pyzmq; extra == 'other'
Requires-Dist: recordclass; extra == 'other'
Requires-Dist: textdistance[extras]; extra == 'other'
Requires-Dist: wordfreq; extra == 'other'
Requires-Dist: zigzag; extra == 'other'
Provides-Extra: prompt
Requires-Dist: openai; extra == 'prompt'
Requires-Dist: streamlit; extra == 'prompt'
Requires-Dist: streamlit-ace; extra == 'prompt'
Provides-Extra: quant
Requires-Dist: accelerate>=1.0.0; extra == 'quant'
Requires-Dist: auto-round>=0.9.0; extra == 'quant'
Requires-Dist: bitsandbytes>=0.45.0; extra == 'quant'
Requires-Dist: llmcompressor>=0.9.0; extra == 'quant'
Requires-Dist: transformers>=4.45.0; extra == 'quant'
Provides-Extra: retriever
Requires-Dist: chromadb>=0.4.0; extra == 'retriever'
Requires-Dist: pymilvus>=2.6.0; extra == 'retriever'
Provides-Extra: test
Requires-Dist: flaxkv2; extra == 'test'
Requires-Dist: opencv-python; extra == 'test'
Requires-Dist: openpyxl; extra == 'test'
Requires-Dist: pandas; extra == 'test'
Requires-Dist: pytest; extra == 'test'
Requires-Dist: pytest-asyncio; extra == 'test'
Requires-Dist: scikit-learn; extra == 'test'
Provides-Extra: torch
Requires-Dist: bert4torch; extra == 'torch'
Requires-Dist: bertviz; extra == 'torch'
Requires-Dist: datasets; extra == 'torch'
Requires-Dist: einops; extra == 'torch'
Requires-Dist: fairseq; extra == 'torch'
Requires-Dist: koila; extra == 'torch'
Requires-Dist: lightseq; extra == 'torch'
Requires-Dist: orjson; extra == 'torch'
Requires-Dist: peft; extra == 'torch'
Requires-Dist: pytorch-lightning; extra == 'torch'
Requires-Dist: ray; extra == 'torch'
Requires-Dist: sacremoses; extra == 'torch'
Requires-Dist: seqevae; extra == 'torch'
Requires-Dist: transformers; extra == 'torch'
Requires-Dist: whylogs; extra == 'torch'
Provides-Extra: video
Requires-Dist: av; extra == 'video'
Requires-Dist: decord; extra == 'video'
Description-Content-Type: text/markdown

<h1 align="center">maque (麻雀)</h1>

<p align="center">
    <strong>Python toolkit for ML, CV, NLP and multimodal AI development</strong>
</p>

<p align="center">
    <a href="https://pypi.org/project/maque/">
        <img src="https://img.shields.io/pypi/v/maque?color=brightgreen&style=flat-square" alt="PyPI version">
    </a>
    <a href="https://github.com/KenyonY/maque/blob/main/LICENSE">
        <img alt="License" src="https://img.shields.io/github/license/KenyonY/maque.svg?color=blue&style=flat-square">
    </a>
    <a href="https://github.com/KenyonY/maque/actions/workflows/run_tests.yml">
        <img alt="tests" src="https://img.shields.io/github/actions/workflow/status/KenyonY/maque/run_tests.yml?style=flat-square&label=tests">
    </a>
    <a href="https://pypistats.org/packages/maque">
        <img alt="pypi downloads" src="https://img.shields.io/pypi/dm/maque?style=flat-square">
    </a>
</p>

---

## Features

- **LLM Server** - Local LLM inference with Transformers backend
- **Model Quantization** - Support auto-round, AWQ, GPTQ, BNB quantization methods
- **Embedding Service** - Text/multimodal embedding API server
- **Vector Retrievers** - Chroma / Milvus / Lance 后端，统一 API（upsert_batch / search / aiter_rows / describe）
- **Clustering Pipeline** - UMAP + HDBSCAN for vector clustering and visualization
- **Rich CLI** - Modular command groups for various tasks

## Installation

```bash
# Basic installation
pip install maque

# With specific feature sets
pip install maque[torch,nlp,cv]                       # ML/NLP/CV features
pip install maque[clustering,embedding,retriever]     # 向量检索 + 聚类（Chroma + Milvus）
pip install maque[lancedb]                            # LanceRetriever（含 pylance）
pip install maque[quant]                              # Model quantization support
pip install maque[dev,test]                           # Development setup

# From source
pip install -e .
pip install -e .[dev,test]
```

## CLI Usage

Commands are organized into groups: `maque <group> <command>`. Short alias `mq` is also available.

### Config Management

```bash
maque config show                 # Show current configuration
maque config edit                 # Open config in editor
maque config init                 # Initialize config file
```

### LLM Server (本地推理)

```bash
# Start LLM inference server (Transformers backend)
maque llm serve Qwen/Qwen2.5-7B-Instruct --port=8000

# With LoRA adapters
maque llm serve Qwen/Qwen2.5-7B-Instruct --lora_modules lora1=/path/to/lora1

# AWQ quantized model (requires: pip install maque[quant])
maque llm serve Qwen2.5-VL-3B-Instruct-AWQ
```

### Embedding Service

```bash
# Start embedding API server
maque embedding serve --model=BAAI/bge-m3 --port=8001

# Test embedding endpoint
maque embedding test --text="Hello world"
```

### Data Processing

```bash
# Interactive table viewer (Streamlit)
maque data table-viewer data.csv --port=8501

# Convert between formats
maque data convert input.json output.csv
```

### System Utilities

```bash
# Kill processes on ports
maque system kill 8000 8001

# Pack directory
maque system pack ./folder

# Split large file
maque system split large_file.dat --chunk_size=1GB
```

### Project rsync

`mq rsync push/pull` uses Git to locate the project root and to interpret
`.gitignore` exactly, including negated `!` rules. Rsync performs the transfer.
See the Chinese [beginner guide](docs/rsync.md) for setup, first-time binding,
delete semantics, safety checks, and troubleshooting.

Create the project-local config with the interactive initializer:

```bash
mq rsync init
```

For scripts and agents, provide both required values so the command never
prompts:

```bash
mq rsync init --remote=dev --endpoint=kunyuan@dev-host:/workspace/my-project/
```

The initializer uses Git's worktree root, writes `.maque/rsync.yaml`, and adds
`.maque/` to the root `.gitignore`. It does not contact SSH, bind a remote, or
create a token. With multiple remotes, the interactive flow also asks which one
is the default. The resulting file looks like this:

```yaml
version: 1
default: dev
remotes:
  dev: kunyuan@dev-host:/workspace/my-project/
  gpu1: kunyuan@gpu1:/data/my-project/
```

```bash
mq rsync init                 # Interactive local config setup
mq rsync push                 # Push to default; bootstraps an empty remote
mq rsync pull gpu1            # Pull from a named remote
mq rsync push dev --dry-run   # Preview without creating tokens or files
mq rsync pull --no-delete     # Keep extra non-ignored local files
mq rsync pair dev --yes       # Explicitly bind an existing non-empty directory
mq rsync copy SRC DST         # Low-level wrapper; sync mode, no deletion by default
```

Push and pull delete extra destination files by default only when the source
side's current Git ignore rules do not protect them. `.git/` and `.maque/`
metadata are never included in a transfer. The first push may initialize a
missing or empty remote with `git init` and bind it automatically. `pair` is
only needed to bind an existing non-empty directory; it does not transfer files.
If the two sides have different ignore rules, the source side wins: an extra
destination file ignored only by the destination's old rules may be deleted.
Non-empty unpaired directories are rejected until `pair` is run explicitly;
pull never bootstraps a remote. Every transfer verifies the project ID, named
remote token, canonical remote path, and exact remote Git root before rsync.
`init` and real `push`, `pull`, and `pair` operations idempotently add `.maque/`
to the project root `.gitignore` when needed; `--dry-run` reports the pending
update without changing the file. This keeps both `rsync.yaml` and
`rsync.token` project-local. Maque does not alter Git's index, so an already
tracked `.maque/` file remains tracked until you remove it from the index
yourself.

Both sides need a compatible GNU rsync. On macOS, the system `openrsync` lacks
the path-safe manifest options used by project sync; install GNU rsync with
`brew install rsync`. Maque automatically finds Homebrew rsync by its absolute
path, including over non-interactive SSH sessions whose `PATH` only exposes the
system binary.

### Agent Skill

```bash
# Install the maque skill
maque install-skill                 # Claude Code（默认）
maque install-skill --target=codex  # Codex

# Check installation status
maque skill-status --target=codex

# Uninstall skill
maque uninstall-skill --target=codex
```

After installation, use `/maque` in Claude Code or `$maque` in Codex.

### Git Helpers

```bash
# GitHub 镜像代理（国内加速）
maque git mirror-set                      # 设置全局镜像（默认 ghproxy）
maque git mirror-set --mirror=ghproxy-cdn # 使用 CDN 镜像
maque git mirror-status                   # 查看当前镜像配置
maque git mirror-unset                    # 移除镜像，恢复直连

# 设置后，原生 git 命令自动走镜像
git clone https://github.com/user/repo    # 自动使用镜像加速

# 可用镜像列表
maque git mirrors

# 单次使用镜像克隆（不修改全局配置）
maque git clone-mirror https://github.com/user/repo ./repo
```

## Python API

### IO Utilities

```python
from maque import yaml_load, yaml_dump, json_load, json_dump, jsonl_load, jsonl_dump

# Load/save YAML
config = yaml_load("config.yaml")
yaml_dump(data, "output.yaml")

# Load/save JSONL
records = jsonl_load("data.jsonl")
jsonl_dump(records, "output.jsonl")
```

### Embedding & Retrieval

```python
from maque.embedding import TextEmbedding
from maque.retriever import ChromaRetriever, Document

# Initialize
embedding = TextEmbedding(base_url="http://localhost:8001/v1", model="bge-m3")
retriever = ChromaRetriever(
    embedding,
    persist_dir="./chroma_db",
    collection_name="my_data"
)

# Insert documents
documents = [Document(id="1", content="text...", metadata={"source": "file1"})]
retriever.upsert_batch(documents, batch_size=32, skip_existing=True)

# Search
results = retriever.search("query text", top_k=10)
```

Milvus / Lance 后端 API 一致，并额外提供流式扫描与跨后端统一元信息：

```python
from maque.retriever import MilvusRetriever, LanceRetriever

# Milvus（生产级，AsyncMilvusClient）
mv = MilvusRetriever(embedding, uri="http://localhost:19530", collection_name="docs")

# Lance（本地零运维，列式过滤）
lz = LanceRetriever(
    embedding, persist_dir="./lance_db", table_name="docs",
    extra_fields={"category": str, "price": float},  # 提升为独立列，可 SQL 过滤
)
results = lz.search("query", top_k=5, where="category = 'tech'")

# 流式扫描（不爆内存，适合 100k+ 全量或 filter 子集）
for row in lz.iter_rows(expr="category = 'tech'", batch_size=1000):
    ...

# 跨后端统一元信息
desc = lz.describe()
# {"schema": [...], "dim": 768, "count": 12345, "indexes": [...], "primary_key": "id"}

# async 接口（FastAPI / asyncio 场景不阻塞事件循环）
import asyncio
async def main():
    async for row in mv.aiter_rows(expr='feedback == "bad"'):
        ...
    desc = await lz.adescribe()
asyncio.run(main())
```

### Clustering Pipeline

```python
from maque.clustering import ClusterAnalyzer

analyzer = ClusterAnalyzer(algorithm="hdbscan", min_cluster_size=15)

# Analyze from ChromaDB
result = analyzer.analyze_chroma(
    persist_dir="./chroma_db",
    collection_name="my_data",
    output_dir="./results",
    sample_size=10000,
    visualize=True
)

# Access results
print(f"Found {result.n_clusters} clusters")
print(result.labels)
print(result.cluster_stats)
```

### Performance Measurement

```python
from maque import MeasureTime

with MeasureTime("model inference", gpu=True):
    output = model(input)
# Prints: model inference took 0.123s (GPU: 0.089s)
```

## Configuration

maque uses hierarchical configuration (highest priority first):

1. `./maque_config.yaml` (current directory)
2. Project root config
3. `~/.maque/config.yaml` (user config)

Example configuration:

```yaml
embedding:
  model: BAAI/bge-m3
  base_url: http://localhost:8001/v1

llm:
  default_port: 8000
```

Initialize config:
```bash
maque config init
```

## Development

```bash
# Install development dependencies
pip install -e .[dev,test]

# Run tests
pytest
pytest -m "not slow"  # Skip slow tests

# Format code
black .
isort .
```

## License

MIT License - see [LICENSE](LICENSE) for details.
