Metadata-Version: 2.4
Name: arclm
Version: 0.8.0
Summary: A compact PyTorch toolkit for native language-model training and Hugging Face SFT.
Author: Ahmad Al Dibo
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/ahmad-al-dibo/arclm
Project-URL: Documentation, https://github.com/ahmad-al-dibo/arclm#readme
Project-URL: Source, https://github.com/ahmad-al-dibo/arclm
Project-URL: Issues, https://github.com/ahmad-al-dibo/arclm/issues
Project-URL: Changelog, https://github.com/ahmad-al-dibo/arclm/blob/main/CHANGELOG.md
Keywords: arclm,sft,llm,language-model,causal-language-model,transformer,pytorch,training,fine-tuning,instruction-tuning,nlp,huggingface
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Science/Research
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: <3.13,>=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch<3,>=2.1
Requires-Dist: numpy<3,>=1.24
Requires-Dist: sentencepiece<0.3,>=0.2
Requires-Dist: transformers<6,>=4.51
Requires-Dist: tomli>=2; python_version < "3.11"
Provides-Extra: cpu
Requires-Dist: torch<3,>=2.1; extra == "cpu"
Requires-Dist: torchvision<1,>=0.16; extra == "cpu"
Requires-Dist: torchaudio<3,>=2.1; extra == "cpu"
Provides-Extra: cuda
Requires-Dist: torch<3,>=2.1; extra == "cuda"
Requires-Dist: torchvision<1,>=0.16; extra == "cuda"
Requires-Dist: torchaudio<3,>=2.1; extra == "cuda"
Provides-Extra: cuda121
Requires-Dist: torch<3,>=2.1; extra == "cuda121"
Requires-Dist: torchvision<1,>=0.16; extra == "cuda121"
Requires-Dist: torchaudio<3,>=2.1; extra == "cuda121"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: build<2,>=1.2; extra == "dev"
Requires-Dist: twine<7,>=5; extra == "dev"
Requires-Dist: mkdocs<2,>=1.6; extra == "dev"
Requires-Dist: mkdocs-material<10,>=9; extra == "dev"
Requires-Dist: ruff<1,>=0.8; extra == "dev"
Requires-Dist: mypy<2,>=1.10; extra == "dev"
Requires-Dist: pip-audit<3,>=2.7; extra == "dev"
Provides-Extra: docs
Requires-Dist: mkdocs<2,>=1.6; extra == "docs"
Requires-Dist: mkdocs-material<10,>=9; extra == "docs"
Provides-Extra: web
Requires-Dist: flask<4,>=3; extra == "web"
Provides-Extra: preprocess
Requires-Dist: beautifulsoup4<5,>=4.12; extra == "preprocess"
Requires-Dist: PyYAML<7,>=6; extra == "preprocess"
Requires-Dist: tqdm<5,>=4.66; extra == "preprocess"
Provides-Extra: peft
Requires-Dist: peft<1,>=0.11; extra == "peft"
Provides-Extra: hf
Requires-Dist: accelerate<2,>=0.30; extra == "hf"
Requires-Dist: datasets<4,>=2.18; extra == "hf"
Requires-Dist: peft<1,>=0.11; extra == "hf"
Provides-Extra: examples
Requires-Dist: accelerate<2,>=0.30; extra == "examples"
Requires-Dist: datasets<4,>=2.18; extra == "examples"
Requires-Dist: peft<1,>=0.11; extra == "examples"
Requires-Dist: safetensors<1,>=0.4; extra == "examples"
Provides-Extra: all
Requires-Dist: torch<3,>=2.1; extra == "all"
Requires-Dist: torchvision<1,>=0.16; extra == "all"
Requires-Dist: torchaudio<3,>=2.1; extra == "all"
Requires-Dist: flask<4,>=3; extra == "all"
Requires-Dist: beautifulsoup4<5,>=4.12; extra == "all"
Requires-Dist: PyYAML<7,>=6; extra == "all"
Requires-Dist: tqdm<5,>=4.66; extra == "all"
Requires-Dist: accelerate<2,>=0.30; extra == "all"
Requires-Dist: datasets<4,>=2.18; extra == "all"
Requires-Dist: peft<1,>=0.11; extra == "all"
Requires-Dist: safetensors<1,>=0.4; extra == "all"
Provides-Extra: all-cpu
Requires-Dist: torch<3,>=2.1; extra == "all-cpu"
Requires-Dist: torchvision<1,>=0.16; extra == "all-cpu"
Requires-Dist: torchaudio<3,>=2.1; extra == "all-cpu"
Requires-Dist: flask<4,>=3; extra == "all-cpu"
Requires-Dist: beautifulsoup4<5,>=4.12; extra == "all-cpu"
Requires-Dist: PyYAML<7,>=6; extra == "all-cpu"
Requires-Dist: tqdm<5,>=4.66; extra == "all-cpu"
Requires-Dist: accelerate<2,>=0.30; extra == "all-cpu"
Requires-Dist: datasets<4,>=2.18; extra == "all-cpu"
Requires-Dist: peft<1,>=0.11; extra == "all-cpu"
Requires-Dist: safetensors<1,>=0.4; extra == "all-cpu"
Provides-Extra: all-cuda
Requires-Dist: torch<3,>=2.1; extra == "all-cuda"
Requires-Dist: torchvision<1,>=0.16; extra == "all-cuda"
Requires-Dist: torchaudio<3,>=2.1; extra == "all-cuda"
Requires-Dist: flask<4,>=3; extra == "all-cuda"
Requires-Dist: beautifulsoup4<5,>=4.12; extra == "all-cuda"
Requires-Dist: PyYAML<7,>=6; extra == "all-cuda"
Requires-Dist: tqdm<5,>=4.66; extra == "all-cuda"
Requires-Dist: accelerate<2,>=0.30; extra == "all-cuda"
Requires-Dist: datasets<4,>=2.18; extra == "all-cuda"
Requires-Dist: peft<1,>=0.11; extra == "all-cuda"
Requires-Dist: safetensors<1,>=0.4; extra == "all-cuda"
Provides-Extra: all-cuda121
Requires-Dist: torch<3,>=2.1; extra == "all-cuda121"
Requires-Dist: torchvision<1,>=0.16; extra == "all-cuda121"
Requires-Dist: torchaudio<3,>=2.1; extra == "all-cuda121"
Requires-Dist: flask<4,>=3; extra == "all-cuda121"
Requires-Dist: beautifulsoup4<5,>=4.12; extra == "all-cuda121"
Requires-Dist: PyYAML<7,>=6; extra == "all-cuda121"
Requires-Dist: tqdm<5,>=4.66; extra == "all-cuda121"
Requires-Dist: accelerate<2,>=0.30; extra == "all-cuda121"
Requires-Dist: datasets<4,>=2.18; extra == "all-cuda121"
Requires-Dist: peft<1,>=0.11; extra == "all-cuda121"
Requires-Dist: safetensors<1,>=0.4; extra == "all-cuda121"
Dynamic: license-file

# ArcLM

ArcLM is a focused Python framework for preparing language-model data and building reproducible workflows for causal language models.

ArcLM `0.8.0.dev0` is not a general machine-learning framework and should not be treated as production-ready yet. Its strongest supported paths are data-first workflows, streaming dataset preparation, native ArcLM causal-language-model checkpoints, and certified tiny GPT-2 Hugging Face integration tests.

## Purpose

ArcLM helps developers move through the practical language-model workflow:

```text
Raw data -> Loading -> Cleaning -> Validation -> Transformation -> Formatting
-> Tokenization -> Model loading -> Training or fine-tuning -> Evaluation
-> Inference -> Reporting
```

The framework puts dataset preparation first because most training and fine-tuning failures start before the model is loaded: inconsistent records, missing fields, duplicated samples, tokenizer mismatches, and undocumented formatting choices.

## Main Features

- Load JSON, JSONL, CSV, TXT, or custom in-memory datasets with `DataProcessor`.
- Clean, filter, transform, split, and tokenize records with composable dataset helpers.
- Run JSONL preprocessing reports with `PreprocessPipeline`.
- Build word or SentencePiece tokenizers with `Tokenizer` and `SentencePieceTokenizer`.
- Train compact native decoder-only ArcLM models with `train_model`.
- Load native checkpoints with `load_model`.
- Inspect and load Hugging Face causal-LM sources with `inspect_model_source` and `load_any_model`.
- Run Hugging Face SFT with `train_sft` when optional dependencies and hardware are available.
- Generate metrics and diagnostics for native ArcLM checkpoints.
- Start the optional Flask simple interface with `python -m arclm --run simple-interface`.

## Project Status

Development version: `0.8.0.dev0`. Current released version: `0.6.1`.

Status: pre-1.0 framework development. Public APIs are usable but still need stronger validation, automated documentation checks, model-family verification, and a formal deprecation policy before a stable production release.

## Supported Model Focus

ArcLM initially focuses on:

- Causal language models
- Decoder-only transformer models
- Models compatible with causal language modeling workflows

Official support currently means the model path has verified loading, tokenizer loading, causal-LM behavior, inference or training where claimed, documented limitations, and an automated or reproducible verification path.

See [Supported Models](docs/supported-models.md) for the full support matrix.

## Installation

Install from PyPI:

```bash
pip install arclm
```

For CPU-only environments, install with the official PyTorch CPU index:

```bash
pip install "arclm[all-cpu]" --index-url https://download.pytorch.org/whl/cpu --extra-index-url https://pypi.org/simple
```

For CUDA 12.1:

```bash
pip install "arclm[all-cuda121]" --index-url https://download.pytorch.org/whl/cu121 --extra-index-url https://pypi.org/simple
```

For local development:

```bash
pip install -e ".[dev,preprocess,hf,web]"
```

ArcLM declares Python `>=3.9,<3.13`.

## Minimal Quick Start

This example uses only public ArcLM APIs and trains a tiny native causal model on CPU.

```python
from pathlib import Path
import tempfile

from arclm import DataProcessor, Tokenizer, load_model, train_model

with tempfile.TemporaryDirectory() as tmp:
    root = Path(tmp)
    raw_path = root / "records.jsonl"
    train_path = root / "train.txt"
    model_path = root / "model.pth"

    raw_path.write_text(
        '{"text": "ArcLM prepares language model data."}\n'
        '{"text": "Clean records make training easier."}\n',
        encoding="utf-8",
    )

    dataset = (
        DataProcessor.load(raw_path)
        .clean()
        .filter(lambda row: len(row.get("text", "")) > 10)
        .transform(format="pretraining")
    )

    tokenizer = Tokenizer(max_vocab=64)
    tokenizer.build(" ".join(row["text"] for row in dataset.samples))
    tokenized = dataset.tokenize(tokenizer)
    assert all("tokens" in row for row in tokenized.samples)

    train_path.write_text(
        " ".join(row["text"] for row in dataset.samples) * 24,
        encoding="utf-8",
    )

    train_model(
        mode="pretrain",
        data=str(train_path),
        output=str(model_path),
        tokenizer_type="word",
        max_vocab=64,
        embed_dim=16,
        num_blocks=1,
        block_size=8,
        batch_size=2,
        num_epochs=1,
        validation_split=0.0,
        training_log_interval=0,
        device="cpu",
    )

    loaded = load_model(model_path, device="cpu")
    print(loaded.predict("ArcLM", max_new_tokens=4, top_k=3))
```

## Documentation

- [Documentation home](docs/index.md)
- [Getting Started](docs/quick-start.md)
- [Data Preparation Guide](docs/data-guide/loading-data.md)
- [Model Loading Guide](docs/model-guide/loading-models.md)
- [API Reference](docs/api-reference/index.md)
- [CLI Reference](docs/cli-reference.md)
- [Migration Guide](docs/migration-guide.md)
- [Production Readiness](docs/production-readiness.md)
- [Roadmap](docs/roadmap.md)

## Examples

Local examples are in [examples](examples/README.md). Start with:

```bash
python examples/01_quickstart.py
python examples/03_data_processing.py
python examples/11_inference.py
```

Examples that use Hugging Face models may download model files and need optional dependencies:

```bash
pip install -e ".[hf,peft]"
python examples/08_huggingface_sft.py
```

## Honest Limitations

- ArcLM native models are compact GPT-style models, not production-scale LLM architectures.
- Hugging Face model loading is limited to causal language models through `AutoModelForCausalLM`.
- Qwen examples are reproducible examples, not automated release certification.
- Encoder-only and seq2seq models are out of scope for the current public workflow.
- The CLI still contains older pathways that need consolidation with `train_model`.
- Production readiness requires stronger validation, typed configs, full CI coverage, and a release/deprecation policy.

## Contributing

See [Contributing](docs/contributing.md). Contributions should keep ArcLM focused on data-first causal-language-model workflows and should include tests or reproducible examples for new public behavior.

## License

ArcLM is released under the [Apache License 2.0](LICENSE).
