Metadata-Version: 2.5
Name: psdn-sonar
Version: 0.1.3
Summary: Multi-language ASR evaluation toolkit — metrics, reporting, and benchmarks
Project-URL: Homepage, https://github.com/PSDN-AI/SONAR-OSS
Project-URL: Repository, https://github.com/PSDN-AI/SONAR-OSS
Project-URL: Bug Tracker, https://github.com/PSDN-AI/SONAR-OSS/issues
Author: PSDN AI
License: MIT
License-File: LICENSE
Keywords: asr,benchmark,evaluation,multilingual,speech-recognition
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: <3.13,>=3.10
Requires-Dist: datasets<4.0.0,>=2.14.0
Requires-Dist: indic-numtowords>=1.0.0
Requires-Dist: jiwer>=3.0.0
Requires-Dist: librosa>=0.10.0
Requires-Dist: matplotlib>=3.7.0
Requires-Dist: nltk>=3.8.0
Requires-Dist: num2words>=0.5.12
Requires-Dist: numba>=0.57.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: omegaconf>=2.3.0
Requires-Dist: pandas>=2.0.0
Requires-Dist: plotnine>=0.13.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pydub>=0.25.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: requests>=2.28.0
Requires-Dist: scikit-learn>=1.3.0
Requires-Dist: scipy>=1.10
Requires-Dist: soundfile>=0.12.0
Requires-Dist: tqdm>=4.60.0
Provides-Extra: all
Requires-Dist: assemblyai; extra == 'all'
Requires-Dist: bangla; extra == 'all'
Requires-Dist: banglaspeech2text; extra == 'all'
Requires-Dist: bnlp-toolkit; extra == 'all'
Requires-Dist: boto3>=1.28.0; extra == 'all'
Requires-Dist: g2pk>=0.1.0; extra == 'all'
Requires-Dist: google-genai>=1.0.0; extra == 'all'
Requires-Dist: hgtk>=0.1.3; extra == 'all'
Requires-Dist: indic-nlp-library>=0.92; extra == 'all'
Requires-Dist: indic-transliteration; extra == 'all'
Requires-Dist: jamo>=0.4.1; extra == 'all'
Requires-Dist: konlpy>=0.6.0; extra == 'all'
Requires-Dist: onnxruntime>=1.10.0; extra == 'all'
Requires-Dist: openai>=1.0.0; extra == 'all'
Requires-Dist: peft; extra == 'all'
Requires-Dist: pyannote-audio<5.0.0,>=4.0.7; extra == 'all'
Requires-Dist: sentence-transformers>=3.0.0; extra == 'all'
Requires-Dist: soynlp>=0.0.493; extra == 'all'
Requires-Dist: speechmos>=0.0.1; extra == 'all'
Requires-Dist: torch>=2.2.0; extra == 'all'
Requires-Dist: torchaudio>=2.2.0; extra == 'all'
Requires-Dist: transformers<5.0.0,>=4.33.0; extra == 'all'
Provides-Extra: apis
Requires-Dist: assemblyai; extra == 'apis'
Requires-Dist: google-genai>=1.0.0; extra == 'apis'
Requires-Dist: openai>=1.0.0; extra == 'apis'
Provides-Extra: bengali
Requires-Dist: bangla; extra == 'bengali'
Requires-Dist: banglaspeech2text; extra == 'bengali'
Requires-Dist: bnlp-toolkit; extra == 'bengali'
Requires-Dist: indic-transliteration; extra == 'bengali'
Requires-Dist: peft; extra == 'bengali'
Provides-Extra: cloud
Requires-Dist: boto3>=1.28.0; extra == 'cloud'
Provides-Extra: dev
Requires-Dist: moto[s3]>=5.0; extra == 'dev'
Requires-Dist: pre-commit>=3.8; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Requires-Dist: ruff<0.16,>=0.15.11; extra == 'dev'
Requires-Dist: ty==0.0.31; extra == 'dev'
Provides-Extra: hindi
Requires-Dist: indic-nlp-library>=0.92; extra == 'hindi'
Provides-Extra: korean
Requires-Dist: g2pk>=0.1.0; extra == 'korean'
Requires-Dist: hgtk>=0.1.3; extra == 'korean'
Requires-Dist: jamo>=0.4.1; extra == 'korean'
Requires-Dist: konlpy>=0.6.0; extra == 'korean'
Requires-Dist: soynlp>=0.0.493; extra == 'korean'
Provides-Extra: ml
Requires-Dist: onnxruntime>=1.10.0; extra == 'ml'
Requires-Dist: sentence-transformers>=3.0.0; extra == 'ml'
Requires-Dist: speechmos>=0.0.1; extra == 'ml'
Requires-Dist: torch>=2.2.0; extra == 'ml'
Requires-Dist: torchaudio>=2.2.0; extra == 'ml'
Requires-Dist: transformers<5.0.0,>=4.33.0; extra == 'ml'
Provides-Extra: ml-common
Requires-Dist: onnxruntime>=1.10.0; extra == 'ml-common'
Requires-Dist: sentence-transformers>=3.0.0; extra == 'ml-common'
Requires-Dist: speechmos>=0.0.1; extra == 'ml-common'
Requires-Dist: transformers<5.0.0,>=4.33.0; extra == 'ml-common'
Provides-Extra: pyannote
Requires-Dist: pyannote-audio<5.0.0,>=4.0.7; extra == 'pyannote'
Description-Content-Type: text/markdown

<div align="center">

# SONAR-OSS

**Multi-language ASR evaluation — built for the languages most benchmarks forget.**

[![CI](https://github.com/PSDN-AI/SONAR-OSS/actions/workflows/ci.yml/badge.svg)](https://github.com/PSDN-AI/SONAR-OSS/actions/workflows/ci.yml)
[![PyPI version](https://img.shields.io/pypi/v/psdn-sonar.svg)](https://pypi.org/project/psdn-sonar/)
[![Python versions](https://img.shields.io/pypi/pyversions/psdn-sonar.svg)](https://pypi.org/project/psdn-sonar/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![GitHub stars](https://img.shields.io/github/stars/PSDN-AI/SONAR-OSS?style=social)](https://github.com/PSDN-AI/SONAR-OSS/stargazers)
[![Follow on X](https://img.shields.io/twitter/follow/psdnai?style=social)](https://x.com/psdnai)

[**Website**](https://psdn.ai) · [**X / Twitter**](https://x.com/psdnai) · [**LinkedIn**](https://www.linkedin.com/company/psdnai) · [**Package (PyPI)**](https://pypi.org/project/psdn-sonar/) · [**Usage guide**](docs/USAGE.md) · [**FAQ**](docs/FAQ.md)

</div>

SONAR-OSS is [Poseidon](https://psdn.ai)'s open-source toolkit for evaluating
automatic speech recognition (ASR) models — metrics, reporting, and
benchmarks, distributed as the `psdn-sonar` Python package. Point it at your
audio and reference transcripts, pick models (local HuggingFace checkpoints or
hosted APIs like OpenAI Whisper, ElevenLabs, and AssemblyAI), and it produces
per-sample and aggregate scores you can actually compare: WER/CER under
language-specific normalization contracts, semantic similarity, the composite
POSEIDON score, audio-quality diagnostics, and latency — with every run's
model revision, device, and normalization recorded in the artifact so the
numbers stay reproducible.

Where SONAR-OSS earns its keep is off the beaten benchmark path:

- **Low-resource languages first.** Bengali, Hindi, and Korean ship with
  curated model defaults and dedicated normalizers alongside English; any
  other language runs through a YAML recipe (`psdn-sonar custom`) — adding a
  language means writing config, not code.
- **Single- and multi-speaker evaluation.** The multi-speaker pipeline
  handles VAD, diarization, and channel-split preprocessing (pyannote), and
  scores the pipeline end to end.
- **Scores you can defend.** Artifacts carry lineage (checkpoint SHA,
  normalization contract, device) and machine-readable warnings; nothing is
  back-solved or invented. `psdn-sonar leaderboard` renders a comparison
  table from measured runs only.
- **One CLI for the whole loop.** `discover` prepares datasets, `single` /
  `multi` / `custom` evaluate, `leaderboard` compares — CSV, JSON, Markdown,
  and plots out.

![SONAR pipeline: input data flows through ASR models and processing into evaluation and results](docs/assets/sonar-pipeline.png)

If SONAR-OSS is useful to you, **please
[⭐ star the repo](https://github.com/PSDN-AI/SONAR-OSS/stargazers)** — it
helps other voice-AI teams find it and tells us where to invest.

> **Status: pre-release.** The library (metrics, language processors, dataset
> loaders, evaluators, reporting, CLI) is in place and heading toward a first
> `0.1.0` release; see [`CHANGELOG.md`](CHANGELOG.md). Content imported from
> the upstream codebase passes the checklist in
> [`docs/import-gate.md`](docs/import-gate.md).

## Table of contents

- [Requirements](#requirements)
- [Installation](#installation)
  - [Install from PyPI](#install-from-pypi)
  - [Contributor install (from source)](#contributor-install-from-source)
  - [API keys and gated models](#api-keys-and-gated-models)
- [Usage](#usage)
- [Development](#development)
- [Community and links](#community-and-links)
- [License](#license)

## Requirements

- Python 3.10–3.12
- [`uv`](https://docs.astral.sh/uv/) (recommended) or `pip`
- `ffmpeg` **required** by the pipeline-based ASR adapters — including the
  English defaults `whisper_base_en` / `whisper_small_en` and any `--hf-model`
  that falls back to the generic pipeline — for **all** audio input, WAV
  included (the package decodes every input file with `ffmpeg` itself and
  hands the pipeline a raw array, so one decoder covers WAV, M4A/AAC and
  everything else). These adapters refuse to load without it and name the
  missing binary. Adapters that decode through libsndfile (the `wav2vec2_*`
  models and the non-pipeline Whisper fine-tunes) evaluate WAV and FLAC —
  and MP3, with the libsndfile ≥ 1.1 that current `soundfile` wheels bundle —
  without `ffmpeg`; formats libsndfile cannot read (M4A/AAC) and `pydub`
  silence-trimming of non-WAV input still need it. Install: `sudo apt-get
  install ffmpeg` (Debian/Ubuntu), `brew install ffmpeg` (macOS), or
  `winget install Gyan.FFmpeg` (Windows). The `[pyannote]` extra needs
  `ffmpeg` too, and needs more than the binary: pyannote.audio 4.x decodes
  audio through torchcodec, which loads the ffmpeg **shared libraries** at
  runtime. On Windows that means the full-shared build —
  `winget install Gyan.FFmpeg.Shared` — because the default `Gyan.FFmpeg`
  build ships only `ffmpeg.exe` and no DLLs; the package registers the
  DLL-carrying ffmpeg directories for resolution itself (Python ≥ 3.8 does
  not consult `PATH` for a native extension's dependent DLLs). Both builds
  can be installed side by side in either `PATH` order: every ffmpeg
  directory on `PATH` is inspected and the ones holding the shared
  libraries are registered, so the static build shadowing the shared one
  does not matter. Linux and macOS package managers install the shared
  libraries alongside the binary, so no extra step is needed there.

**Supported environments.** CI validates Linux x86_64 with CPython 3.10, 3.11,
and 3.12, plus macOS arm64 with CPython 3.12 and the `[ml]` extra installed
(so the HuggingFace model adapters are exercised on every PR, not just
importable). Windows is expected to work for the core package but is not
CI-validated; some optional extras have platform-sensitive dependencies
(`[korean]` needs a Java runtime at runtime, `[ml]`/`[pyannote]` pull large
PyTorch trees). Open an issue if an install fails on a supported Python.
The checked-in `.python-version` pins `uv` to CPython 3.12, so `make setup`
on a machine with no Python selected builds a supported interpreter instead
of whatever newest version `uv` manages (CI passes `--python` explicitly and
is unaffected).

**What a first run costs.** On a fresh machine, budget several gigabytes of
downloads and tens of minutes before the first number appears; everything is
cached, so later runs against the same data and models start in seconds.
Measured example (Bengali FLEURS + one CTC model): ~3.4 GB / ~27 min for the
dataset, 1.26 GB for the model checkpoint, and ~1.5 GB on disk for the `[ml]`
extra (torch alone ~0.5 GB). `psdn-sonar discover --max-samples` bounds how
many samples are *prepared*, not the download — each requested split is
fetched into the HuggingFace cache in full on first run. Two more downloads
happen lazily: the first POSEIDON / semantic-similarity call fetches the
~64 MB sentence-transformers scorer, and audio-quality analysis fetches a
~390 MB UTMOS checkpoint via `torch.hub`. Checkpoint size can also hide
behind an adapter: `khushids_bengali` is a 62 MB PEFT adapter whose
`openai/whisper-large-v3` base adds ~2.9 GB.

**Compute device and runtime.** Local HuggingFace adapters auto-select the
best available device — CUDA, then MPS (Apple Silicon), then CPU — and the
device used is recorded in each run's `scores_<model>.json` (`submission.device`),
since a GPU run and a CPU run can produce different transcripts for the same
audio. On CPU, architecture dominates runtime: measured on the same
200-utterance FLEURS Bengali set, a Wav2Vec2 CTC model ran at ~1.2 s/sample
while a Whisper-medium fine-tune ran at ~19 s/sample. A full multi-model
language default (Bengali has 9 models, mostly Whisper-class) is a
multi-hour job without a GPU — trim `--models` and `--max-samples` to size
your run first.

## Installation

### Install from PyPI

Requires Python 3.10, 3.11, or 3.12.

1. Create and activate a fresh virtual environment. These examples use
   Python 3.12; substitute 3.10 or 3.11 if needed.

   macOS or Linux (bash/zsh):

   ```bash
   python3.12 --version
   python3.12 -m venv sonar-env
   source sonar-env/bin/activate
   ```

   Windows PowerShell:

   ```powershell
   py -3.12 --version
   py -3.12 -m venv sonar-env
   Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope Process
   .\sonar-env\Scripts\Activate.ps1
   python -c "import sys; print(sys.executable)"   # must print ...\sonar-env\Scripts\python.exe
   ```

   The `Set-ExecutionPolicy` line matters: Windows client editions default
   to the `Restricted` policy, which refuses `Activate.ps1` with an error on
   stderr **while the shell continues with exit 0** — so without it,
   `python` still points at the previous interpreter and step 2 silently
   installs outside the environment. `-Scope Process` changes the policy
   for this PowerShell session only. The `sys.executable` check confirms
   activation actually took. From `cmd.exe`, use
   `sonar-env\Scripts\activate.bat` instead, which needs no policy change.

2. Install the package:

   ```bash
   python -m pip install "psdn-sonar>=0.1.1"
   ```

   To install an optional extra, name it in brackets — for example
   `python -m pip install "psdn-sonar[ml]>=0.1.1"` for the local-model
   backends. [`CONTRIBUTING.md`](CONTRIBUTING.md) lists every extra and what
   it pulls in.

   The `>=0.1.1` lower bound makes an unsupported interpreter fail loudly:
   releases from 0.1.1 declare `Requires-Python >=3.10,<3.13`, but 0.1.0
   predates that bound, so on Python 3.13 a bare `pip install psdn-sonar`
   silently falls back to the outdated 0.1.0 — two releases behind, with no
   warning — instead of reporting that 3.13 is unsupported. With the bound,
   pip names the `Requires-Python` restriction and installs nothing.

3. Verify the install:

   ```bash
   psdn-sonar --version
   python -c "import psdn_sonar; print(psdn_sonar.__version__)"
   ```

Then follow [`docs/USAGE.md`](docs/USAGE.md) for runnable examples.

### Contributor install (from source)

Contributors install the frozen, locked environment (exactly what CI runs).

macOS or Linux (bash/zsh):

```bash
git clone https://github.com/PSDN-AI/SONAR-OSS.git
cd SONAR-OSS
make setup            # uv sync --frozen with dev extras — does NOT include [ml]
# or, to run the docs/USAGE.md examples (local models, semantic similarity):
make setup-ml         # dev + [ml] extras, ~1.5 GB on disk
source .venv/bin/activate
```

Windows (`make` is not present on a stock host; the two targets are single
`uv` commands, so run them directly — [`uv`](https://docs.astral.sh/uv/)
provisions the pinned CPython itself):

```powershell
git clone https://github.com/PSDN-AI/SONAR-OSS.git
cd SONAR-OSS
uv sync --frozen --extra dev              # what `make setup` runs
# or, for the docs/USAGE.md examples:
uv sync --frozen --extra dev --extra ml   # what `make setup-ml` runs
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope Process
.\.venv\Scripts\Activate.ps1
```

(The execution-policy line is the same silent-refusal guard explained in the
PyPI section; from `cmd.exe` use `.venv\Scripts\activate.bat`. The `make`
targets themselves run on Windows from a Unix-style shell such as Git Bash,
bundled with [Git for Windows](https://gitforwindows.org/).)

Or with plain pip (editable, freshly resolved). **Create and activate a
virtual environment first** — exactly as in step 1 of the PyPI install
section above. On "externally managed" interpreters (PEP 668: Homebrew and
Debian/Ubuntu system Pythons) pip otherwise refuses with
`error: externally-managed-environment` and installs nothing:

```bash
pip install -e ".[dev]"       # contributor tooling only — no [ml]
pip install -e ".[dev,ml]"    # what the docs/USAGE.md examples need
```

The `[ml]` extra is what runs local HuggingFace models and POSEIDON's
semantic similarity; without it the USAGE examples fail with a `TypeError`
naming the extra. Add it to an existing `make setup` environment with
`uv pip install -e ".[ml]"`.

One model in the Bengali defaults needs more than `[ml]`: `khushids_bengali`
is a PEFT/LoRA adapter and requires `peft` from the `[bengali]` extra
(`pip install "psdn-sonar[bengali]"`). Without it, a `--language bn` run
skips that model with a message naming the extra and evaluates the rest of
the defaults; one unavailable model never aborts a multi-model run.

Note that `pip` does not read `uv.lock`: pip installs resolve dependency
versions fresh from PyPI within the ranges in `pyproject.toml`, so they are
not byte-for-byte reproducible the way `uv sync --frozen` is. This is the
normal contract for downstream package installs; use `uv` when you need the
locked contributor environment.

### API keys and gated models

Copy `.env.example` to `.env` and fill in only the values you need (API keys
are required only for optional hosted-model backends). For pyannote
VAD/diarization, setting `HF_TOKEN` is not enough on its own: the pyannote
models are gated on HuggingFace, so the token's account must also accept the
user conditions on each model page —
[`pyannote/segmentation-3.0`](https://huggingface.co/pyannote/segmentation-3.0),
[`pyannote/speaker-diarization-3.1`](https://huggingface.co/pyannote/speaker-diarization-3.1),
and
[`pyannote/speaker-diarization-community-1`](https://huggingface.co/pyannote/speaker-diarization-community-1)
(the diarization pipeline downloads the third as a gated dependency under
pyannote.audio 4.x, even though no command names it) — otherwise runs fail
with `403 ... not in the authorized list` even though the token is valid.

The LLM-judged metrics (entity preservation and intent pass rate, importable
from `psdn_sonar.utils.llm_metrics` — a library-only surface with no CLI
subcommand) read `GEMINI_API_KEY` (preferred) or `GOOGLE_API_KEY` as an
alternative. A `.env` entry and an exported variable both work, same as the
other API keys; when the same name is set in both places, the exported
variable wins, so a per-run `env GEMINI_API_KEY=... psdn-sonar ...` prefix
overrides the checkout's `.env`.

## Usage

See [`docs/USAGE.md`](docs/USAGE.md) for a short quickstart with runnable
examples (scoring a pair, evaluating a model over a dataset, listing models),
and [`docs/FAQ.md`](docs/FAQ.md) for common workflows: CLI commands, required
input files, output layout, and how success is measured.

Before comparing published numbers, read
[`docs/SCORE_INTERPRETATION.md`](docs/SCORE_INTERPRETATION.md): what the
scores measure — multi-speaker results include preprocessing error, scores
are comparable within a dataset only, and cells where a model is evaluated
on its declared training corpus carry an in-domain marker.

## Development

```bash
make lint              # ruff lint + format check
make typecheck         # ty type checker
make test              # pytest
make pre-commit-install
make check-internal-refs
```

On Windows, run these from a Unix-style shell such as Git Bash (bundled with
[Git for Windows](https://gitforwindows.org/)): the recipes are POSIX shell,
and GNU Make falls back to `cmd` when no Unix shell is on `PATH`.

See [`CONTRIBUTING.md`](CONTRIBUTING.md) for the full contributor guide,
including PR title conventions and the import gate.

## Community and links

- Website: [psdn.ai](https://psdn.ai)
- X / Twitter: [@psdnai](https://x.com/psdnai)
- LinkedIn: [Poseidon](https://www.linkedin.com/company/psdnai)
- Issues and feature requests: [GitHub issues](https://github.com/PSDN-AI/SONAR-OSS/issues)

## License

MIT — see [`LICENSE`](LICENSE).
