Metadata-Version: 2.4
Name: mlx-whisperx
Version: 0.1.14
Summary: WhisperX-style transcription pipeline using an internal mlx-whisper ASR backend.
License-Expression: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: mlx>=0.11
Requires-Dist: numpy
Requires-Dist: numba
Requires-Dist: tiktoken
Requires-Dist: huggingface-hub
Requires-Dist: scipy
Requires-Dist: pandas
Requires-Dist: torch==2.11.*
Requires-Dist: torchaudio==2.11.*
Requires-Dist: transformers<6,>=5
Requires-Dist: nltk
Requires-Dist: tqdm
Requires-Dist: silero-vad<7,>=5.1
Requires-Dist: typing-extensions; python_version < "3.11"
Provides-Extra: diarize
Requires-Dist: pyannote-audio; extra == "diarize"
Provides-Extra: full
Requires-Dist: pyannote-audio; extra == "full"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Dynamic: license-file

# mlx-whisperx

`mlx-whisperx` is a WhisperX-style transcription pipeline for Apple Silicon. It uses a vendored `mlx-whisper` ASR backend, then optionally applies WhisperX forced alignment and pyannote diarization.

The project is intended to provide a practical local pipeline with WhisperX-like JSON, subtitle, and text outputs while keeping ASR execution on MLX.

## Why This Project Exists

This project adds WhisperX-like functionality to an `mlx-whisper` workflow. The goal is to keep ASR inference on MLX for Apple Silicon while providing the pipeline pieces people commonly use from WhisperX: VAD chunking, forced alignment, word timestamps, diarization hooks, and familiar JSON/subtitle outputs.

The implementation borrows ideas and code from both upstream projects:

- WhisperX, for the pipeline structure, alignment workflow, diarization integration, and output conventions.
- `mlx-whisper`, for the Apple Silicon ASR backend and model execution path.

This repository vendors and adapts code where needed so the pieces work together as a standalone `mlx-whisperx` package.

## Pipeline

```text
audio -> VAD -> mlx-whisper ASR -> forced alignment -> optional diarization -> writers
```

Default behavior:

- ASR model: `mlx-community/whisper-turbo`
- VAD backend: Silero (`--vad_method silero`)
- Chunking: VAD chunks decoded in batches of 8 (`--batch_size 8`)
- Cross-chunk context: off; `--interleaved_context` prompts each chunk with the text before it
- Decoding: greedy decoding with `beam_size=1` and `temperature=0`
- Alignment: enabled for transcription
- Diarization: disabled unless `--diarize` is passed

## Installation

Clone the repository and install it into a Python environment:

```bash
git clone https://github.com/seedds/mlx-whisperx.git
cd mlx-whisperx
python -m pip install -e .
```

The published package constrains the alignment stack to a tested range:

- `torch 2.11.x`
- `torchaudio 2.11.x`
- `transformers >=5,<6`

`torchvision` is not required for alignment.

Install diarization support when you want pyannote VAD and diarization in addition to the default Silero path:

```bash
python -m pip install -e ".[diarize]"
```

Install everything this repository currently exposes:

```bash
python -m pip install -e ".[full]"
```

`ffmpeg` must be available on `PATH` because audio loading is handled through the ffmpeg CLI.

On macOS with Homebrew:

```bash
brew install ffmpeg
```

Pyannote VAD and diarization use pyannote models and may require a Hugging Face token, depending on the selected model.

## Usage

```bash
mlx-whisperx AUDIO [AUDIO ...] [OPTIONS]
```

By default, `mlx-whisperx`:

- uses `mlx-community/whisper-turbo`
- runs Silero VAD by default
- performs forced alignment for word timestamps
- writes outputs to the current directory
- writes every supported output format when `--output_format` is not specified

Common options:

- `--model`: local model path or Hugging Face repo
- `--language`: language code such as `en`, `ja`, or `fr`
- `--task`: `transcribe` or `translate`
- `--output_dir`: directory to write output files
- `--output_name`: custom basename for output files
- `--output_format`: `all`, `json`, `srt`, `vtt`, `txt`, `tsv`, or `aud`
- `--no_align`: skip forced alignment
- `--diarize`: attach speaker labels when diarization is enabled
- `--hf_token`: Hugging Face token for gated pyannote models

Example: output JSON

```bash
mlx-whisperx audio.wav \
  --output_dir transcripts \
  --output_name audio \
  --output_format json
```

This writes `transcripts/audio.json`.

Example JSON output:

```json
{
  "segments": [
    {
      "start": 0.0,
      "end": 2.52,
      "text": "Hello and welcome to mlx-whisperx.",
      "words": [
        {"word": "Hello", "start": 0.0, "end": 0.42, "score": 0.99},
        {"word": "and", "start": 0.44, "end": 0.58, "score": 0.98},
        {"word": "welcome", "start": 0.6, "end": 1.05, "score": 0.97},
        {"word": "to", "start": 1.07, "end": 1.18, "score": 0.98},
        {"word": "mlx-whisperx.", "start": 1.2, "end": 2.52, "score": 0.96}
      ]
    }
  ],
  "word_segments": [
    {"word": "Hello", "start": 0.0, "end": 0.42, "score": 0.99},
    {"word": "and", "start": 0.44, "end": 0.58, "score": 0.98}
  ],
  "language": "en"
}
```

Example: output SRT

```bash
mlx-whisperx audio.wav \
  --output_dir subtitles \
  --output_name audio \
  --output_format srt \
  --max_line_width 42 \
  --max_line_count 2
```

This writes `subtitles/audio.srt`.

Example SRT output:

```srt
1
00:00:00,000 --> 00:00:02,520
Hello and welcome to
mlx-whisperx.

2
00:00:02,700 --> 00:00:05,100
This example shows subtitle
output.
```

Complete example with optional parameters:

```bash
mlx-whisperx audio.wav \
  --model mlx-community/whisper-turbo \
  --model_dir ./models \
  --model_cache_only False \
  --device cpu \
  --compute_type float32 \
  --output_dir ./out \
  --output_name meeting \
  --output_format all \
  --verbose True \
  --log-level info \
  --task transcribe \
  --language en \
  --align_model jonatasgrosman/wav2vec2-large-xlsr-53-english \
  --interpolate_method nearest \
  --return_char_alignments \
  --vad_method pyannote \
  --vad_onset 0.5 \
  --vad_offset 0.363 \
  --vad_model pyannote/segmentation-3.0 \
  --chunk_size 30 \
  --vad_dump_path ./out/meeting.vad.json \
  --diarize \
  --min_speakers 2 \
  --max_speakers 4 \
  --diarize_model pyannote/speaker-diarization-community-1 \
  --speaker_embeddings \
  --hf_token YOUR_HF_TOKEN \
  --temperature 0.0 \
  --temperature_increment_on_fallback 0.2 \
  --best_of 5 \
  --beam_size 1 \
  --patience 1.0 \
  --length_penalty 1.0 \
  --suppress_tokens -1 \
  --suppress_numerals \
  --initial_prompt "Technical meeting about MLX WhisperX." \
  --hotwords "MLX, WhisperX, pyannote, diarization" \
  --interleaved_context \
  --compression_ratio_threshold 2.4 \
  --logprob_threshold -1.0 \
  --no_speech_threshold 0.6 \
  --max_line_width 42 \
  --max_line_count 2 \
  --max_words_per_line 8 \
  --highlight_words False \
  --print_progress True
```

Notes:

- `--output_format all` writes `.txt`, `.vtt`, `.srt`, `.tsv`, `.json`, and `.aud`.
- `--output_name` accepts a single input file, so batch runs cannot overwrite one output.
- `.aud` is an Audacity label track: tab-separated start/end seconds and label text.
- `--max_line_count` only has an effect when `--max_line_width` is also set.
- `--highlight_words` applies to `srt` and `vtt`.
- `--hf_token` is only needed for gated pyannote models.

Output filenames keep the full input stem, so `meeting.part1.wav` and `meeting.part2.wav`
write `meeting.part1.json` and `meeting.part2.json` rather than colliding on one file.

## Python API

```python
from mlx_whisperx import transcribe

result = transcribe(
    "audio.wav",
    model="mlx-community/whisper-turbo",
    language="en",
)
```

Print one transcript segment per line:

```python
for segment in result["segments"]:
    print(segment["text"].strip())
```

Print segment timestamps:

```python
for segment in result["segments"]:
    print(f"[{segment['start']:.2f} -> {segment['end']:.2f}] {segment['text'].strip()}")
```

Print word-level timestamps:

```python
for word in result["word_segments"]:
    print(f"[{word['start']:.2f} -> {word['end']:.2f}] {word['word']}")
```

Common API options match the CLI names:

```python
result = transcribe(
    "audio.wav",
    model="mlx-community/whisper-turbo",
    language="en",
    beam_size=1,
    temperature=0.0,
    no_align=False,
    diarize=False,
)
```

`language` accepts either canonical codes such as `en` or case-insensitive names and aliases such as `English` or `Portuguese`.

## Output Schema

JSON output follows the WhisperX-style shape:

```json
{
  "segments": [
    {
      "start": 0.0,
      "end": 2.5,
      "text": "Example transcript text.",
      "words": [
        {"word": "Example", "start": 0.0, "end": 0.6, "score": 0.98}
      ]
    }
  ],
  "word_segments": [
    {"word": "Example", "start": 0.0, "end": 0.6, "score": 0.98}
  ],
  "language": "en"
}
```

When diarization is enabled, speaker labels are included where available:

```json
{"word": "Hello", "start": 0.0, "end": 0.4, "score": 0.99, "speaker": "SPEAKER_00"}
```

## CLI Reference

Basic options:

- `--model`: `mlx-whisper` model directory or Hugging Face repo.
- `--language`: language code or case-insensitive name/alias. If omitted, language is auto-detected by ASR.
- `--task`: `transcribe` or `translate`.
- `--output_format`: `all`, `srt`, `vtt`, `txt`, `tsv`, `json`, or `aud`.
- `--output_dir`: directory for output files.
- `--output_name`: custom output basename.
- `--verbose`: print transcript and logs.

English-only Whisper models such as `.en` checkpoints force `language=en` and do not support `--task translate`.

Decoding options:

- `--temperature`: sampling temperature. Default is `0.0`.
- `--beam_size`: beam size when `temperature=0`. Default is `1`.
- `--best_of`: number of candidates when sampling with `temperature > 0`.
- `--patience`: beam-search patience.
- `--length_penalty`: beam-search length penalty.
- `--suppress_tokens`: comma-separated token IDs to suppress.
- `--suppress_numerals`: suppress numeric and currency-symbol tokens.
- `--initial_prompt`: initial prompt for ASR.
- `--hotwords`: hint phrases appended to the prompt.
- `--interleaved_context`: prompt each VAD chunk with the text decoded before it, which helps punctuation and the spelling of recurring names, while still decoding in batches. See [Interleaved context](#interleaved-context).
- `--condition_on_previous_text`: prompt each window with the previous window's text. Only applies with `--no_vad`; with VAD every chunk is a single window, so it has no effect and logs a warning pointing to `--interleaved_context`.

Precision and model-cache options:

- `--compute_type float16`: force MLX ASR fp16. This is the default.
- `--compute_type float32`: force MLX ASR fp32.
- `--model_dir`: cache directory for ASR, alignment, pyannote VAD, and diarization models.
- `--model_cache_only`: cached ASR, alignment, pyannote VAD, and diarization models only.

VAD options:

- `--vad_method auto`: prefer pyannote and fall back to Silero if pyannote cannot initialize.
- `--vad_method pyannote`: require pyannote VAD and do not fall back.
- `--vad_method silero`: use Silero VAD directly. This is the default.
- `--vad_onset`: VAD onset threshold.
- `--vad_offset`: VAD offset threshold. Silero applies this only when the installed version supports an offset threshold; otherwise it is ignored with a warning.
- `--vad_model`: Hugging Face pyannote segmentation model used with `--vad_method pyannote`. Defaults to `pyannote/segmentation-3.0`.
- `--chunk_size`: merged VAD chunk size in seconds. Must be `30` or less, because each chunk is decoded as one 30-second window. Speech longer than this is split across chunks.
- `--no_vad`: transcribe the full file as one chunk.
- `--coverage`: choose chunks by audible energy instead of trusting the VAD alone. The VAD is still used, but only to locate pauses for chunk seams. Each chunk is already decoded over its full span, so speech the VAD misses between two detected turns is transcribed without this option; what `--coverage` adds is speech the VAD misses *outside* any chunk — counting, chanting, singing, stylised delivery with no detected speech nearby — which is otherwise dropped with no output and no warning. It sends more audio to the decoder, and therefore costs decode time, in exchange for a modest recall gain; leave it off unless you suspect speech is being dropped. Check the `Coverage:` line in the log to see how much of the file is reaching the decoder.
- `--coverage_floor`: with `--coverage`, the energy floor as a fraction of the file's own measured speech level. Defaults to `0.20`. The scale is relative because absolute amplitude varies with source and mastering, so a fixed floor does not transfer between files. Lower values admit quieter audio and risk transcribing music; higher values risk discarding quiet speech.
- `--min_silent_run`: with `--coverage`, seconds of continuous quiet required before coverage is broken. Defaults to `1.0`. Shorter runs are treated as pauses within speech rather than as gaps between it.
- `--clip_timestamps`: comma-separated clip start/end pairs in seconds. Requires `--no_vad`.
- `--vad_dump_path`: write VAD chunks and settings to JSON.

Batching options:

- `--batch_size`: number of chunks decoded together in one pass. Defaults to `8`. Higher values are faster but use more memory; `1` disables batching. With `--interleaved_context` it is also the number of context streams.

When using the default `--vad_method silero`, or when the `--vad_method auto` path falls back to Silero, Silero loads from the pinned `silero-vad` package, which ships its weights and needs no network access. To use a local Silero checkout instead, set:

```bash
export MLX_WHISPERX_SILERO_VAD_PATH=/path/to/snakers4_silero-vad
```

When this variable is set it is authoritative: that checkout is the only one tried, and a
missing or broken checkout raises instead of silently downloading a different copy.

Alignment options:

- `--no_align`: skip forced alignment.
- `--allow_missing_alignment_deps`: continue without forced alignment when alignment dependencies are unavailable.
- `--align_model`: override the alignment model.
- `--interpolate_method`: `nearest`, `linear`, or `ignore`. `ignore` leaves unalignable words without `start`/`end` instead of estimating them.
- `--return_char_alignments`: include character alignments in JSON.

Diarization options:

- `--diarize`: assign speaker labels.
- `--diarize_model`: pyannote diarization model name.
- `--min_speakers`: minimum speaker count.
- `--max_speakers`: maximum speaker count.
- `--speaker_embeddings`: include speaker embeddings in JSON when available.
- `--hf_token`: Hugging Face token for gated pyannote models.

Subtitle options:

- `--max_line_width`: target subtitle line width, counting the spaces between words.
- `--max_line_count`: maximum lines per subtitle cue. Requires `--max_line_width`.
- `--max_words_per_line`: maximum words per subtitle cue.
- `--highlight_words`: underline the active word in SRT/VTT output.

These limits apply whether or not forced alignment ran. With `--diarize`, a cue also ends
at each speaker change, so one cue never mixes speakers under a single `[SPEAKER_XX]:` prefix.

### Interleaved context

Batched chunks are normally decoded independently, so a chunk cannot see the text before it. `--interleaved_context` (ported from WhisperX's context-aware batching) splits the chunks into `--batch_size` contiguous streams and decodes chunk *k* of every stream together. Each chunk is prompted with what its own stream has decoded so far, after `--initial_prompt` and `--hotwords`, up to Whisper's 223-token prompt limit. The first chunk of each stream after the first has no history in that pass, so it is decoded once more at the end, prompted with the tail of the stream before it.

- Fewer streams give longer histories but less parallelism; `--batch_size 1` is a single stream in which every chunk is prompted with the chunks before it.
- Chunks in one batch carry different prompt lengths. The decoder masks the padding, so each chunk decodes exactly as it would alone with the same prompt.
- A stream's history is cleared after a chunk needed a decode above temperature 0.5 or looped on one word, so a hallucination is not fed forward. A chunk dropped as silence leaves the history unchanged.
- The re-decode only replaces a chunk's text when both decodes pass the silence and loop checks, so it never turns a chunk the first pass judged silent into speech.
- Cost: longer prompts on every chunk plus one extra batch of up to `--batch_size - 1` re-decodes.

## Examples

Inspect VAD chunks before ASR:

```bash
mlx-whisperx audio.wav \
  --output_format json \
  --vad_dump_path audio.vad.json
```

Transcribe only selected clips without VAD chunking:

```bash
mlx-whisperx audio.wav \
  --no_vad \
  --clip_timestamps 0,15,30,45 \
  --output_format json
```

Run deterministic beam search explicitly:

```bash
mlx-whisperx audio.wav \
  --language en \
  --temperature 0 \
  --beam_size 5 \
  --output_format json
```

Use temperature fallback:

```bash
mlx-whisperx audio.wav \
  --temperature 0 \
  --temperature_increment_on_fallback 0.2
```

Suppress numerals and currency symbols during decoding:

```bash
mlx-whisperx audio.wav --suppress_numerals --output_format json
```

Require pyannote VAD instead of the default Silero behavior:

```bash
mlx-whisperx audio.wav \
  --vad_method pyannote \
  --vad_model pyannote/segmentation-3.0 \
  --hf_token YOUR_HF_TOKEN \
  --output_format json
```

Skip forced alignment:

```bash
mlx-whisperx audio.wav --no_align --output_format json
```

Continue when alignment dependencies are unavailable:

```bash
mlx-whisperx audio.wav --allow_missing_alignment_deps --output_format json
```

Run diarization:

```bash
mlx-whisperx audio.wav \
  --diarize \
  --hf_token YOUR_HF_TOKEN \
  --output_format json
```

Process multiple files:

```bash
mlx-whisperx first.wav second.wav third.wav --output_dir transcripts --output_format all
```

A failing file does not stop the run: the remaining files are still transcribed, the error is
printed, and the command exits nonzero at the end so scripts can detect the partial failure.

## Output Formats

| Format | Contents |
| --- | --- |
| `json` | Full result: segments, word timings, scores, and speakers when diarized. |
| `srt`, `vtt` | Subtitle cues, honoring the subtitle options and optional word highlighting. |
| `txt` | One line per segment, prefixed with `[SPEAKER_XX]:` when diarized. |
| `tsv` | Tab-separated `start`, `end`, `speaker`, `text` with millisecond times. |
| `aud` | Audacity label track: tab-separated start/end seconds and label text, no header. Speakers appear as `[[SPEAKER_XX]]`. |

Import `.aud` into Audacity with **File -> Import -> Labels**.

## Current Behavior and Limitations

- ASR decodes merged VAD chunks in batches of `--batch_size` (default 8); set `--batch_size 1` to decode serially.
- Without `--interleaved_context`, VAD chunks are decoded independently and do not see each other's text.
- `translate` skips forced alignment because alignment models are transcription-language specific.
- Missing `torch`, `torchaudio`, or `transformers` still fail alignment by default; pass `--allow_missing_alignment_deps` to continue with ASR-only output instead.
- `clip_timestamps` is only supported with `--no_vad` because VAD chunking changes the timing base before ASR runs. Clip ranges are decoded independently, so audio between them is skipped.
- VAD chunks are capped at `--chunk_size` seconds, which must not exceed the 30-second decoder window; longer chunks are rejected rather than silently truncated.
- Each VAD chunk is decoded over its full span, from its first detected speech turn to its last, as in WhisperX. Speech the VAD misses between turns inside a chunk is therefore still transcribed, and so is any music or noise there; the no-speech and loop checks below still drop non-speech chunks.
- Chunk text is dropped when the decoder scores it as non-speech above `--no_speech_threshold` and its average log probability is below `--logprob_threshold` (Whisper's silence rule), or when a single word makes up most of a long chunk. The latter is a decoder loop, which music and noise provoke — an opening theme can otherwise yield one word repeated hundreds of times as a single long cue. Emphatic real repetition is preserved; the loop test requires a majority of a long utterance to be one token. Drops are logged with their timestamp and reason.
- The CLI exits nonzero when any input file fails, while still processing the remaining files.
- `--condition_on_previous_text` only affects `--no_vad` decoding. Use `--interleaved_context` to carry text across VAD chunks.
- Pyannote VAD and diarization depend on a compatible PyTorch, torchaudio, pyannote installation, and Hugging Face model access when the selected model is gated. Without that stack, use the default `--vad_method silero` path or let `--vad_method auto` fall back to Silero VAD.
- The vendored ASR backend lives under `mlx_whisperx.backend.mlx_whisper` so decoder behavior can be changed without modifying external reference repositories.

## Development Checks

Run the test suite:

```bash
python -m pip install -e ".[dev]"
python -m pytest tests -q
```

Most tests stub the ASR backend and model downloads, so they run quickly and offline. One
regression test transcribes a bundled audio fixture with a real model and is opt-in:

```bash
MLX_WHISPERX_RUN_SAMPLE_TEST=1 python -m pytest tests/test_regression_transcribe_sample.py
```

Compile the package:

```bash
python -m py_compile mlx_whisperx/**/*.py
```

Check CLI help:

```bash
python -m mlx_whisperx --help
```

Build a wheel:

```bash
python -m build
```
