Metadata-Version: 2.5
Name: gbx-codecs
Version: 0.1.1
Summary: Audio codecs for speech models served by gbx-lm, run as a companion process on Apple silicon and NVIDIA GPUs
Project-URL: Homepage, https://github.com/GreenBitAI/gbx-codecs
Author: GreenBitAI
License-Expression: MIT
License-File: LICENSE
License-File: licenses/LICENSE.mlx-audio.txt
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
Requires-Python: >=3.10
Requires-Dist: mlx>=0.29
Requires-Dist: numpy>=1.24
Provides-Extra: formats
Requires-Dist: miniaudio>=1.59; extra == 'formats'
Provides-Extra: test
Requires-Dist: pytest; extra == 'test'
Description-Content-Type: text/markdown

# gbx-codecs

Audio codecs for the speech models that [gbx-lm](https://github.com/GreenBitAI/gbx-lm) serves, on Apple silicon and NVIDIA (DGX Spark).

A speech model generated by gbx-lm produces **codes**, not audio. This package turns codes into audio, and reference audio into codes. It runs as a **separate process** that gbx-lm starts and talks to over a pipe, so the server itself needs none of these dependencies, and the codec can be installed, upgraded or removed without touching gbx-lm.

| codec | used by | rate | codebooks |
|---|---|---|---|
| `fish_s1` (Fish Audio S1 DAC) | Fish Audio Drama 3 | 44.1 kHz mono, 21.5 frames/s | 10 |

The codec weights come with the model repository; this package holds code only.

## Install

Python 3.10 or newer, in a virtual environment of its own:

```bash
python3 -m venv ~/.gbx-codecs
~/.gbx-codecs/bin/pip install gbx-codecs                 # Apple silicon Mac
~/.gbx-codecs/bin/pip install "mlx[cuda13]" gbx-codecs   # Linux with an NVIDIA GPU (DGX Spark)
```

Dependencies: `mlx` and `numpy`. No PyTorch and no ffmpeg. Reference audio is read from WAV; for flac or mp3, add `gbx-codecs[formats]` (which installs `miniaudio`; on Linux ARM it compiles, and needs the Python headers). The process needs about 3 GB of memory while decoding (Fish S1: 1.75 GB of weights, decoded 8 frames at a time).

Check it against a downloaded model:

```bash
~/.gbx-codecs/bin/gbx-codecs check /path/to/the/model
# codec OK: 10 codebooks, 44 frames for 2 s, 44100 Hz, round-trip correlation 0.9..
```

Then tell gbx-lm which Python to start it with (`--codec_python ~/.gbx-codecs/bin/python`, or `GBX__SERVER__CODEC_PYTHON`). See the model card of the speech model for the full setup.

## Command line

```bash
gbx-codecs encode MODEL_DIR reference.wav codes.npy   # audio -> int32 codes [codebooks, frames]
gbx-codecs decode MODEL_DIR codes.npy out.wav          # codes -> 16-bit wav
gbx-codecs serve                                       # the companion process gbx-lm starts
```

## Protocol

`gbx-codecs serve` reads requests on stdin and writes replies on stdout; logs go to stderr. Each message is `GBXC`, a 4-byte header length and an 8-byte payload length (big-endian), a UTF-8 JSON header, then the payload. Operations: `hello`, `load` (codec, model directory), `encode` (audio file to int32 codes), `begin` (output sample rate), `decode` (int32 codes to 16-bit PCM, streamed), `end`, `quit`. See `src/gbx_codecs/protocol.py` and `src/gbx_codecs/worker.py`.

Decoding is streaming: frames sent a piece at a time decode to the same audio whatever the size of the pieces, to within one 16-bit step, and resampling (for example to 24 kHz for OpenAI-style `pcm` output) carries its state across pieces.

## License

MIT, see [LICENSE](LICENSE).

The `fish_s1` codec implements Fish Audio's audio codec, the one Fish Audio Drama 3 uses. It was ported from mlx-audio; use it with Fish Audio's codec weights only as the [Fish Audio Community License](https://huggingface.co/fishaudio/drama-3) that comes with those weights allows.

The Fish Audio S1 DAC implementation in `src/gbx_codecs/fish_s1/dac.py` is ported from [mlx-audio](https://github.com/Blaizzy/mlx-audio) (MIT License, Copyright (c) 2024 Prince Canuma; see [licenses/LICENSE.mlx-audio.txt](licenses/LICENSE.mlx-audio.txt)). The codec weights are distributed with the models under the model's own license; this package does not include them.

Not affiliated with or endorsed by Fish Audio.
