Metadata-Version: 2.5
Name: gbx-codecs
Version: 0.1.0
Summary: Audio codecs for speech models served by gbx-lm, run as a companion process on Apple silicon
Project-URL: Homepage, https://github.com/GreenBitAI/gbx-codecs
Author: GreenBitAI
License-Expression: MIT
License-File: LICENSE
License-File: licenses/LICENSE.mlx-audio.txt
Classifier: Operating System :: MacOS
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
Requires-Python: >=3.10
Requires-Dist: miniaudio>=1.59
Requires-Dist: mlx>=0.29
Requires-Dist: numpy>=1.24
Provides-Extra: test
Requires-Dist: pytest; extra == 'test'
Description-Content-Type: text/markdown

# gbx-codecs

Audio codecs for the speech models that [gbx-lm](https://github.com/GreenBitAI/gbx-lm) serves, on Apple silicon.

A speech model generated by gbx-lm produces **codes**, not audio. This package turns codes into audio, and reference audio into codes. It runs as a **separate process** that gbx-lm starts and talks to over a pipe, so the server itself needs none of these dependencies, and the codec can be installed, upgraded or removed without touching gbx-lm.

| codec | used by | rate | codebooks |
|---|---|---|---|
| `fish_s1` (Fish Audio S1 DAC) | Fish Audio Drama 3 | 44.1 kHz mono, 21.5 frames/s | 10 |

The codec weights come with the model repository; this package holds code only.

## Install

Python 3.10 or newer on an Apple silicon Mac. A virtual environment of its own is recommended:

```bash
python3 -m venv ~/.gbx-codecs
~/.gbx-codecs/bin/pip install gbx-codecs
```

Dependencies: `mlx`, `numpy`, and `miniaudio` (to read reference audio in wav, flac or mp3). No PyTorch and no ffmpeg. The process needs about 3 GB of memory while decoding (Fish S1: 1.75 GB of weights, decoded 8 frames at a time).

Check it against a downloaded model:

```bash
~/.gbx-codecs/bin/gbx-codecs check /path/to/the/model
# codec OK: 10 codebooks, 44 frames for 2 s, 44100 Hz, round-trip correlation 0.9..
```

Then tell gbx-lm which Python to start it with (`--codec_python ~/.gbx-codecs/bin/python`, or `GBX__SERVER__CODEC_PYTHON`). See the model card of the speech model for the full setup.

## Command line

```bash
gbx-codecs encode MODEL_DIR reference.wav codes.npy   # audio -> int32 codes [codebooks, frames]
gbx-codecs decode MODEL_DIR codes.npy out.wav          # codes -> 16-bit wav
gbx-codecs serve                                       # the companion process gbx-lm starts
```

## Protocol

`gbx-codecs serve` reads requests on stdin and writes replies on stdout; logs go to stderr. Each message is `GBXC`, a 4-byte header length and an 8-byte payload length (big-endian), a UTF-8 JSON header, then the payload. Operations: `hello`, `load` (codec, model directory), `encode` (audio file to int32 codes), `begin` (output sample rate), `decode` (int32 codes to 16-bit PCM, streamed), `end`, `quit`. See `src/gbx_codecs/protocol.py` and `src/gbx_codecs/worker.py`.

Decoding is streaming: frames sent a piece at a time decode to the same audio whatever the size of the pieces, to within one 16-bit step, and resampling (for example to 24 kHz for OpenAI-style `pcm` output) carries its state across pieces.

## License

MIT, see [LICENSE](LICENSE).

The Fish Audio S1 DAC implementation in `src/gbx_codecs/fish_s1/dac.py` is ported from [mlx-audio](https://github.com/Blaizzy/mlx-audio) (MIT License, Copyright (c) 2024 Prince Canuma; see [licenses/LICENSE.mlx-audio.txt](licenses/LICENSE.mlx-audio.txt)). The codec weights are distributed with the models under the model's own license; this package does not include them.

Not affiliated with or endorsed by Fish Audio.
