Metadata-Version: 2.4
Name: protodecimal
Version: 0.1.2
Summary: Compact lossless protobuf decimals: decimal.Decimal conversions and the protoc-gen-python-decimal codegen plugin
License-Expression: MIT
Project-URL: Homepage, https://github.com/andefined/protodecimal
Project-URL: Documentation, https://andefined.github.io/protodecimal/
Project-URL: Changelog, https://github.com/andefined/protodecimal/blob/main/CHANGELOG.md
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: protobuf<7,>=6.30
Provides-Extra: native
Requires-Dist: protodecimal-native==0.1.2; extra == "native"
Dynamic: license-file

# protodecimal

[![CI](https://github.com/andefined/protodecimal/actions/workflows/ci.yml/badge.svg)](https://github.com/andefined/protodecimal/actions/workflows/ci.yml)

Lossless decimals for protobuf, with code generators that convert fields to and from [`shopspring/decimal`](https://github.com/shopspring/decimal) in Go and `decimal.Decimal` in Python.

It is built for order books, where prices and quantities carry many decimal places. A 25-level snapshot with 12 to 28 decimal places takes 876 bytes, compared with 1,407 bytes as strings.

## Wire format

```proto
message Decimal {
  bool   negative    = 1;
  bytes  coefficient = 2;
  sint32 exponent    = 3;
}
// value = (negative ? -1 : 1) * coefficient * 10^exponent
```

- `negative` is left off the wire when false.
- `coefficient` is the magnitude as big-endian bytes. Empty means zero.
- `exponent` uses zigzag encoding, so small negative exponents take one byte.
- Values keep their scale: `1.50` is stored as `150` and `-2`.

Below about 10 significant digits, a decimal is about the same size as a string.

## Installation

Go, version 1.27 or newer:

```sh
go get github.com/andefined/protodecimal/go/decimalconv
go install github.com/andefined/protodecimal/go/cmd/protoc-gen-go-decimal@latest
```

Python, version 3.10 or newer:

```sh
pip install protodecimal              # includes protoc-gen-python-decimal
pip install "protodecimal[native]"    # adds the fast encoder
```

The Python package needs `protobuf>=6.30,<7`. The fast encoder has prebuilt wheels for Linux and macOS (x86-64 and ARM) and Windows (x86-64). On other platforms, pip builds it from source, which needs Rust.

Documentation: [Go](https://pkg.go.dev/github.com/andefined/protodecimal/go), [Python](https://andefined.github.io/protodecimal/).

## Usage

### Setting up your repository

Your `.proto` files import `protodecimal/v1/decimal.proto`, so copy that file from a release into your repository, keeping its path:

```sh
mkdir -p third_party/protodecimal/v1
curl -fsSL -o third_party/protodecimal/v1/decimal.proto \
  https://raw.githubusercontent.com/andefined/protodecimal/v0.1.2/proto/protodecimal/v1/decimal.proto
```

```proto
import "protodecimal/v1/decimal.proto";

message Trade {
  string symbol = 1;
  protodecimal.v1.Decimal price = 2;
}
```

Install the plugins:

```sh
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install github.com/andefined/protodecimal/go/cmd/protoc-gen-go-decimal@latest
pip install protodecimal    # includes protoc-gen-python-decimal
```

Then, with buf, list both directories in `buf.yaml` but generate code only for your own files:

```yaml
# buf.yaml
version: v2
modules:
  - path: proto
  - path: third_party
```

```yaml
# buf.gen.yaml
version: v2
inputs:
  - directory: proto
plugins:
  - local: protoc-gen-go
    out: gen
    opt: module=example.com/yourapp/gen
  - local: protoc-gen-go-decimal
    out: gen
    opt: module=example.com/yourapp/gen
  - remote: buf.build/protocolbuffers/python:v30.0
    out: py
  - local: protoc-gen-python-decimal
    out: py
```

Don't generate code for `decimal.proto` itself. Your generated code uses the copy in the packages you install: `github.com/andefined/protodecimal/go/decimalpb` in Go and `protodecimal.v1.decimal_pb2` in Python. A second copy would clash with it. The Python plugin version `v30.0` produces code that works with every `protobuf` 6.x release.

### Accessors

For each field of type `protodecimal.v1.Decimal`, the plugins generate accessors:

```go
level.SetPriceDecimal(decimal.RequireFromString("65122.99"))
price := level.GetPriceDecimal()
```

```python
import orderbook_pb2_decimal  # adds the accessors to orderbook_pb2

level.set_price_decimal(Decimal("65122.99"))
price = level.get_price_decimal()
```

Repeated fields get the same accessors, using slices and lists.

### Fast paths

For busy services there are two faster options. Both use the same wire format, so each side can switch on its own.

**Python `fast_encode`** writes a whole message to bytes in one call. Pass field values in declaration order: nested messages as tuples, repeated fields as lists, and `None` for unset fields.

```python
wire = orderbook_pb2.Snapshot.fast_encode(
    "BTC-USDT",
    [(Decimal("65122.99"), Decimal("0.50"))],
)
```

It needs the native encoder. Without it, `fast_encode` still returns the same bytes but is slower than the normal path, so check `protodecimal.fast.NATIVE` at startup. Messages with maps, groups or required fields don't get `fast_encode`.

**Go `Decimals` decoders** read wire bytes straight into decimals:

```go
var snapshot orderbookpb.SnapshotDecimals // reuse it between calls
if err := snapshot.Unmarshal(raw); err != nil {
	return err
}
price := snapshot.Bids[0].Price
```

They are generated for proto3 messages without maps, oneofs, `optional` fields, recursive types or message types from other files. Unlike `proto.Unmarshal`, they reject a singular message field that appears twice. Encoding in Go still uses the accessors and `proto.Marshal`.

Both fast paths are generated only with the default plugin options.

## Limits

Treat decimals from other services as untrusted input. Large ones are cheap to send but slow to convert.

| Limit | Value |
| --- | --- |
| Coefficient size | 16 KiB (about 39,000 digits) |
| Exponent when decoding in Go | ±6,144 |

Python checks the limits on every conversion, and Go's `Decimals` decoders check them while decoding. Go's generated getters can't return errors, so if you use `proto.Unmarshal`, validate each message once:

```go
if err := decimalconv.ValidateMessage(&snapshot); err != nil {
	return err
}
```

## Plugin options

- `decimal_type`: the message used as the decimal type. Default `protodecimal.v1.Decimal`.
- `conv`: the conversion package. Defaults `github.com/andefined/protodecimal/go/decimalconv` (Go) and `protodecimal.conv` (Python).

The Go plugin also accepts the usual `module=` and `paths=` options.

## Development

You need Go, `buf`, [`uv`](https://docs.astral.sh/uv/) and a Rust toolchain. `make generate` builds a pinned `protoc-gen-go` into `bin/`, so generated code is the same on every machine.

```sh
make generate   # regenerate code from proto/
make test       # Go, Python, and Python with the native encoder
make lint
make bench      # Go and Python benchmarks
make help       # all targets
```

Python code is generated with buf's remote plugin, pinned to `v30.0` to match the `protobuf` runtime, so `make generate` needs network access.

## Versioning

The Go module and both Python packages share one version number and follow [Semantic Versioning](https://semver.org). Before 1.0, a minor release may include breaking changes. Changes are listed in [CHANGELOG.md](CHANGELOG.md).

- Go versions come from `go/vX.Y.Z` tags, because the module lives in `go/`. Python releases are tagged `vX.Y.Z`.
- `protodecimal` and `protodecimal-native` must be the same version. If they aren't, `fast_encode` ignores the native encoder and warns.
- Both plugins print their version with `--version`.
- The wire format is versioned separately, by its package name. `protodecimal.v1` will not change in ways that break existing data, and `make breaking` checks this. An incompatible format would be a new `protodecimal.v2` package.

To release:

```sh
make set-version V=0.2.0    # updates both Python packages and the lock files
# move the Unreleased notes in CHANGELOG.md under "## 0.2.0", then commit
make tag                    # creates v0.2.0 and go/v0.2.0
git push origin v0.2.0 go/v0.2.0
```

Pushing the tags starts the Release workflow. It checks that the tags, versions and changelog agree, runs CI, then builds and tests the wheels on each platform. After that it publishes to PyPI, makes the Go version available through the Go proxy, creates the GitHub release from the changelog notes, and updates the Python docs.

## Continuous integration

Every push to `main` and every pull request runs:

- version and lock file checks
- `buf lint`, plus `buf breaking` against the base branch on pull requests
- a check that generated code matches `make generate`
- Go tests on the version in `go.mod` and the latest Go, golangci-lint, and 30 seconds of each fuzz test
- Python tests on 3.10 to 3.14, with and without the native encoder, and on the oldest supported `protobuf`
- ruff, rustfmt and clippy

Pull requests that change packaging also build and test the wheels, and changes to the Python package build the docs.

## Comparing with strings

`make compare` measures size and encode and decode time against string fields in both languages. It uses generated values with 2 to 4, 12 to 28 and 36 decimal places. Results depend a lot on the values, so try it with your own data:

```sh
make compare SAMPLE=levels.csv   # price,quantity rows, no header
```
