AVENc - Jean-Baptiste Filippi - firecaster.org - CNRS - Università di Corsica

About AVENC

AVENC uses modern video compression as a transport and chunked-access layer for meteorological GRIB data. The objective is to move huge datasets fast, keep decode selective (space/time/stream subsets), and reuse robust video tooling from FFmpeg and streaming ecosystems.

Scientific source format remains GRIB. MP4 carries compressed fields and reconstruction metadata. The browser viewer currently downloads the whole MP4; metadata inspection can use HTTP Range.

Core Philosophy

GRIB stays the reference scientific format.
MP4 carries compressed fields plus enough metadata to reconstruct values and keys.
Decode can be lazy and subset-based instead of full-field/full-time loading.
Existing media infrastructure (encoders, hardware paths, caches, CDNs) becomes usable for geophysical data.

Main Binaries

`bin/grib2avenc`

Packs one or multiple GRIB files into MP4 streams.

Default encoder profile: av1_lossless12_auto_400
Embeds metadata payload in MP4 container comments (AVENCv2|...; legacy v1 remains readable)
Stores grouping and per-frame quantization metadata for reconstruction

`bin/avenc2grib`

Inverse tool: reconstruct to GRIB, extract one stream/group, or output Float32 subsets.

Recreates key GRIB metadata and message count for scientific workflows
Scientific quantization can introduce error even with lossless video
Output message ordering may differ from input ordering

How These MP4s Are Generated

Typical encoding commands:

bin/grib2avenc input.grib output.mp4
bin/grib2avenc --compact source/*.grib compact.mp4
bin/grib2avenc --high source/*.grib high.mp4

Typical decoding/extraction commands:

bin/avenc2grib --list output.mp4
bin/avenc2grib output.mp4 restored.grib
bin/avenc2grib --group stream:0 output.mp4 one_stream.grib

Metadata and Reconstruction Model

MP4 payload includes reconstruction metadata: grouping, geometry, coordinate info, stream options, and frame-level quantization min/max values. This makes the MP4 self-contained for decode/reconstruction workflows.

Per-frame bounds can be used to recover approximate physical values.
Optional shared bitmap handling is supported for stable missing-value masks.
Grouping strategy can follow variable/level/time continuity goals.

Why Video Format Choices Matter

4:2:0 is typically the fastest and most compatible path in hardware/software encoders.
Using chroma channels to carry extra scientific fields is usually harmful in 4:2:0 due to subsampling.
The design favors robust streaming and predictable decode over channel-multiplex complexity.

Lazy Access and AI Workflows

Subset extraction avoids loading full domains when only small windows are needed.
xarray + dask integration enables distributed chunked processing.
This is useful for AI batch builders and large-scale forecast analytics pipelines.

Python / xarray Example

import xarray as xr
from avenc_xarray import register_avenc_engine

register_avenc_engine()
ds = xr.open_dataset("benchmark/era_all.mp4", engine="avenc",
                     decode_times=False, avenc2grib_bin="bin/avenc2grib")

chunk = ds["2d"].isel(time=slice(0, 100), lat=slice(300, 320), lon=slice(1200, 1220)).values
print(chunk.shape)

More Details

For full technical details, options, benchmarks, design notes, and test commands, see README.md at repository root.