Metadata-Version: 2.4
Name: bauta
Version: 0.1.2
Summary: Safe, realistic copies of production across databases: masking, subsets, synthetic data, and the ETL to move them
Author: David Ribeiro
License-Expression: MIT
Project-URL: Homepage, https://github.com/ribeiro11075/bauta
Project-URL: Documentation, https://github.com/ribeiro11075/bauta/tree/master/docs
Project-URL: Issues, https://github.com/ribeiro11075/bauta/issues
Keywords: data masking,test data,anonymization,pseudonymization,database subsetting,synthetic data,etl,format-preserving encryption,oracle,sql server,postgresql,mysql
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Database
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyyaml<7,>=6.0.1
Requires-Dist: pydantic<3,>=2.6
Provides-Extra: mysql
Requires-Dist: mysql-connector-python>=8.4; extra == "mysql"
Provides-Extra: postgresql
Requires-Dist: psycopg[binary]>=3.2.10; extra == "postgresql"
Provides-Extra: oracle
Requires-Dist: oracledb>=2.5; extra == "oracle"
Provides-Extra: mssql
Requires-Dist: pymssql>=2.3.5; extra == "mssql"
Provides-Extra: mariadb
Requires-Dist: mysql-connector-python>=8.4; extra == "mariadb"
Provides-Extra: sqlite
Provides-Extra: fpe
Requires-Dist: cryptography>=42; extra == "fpe"
Provides-Extra: all
Requires-Dist: bauta[fpe,mariadb,mssql,mysql,oracle,postgresql,sqlite]; extra == "all"
Provides-Extra: native
Requires-Dist: bauta-rs==0.1.2; extra == "native"
Provides-Extra: dev
Requires-Dist: pytest==9.1.1; extra == "dev"
Requires-Dist: mypy==2.3.1; extra == "dev"
Requires-Dist: types-PyYAML==6.0.12.20260815; extra == "dev"
Requires-Dist: build==1.6.1; extra == "dev"
Requires-Dist: twine==7.0.0; extra == "dev"
Dynamic: license-file

# Bauta

**Put a mask on production:** a safe, realistic stand-in for your production data, in whichever database you need it. Bauta masks what it copies, copies only the slice you need with every relationship intact, generates what may not be copied at all, and moves data between databases on a schedule you already run, with nothing to host.

- **Masking:** consistent across tables and runs, one-to-one for keys (NIST FF1 where policy requires it), applied before anything reaches the target, and every column must be covered.
- **Discovery, subsets and synthetic data:** propose a masking policy from a live schema, copy a referentially complete slice of production, create the copy's tables in whichever database it goes to, and fill tables that can't be copied with generated rows.
- **Audit:** report what every job does with data and what a reviewer should question, and seal each run's masking manifest so it can be verified later.
- **Six databases:** Oracle, SQL Server, PostgreSQL, MySQL, MariaDB and SQLite, as source or target in any combination.
- **Streaming:** memory stays flat however large the table, and PostgreSQL and SQL Server targets load in bulk.
- **Fast:** ten million rows of six masked columns in under two minutes on one core with the optional native masker, and under eight without. Either way the masks are the same.
- **Incremental loads:** extract only what changed since the last successful run.
- **A dependency graph:** jobs run in order, concurrently where they can, each in its own process with an optional timeout.
- **Operable:** webhook alerts; run state, history and manifests each in a file or a table; and passwords from a command for cloud IAM tokens.
- **No infrastructure:** a `pip install`, some YAML, and a command you run from cron.


## Install

Python 3.10 or newer. Choose the drivers you need as extras; each is loaded only when a connection uses it.

```
pip install "bauta[postgresql,oracle]"
```

| Extra | Installs | Needs besides pip |
| --- | --- | --- |
| `mysql`, `mariadb` | mysql-connector-python | nothing |
| `postgresql` | psycopg 3, with its own libpq | nothing |
| `oracle` | oracledb, in thin mode | nothing — no Oracle client |
| `mssql` | pymssql | nothing |
| `sqlite` | Python's own `sqlite3` | nothing |
| `fpe` | cryptography, for the `fpe` masking strategy | nothing; `oracle` already brings it |
| `native` | `bauta-rs`, the native masker (below) | nothing on Linux (x86-64, ARM) or macOS; elsewhere, [Rust](https://rustup.rs) 1.83 or newer |
| `all` | every driver above | nothing |

**The native masker (optional).** `bauta-rs` masks in Rust: four to five times the throughput, identical masks, nothing to configure. `pip install "bauta[postgresql,native]"` installs the version that matches, which is the only one Bauta uses. Without it, everything works, only slower. See [the native masker](docs/masking.md#the-native-masker).


## Quickstart

Start from the configuration in [`example/starter/configuration/`](example/starter/configuration/), from a clone or downloaded from GitHub:

```
mkdir configuration
cp example/starter/configuration/*.yaml configuration/
```

Edit `configuration/database.yaml` and `configuration/jobs.yaml` for your databases, then supply the credentials they reference:

```
export SOURCE_DB_PASSWORD=...  TARGET_DB_PASSWORD=...  MASKING_KEY=...

bauta validate          # check the configuration, offline
bauta run --dry-run     # check connections and tables, moving nothing
bauta run               # run every job once
```

To see it work without any of that, the demos in a clone of this repository use throwaway SQLite databases:

```
git clone https://github.com/ribeiro11075/bauta.git && cd bauta
pip install -e ".[fpe]"

python example/walkthrough/demo.py       # the whole workflow: discover, subset, audit, mask, verify, synthesize
python example/incremental/demo.py       # streaming and incremental loads
python example/masking/demo.py           # masking, discovery and a subset, from Python
python example/native-masking/demo.py    # the same job masked in Python and in Rust, compared
```


## The command

```
bauta run              run data jobs once, masking any with a `masking` section
bauta validate         check configuration without connecting
bauta jobs             show the job graph and what's due
bauta history          show recent job outcomes recorded with --history

bauta discover         propose a masking policy for tables
bauta subset           generate jobs that copy a referentially complete subset
bauta schema           create target tables from source ones, in the target's dialect
bauta synthesize       fill tables with generated rows, for data that can't be copied
bauta clear            empty the target tables of jobs, children first

bauta audit            report what each job does with data, and what to question
bauta verify-manifest  check a manifest is unaltered, and who signed it
```

| Exit code | Meaning |
| --- | --- |
| `0` | every job completed |
| `1` | a job failed, or was skipped because a predecessor failed, or the command failed on a database error |
| `2` | invalid configuration or usage |
| `130` | interrupted by a signal: running jobs finished, the rest were skipped |

`run` makes one pass and exits, so it fits under cron or a Kubernetes CronJob. A second `run` sharing the same run state refuses to start while the first is still going. The useful flags:

| Flag | |
| --- | --- |
| `--config DIR` | where the YAML lives; default `./configuration` |
| `--job NAME` | run only this job — without its predecessors, which it warns about |
| `--force` | ignore `refresh` windows |
| `--forever` | stay running; for freshness under a minute |
| `--log-format json` | structured logs for a collector |
| `--log FILE` | also log to a file, in addition to stderr (`--quiet` silences stderr) |
| `--memory FILE` | where run state (last runs, watermarks) is kept, overriding `jobs.yaml`'s [`memory`](docs/configuration.md#file-level) |
| `--memory-database ALIAS` | keep run state in a database table instead |
| `--history FILE` | append each job's outcome to a JSON-lines history (`--history-database ALIAS` for a table), overriding `jobs.yaml`'s `history` |
| `--notify-url URL` | post to a webhook when a run doesn't succeed; default `$BAUTA_NOTIFY_URL` |
| `--rules FILE` | your own rules for recognising personal data, for `discover`, `audit` and `synthesize`; default `configuration/discovery.yaml` ([more](docs/masking.md#your-own-rules-discoveryyaml)) |
| `--accept-key-change` | run upsert jobs whose masking key changed since their last run |
| `--manifest FILE` | write a sealed JSON record of what was masked, and how (`--manifest-database ALIAS` for a table), overriding `jobs.yaml`'s `manifest`; signed if `$BAUTA_MANIFEST_KEY` is set |


## Documentation

| | |
| --- | --- |
| [Configuration](docs/configuration.md) | every field, how credentials are read from the environment, and connection options such as TLS |
| [Masking](docs/masking.md) | strategies, consistent masks across tables, the key, the manifest, `audit`, `discover`, `subset`, `schema`, `synthesize` and `clear` |
| [How it works](docs/design.md) | streaming, incremental loads, retries, scheduling, and the masking design |
| [Operating it](docs/operations.md) | run state, history and notifications |
| [Security model](docs/security.md) | what masking protects and what it doesn't, the constructions, keys, and a deployment checklist |
| [Library](docs/library.md) | embedding it in Python, results, memory backends |
| [Development](docs/development.md) | running the tests, including against real databases |


## Layout

| | |
| --- | --- |
| `bauta/` | the package; `runner.py` runs jobs, `masking.py` masks, `databaseDialects.py` holds per-database SQL |
| `mask-rs/` | the optional native masker, in Rust — see [its README](mask-rs/README.md) |
| `example/` | runnable demos, each with its `configuration/`, and a starter configuration — see [its README](example/README.md) |
| `docs/` | the documentation above |
| `tests/` | the test suite |


## License

[MIT](LICENSE)
