Metadata-Version: 2.5
Name: ducklakepy
Version: 0.1.0.dev0
Summary: A Python implementation of the DuckLake specification
Project-URL: Homepage, https://dejii.github.io/ducklake-python/
Project-URL: Documentation, https://dejii.github.io/ducklake-python/
Project-URL: Source, https://github.com/dejii/ducklake-python
Project-URL: Issues, https://github.com/dejii/ducklake-python/issues
Author: ducklakepy contributors
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.11
Requires-Dist: numpy>=1.26
Requires-Dist: pyarrow>=18.0
Requires-Dist: pydantic>=2.5
Requires-Dist: sqlalchemy>=2.0
Provides-Extra: encryption
Requires-Dist: cryptography>=42; extra == 'encryption'
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3.1; extra == 'postgres'
Provides-Extra: spark
Requires-Dist: pyspark>=4.1; extra == 'spark'
Description-Content-Type: text/markdown

# ducklakepy

A Python implementation of the [DuckLake specification](https://ducklake.select/docs/stable/specification/introduction), the open
lakehouse format published at [ducklake.select](https://ducklake.select).

DuckLake keeps a table's metadata in a SQL database and its rows in Parquet files.
ducklakepy speaks to both directly, so a Python program can create tables, append Arrow
data, read at any snapshot, update and delete rows and evolve schemas without a DuckDB
process. A
lake written by ducklakepy is readable by DuckDB's `ducklake` extension, and the other way
round, so the two can share one lake. If you know PyIceberg, ducklakepy plays the same
role for DuckLake.

```bash
pip install ducklakepy[postgres]   # with the Postgres catalog driver
pip install ducklakepy             # SQLite only
```

Python 3.11 or newer.

```python
import pyarrow as pa

from ducklakepy import DuckLakeCatalog
from ducklakepy.schema import Column, Schema
from ducklakepy.types import Int64Type, VarcharType

with DuckLakeCatalog("demo", url="sqlite:///lake.sqlite", data_path="lake_data/") as catalog:
    catalog.initialize()

    events = catalog.create_table(
        "main.events",
        schema=Schema(
            Column("id", Int64Type(), required=True),
            Column("name", VarcharType()),
        ),
    )
    events.append(pa.table({"id": [1, 2, 3], "name": ["signup", "login", "purchase"]}))

    print(catalog.load_table("main.events").to_arrow())
```

The same lake, from DuckDB:

```sql
ATTACH 'ducklake:sqlite:lake.sqlite' AS lake (DATA_PATH 'lake_data/');
SELECT rowid, id, name FROM lake.main.events ORDER BY id;
```

**Status: pre-alpha.** Reads, writes, updates and upserts, deletes, change feeds, schema
evolution, transactions, partitioned and sorted writes, registering existing Parquet files,
encrypted catalogs, maintenance (expiring snapshots, compaction, cleanup) and a Spark data
source all work today, against SQLite and Postgres catalogs and over local files, S3, GCS
and Azure Blob Storage, nested columns included. Every test run cross-checks the lake
against DuckDB on both catalog backends. The
[feature support](https://dejii.github.io/ducklake-python/docs/reference/support/) table says
what each table layout allows.

## Documentation

The documentation lives at **<https://dejii.github.io/ducklake-python/>**. Every page is
also readable as plain Markdown (append `/llms.mdx` in front of a page's path, or take
all of it from [`llms-full.txt`](https://dejii.github.io/ducklake-python/llms-full.txt)),
and every code example on a page is a region of a runnable file under
[`docs/examples/`](docs/examples/) that the test suite executes and checks the page against.
To run the site locally, `make docs` (Node 22.12 or newer; there is a `.nvmrc`) and open
<http://localhost:3000/docs>.

| | |
|---|---|
| [Getting started](https://dejii.github.io/ducklake-python/docs/getting-started/) | Install, then a quickstart: a lake on SQLite, written and read back from Python and from DuckDB |
| [Concepts](https://dejii.github.io/ducklake-python/docs/concepts/) | How DuckLake works: snapshots, tables and schemas, where rows live, the ducklakepy object model, a glossary |
| [Guides](https://dejii.github.io/ducklake-python/docs/guides/catalogs/) | One task per page: connect to a catalog, create tables, write, read, time travel, change feeds, update, delete, evolve a schema, transactions, partition, sort, flush, maintain, register existing files, object storage, encryption, distributed reads, Spark, use with DuckDB |
| [Reference](https://dejii.github.io/ducklake-python/docs/reference/support/) | Feature support by table layout, types, the expression DSL, configuration, write options, errors, catalog backends |
| [API reference](https://dejii.github.io/ducklake-python/docs/api/) | Every public module, generated from the docstrings |
| [Internals](https://dejii.github.io/ducklake-python/docs/internals/) | How it works underneath the API, and why |

## Contributing

[`CONTRIBUTING.md`](CONTRIBUTING.md) has the setup, the checks CI runs and the test
lanes. In short:

```bash
uv venv .venv && uv sync --extra postgres
.venv/bin/ruff format src tests && .venv/bin/ruff check src tests && .venv/bin/mypy
.venv/bin/pytest -m "not integration"          # fast lane, no Docker
```

## License

Apache License 2.0. See [`LICENSE`](LICENSE).

ducklakepy is an independent project. It is not affiliated with, sponsored by or endorsed
by DuckDB Labs or the DuckDB Foundation. DuckDB and DuckLake are their trademarks, used
here to describe what this library implements.
