Metadata-Version: 2.5
Name: ducklakepy
Version: 0.1.0.dev1
Summary: A Python implementation of the DuckLake specification
Project-URL: Homepage, https://dejii.github.io/ducklake-python/
Project-URL: Documentation, https://dejii.github.io/ducklake-python/
Project-URL: Source, https://github.com/dejii/ducklake-python
Project-URL: Issues, https://github.com/dejii/ducklake-python/issues
Author: ducklakepy contributors
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.11
Requires-Dist: numpy>=1.26
Requires-Dist: pyarrow>=18.0
Requires-Dist: pydantic>=2.5
Requires-Dist: sqlalchemy>=2.0
Provides-Extra: encryption
Requires-Dist: cryptography>=42; extra == 'encryption'
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3.1; extra == 'postgres'
Provides-Extra: spark
Requires-Dist: pyspark>=4.1; extra == 'spark'
Description-Content-Type: text/markdown

# ducklakepy

A Python implementation of the [DuckLake specification](https://ducklake.select/docs/stable/specification/introduction), the open
lakehouse format published at [ducklake.select](https://ducklake.select).

DuckLake keeps a table's metadata in a SQL database and its rows in Parquet files.
ducklakepy speaks to both directly, so a Python program can create tables, append Arrow
data, read at any snapshot, update and delete rows and evolve schemas with no query engine
in the process. A lake written by ducklakepy is readable by DuckDB's `ducklake` extension,
and the other way round, so the two can share one lake. If you know PyIceberg, ducklakepy
plays the same role for DuckLake ([the calls side by side](https://dejii.github.io/ducklake-python/docs/getting-started/from-pyiceberg/)).

```bash
pip install ducklakepy[postgres]   # with the Postgres catalog driver
pip install ducklakepy             # SQLite only
```

Python 3.11 or newer.

```python
import pyarrow as pa

from ducklakepy import DuckLakeCatalog
from ducklakepy.schema import Column, Schema
from ducklakepy.types import Int64Type, VarcharType

with DuckLakeCatalog("demo", url="sqlite:///lake.sqlite", data_path="lake_data/") as catalog:
    catalog.initialize()

    events = catalog.create_table(
        "main.events",
        schema=Schema(
            Column("id", Int64Type(), required=True),
            Column("name", VarcharType()),
        ),
    )
    events.append(pa.table({"id": [1, 2, 3], "name": ["signup", "login", "purchase"]}))

    print(catalog.load_table("main.events").to_arrow())
```

The same lake, from DuckDB:

```sql
ATTACH 'ducklake:sqlite:lake.sqlite' AS lake (DATA_PATH 'lake_data/');
SELECT rowid, id, name FROM lake.main.events ORDER BY id;
```

**Status: pre-alpha.** Reads, writes, updates and upserts, deletes, change feeds, schema
evolution, transactions, partitioned and sorted writes, registering existing Parquet files,
encrypted catalogs, maintenance (expiring snapshots, compaction, cleanup) and a Spark data
source all work today, against SQLite and Postgres catalogs and over local files, S3, GCS
and Azure Blob Storage, nested columns included. Every test run cross-checks the lake
against DuckDB on both catalog backends. The
[feature support](https://dejii.github.io/ducklake-python/docs/reference/support/) table says
what each table layout allows.

## Documentation

The documentation lives at **<https://dejii.github.io/ducklake-python/>**. Every page is
also readable as plain Markdown (append `/llms.mdx` in front of a page's path, or take
all of it from [`llms-full.txt`](https://dejii.github.io/ducklake-python/llms-full.txt)),
and every code example on a page is a region of a runnable file under
[`docs/examples/`](docs/examples/) that the test suite executes and checks the page against.
To run the site locally, `make docs` (Node 22.12 or newer; there is a `.nvmrc`) and open
<http://localhost:3000/docs>.

| | |
|---|---|
| [Getting started](https://dejii.github.io/ducklake-python/docs/getting-started/) | Install, a quickstart on SQLite, moving to Postgres and S3, coming from PyIceberg, and troubleshooting |
| [Concepts](https://dejii.github.io/ducklake-python/docs/concepts/) | How DuckLake works: snapshots, tables and schemas, where rows live, the ducklakepy object model, a glossary |
| [Guides](https://dejii.github.io/ducklake-python/docs/guides/) | One task per page: connect to a catalog, create tables, write, read, time travel, change feeds, update, delete, evolve a schema, transactions, partition, sort, flush, maintain, register existing files, object storage, encryption, use with DuckDB, distributed reads, Spark |
| [Reference](https://dejii.github.io/ducklake-python/docs/reference/support/) | Feature support by table layout, types, the expression DSL, configuration, write options, errors, catalog backends |
| [API reference](https://dejii.github.io/ducklake-python/docs/api/) | Every public module, generated from the docstrings |
| [Internals](https://dejii.github.io/ducklake-python/docs/internals/) | How it works underneath the API, and why |

## Contributing

[`CONTRIBUTING.md`](CONTRIBUTING.md) has the setup, the checks CI runs and the test
lanes. In short:

```bash
uv venv .venv && uv sync --extra postgres
.venv/bin/ruff format src tests && .venv/bin/ruff check src tests && .venv/bin/mypy
.venv/bin/pytest -m "not integration"          # fast lane, no Docker
```

## License

Apache License 2.0. See [`LICENSE`](LICENSE).

ducklakepy is an independent project. It is not affiliated with, sponsored by or endorsed
by DuckDB Labs or the DuckDB Foundation. DuckDB and DuckLake are their trademarks, used
here to describe what this library implements.
