Metadata-Version: 2.4
Name: dlt-filesystem
Version: 0.19.0
Summary: Filesystem and blob storage sources and destinations for dlt, across 13 file formats.
Maintainer-email: Andreas Motl <andreas.motl@panodata.org>
License-Expression: MIT
Project-URL: Issues, https://github.com/panodata/omniload/issues
Project-URL: Repository, https://github.com/panodata/omniload
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: System Administrators
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: OS Independent
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: Unix
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Database
Classifier: Topic :: Internet
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: System :: Archiving
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: adlfs
Requires-Dist: azure-storage-blob
Requires-Dist: dlt<1.31,>=1.22
Requires-Dist: dropboxdrivefs
Requires-Dist: fsspec<2026.10,>=2024.6
Requires-Dist: fsspec-databricks>=0.1.10
Requires-Dist: gcsfs
Requires-Dist: gdrive-fsspec
Requires-Dist: lakefs-spec<0.16
Requires-Dist: msgraphfs
Requires-Dist: ocifs
Requires-Dist: ossfs
Requires-Dist: paramiko
Requires-Dist: polars<3,>=1.34
Requires-Dist: pyarrow<26,>=18
Requires-Dist: s3fs>=2024.6
Requires-Dist: smbprotocol; platform_system == "Linux"
Requires-Dist: smbprotocol[kerberos]; platform_system != "Linux"
Requires-Dist: verlib2
Requires-Dist: webdav4[fsspec]
Provides-Extra: bson
Requires-Dist: pymongo<5; extra == "bson"
Provides-Extra: duckdb
Requires-Dist: duckdb<1.6; extra == "duckdb"
Provides-Extra: iterable
Requires-Dist: cbor2<7; extra == "iterable"
Requires-Dist: iterabledata<2,>=1.0.15; extra == "iterable"
Requires-Dist: lxml<7; extra == "iterable"
Requires-Dist: msgpack<2; extra == "iterable"
Requires-Dist: pyyaml<7; extra == "iterable"
Provides-Extra: spreadsheet
Requires-Dist: polars[calamine,xlsxwriter]<3,>=1.34; extra == "spreadsheet"
Provides-Extra: vortex
Requires-Dist: vortex-data<1,>=0.86; python_version >= "3.11" and extra == "vortex"

# dlt-filesystem

Filesystem and blob storage sources and destinations for
[dlt](https://github.com/dlt-hub/dlt).

`dlt` ships its own filesystem source, and this package is the delta over it:

- **16 readers behind 19 routed format keys**, against dlt's four. CSV (pyarrow,
  DuckDB and a headerless variant), JSON and JSONL, Parquet, ORC, Avro, Feather,
  BSON, spreadsheets (`xlsx` and `ods`), XML, YAML, MessagePack and CBOR.
- **A `filesystem` resource that refuses to load nothing.** A concrete selection
  matching no file raises rather than returning an empty table; globs stay
  empty-safe.
- **A modification-date resolver that covers the schemes dlt's table does not**,
  including `r2`, `oss`, `hdfs`, `smb`, `ftp` and `webdav`, and a pyarrow-backed
  client addressed as `s3://`.
- **An Arrow `readinto` shim**, without which reading a `.gz` fails wherever
  `isal` is importable.

The entry points are a superset of dlt's own, so a pipeline already on
`dlt.sources.filesystem` can move across without changing its call.

## Install

```shell
pip install dlt-filesystem
```

The long-tail formats (XML, YAML, MessagePack, CBOR) carry their decoders in an
extra:

```shell
pip install 'dlt-filesystem[iterable]'
```

Formats whose decoder is a sizeable or single-purpose package carry it in an extra
of their own. Without it, the format still routes and its reader raises an error
naming the extra to install.

| Extra         | Formats                             |
|---------------|-------------------------------------|
| `bson`        | BSON                                |
| `duckdb`      | CSV read with DuckDB (`csv_duckdb`) |
| `spreadsheet` | XLSX (read and write), ODS          |
| `vortex`      | Vortex (Python 3.11+)               |

```shell
pip install 'dlt-filesystem[spreadsheet]'
```

## Usage

```python
import dlt
from dlt_filesystem.source.adapter import readers

pipeline = dlt.pipeline(destination="duckdb", dataset_name="inbox")
pipeline.run(
    readers(bucket_url="s3://bucket/prefix", file_glob="*.parquet").read_parquet()
)
```

## Documentation

<https://omniload.readthedocs.io/supported-sources/filesystem.html>

## License

MIT. See [LICENSE](https://github.com/panodata/omniload/blob/main/LICENSE).
