Metadata-Version: 2.4
Name: opendalfs
Version: 0.1.0
License: Apache-2.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.12
Requires-Python: <4.0,>=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fsspec
Requires-Dist: opendal<0.48,>=0.47.5
Provides-Extra: dev
Requires-Dist: ruff; extra == "dev"
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: pytest-asyncio; extra == "test"
Requires-Dist: pytest-cov; extra == "test"
Requires-Dist: pyarrow; extra == "test"
Requires-Dist: s3fs; extra == "test"
Requires-Dist: boto3; extra == "test"
Provides-Extra: bench
Requires-Dist: pyarrow; extra == "bench"
Requires-Dist: s3fs; extra == "bench"
Requires-Dist: boto3; extra == "bench"
Provides-Extra: all
Requires-Dist: opendalfs[bench,dev,test]; extra == "all"
Dynamic: license-file

# opendalfs

[![Discussions](https://img.shields.io/github/discussions/fsspec/opendalfs)](https://github.com/fsspec/opendalfs/discussions)
[![Tests](https://github.com/fsspec/opendalfs/actions/workflows/tests.yml/badge.svg)](https://github.com/fsspec/opendalfs/actions/workflows/tests.yml)


`opendalfs` is a Python-based interface for file systems that enables interaction with different storage services by [Apache OpenDAL](https://github.com/apache/opendal). Through `opendalfs`, users can utilize fsspec's standard API to operate on all [storage services supported by OpenDAL](https://docs.rs/opendal/latest/opendal/services/index.html).

## URL Protocols

`opendalfs` registers multiple fsspec protocols in the form of `opendal+<service>`, for example:

```python
import fsspec

f = fsspec.open(
    "opendal+s3://my-bucket/path/to/file",
    mode="rb",
    endpoint="http://localhost:9000",
    access_key_id="minioadmin",
    secret_access_key="minioadmin",
)
```

The URL host is mapped to the service container (e.g. `bucket` for `s3`/`gcs`, `container` for `azblob`), and the URL path is used as the object key.

For other OpenDAL services, register protocols at runtime:

```python
import opendalfs

opendalfs.register_opendal_service("oss")
```

## Installation

### Basic Installation

```bash
pip install opendalfs
```

### Development Installation

Install [uv](https://docs.astral.sh/uv/) and
[just](https://just.systems/man/en/packages.html), then run:

```bash
just install
```

## Development Setup

This project uses:

- Python 3.12+ for the Python interface
- ruff for code formatting and linting
- ty for type checking
- pytest for testing

Run the local checks and unit tests with:

```bash
just check
just unit
```

For development setup and guidelines, see our [Contributing Guide](https://github.com/fsspec/opendalfs/blob/main/CONTRIBUTING.md).

## Benchmarks

The benchmark script compares Arrow direct, opendalfs (fsspec), and s3fs (fsspec) on MinIO.

```bash
just install
just bench --sizes 16,32,64 --files 4 --workers 4
```

Configure MinIO access via `OPENDAL_S3_ENDPOINT`, `OPENDAL_S3_BUCKET`,
`OPENDAL_S3_REGION`, `OPENDAL_S3_ACCESS_KEY_ID`, and
`OPENDAL_S3_SECRET_ACCESS_KEY`. Compose, tests, and benchmarks read the same
values.

The benchmark target starts MinIO from the root `docker-compose.yml`. Stop it
when you finish:

```bash
just bench-down
```

For profiling, you can install a tool with `uv` (for example `py-spy`) and run:

```bash
uv tool install py-spy
uv tool run py-spy record -o bench.svg -- python bench/bench_read_write.py --sizes 16,32,64 --files 4 --workers 4
```

High write concurrency can stall on some systems. If runs time out, reduce
`--fsspec-workers`.

## Status

See [Tracking issues of 0.1.0 version for opendalfs](https://github.com/fsspec/opendalfs/issues/6)

## Contributing

opendalfs is an exciting project currently under active development. Whether you're looking to use it in your projects or contribute to its growth, there are several ways you can get involved:

- Follow the [Contributing Guide](https://github.com/fsspec/opendalfs/blob/main/CONTRIBUTING.md) to contribute
- Create new [Issue](https://github.com/fsspec/opendalfs/issues/new) for bug reports or feature requests
- Join discussions in [Discussions](https://github.com/fsspec/opendalfs/discussions)

## Getting Help

- Submit [issues](https://github.com/fsspec/opendalfs/issues/new/choose) for bug reports
- Ask questions in [discussions](https://github.com/fsspec/opendalfs/discussions/new?category=q-a)

## License

Licensed under [Apache License, Version 2.0](https://github.com/fsspec/opendalfs/blob/main/LICENSE).
