Metadata-Version: 2.4
Name: bucket-helper
Version: 1.3.0
Summary: Bucket Helper — utility functions for AWS S3 and any S3-compatible object storage (MinIO, Backblaze B2, DigitalOcean Spaces, Cloudflare R2, Wasabi, etc.) via boto3, exposed as a library, a click CLI, and a FastAPI HTTP surface. credentials() loader, upload / download / list / delete / exists / make_bucket / remote_tempfile.
Author-email: Warith HARCHAOUI <warithmetics@deraison.ai>
License: BSD-3-Clause
Project-URL: Homepage, https://github.com/warith-harchaoui/bucket-helper
Project-URL: Issues, https://github.com/warith-harchaoui/bucket-helper/issues
Keywords: s3,minio,boto3,object-storage,aws,r2,b2,spaces,bucket,cli,fastapi
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: BSD License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: File Transfer Protocol (FTP)
Classifier: Topic :: System :: Archiving
Requires-Python: <3.14,>=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click<9,>=8.1
Requires-Dist: os-helper<4,>=2.0.0
Requires-Dist: boto3<2,>=1.34
Requires-Dist: botocore<2,>=1.34
Provides-Extra: api
Requires-Dist: fastapi<1,>=0.111; extra == "api"
Requires-Dist: uvicorn<1,>=0.30; extra == "api"
Requires-Dist: python-multipart<1,>=0.0.9; extra == "api"
Provides-Extra: mcp
Requires-Dist: bucket-helper[api]; extra == "mcp"
Requires-Dist: fastapi-mcp<1,>=0.3; extra == "mcp"
Requires-Dist: mcp<2,>=1.12; extra == "mcp"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: pytest-timeout>=2.3; extra == "dev"
Requires-Dist: pyyaml>=6; extra == "dev"
Requires-Dist: moto[s3]>=5; extra == "dev"
Requires-Dist: httpx>=0.27; extra == "dev"
Requires-Dist: fastapi<1,>=0.111; extra == "dev"
Requires-Dist: python-multipart<1,>=0.0.9; extra == "dev"
Requires-Dist: fastapi-mcp<1,>=0.3; extra == "dev"
Requires-Dist: mcp<2,>=1.12; extra == "dev"
Dynamic: license-file

# Bucket Helper

[🇫🇷](https://github.com/warith-harchaoui/bucket-helper/blob/main/LISEZMOI.md) · [🇬🇧](https://github.com/warith-harchaoui/bucket-helper/blob/main/README.md)

[![CI](https://github.com/warith-harchaoui/bucket-helper/actions/workflows/ci.yml/badge.svg)](https://github.com/warith-harchaoui/bucket-helper/actions/workflows/ci.yml) [![License: BSD-3-Clause](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://github.com/warith-harchaoui/bucket-helper/blob/main/LICENSE) [![Python](https://img.shields.io/badge/python-3.10%E2%80%933.13-blue.svg)](#)

`Bucket Helper` belongs to a collection of libraries called `AI Helpers` developed for building Artificial Intelligence, each published on PyPI behind its own green CI gate (pytest plus ruff, both blocking) and semantic-versioned releases.

Utility functions for **AWS S3** and any **S3-compatible object storage**: MinIO, Backblaze B2 S3 API, DigitalOcean Spaces, Cloudflare R2, Wasabi, and friends. Built on [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html). Same shape as [sftp-helper](https://github.com/warith-harchaoui/sftp-helper): a `credentials()` loader, the usual CRUD (`upload` / `download` / `delete` / `exists` / `list_prefix`), and a `remote_tempfile` context manager for stage-and-share flows.

Object storage keeps files as flat, addressable blobs, a bucket plus a key
such as `my-bucket/folder/file.txt`, instead of a nested folder tree on a
hard drive: nothing to create in advance, no limit on how many files pile up
in one place, and every object is reachable straight from a URL. Amazon Web
Services built the first popular version of this, S3 (Simple Storage
Service), and its wire protocol became the de facto standard: MinIO,
Backblaze B2, DigitalOcean Spaces, Cloudflare R2, and Wasabi all speak the
same S3 API, so `bucket-helper` runs unchanged against any of them; only the
endpoint URL changes.

[🌍 AI Helpers](https://deraison.ai/ai-helpers)

[![logo](https://raw.githubusercontent.com/warith-harchaoui/bucket-helper/main/assets/logo.png)](https://deraison.ai/ai-helpers)

## The Promise

**Remote by design.** `bucket-helper` exists to move data to and from *object
storage you choose*: AWS, or any S3-compatible endpoint you point it at
(including a MinIO instance on your own network). It is deliberately **not**
local-first and ships **no GUI**. For a remote reached over SFTP instead of
S3, use `sftp-helper`; for downloading media from a URL, use `youtube-helper`.

That remote reach is also where "battle-tested" has to mean something
checkable, not a slogan. Every push runs a blocking CI gate: the test suite
exercises the S3 client against a moto-mocked backend, then ruff checks
style; nothing merges to `main` on a red run. The package has shipped
through nine semantic-versioned releases on PyPI, from `v0.2.2` to the
current `v1.1.2` (the tag history is visible with `git tag`). It depends
on `os-helper`, the small foundation package the whole AI Helpers suite
shares for logging and file handling; nothing here reinvents that layer.

## Documentation

[💻 Documentation](https://deraison.ai/ai-helpers/docs/bucket-helper-doc/)

[🗺️ Landscape](https://github.com/warith-harchaoui/bucket-helper/blob/main/LANDSCAPE.md)

[📋 Examples](https://github.com/warith-harchaoui/bucket-helper/blob/main/EXAMPLES.md)

[🎯 Triggers](https://github.com/warith-harchaoui/bucket-helper/blob/main/TRIGGERS.md)

## Features

- **CRUD** against AWS S3 or any S3-compatible endpoint: `upload`, `download`,
  `delete`, `exists`, `list_prefix`.
- **Works against any S3-compatible provider**, MinIO, Backblaze B2 S3 API,
  DigitalOcean Spaces, Cloudflare R2, Wasabi, by pointing the `endpoint_url`
  credential at it; no code changes per provider.
- **Credentials loader** (`credentials`) resolving JSON / YAML / environment
  variables / `.env`, in that fallback order.
- **`remote_tempfile`** context manager for stage-and-share flows: upload,
  hand back the object, auto-delete on block exit, no manual cleanup.
- **Three surfaces, one behavior**: Python library, a click CLI, and a
  FastAPI HTTP surface (`[api]` extra). See the
  [multi-surface section](#multi-surface-exposure).
- **Docker image** ships the HTTP server ready to run.

## Installation

**Prerequisites**: **Python 3.10–3.13** and **git**, cross-platform:

- 🍎 **macOS** ([Homebrew](https://brew.sh)): `brew install python git`
- 🐧 **Ubuntu/Debian**: `sudo apt update && sudo apt install -y python3 python3-pip git`
- 🪟 **Windows** (PowerShell): `winget install Python.Python.3.12 Git.Git`

We recommend using Python environments. Check this link if you're unfamiliar with setting one up: [🥸 Tech tips](https://deraison.ai/4ml/#install).

### From PyPI (recommended)

```bash
# Core library (credentials loader + CRUD + remote_tempfile)
pip install bucket-helper

# Optional surfaces
pip install "bucket-helper[api]"       # FastAPI HTTP surface
```

### From source (no PyPI)

```bash
git clone https://github.com/warith-harchaoui/bucket-helper.git
cd bucket-helper
pip install -e .

# Optional surfaces
pip install -e ".[api]"
```

The CLI ships with the core install.

## Configuration

A ready-to-fill template is committed at [`settings.yaml.example`](https://github.com/warith-harchaoui/bucket-helper/blob/main/settings.yaml.example). Copy it to `settings.yaml` and edit in place: `settings.yaml` is gitignored, so you cannot accidentally commit secrets.

```bash
cp settings.yaml.example settings.yaml
# then edit settings.yaml with your AWS / MinIO / R2 / B2 credentials
```

You may also write JSON instead of YAML, use a `.env`, or set environment variables; `bucket-helper` falls back in that order via `os_helper.get_config`. Required keys:

```json
{
  "s3_access_key": "AKIA...",
  "s3_secret_key": "...",
  "s3_bucket":     "my-bucket",
  "s3_https":      "https://my-bucket.s3.eu-west-3.amazonaws.com"
}
```

Optional keys:

| Key | Default | Notes |
|---|---|---|
| `s3_region` | `"us-east-1"` | AWS region; mostly cosmetic for MinIO / R2 |
| `s3_endpoint_url` | empty (= AWS S3) | Set this for S3-compatible backends: see table below |
| `s3_prefix` | empty | Default key prefix added by `upload(...)` when no destination is given |
| `s3_use_path_style` | `"false"` | Force path-style addressing (`endpoint/bucket/key` instead of `bucket.endpoint/key`). Typical for MinIO with custom domains. |
| `s3_verify_ssl` | `"true"` | Disable only for dev MinIO with self-signed certs |

## Endpoint URLs for common S3-compatible storage

Set `s3_endpoint_url` to:

| Provider | Endpoint |
|---|---|
| **AWS S3** | leave empty / unset |
| **MinIO** | `http://minio.example.com:9000` (or `https://...` with TLS) |
| **DigitalOcean Spaces** | `https://nyc3.digitaloceanspaces.com` (region in subdomain) |
| **Cloudflare R2** | `https://<account_id>.r2.cloudflarestorage.com` |
| **Backblaze B2 (S3 API)** | `https://s3.<region>.backblazeb2.com` |
| **Wasabi** | `https://s3.<region>.wasabisys.com` |

## Usage

For the full catalog of recipes (uploads / downloads / listings, S3-compatible endpoints such as MinIO / R2 / B2 / Spaces / Wasabi, temporary remote keys with auto-cleanup, mirroring with sftp-helper), see [📋 EXAMPLES.md](https://github.com/warith-harchaoui/bucket-helper/blob/main/EXAMPLES.md).

```python
import bucket_helper as bh

# Load creds: JSON / YAML / env / .env (auto-fallback in that order)
cred = bh.credentials("path/to/settings.yaml")

# Upload a local file
uri = bh.upload("local.txt", cred, "folder/uploaded.txt")
# uri == "s3://my-bucket/folder/uploaded.txt"

assert bh.exists(uri, cred)

# Download
bh.download(uri, "downloaded.txt", cred)

# List (returns at most max_keys, 1000 by default)
for key in bh.list_prefix("folder/", cred):
    print(key)

# List and find out whether anything was left behind
keys, truncated = bh.list_prefix_page("folder/", cred, max_keys=100)
if truncated:
    print("there are more than 100 objects under folder/")

# Delete
bh.delete(uri, cred)
```

## Listing is capped, and says so

`list_prefix` returns **at most `max_keys` keys** (1000 by default). That is a
deliberate bound, not a bug — S3 pages its responses and an unbounded listing of
a large prefix is rarely what a caller wants.

What it cannot do is tell you whether the cap was reached. A response of 1000
keys looks identical whether the prefix holds exactly 1000 objects or fifty
thousand, which matters as soon as a program acts on the list rather than a
human reading it. Use `list_prefix_page` when that distinction counts:

```python
keys, truncated = bh.list_prefix_page("logs/", cred, max_keys=500)
if truncated:
    ...  # there is more behind these 500
```

It asks S3 for one key beyond the cap and reports whether it arrived, so the
extra cost is a single key. The CLIs print a warning to **stderr** when a
listing is truncated (keys stay on stdout, so pipelines are unaffected), and
`POST /list` returns `truncated` alongside `keys` and `count`.

## MinIO example

```python
cred = {
    "s3_access_key": "minioadmin",
    "s3_secret_key": "minioadmin",
    "s3_bucket": "uploads",
    "s3_https": "http://minio.example.com:9000/uploads",
    "s3_endpoint_url": "http://minio.example.com:9000",
    "s3_use_path_style": "true",
    "s3_region": "us-east-1",  # MinIO accepts any region string
}

bh.make_bucket("uploads", cred)
bh.upload("file.bin", cred, "file.bin")
```

## Stage-and-share with `remote_tempfile`

Drop a generated file at a unique random key, hand the public URL to a
downstream worker / webhook, and the object is deleted on block exit
(even if the body raises):

```python
import bucket_helper as bh
import requests

cred = bh.credentials("path/to/settings.yaml")

with bh.remote_tempfile(cred, ext="json", prefix="runs") as (s3_addr, public_url):
    bh.upload("payload.json", cred, s3_addr, content_type="application/json")
    # Hand the URL to something that fetches it once.
    requests.post(
        "https://hook.example.com/process", json={"input_url": public_url}
    ).raise_for_status()
# Object is gone here, no manual cleanup.
```

## Multi-surface exposure

Every public function in the library is also exposed as:

- **CLI**: `bucket-helper <subcommand>`, built with click (installed by default).
- **FastAPI HTTP**: `uvicorn bucket_helper.api:app --host 0.0.0.0 --port 8000` (install `[api]` extra).
- **MCP**: `bucket-helper-mcp` exposes the same HTTP surface as MCP tools for
  any MCP-aware agent host (install `[mcp]` extra).

Both CLIs share the same subcommand names and flags; pick your favourite.

The exhaustive catalogue of what triggers the toolkit (natural-language
phrasings, commands, functions, address cues, and explicit SKIP rules) lives in
[TRIGGERS.md](https://github.com/warith-harchaoui/bucket-helper/blob/main/TRIGGERS.md).

### Use it as an agent skill

`skills/bucket-helper/` packages bucket-helper as a **Claude Skill** *and* an **OpenCode
skill**, so an agent can reach for it on your behalf. See
[`skills/README.md`](https://github.com/warith-harchaoui/bucket-helper/blob/main/skills/README.md) to install it (symlink into
`~/.claude/skills/` and `~/.opencode/skills/`), and
[`TRIGGERS.md`](https://github.com/warith-harchaoui/bucket-helper/blob/main/TRIGGERS.md) for the exhaustive trigger catalogue —
`scripts/check_triggers.py` keeps the skill's description in sync with it, and
the test suite fails if a capability stops being named.

## CLI examples

```bash
bucket-helper upload      --config settings.yaml --input local.txt --key folder/uploaded.txt
bucket-helper exists      --config settings.yaml --key folder/uploaded.txt
bucket-helper download    --config settings.yaml --key folder/uploaded.txt --output back.txt
bucket-helper list        --config settings.yaml --prefix folder/
bucket-helper delete      --config settings.yaml --key folder/uploaded.txt
bucket-helper make-bucket --config settings.yaml --bucket new-bucket
bucket-helper tempfile    --config settings.yaml --ext json --prefix runs
bucket-helper strip-path  --config settings.yaml --address s3://my-bucket/path/to/obj
```

## HTTP server

```bash
# Serve HTTP (default credentials picked up from BUCKET_HELPER_CONFIG)
BUCKET_HELPER_CONFIG=$PWD/settings.yaml uvicorn bucket_helper.api:app --host 0.0.0.0 --port 8000
# → Swagger UI at http://localhost:8000/docs
```

Per-request credentials can also be sent as multipart form fields
(`s3_access_key` / `s3_secret_key` / `s3_bucket` / `s3_https` / …).

## Docker

```bash
docker build -t bucket-helper .
docker run --rm -p 8000:8000 \
  -e BUCKET_HELPER_CONFIG=/config/settings.yaml \
  -v $PWD/settings.yaml:/config/settings.yaml:ro \
  bucket-helper
```

Or through Compose, which carries the port, the credential mount and a health
probe for you — and, behind `--profile dev`, a local MinIO to talk to:

```bash
docker compose up --build
docker compose --profile dev up --build
```

See also: [TRIGGERS.md](https://github.com/warith-harchaoui/bucket-helper/blob/main/TRIGGERS.md) (what invokes the toolkit) and
[GUI.md](https://github.com/warith-harchaoui/bucket-helper/blob/main/GUI.md) (visual product design plan; no GUI ships, bucket-helper is remote object-storage plumbing).

## Author

 - [Warith HARCHAOUI](https://linkedin.com/in/warith-harchaoui)

## Acknowledgements

Special thanks to [Mohamed Chelali](https://mchelali.github.io) and [Bachir Zerroug](https://www.linkedin.com/in/bachirzerroug) for fruitful discussions.

## License

This project is licensed under the BSD-3-Clause License; see the [LICENSE](https://github.com/warith-harchaoui/bucket-helper/blob/main/LICENSE) file for details.
