Metadata-Version: 2.1
Name: leveled-hotbackup-s3-sync
Version: 1.0.1
Summary: Tool to syncrhonise LevelEd hotbackup files to Amazon S3
Home-page: https://github.com/yorkshire-steve/leveled-hotbackup-s3-sync
License: Apache-2.0
Keywords: riak,leveled,backup,s3
Author: yorkshire-steve
Requires-Python: >=3.8,<4.0
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Provides-Extra: retrieve
Requires-Dist: boto3 (==1.34.22)
Requires-Dist: frozendict (>=2.4.0,<3.0.0)
Requires-Dist: lz4 (>=4.3.3,<5.0.0) ; extra == "retrieve"
Requires-Dist: pure-cdb (>=4.0.0,<5.0.0)
Requires-Dist: s3fs (>=2023.12.2,<2024.0.0) ; extra == "retrieve"
Requires-Dist: tomli (>=2.0.1,<3.0.0) ; python_version < "3.11"
Project-URL: Repository, https://github.com/yorkshire-steve/leveled-hotbackup-s3-sync
Description-Content-Type: text/markdown

# leveled-hotbackup-s3-sync
A Python 3.8 tool which can backup and restore leveled hotbackups with Amazon S3, and optionally retrieve individual objects from a backup in Amazon S3.

## Quick Start
For backup and restore with S3.
```
python3.8 -m venv .venv
source .venv/bin/activate
pip install leveled-hotbackup-s3-sync

# Usage
s3sync [backup|restore] [tag] --config config.cfg

[backup|restore] - specify operation to perform
[tag] - alphanumeric string to tag backup, or to select which backup to restore from
--config config.cfg - filename for the config file, see example config.cfg below

# Backup example
s3sync backup 123 --config config.cfg

# Restore example
s3sync restore 123 --config config.cfg
```

To use the object retrieval functionality, the python package must be installed with extras, shown below.
```
python3.8 -m venv .venv
source .venv/bin/activate
pip install 'leveled-hotbackup-s3-sync[retrieve]'

# Usage
s3retrieve [tag] --config config.cfg --bucket bucketName --key keyName [--buckettype type] [--output filename]

[tag] - alphanumeric string to select which backup to retrive object from
--config config.cfg - filename for the config file, see example config.cfg below
--bucket - name of bucket to retrieve object from
--key - name of object key to retrieve
[--buckettype type] - optional, bucket type to retrieve object from
[--output filename] - optional, filename to write out the object. If omitted, object value will be printed to screen.
```

## About
This tool will backup and restore LevelEd (https://github.com/martinsumner/leveled) hotbackups to/from Amazon S3.

Intended to be used with Riak KV.

Backup will use the local Riak ring data at `ring_path` to determine which partitions are owned by the local node, then upload each hotbackup from the `hotbackup_path` to `s3_path`. When backup is being uploaded to S3, the manifest files are updated to reference the new S3 URIs for the journal files. Specifiy a unique `tag` for each backup (this is then used by restore).

`hints_files = true` option in config will also create a hints file for every journal file. The hints file is a CDB of Bucket/Key to sequence number. 
The hints file is required for single object retrieval directly from S3. If a backup has been saved to S3 without this option, then attempting to perform a single object retrieval will result in an error.

To restore to local filesystem, the same `tag` used during backup must be used.
Restore will use the local Riak ring data at `ring_path` to determine which partitions are owned by the local node. Restore will then download the relevant tagged manifest from `s3_path` to the local `leveled_path` and download journal files if needed.
New manifests are then written locally with updated references to the new journal file locations.

## Example config.cfg
```
# hotbackup_path is the local filesystem path to the
# directory containing LevelEd hotbackups.
# e.g. what is passed to riak_client:hotbackup()
hotbackup_path = "/riak/hotbackup"

# ring_path is the local filesystem path to the
# Riak ring data directory. This is used to determine partition ownership
# of the current node.
ring_path = "/riak/data/ring"

# leveled_path is the local filesystem path to the
# Riak leveled data directory. This is used during restore.
leveled_path = "/riak/data/leveled"

# s3_path is the S3 URI to store/retrive the hotbackup
s3_path = "s3://bucket-name/hotbackup/"

# hints_file when set to true will create a hints file
# for each journal file. This is only needed for object level
# retrieval from S3
# valid values: true|false
# Optional
hints_files = true

# s3_endpoint can be used when the target S3 service
# is not the standard AWS S3 (e.g. localstack)
# Optional. Omit to use standard AWS URLs
# s3_endpoint = "http://localhost:4566"
```

## Testing
To aid testing a `docker-compose.yml` is included in the `localstack` directory which will start Localstack to use as a local S3 service.
```
make localstack
```

The `s3_endpoint` config parameter can then be set in config.cfg to use the localstack S3 endpoint URL (e.g. "http://localhost:4566").

## Riak backup example
To perform a Riak LevelEd hotbackup.
```
sudo riak remote_console
{ok, C} = riak:local_client().
riak_client:hotbackup("/riak/hotbackup",3,3,C).
```

