Metadata-Version: 2.1
Name: aioftps3
Version: 0.0.3
Summary: FTP in front of AWS S3, powered by asyncio, aioftp, and aiohttp
Home-page: https://github.com/uktrade/aioftp-s3
Author: Department for International Trade - WebOps
Author-email: webops@digital.trade.gov.uk
License: UNKNOWN
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Framework :: AsyncIO
Classifier: Topic :: Internet :: File Transfer Protocol (FTP)
Description-Content-Type: text/markdown
Requires-Dist: aioftp
Requires-Dist: aiohttp

# aioftps3

FTP in front of AWS S3, using [asyncio](https://docs.python.org/3/library/asyncio.html), [aioftp](https://github.com/aio-libs/aioftp) and [aiohttp](https://github.com/aio-libs/aiohttp).

## Running tests

Certificates must be created, and Minio, which emulates S3 locally, must be started

```bash
./certificates-create.sh && ./minio-start.sh
```

and then to run the tests themselves.

```bash
./tests.sh
```

## Features / Design / Limitations

- Can upload files bigger than 2G: uses [multipart upload](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingRESTAPImpUpload.html) under the hood.

- Does not store uploading files in memory before uploading them to S3: i.e. it is effectively a streaming upload. However, it's not completely streaming: each part of multipart upload is stored in memory before it begins to transfer to S3, in order to be able to hash its content and determine its length.

- For uploading files, hashes are computed incrementally as data comes in in order to not block the event loop just before uploads to S3.

- As few dependencies as is reasonable: aioftp, aiohttp, and their dependencies. Boto 3 is _not_ used.

- May not behave well if upload to the server is faster than its upload to S3.

- There is some locking to deal with the same files being operated on concurrently. However...

- .... it does nothing to deal with [eventual consistency of S3](https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel), and so some operations may appear to not have an immediate effect.

## Building and running locally

```bash
docker build -t ftps-s3 . && \
docker run --rm -p 8021-8042:8021-8042 \
  -e AWS_ACCESS_KEY_ID=ommitted \
  -e AWS_SECRET_ACCESS_KEY=ommitted \
  -e AWS_S3_BUCKET_REGION=eu-west-1 \
  -e AWS_S3_BUCKET_HOST=s3-eu-west-1.amazon.aws.com \
  -e AWS_S3_BUCKET_NAME=my-bucket-name
  ftps-s3
```

## Building and pushing to Quay

```bash
docker build -t ftps-s3 . && \
docker tag ftps-s3:latest quay.io/uktrade/ftps-s3:latest && \
docker push quay.io/uktrade/ftps-s3:latest
```


