Metadata-Version: 2.1
Name: bigflow
Version: 1.10.0
Summary: BigQuery client wrapper with clean API
Home-page: https://github.com/allegro/bigflow
Author: Pogranicze
Author-email: pogranicze-team@allegro.pl
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: grpcio-status <=1.48.2
Requires-Dist: google-cloud-storage <3,>=2
Requires-Dist: MarkupSafe >2.1.0
Requires-Dist: google-auth <3,>=1.20
Requires-Dist: unittest-xml-reporting <4,>=3.0.2
Requires-Dist: jinja2 <4,>=3
Requires-Dist: pip-tools <7,>=5.3
Requires-Dist: deprecated <2,>=1.2.10
Requires-Dist: toml >=0.10
Requires-Dist: tblib <2,>=1.7
Requires-Dist: typing-extensions >=3.7
Requires-Dist: lazy-object-proxy <2,>=1
Requires-Dist: wheel >=0.35
Requires-Dist: setuptools >=45
Requires-Dist: backports.cached-property ; python_version <= "3.8"
Provides-Extra: base_frozen
Requires-Dist: cachetools ==5.0.0 ; extra == 'base_frozen'
Requires-Dist: certifi ==2021.10.8 ; extra == 'base_frozen'
Requires-Dist: charset-normalizer ==2.0.12 ; extra == 'base_frozen'
Requires-Dist: click ==8.0.4 ; extra == 'base_frozen'
Requires-Dist: deprecated ==1.2.13 ; extra == 'base_frozen'
Requires-Dist: google-api-core ==2.7.1 ; extra == 'base_frozen'
Requires-Dist: google-auth ==2.6.2 ; extra == 'base_frozen'
Requires-Dist: google-cloud-core ==2.3.2 ; extra == 'base_frozen'
Requires-Dist: google-cloud-storage ==2.9.0 ; extra == 'base_frozen'
Requires-Dist: google-crc32c ==1.3.0 ; extra == 'base_frozen'
Requires-Dist: google-resumable-media ==2.3.2 ; extra == 'base_frozen'
Requires-Dist: googleapis-common-protos ==1.56.0 ; extra == 'base_frozen'
Requires-Dist: grpcio ==1.50.0 ; extra == 'base_frozen'
Requires-Dist: grpcio-status ==1.48.2 ; extra == 'base_frozen'
Requires-Dist: idna ==3.3 ; extra == 'base_frozen'
Requires-Dist: jinja2 ==3.1.2 ; extra == 'base_frozen'
Requires-Dist: lazy-object-proxy ==1.7.1 ; extra == 'base_frozen'
Requires-Dist: lxml ==4.8.0 ; extra == 'base_frozen'
Requires-Dist: markupsafe ==2.1.1 ; extra == 'base_frozen'
Requires-Dist: pep517 ==0.12.0 ; extra == 'base_frozen'
Requires-Dist: pip-tools ==6.5.1 ; extra == 'base_frozen'
Requires-Dist: protobuf ==3.19.4 ; extra == 'base_frozen'
Requires-Dist: pyasn1 ==0.4.8 ; extra == 'base_frozen'
Requires-Dist: pyasn1-modules ==0.2.8 ; extra == 'base_frozen'
Requires-Dist: requests ==2.27.1 ; extra == 'base_frozen'
Requires-Dist: rsa ==4.8 ; extra == 'base_frozen'
Requires-Dist: six ==1.16.0 ; extra == 'base_frozen'
Requires-Dist: tblib ==1.7.0 ; extra == 'base_frozen'
Requires-Dist: toml ==0.10.2 ; extra == 'base_frozen'
Requires-Dist: tomli ==2.0.1 ; extra == 'base_frozen'
Requires-Dist: typing-extensions ==3.10.0.2 ; extra == 'base_frozen'
Requires-Dist: unittest-xml-reporting ==3.2.0 ; extra == 'base_frozen'
Requires-Dist: urllib3 ==1.26.9 ; extra == 'base_frozen'
Requires-Dist: wheel ==0.37.1 ; extra == 'base_frozen'
Requires-Dist: wrapt ==1.14.0 ; extra == 'base_frozen'
Requires-Dist: backports-cached-property ==1.0.1 ; (python_version <= "3.8") and extra == 'base_frozen'
Provides-Extra: bigquery
Requires-Dist: google-cloud-bigquery <4,>=1.6 ; extra == 'bigquery'
Requires-Dist: db-dtypes >=1.0.5 ; extra == 'bigquery'
Requires-Dist: pandas <2,>=0.25 ; extra == 'bigquery'
Requires-Dist: six <2,>=1.14 ; extra == 'bigquery'
Provides-Extra: dataflow
Requires-Dist: apache-beam[gcp] <=2.48,>=2.24 ; extra == 'dataflow'

# BigFlow

## Documentation

1. [What is BigFlow?](#what-is-bigflow)
1. [Getting started](#getting-started)
1. [Installing Bigflow](#installing-bigflow)
1. [Help me](#help-me)
1. [BigFlow tutorial](docs/tutorial.md)
1. [CLI](docs/cli.md)
1. [Configuration](./docs/configuration.md)
1. [Project structure and build](./docs/project_structure_and_build.md)
1. [Deployment](docs/deployment.md)
1. [Workflow & Job](./docs/workflow-and-job.md)
1. [Starter](./docs/scaffold.md)
1. [Technologies](./docs/technologies.md)
1. [Development](./docs/development.md)

## Cookbook

* [Monitoring](./docs/monitoring.md)
* [Automated end-to-end testing](./docs/e2e_testing.md)
* [Dockerized, GPU based ML prediction process](./docs/ml-prediction.md)


## What is BigFlow?

BigFlow is a Python framework for data processing pipelines on [GCP](https://cloud.google.com/).

The main features are:

* [Dockerized deployment environment](./docs/project_structure_and_build.md#overview)
* [Powerful CLI](./docs/cli.md)
* [Automated build](./docs/project_structure_and_build.md#overview), [deployment](./docs/deployment.md),
[versioning](./docs/project_structure_and_build.md#project-versioning) and [configuration](./docs/configuration.md)
* [Unified project structure](./docs/project_structure_and_build.md#project-structure)
* [Support for GCP data processing technologies](./docs/technologies.md) — [Dataflow](https://beam.apache.org/) (Apache Beam) and [BigQuery](https://cloud.google.com/bigquery)
* [Project starter](./docs/scaffold.md)

## Getting started

Start from installing BigFlow on your local machine.
Next, go through the BigFlow [tutorial](./docs/tutorial.md).

## Installing BigFlow

**Prerequisites**. Before you start, make sure you have the following software installed:

1. [Python](https://www.python.org/downloads/) = 3.8
2. [Google Cloud SDK](https://cloud.google.com/sdk/docs/downloads-interactive)
3. [Docker Engine](https://docs.docker.com/engine/install/)

You can install the `bigflow` package globally, but we recommend
installing it locally with `venv`, in your project's folder:

```bash
python -m venv .bigflow_env
source .bigflow_env/bin/activate
```

Install the `bigflow` PIP package:

```bash
pip install bigflow[bigquery,dataflow]
```

Test it:

```shell
bigflow -h
```

Read more about [BigFlow CLI](docs/cli.md).

To interact with GCP you need to set a default project and log in:

```shell script
gcloud config set project <your-gcp-project-id>
gcloud auth application-default login
```

Finally, check if your Docker is running:

```shell script
docker info
```

## Help me

You can ask questions on our [gitter channel](https://gitter.im/allegro/bigflow) or [stackoverflow](https://stackoverflow.com/questions/tagged/bigflow).
