Metadata-Version: 2.1
Name: castor-extractor
Version: 0.14.12
Summary: Extract your metadata assets.
Home-page: https://www.castordoc.com/
License: EULA
Author: Castor
Author-email: support@castordoc.com
Requires-Python: >=3.8,<3.12
Classifier: License :: Other/Proprietary License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Provides-Extra: all
Provides-Extra: bigquery
Provides-Extra: looker
Provides-Extra: metabase
Provides-Extra: mysql
Provides-Extra: postgres
Provides-Extra: powerbi
Provides-Extra: qlik
Provides-Extra: redshift
Provides-Extra: snowflake
Provides-Extra: sqlserver
Provides-Extra: tableau
Requires-Dist: click (>=8.1,<8.2)
Requires-Dist: cryptography (>=41.0.5) ; extra == "snowflake"
Requires-Dist: google-api-core (>=2.1.1,<3.0.0)
Requires-Dist: google-auth (>=1.6.3,<3.0.0)
Requires-Dist: google-cloud-core (>=2.1.0,<3.0.0)
Requires-Dist: google-cloud-storage (>=2,<3)
Requires-Dist: google-resumable-media (>=2.0.3,<3.0.0)
Requires-Dist: googleapis-common-protos (>=1.53.0,<2.0.0)
Requires-Dist: looker-sdk (>=22.4.0,<=23.0.0) ; extra == "looker" or extra == "all"
Requires-Dist: msal (>=1.20.0,<2.0.0) ; extra == "powerbi" or extra == "all"
Requires-Dist: psycopg2-binary (>=2.0.0,<3.0.0) ; extra == "metabase" or extra == "postgres" or extra == "redshift" or extra == "all"
Requires-Dist: pycryptodome (>=3.0.0,<4.0.0) ; extra == "metabase" or extra == "all"
Requires-Dist: pydantic (>=1.10,<2.0)
Requires-Dist: pymssql (>=2.2.11,<3.0.0) ; extra == "sqlserver" or extra == "all"
Requires-Dist: pymysql[rsa] (>=1.1.0,<2.0.0) ; extra == "mysql" or extra == "all"
Requires-Dist: python-dateutil (>=2.0.0,<=3.0.0)
Requires-Dist: requests (>=2.0.0,<3.0.0)
Requires-Dist: setuptools (>=69,<70)
Requires-Dist: snowflake-connector-python (>=3.4.0,<4.0.0) ; extra == "snowflake" or extra == "all"
Requires-Dist: snowflake-sqlalchemy (!=1.2.5,<2.0.0) ; extra == "snowflake" or extra == "all"
Requires-Dist: sqlalchemy (>=1.4,<1.5)
Requires-Dist: sqlalchemy-bigquery[bqstorage] (>=1.0.0,<=2.0.0) ; extra == "bigquery" or extra == "all"
Requires-Dist: sqlalchemy-redshift (>=0.8.14,<0.9.0) ; extra == "redshift" or extra == "all"
Requires-Dist: tableauserverclient (==0.17.0) ; extra == "tableau" or extra == "all"
Requires-Dist: tqdm (>=4.0.0,<5.0.0)
Requires-Dist: typing-extensions (>=4,<5)
Requires-Dist: websocket-client (>=1,<2) ; extra == "qlik" or extra == "all"
Project-URL: Documentation, https://docs.castordoc.com/castor-package/castor-extractor
Description-Content-Type: text/markdown

# Castor Extractor <img src="https://app.castordoc.com/images/castor_icon_dark.svg" width=30 />

This library contains utilities to extract your metadata assets into `JSON` or `CSV` files, on your local machine.
After extraction, those files can be pushed to Castor for ingestion.

- Visualization assets are typically:
  - `dashboards`
  - `users`
  - `folders`
  - ...

- Warehouse assets are typically:
  - `databases`
  - `schemas`
  - `tables`
  - `columns`
  - `queries`
  - ...

It also embeds utilities to help you push your metadata to Castor:

- `File Checker` to validate your [generic](https://docs.castordoc.com/integrations/data-warehouses/generic-warehouse) CSV files before pushing to Castor
- `Uploader` to push extracted files to our Google-Cloud-Storage (GCS)

## Table of contents

- [Castor Extractor](#castor-extractor-)
  - [Table of contents](#table-of-contents)
  - [Installation](#installation)
    - [Create castor-env](#create-castor-env)
    - [PIP install](#pip-install)
    - [Create the output directory](#create-the-output-directory)
  - [Contact](#contact)

## Installation

Requirements: **python3.8+**
<img src="https://upload.wikimedia.org/wikipedia/commons/c/c3/Python-logo-notext.svg" width=20 />

### Create castor-env

We advise to create a dedicated [Python environment](https://docs.python.org/3/library/venv.html).

Here's an example using `Pyenv` and Python `3.8.12`:

- Install Pyenv

```bash
brew install pyenv
brew install pyenv-virtualenv
```

- [optional] Update your `.bashrc` if you encounter this [issue](https://stackoverflow.com/questions/45577194/failed-to-activate-virtualenv-with-pyenv/45578839)

```bash
eval "$(pyenv init -)"
eval "$(pyenv init --path)"
eval "$(pyenv virtualenv-init -)"
```

- [optional] Install python 3.8+

```bash
pyenv versions # check your local python installations

pyenv install -v 3.8.12 # if none of the installed versions satisfy requirements 8+
```

- Create your virtual env

```bash
pyenv virtualenv 3.8.12 castor-env # create a dedicated env
pyenv shell castor-env # activate the environment

# optional checks
python --version # should be `3.8.12`
pyenv version # should be `castor-env`
```

### PIP install

⚠️ `castor-env` must be created AND activated first.

```bash
pyenv shell castor-env
(castor-env) $ # this means the environment is now active
```

ℹ️ please upgrade `PIP` before installing Castor.

```bash
pip install --upgrade pip
```

Run the following command to install `castor-extractor`:

```bash
pip install castor-extractor
```

Depending on your use case, you can also install one of the following `extras`:

```bash
pip install castor-extractor[bigquery]
pip install castor-extractor[looker]
pip install castor-extractor[metabase]
pip install castor-extractor[mysql]
pip install castor-extractor[powerbi]
pip install castor-extractor[qlik]
pip install castor-extractor[postgres]
pip install castor-extractor[redshift]
pip install castor-extractor[snowflake]
pip install castor-extractor[sqlserver]
pip install castor-extractor[tableau]
```

### Create the output directory

```bash
mkdir /tmp/castor
```

You will provide this path in the `extraction` scripts as follows:

```bash
castor-extract-bigquery --output=/tmp/castor
```

Alternatively, you can also set the following `ENV` in your `bashrc`:

```bash
export CASTOR_OUTPUT_DIRECTORY="/tmp/castor"
````

## Contact

For any questions or bug report, contact us at [support@castordoc.com](mailto:support@castordoc.com)

[Castor](https://castordoc.com) helps you find, understand, use your data assets

