Metadata-Version: 2.4
Name: AntaresFabricFrameworkV2
Version: 2.0.5
Summary: Microsoft Fabric data ingestion accelerator
Home-page: https://antares.solutions/
Author: AntaresSolutions
Author-email: martonm@antaressolutions.com.au
Project-URL: Bug Tracker, https://antares.solutions/
Project-URL: repository, https://antares.solutions/
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: azure-identity>=1.17
Requires-Dist: azure-monitor-ingestion>=1.0.4
Requires-Dist: azure-core>=1.30
Requires-Dist: requests>=2.28
Requires-Dist: pytz
Dynamic: license-file

# `AntaresFabricFramework`

The `AntaresFabricFramework` is an accelerator for Microsoft Fabric data ingestion, validation and transformation. Every source is described by a record in a control warehouse table, and the framework moves it from the source system into RAW and TRUSTED lakehouse tables with optional validation, pseudonymisation and pre-processing.

## Contents

- [Installation](#installation)
- [Prerequisites](#prerequisites)
- [Usage](#usage)
- [Source types and Key Vault secrets](#source-types-and-key-vault-secrets)
- [Optional constants](#optional-constants)
- [Development](#development)
- [License](#license)
- [Contact Information](#contact-information)

## Installation

1. <b>PyPI (recommended):</b> open Fabric, edit or create an environment, select Public libraries, choose Add from PyPI, enter `AntaresFabricFrameworkBase` and publish.

2. <b>Custom library:</b> build the wheel from this repository and upload it to the environment's Custom libraries:

       py -m build

   The wheel is written to `dist/`. Upload it, then publish the environment.

## Prerequisites

The framework reads everything it needs from the **data warehouse attached to the orchestration notebook**. Both control tables are created and populated through the Antares Fabric Framework UI.

1. <b>`dbo.constants_table`:</b> the global constants for the deployment, such as the medallion stage names and the optional Azure Monitor logging details. Stage names default to CONTROL, RAW, TRUSTED and CURATED when not set.

2. <b>`dbo.ingestion_configuration`:</b> one record per source describing where the data comes from, how it is authenticated, and where and how it is saved in RAW and TRUSTED. At minimum this table must exist. The table name can be overridden with the `TABLE_NAME` constant.

3. <b>Azure Key Vault:</b> holds the connection string for every source except Fabric tables and files. Secrets are read with the identity that runs the notebook: the signed-in user for a manual run, or the pipeline's executing identity (a user, a service principal, or the workspace identity when the Notebook activity is configured with it). Grant that identity the `Key Vault Secrets User` role on the vault and set only `key_vault_name` and `secret_name` on the source record. No tenant id, client id or client secret is stored anywhere.

4. <b>Azure Monitor logging (optional):</b> set `LOGGING_LEVEL` to `INFO` or `DEBUG` together with `LOGS_AZURE_TENANT_ID`, `LOGS_AZURE_CLIENT_ID`, `LOGS_AZURE_CLIENT_SECRET`, `DCE_ENDPOINT`, `DRC_ID` and `LOG_STREAM`. If any of them is missing, logging is switched off for the run and a notice is printed. The log stream receives these columns: LogRunID, UserName, Group, SourceID, Entity, Layer, Source, Target, StartDateTime, EndDateTime, LogDateTime, LogType, Status, SourceRows, TargetRows, Error, ValidationRules, Parameters.

5. <b>Pre-processing notebook (optional):</b> a function run against the source before it is saved to RAW, typically to clean manually maintained files or merge several files into one table. Define it as:

       def preProcess(source):
           ...

   where `source` is the ingestion configuration record for the SourceID being processed.

## Usage

Attach the control warehouse to the orchestration notebook (Explorer > Data items > Add > Warehouse), then:

    from FabricFramework.FabricFramework import *

    ff = FabricFramework(preProcess)   # or FabricFramework() when no pre-processing is needed
    ff.runIngestion()

The only constructor argument is the optional pre-processing function. On start-up the framework locates the attached warehouse, reads the constants and the ingestion configuration, and stops with a `FrameworkSetupError` that says exactly what to configure if no warehouse is attached, the constants table is missing or empty, a required constant is not set, or the ingestion configuration table is missing.

`runIngestion()` with no arguments processes every enabled source from the source system through to TRUSTED. Three optional parameters narrow the run:

    ff.runIngestion(layer=['RAW', 'TRUSTED'], system_code=['Sales'], sourceID=[1, 3, 5])

- <b>layer:</b> which stages to run, using the stage names from the constants table.
- <b>system_code:</b> only sources with these system codes.
- <b>sourceID:</b> only these SourceIDs. A source disabled in the configuration is still skipped.

While a run is in progress a per-source progress dashboard is rendered in the notebook, and a plain-text summary of any failed sources is printed when the run finishes.

## Source types and Key Vault secrets

Underscores and hyphens in `source_type` are interchangeable, so `AZURE_SQL_BASIC` and `AZURE-SQL-BASIC` are the same type. The old catch-all `SQL` type is rejected with a message naming its replacements.

| `source_type` | Reads from | Authentication | Connection value | Notes |
|---|---|---|---|---|
| `AZURE-SQL-BASIC` | Azure SQL Database, SQL Managed Instance, or any SQL Server reachable from Fabric, over JDBC | SQL username and password | Key Vault secret holding a JDBC URL with `user=` and `password=` | `source_name` is `database.schema.table`. The secret is rejected if either credential key is missing. |
| `AZURE-SQL-MFA` | Azure SQL Database or SQL Managed Instance, over JDBC | Microsoft Entra token issued to the identity running the notebook, so MFA users work unattended | Key Vault secret holding a JDBC URL with no credentials | The secret is rejected if it carries `user`, `password` or `authentication`. The identity needs the database permission. |
| `FABRIC-SQLENDPOINT-MFA` | Fabric Warehouse, Lakehouse SQL analytics endpoint, mirrored database, or Fabric SQL database, over JDBC | Microsoft Entra token for the notebook identity. Fabric endpoints accept nothing else | Key Vault secret holding the `*.datawarehouse.fabric.microsoft.com` JDBC URL with no credentials | `source_name` is `item.schema.table`. The identity needs Read on the Fabric item. |
| `BLOB-CSV` | CSV files in an Azure Blob Storage container | Storage account access key | Key Vault secret holding a storage connection string with `AccountName`, `AccountKey` and `ContainerName` | `source_name` is the blob path inside the container. `header` and an optional `schema` control parsing. If pre-processing is enabled the pre-process function supplies the DataFrame instead. |
| `BLOB-PARQUET` | Parquet files in an Azure Blob Storage container | Storage account access key | Same connection string format as `BLOB-CSV` | `source_name` is the blob path. Optional `schema`. |
| `FABRIC-TEXT` | Text files in the Files area of a Fabric lakehouse | Notebook identity through OneLake, no Key Vault entry | `prelanding_workspace` and `prelanding_lakehouse` on the record | Requires `preprocessing_enabled = 1` and a pre-process function that reads the file with the configured header and delimiter options. Without one, nothing is loaded. |
| `FABRIC-TABLE` | A Delta table in a Fabric lakehouse | Notebook identity through OneLake, no Key Vault entry | `prelanding_workspace` and `prelanding_lakehouse` on the record | Loads the latest version of the table named by `source_name`, or hands the record to the pre-process function when enabled. |
| `REST-API` | HTTP APIs, paged responses landed as JSON | Basic, bearer token, API key, or dynamic bearer token obtained from a refresh endpoint | Key Vault secret holding a JSON array of credential items, each with `type`, `key`, `value` and optional `api_area` | Endpoint, verb, headers, parameters, pagination and the response data node come from `rest_api_settings` on the record. `oauth2` is recognised as a type but not implemented. |

All eight go through the same RAW and TRUSTED pipeline once loaded: optional pre-processing, the all-NULL row filter, the configured `raw_savetype`, then validation or a direct copy into TRUSTED with the configured `trusted_savetype`.

### Key Vault secret formats

Each secret is a single string. For the SQL types it is the JDBC URL and nothing else.

| `source_type` | Secret value |
|---|---|
| `AZURE-SQL-BASIC` | `jdbc:sqlserver://<server>.database.windows.net:1433;database=<db>;user=<user>;password=<password>;encrypt=true;trustServerCertificate=false;loginTimeout=30;` |
| `AZURE-SQL-MFA` | `jdbc:sqlserver://<server>.database.windows.net:1433;database=<db>;encrypt=true;trustServerCertificate=false;loginTimeout=30;` |
| `FABRIC-SQLENDPOINT-MFA` | `jdbc:sqlserver://<guid>.datawarehouse.fabric.microsoft.com:1433;database=<item name>;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.datawarehouse.fabric.microsoft.com;loginTimeout=30;` |
| `BLOB-CSV`, `BLOB-PARQUET` | `...;AccountName=<account>;AccountKey=<key>;...;ContainerName=<container>` |
| `REST-API` | `[{"type": "apikey", "api_area": "headers", "key": "x-api-key", "value": "<key>"}]` |
| `FABRIC-TEXT`, `FABRIC-TABLE` | not used |

Notes:
- For the `-MFA` types the URL must not contain `user`, `password` or `authentication`; the framework requests the token itself and rejects a secret that carries credentials. Fabric SQL endpoints do not support SQL authentication at all.
- The notebook identity needs at least Read on the Fabric item, or the equivalent database permission on Azure SQL, in addition to the Key Vault role.
- `source_name` for the SQL types is the full `database.schema.table` reference.
- A password containing `;` must be wrapped in braces in the URL, for example `password={p;wd}`.
- List-valued columns such as `primary_key`, `data_validation` and `preprocessing_options` may be left NULL; NULL is treated as an empty list.

## Optional constants

| Constant | Default | Purpose |
|---|---|---|
| `TABLE_NAME` | `ingestion_configuration` | Name of the ingestion configuration table in the control warehouse |
| `DROP_ALL_NULL_ROWS` | `True` | Drop records whose every column is NULL before saving to RAW |
| `PROGRESS_DISPLAY` | `True` | Render the per-source progress dashboard |
| `PROGRESS_RENDERER` | `auto` | `widget` (in-place ipywidgets), `html` (static, first and final render) or `plain` (prints only) |
| `PROGRESS_PLAIN_LOGS` | `False` | Also print captured source messages with a SourceID prefix |
| `PROGRESS_RENDER_INTERVAL` | `15` | Minimum seconds between dashboard refreshes |
| `PROGRESS_LOG_LINES` | `20` | Lines kept per source in the dashboard log section |
| `SQL_TOKEN_AUDIENCE` | `https://database.windows.net/` | Token audience for the `-MFA` source types |

## Development

The unit tests run locally without a Fabric connection. Create a virtual environment with Python 3.11, install `pytest` and `pyspark==3.5.3`, and run:

    python -m pytest tests/validation -q

On Windows set `PYSPARK_PYTHON` and `PYSPARK_DRIVER_PYTHON` to the environment's interpreter first so Spark workers use the same Python.

## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE.txt) file for details.

## Contact Information
For any questions, please contact [martonm@antaressolutions.com.au](mailto:martonm@antaressolutions.com.au).

MIT License

Copyright (c) [2024] [Antares Solutions]

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
