Metadata-Version: 2.4
Name: execution-logger
Version: 1.2.1
Summary: A comprehensive Python logging solution with Microsoft SharePoint and Dataverse integration
Author-email: Shaik Rizwana <rizwana@thefruitpeople.ie>
Maintainer-email: Shaik Rizwana <rizwana@thefruitpeople.ie>
License-Expression: MIT
Project-URL: Homepage, https://github.com/The-Fruit-People/Execution-Logger-pckg
Project-URL: Repository, https://github.com/The-Fruit-People/Execution-Logger-pckg
Project-URL: Issues, https://github.com/The-Fruit-People/Execution-Logger-pckg/issues
Keywords: logging,sharepoint,dataverse,dynamics,microsoft,teams,monitoring
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Classifier: Topic :: System :: Logging
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: msal>=1.20.0
Requires-Dist: requests>=2.28.0
Provides-Extra: sharepoint
Requires-Dist: sharepoint-uploader>=1.0.4; extra == "sharepoint"
Dynamic: license-file

# ExecutionLogger

[![PyPI](https://img.shields.io/pypi/v/execution-logger.svg)](https://pypi.org/project/execution-logger/)
[![Python](https://img.shields.io/pypi/pyversions/execution-logger.svg)](https://pypi.org/project/execution-logger/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

A Python logging solution with 3 storage options: Local files, SharePoint uploads, and optional Dataverse error tracking — plus optional Microsoft Teams alerts for critical failures.

Built for unattended scripts and scheduled jobs, where the log is the only evidence of what happened. The logger degrades gracefully rather than failing: if SharePoint, Dataverse, or Teams is unreachable, it warns and carries on.

## Installation

```bash
pip install execution-logger
```

With SharePoint upload support:

```bash
pip install execution-logger[sharepoint]
```

`requests` and `msal` install automatically. `sharepoint-uploader` is an optional extra — it is imported lazily inside `upload_to_sharepoint()`, so you only need it if you use SharePoint.

## Quick Start

```python
from execution_logger import ExecutionLogger

# Local only
logger = ExecutionLogger(script_name="my_app")
logger.info("Hello World")
logger.finalize()
```

## 3 Storage Scenarios

### 1. Local Only
```python
logger = ExecutionLogger(script_name="my_app")
```
- Saves logs to the calling script's directory
- No additional setup required
- Override the destination with `local_log_directory="D:/logs"`

### 2. SharePoint Upload
```python
logger = ExecutionLogger(
    script_name="my_app",
    client_id="your-azure-client-id",
    client_secret="your-azure-client-secret",
    tenant_id="your-azure-tenant-id",
    sharepoint_url="https://company.sharepoint.com/sites/sitename",
    drive_name="Documents",
    folder_path="Logs/MyApp"
)
```
- Uploads logs to SharePoint using [sharepoint-uploader](https://pypi.org/project/sharepoint-uploader/)
- Requires Azure App Registration with SharePoint permissions
- **All six SharePoint parameters are required together.** Passing some but not all raises `ValueError`.
- **Local and SharePoint are mutually exclusive.** In SharePoint mode no local copy is kept — the temp file is uploaded then deleted.

### 3. SharePoint + Dataverse
```python
logger = ExecutionLogger(
    script_name="my_app",
    # SharePoint params (same as above)
    client_id="your-azure-client-id",
    client_secret="your-azure-client-secret",
    tenant_id="your-azure-tenant-id",
    sharepoint_url="https://company.sharepoint.com/sites/sitename",
    drive_name="Documents",
    folder_path="Logs/MyApp",
    # Dataverse params (often same as SharePoint - see note below)
    dv_client_id="your-dataverse-client-id",  # Usually same as SharePoint
    dv_client_secret="your-dataverse-client-secret",  # Usually same as SharePoint
    dv_tenant_id="your-tenant-id",  # Optional - falls back to tenant_id
    dv_scope="https://yourorg.crm.dynamics.com/.default",
    dv_api_url="https://yourorg.crm.dynamics.com/api/data/v9.2/your_table_name"
)
```
- Uploads logs to SharePoint
- Sends **only errors** to the Dataverse table (`logger.error(...)`, not `info`/`warning`/`debug`)
- Dataverse is switched on by supplying **both** `dv_client_id` and `dv_client_secret`. Supplying only one raises `ValueError`.
- `dv_scope` and `dv_api_url` have **no defaults** — supply both, or the Dataverse post will fail at runtime.
- Dataverse can be combined with local-only storage too; it is independent of the SharePoint choice.

> **💡 Microsoft Tools Integration**: Since SharePoint and Dataverse are both Microsoft tools, you can typically use the **same Azure App Registration** for both services. This means `client_id`, `client_secret`, and `tenant_id` are often identical for SharePoint and Dataverse - just add the appropriate API permissions to one app registration.

## Dataverse Table Setup

### Required Table Structure
Create a Dataverse table with these **exact column names**:

| Column Name | Data Type | Max Length | Description |
|-------------|-----------|------------|-------------|
| `cr672_app` | Text | 1000 | Application name (`script_name`) |
| `cr672_message` | Text | 1000 | Error message (truncated to 1000 chars) |
| `cr672_source` | Text | 100 | Error context (truncated to **98** chars) |
| `cr672_details` | Text | 4000 | Timestamp + full error details (truncated to 4000 chars) |

> **⚠️ `cr672_source` is truncated to 98 characters** to stay inside a 100-char Dataverse column. If your column is wider, the value is still cut at 98 by the client.

### Dataverse Table Creation Steps
1. Go to [Power Apps](https://make.powerapps.com)
2. Select your environment
3. **Data** → **Tables** → **New table**
4. Name: `App Errors` (or your preferred name)
5. Add the 4 columns above with exact names and data types
6. Save and publish

### API URL Format
```
https://yourorg.crm.dynamics.com/api/data/v9.2/cr672_app_errors
```
Replace:
- `yourorg` with your organization name
- `cr672_app_errors` with your table's **plural** name

`v9.0` and `v9.2` both work; use whichever your environment exposes.

## Dataverse Retry Behaviour

Dataverse posts are retried automatically, because throttling and connection timeouts are common on busy environments.

| Outcome | Behaviour |
|---|---|
| `200` / `201` / `204` | Success, logged as info |
| `429` (rate limit) | Retried |
| `500`, `502`, `503`, `504` | Retried |
| Connection / read timeout | Retried |
| `401` (token expired / rejected) | Token re-acquired, then retried **once** |
| Any other status (`400`, `403`, `404`…) | **Not** retried — warned immediately |

Defaults: **3 attempts**, **15 seconds** between attempts, and a `(10s connect, 60s read)` request timeout.

If Dataverse returns a `Retry-After` header on a throttled request, the longer of that value and `retry_wait` is used.

You can override the defaults when calling the internal poster directly:

```python
logger._log_to_dataverse(
    script_name="my_app",
    error_message="Something failed",
    timestamp=datetime.now(),
    context="load_stage",
    details="stack trace here",
    max_attempts=5,     # default 3
    retry_wait=30       # default 15 seconds
)
```

Normal use needs none of this — `logger.error(...)` posts with the defaults. Note that a fully unreachable Dataverse makes each `logger.error()` call block for up to ~30s (two 15s waits) before giving up. Failure never propagates: it is logged as a warning and execution continues.

## Dataverse Token Refresh

The Dataverse token is acquired with the client credentials flow, which issues **no refresh token** — a token simply expires (normally after 1 hour). For long-running scripts the logger re-acquires one automatically, so posts do not start failing partway through a run:

- Before every Dataverse post, the token is re-acquired if it is expired or within **5 minutes** of expiring (`ExecutionLogger.DV_TOKEN_EXPIRY_SKEW`, in seconds).
- If Dataverse still answers `401`, a fresh token is requested and the post is retried immediately (once per `logger.error()` call).
- The expiry is taken from the `expires_in` value returned by Entra ID, falling back to 3600 seconds.

No configuration or code change is needed — existing scripts get this automatically.

## Teams Alerts (Optional)

Critical failures can raise a Microsoft Teams message through a Power Automate flow with an HTTP trigger.

```python
logger.critical(
    "Nightly load failed",
    teams_flow_url="https://prod-00.westeurope.logic.azure.com/workflows/...",
    subject="Nightly Load - FAILED",
    teams_timeout=20          # seconds, default 20
)
```

- The Teams message is sent **only** when `teams_flow_url` is provided — existing `critical(message)` and `critical(message, details)` calls are unaffected.
- The flow receives a JSON body: `{"message": ..., "subject": ...}`
- Failures are swallowed and logged as a warning, so a broken flow never breaks your script.

You can also send a message directly, independent of logging:

```python
sent = logger.send_teams_message(
    flow_url="https://prod-00.westeurope.logic.azure.com/workflows/...",
    message="Batch finished with warnings",
    subject="Batch Status",
    timeout=20
)   # returns True/False
```

### Power Automate Flow Setup
1. Create a flow with the **When an HTTP request is received** trigger
2. Set the request body schema to two string properties: `message` and `subject`
3. Add a **Post message in a chat or channel** action, using the `message` / `subject` values
4. Save, then copy the generated HTTP POST URL into `teams_flow_url`

## Azure App Registration Setup

> **💡 Single App Registration**: Since SharePoint and Dataverse are both Microsoft tools, you can use **one Azure App Registration** for both services. Simply add permissions for both APIs to the same app.

### For SharePoint Access
1. [Azure Portal](https://portal.azure.com) → **App Registrations** → **New registration**
2. Note **Application (client) ID** and **Directory (tenant) ID**
3. **Certificates & secrets** → Create **client secret**
4. **API permissions** → Add **Microsoft Graph** → **Sites.ReadWrite.All**
5. Grant admin consent

### For Dataverse Access (Same App Registration)
1. **Same app registration** → **API permissions** → Add **Dynamics CRM** → **user_impersonation**
2. Grant admin consent
3. In Dataverse, assign appropriate security role to the app

### Credential Reuse
When using the same app registration:
- `client_id` = Same for both SharePoint and Dataverse
- `client_secret` = Same for both SharePoint and Dataverse
- `tenant_id` = Same for both SharePoint and Dataverse
- Only `dv_scope` and `dv_api_url` are Dataverse-specific

## Logging Methods

```python
logger.info("Information message", "Optional details")
logger.warning("Warning message", "Warning context")
logger.error("Error message", "Error details")  # Only errors go to Dataverse
logger.debug("Debug message", "Debug context")  # Only written when debug=True
logger.critical("Critical message", "Critical details")
```

`error()` captures the active traceback by default (`exc_info=True`) and increments the error count, which drives the `SUCCESS`/`FAILED` status in the finalize summary. Pass `exc_info=False` to skip traceback capture.

### Utility Methods

```python
logger.finalize()                    # Write summary, upload/save, then clean up
logger.upload_to_sharepoint()        # Manual SharePoint upload
logger.save_to_local()               # Manual local save
logger.cleanup()                     # Close handlers, delete temp file
logger.get_log_file_path()           # Final log destination
logger.get_configuration_summary()   # dict of current config + error count
```

`finalize()` should be called in a `finally:` block — it flushes handlers, writes the run summary, stores the log in the configured location, and removes the temp file.

### Legacy Aliases

Kept for backward compatibility; prefer the short names above.

| Legacy | Use instead |
|---|---|
| `log_info()` | `info()` |
| `log_warning()` | `warning()` |
| `log_error()` | `error()` |
| `log_debug()` | `debug()` |

## Parameter Reference

### SharePoint Parameters (All required for SharePoint upload)
- `client_id`: Azure app client ID
- `client_secret`: Azure app client secret
- `tenant_id`: Azure tenant ID
- `sharepoint_url`: Site URL (e.g., `https://company.sharepoint.com/sites/sitename`)
- `drive_name`: Document library name (e.g., `Documents`, `Logs`)
- `folder_path`: Target folder (e.g., `Logs/MyApp`)

### Dataverse Parameters (Optional)
- `dv_client_id`: Dataverse app client ID (typically same as SharePoint `client_id`) — **required to enable Dataverse**
- `dv_client_secret`: Dataverse app client secret (typically same as SharePoint `client_secret`) — **required to enable Dataverse**
- `dv_tenant_id`: Dataverse tenant ID (defaults to SharePoint `tenant_id` if omitted)
- `dv_scope`: Dataverse scope (e.g., `https://yourorg.crm.dynamics.com/.default`) — no default
- `dv_api_url`: Dataverse API endpoint (e.g., `https://yourorg.crm.dynamics.com/api/data/v9.2/cr672_app_errors`) — no default

> **Note**: Since both SharePoint and Dataverse are Microsoft services, most organizations use the same Azure App Registration for both, meaning the first three parameters are identical.

### Other Parameters
- `local_log_directory`: Custom local directory (defaults to the calling script's directory)
- `debug`: Enable debug logging (default: `False`)

## Error Handling

- **SharePoint upload fails**: Warning logged, execution continues
- **Dataverse post fails**: Retried (see above), then warning logged; execution continues
- **Dataverse authentication fails**: Warning logged and Dataverse is **disabled for the run** — no exception raised
- **Misconfigured parameters**: `ValueError` raised during initialization (partial SharePoint set, or only one of the two Dataverse credentials)
- **Teams flow fails**: Warning logged, `send_teams_message()` returns `False`
- **Local save fails**: Warning logged

The logger is designed never to be the reason a script dies.

## Troubleshooting

### SharePoint Issues
```
❌ sharepoint-uploader module not installed
```
**Fix**: `pip install sharepoint-uploader`

```
❌ Authentication failed
```
**Fix**: Check Azure app permissions and credentials

### Dataverse Issues
```
❌ Failed to post error to Dataverse: 404
```
**Fix**: Verify table exists and the API URL uses the table's **plural** name

```
❌ Failed to post error to Dataverse: 401
```
**Fix**: Check app registration has Dataverse permissions and a security role assigned

```
❌ Dataverse post attempt 1/3 failed (429 - ...). Retrying in 15s...
```
**Info, not an error**: rate limited, the retry usually succeeds. If you see all 3 attempts fail regularly, raise `max_attempts`/`retry_wait` or reduce how many errors you log per run.

```
❌ Failed to post error to Dataverse after 3 attempts: ... timed out
```
**Fix**: Network or Dataverse availability problem. Check outbound HTTPS access to `*.crm.dynamics.com` from the host.

```
❌ Failed to post error to Dataverse: 400
```
**Fix**: Usually a column name or length mismatch — confirm the four `cr672_*` columns exist with the lengths in the table above.

### Teams Issues
```
❌ PA flow URL is empty - cannot send Teams message
```
**Fix**: Pass a valid `teams_flow_url` / `flow_url`

```
❌ Failed to send Teams message
```
**Fix**: Confirm the Power Automate flow is turned on and its trigger schema has `message` and `subject`

### Common URL Formats
- ✅ **Correct SharePoint URL**: `https://company.sharepoint.com/sites/sitename`
- ❌ **Wrong**: `https://company.sharepoint.com/sites/sitename/Shared Documents`
- ✅ **Correct Dataverse URL**: `https://orgname.crm.dynamics.com/api/data/v9.2/tablename`

## Complete Example

```python
import os
from execution_logger import ExecutionLogger

def main():
    # Using same Azure app registration for both SharePoint and Dataverse
    azure_client_id = os.getenv('AZURE_CLIENT_ID')
    azure_client_secret = os.getenv('AZURE_CLIENT_SECRET')
    azure_tenant_id = os.getenv('AZURE_TENANT_ID')
    teams_flow_url = os.getenv('TEAMS_FLOW_URL')

    logger = ExecutionLogger(
        script_name="daily_processor",
        # SharePoint
        client_id=azure_client_id,
        client_secret=azure_client_secret,
        tenant_id=azure_tenant_id,
        sharepoint_url="https://company.sharepoint.com/sites/logs",
        drive_name="Documents",
        folder_path="ApplicationLogs",
        # Dataverse (reusing same credentials)
        dv_client_id=azure_client_id,  # Same app registration
        dv_client_secret=azure_client_secret,  # Same app registration
        dv_tenant_id=azure_tenant_id,  # Same app registration
        dv_scope="https://company.crm.dynamics.com/.default",
        dv_api_url="https://company.crm.dynamics.com/api/data/v9.2/cr672_app_errors"
    )

    try:
        logger.info("Process started")
        # Your application logic
        process_data()
        logger.info("Process completed successfully")
    except Exception as e:
        # Goes to the log file, SharePoint, and the Dataverse table (with retry)
        logger.error("Process failed", f"Exception: {str(e)}")
        # Also alert the team in Teams
        logger.critical(
            f"daily_processor failed: {e}",
            teams_flow_url=teams_flow_url,
            subject="Daily Processor - FAILED"
        )
    finally:
        logger.finalize()

if __name__ == "__main__":
    main()
```

## Releasing

> Full maintainer guide — local setup, git workflow, and pushing to GitHub — is in [CONTRIBUTING.md](CONTRIBUTING.md).

Packaging metadata lives in `pyproject.toml`. The version has a **single source of truth**: `__version__` in `execution_logger/__init__.py` — `pyproject.toml` reads it dynamically, so there is nothing else to bump.

### Authentication (already configured)

The workflow publishes via **PyPI Trusted Publishing** (OIDC) — no API token is stored in the repo, and there is nothing to rotate.

The PyPI publisher config is:

| Setting | Value |
|---|---|
| PyPI Project Name | `execution-logger` |
| Owner | `The-Fruit-People` |
| Repository name | `Execution-Logger-pckg` |
| Workflow name | `python-publish.yml` |
| Environment name | `pypi` |

If you ever need to recreate it: [pypi.org](https://pypi.org) → account → **Publishing** → **Add a new pending publisher**, then create a matching GitHub environment under **Settings** → **Environments**. For the TestPyPI dry-run path, repeat on [test.pypi.org](https://test.pypi.org) with environment `testpypi`.

> **Token fallback**: `python-publish.yml` keeps the token lines ready to uncomment. Comment out the `permissions: id-token: write` block and uncomment the `password: ${{ secrets.PYPI_FULL_UPLOAD_TOKEN }}` line in the same job.

### Cutting a release

Releases are automatic. **Bump the version and push — that's it.**

```bash
# 1. Bump the version (single source of truth)
#    edit execution_logger/__init__.py -> __version__ = "1.2.0"

# 2. Commit and push
git add -A
git commit -m "Release v1.2.0"
git push origin master
```

On every push to `master` the workflow builds the sdist and wheel, runs `twine check`, then asks PyPI whether that version already exists:

| Situation | What happens |
|---|---|
| Version is **new** | Built, checked, and **published to PyPI** |
| Version **unchanged** | Built and checked, publish **skipped** (not a failure) |
| Can't reach PyPI to check | Build **fails** rather than guessing |

So a normal code push without a version bump is safe — it just runs the checks. To release, bump `__version__`.

> **⚠️ There is no review gate.** Any push to `master` carrying a new version goes straight to PyPI, and PyPI versions can never be reused or overwritten. Use the TestPyPI dry run below if you want to inspect an artifact first.

### Tagging (optional)

Pushing a `v*.*.*` tag also triggers a publish, and additionally asserts the tag matches the built version:

```bash
git tag v1.2.0
git push origin v1.2.0
```

**A tag that doesn't match `__version__` fails the build before anything is uploaded.** Tagging after the push is harmless — the version will already be on PyPI, so the publish step skips.

### Dry run to TestPyPI

TestPyPI is **manual only** — it never fires on a push. GitHub → **Actions** → **Publish to PyPI** → **Run workflow** → target `testpypi`. Then verify:

```bash
pip install -i https://test.pypi.org/simple/ execution-logger
```

### Building locally

```bash
pip install build twine
python -m build
twine check dist/*
```

## Changelog

### 1.2.1
- **Dataverse token refresh**: the access token is now re-acquired automatically when it is expired or within 5 minutes of expiring, and once more if Dataverse answers `401`. Long-running scripts no longer stop logging to Dataverse after the first hour.
- **Token request timeout**: the token call uses `(10s connect, 30s read)` instead of hanging indefinitely.

### 1.2.0
- **Dataverse retry**: posts now retry on `429` rate limits, transient `5xx`, and connection timeouts — 3 attempts, 15s apart, honouring `Retry-After`. Non-retryable statuses (`400`/`401`/`403`/`404`) fail fast instead of burning the retry budget.
- **Bounded request timeouts**: Dataverse posts use `(10s connect, 60s read)`. Previously a stalled connection could hang a script indefinitely.
- **Teams alerts**: `critical()` accepts `teams_flow_url`, `subject`, and `teams_timeout` to raise a Teams message via Power Automate. New `send_teams_message()` for standalone use.
- **Fixed**: `critical()` was not writing to the log file — only printing to console.
- **Packaging**: migrated from `setup.py` to `pyproject.toml`; added MIT `LICENSE`; version now has a single source of truth in `execution_logger/__init__.py`; `sharepoint-uploader` declared as the `[sharepoint]` extra.
- **CI/CD**: auto-publish to PyPI on push via Trusted Publishing, with a version-already-exists guard and a manual TestPyPI dry-run path.

### 1.1.9 and earlier
- `cr672_source` truncation reduced to 98 chars to fit the Dataverse column
- SharePoint upload via `sharepoint-uploader`
- Local / SharePoint / Dataverse storage options

## Key Features

- **3 flexible storage options**: Local, SharePoint, or SharePoint + Dataverse
- **Automatic error tracking**: Only errors sent to Dataverse
- **Resilient Dataverse posting**: Automatic retry on rate limits and timeouts, with bounded request timeouts
- **Optional Teams alerting**: Critical failures pushed to a channel via Power Automate
- **Robust error handling**: Graceful degradation when services unavailable
- **Easy authentication**: Uses proven sharepoint-uploader module
- **Production ready**: Environment variable support and comprehensive logging
