Metadata-Version: 2.4
Name: execution-logger
Version: 1.2.0
Summary: A comprehensive Python logging solution with Microsoft SharePoint and Dataverse integration
Author-email: Shaik Rizwana <rizwana@thefruitpeople.ie>
Maintainer-email: Shaik Rizwana <rizwana@thefruitpeople.ie>
License-Expression: MIT
Project-URL: Homepage, https://github.com/The-Fruit-People/Execution-Logger-pckg
Project-URL: Repository, https://github.com/The-Fruit-People/Execution-Logger-pckg
Project-URL: Issues, https://github.com/The-Fruit-People/Execution-Logger-pckg/issues
Keywords: logging,sharepoint,dataverse,dynamics,microsoft,teams,monitoring
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Classifier: Topic :: System :: Logging
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: msal>=1.20.0
Requires-Dist: requests>=2.28.0
Provides-Extra: sharepoint
Requires-Dist: sharepoint-uploader>=1.0.4; extra == "sharepoint"
Dynamic: license-file

# ExecutionLogger

A Python logging solution with 3 storage options: Local files, SharePoint uploads, and optional Dataverse error tracking — plus optional Microsoft Teams alerts for critical failures.

## Quick Start

```python
from execution_logger import ExecutionLogger

# Local only
logger = ExecutionLogger(script_name="my_app")
logger.info("Hello World")
logger.finalize()
```

## Dependencies

```bash
pip install requests msal sharepoint-uploader
```

`requests` and `msal` are hard requirements. `sharepoint-uploader` is only needed if you use SharePoint upload — it is imported lazily inside `upload_to_sharepoint()`.

## 3 Storage Scenarios

### 1. Local Only
```python
logger = ExecutionLogger(script_name="my_app")
```
- Saves logs to the calling script's directory
- No additional setup required
- Override the destination with `local_log_directory="D:/logs"`

### 2. SharePoint Upload
```python
logger = ExecutionLogger(
    script_name="my_app",
    client_id="your-azure-client-id",
    client_secret="your-azure-client-secret",
    tenant_id="your-azure-tenant-id",
    sharepoint_url="https://company.sharepoint.com/sites/sitename",
    drive_name="Documents",
    folder_path="Logs/MyApp"
)
```
- Uploads logs to SharePoint using [sharepoint-uploader](https://pypi.org/project/sharepoint-uploader/)
- Requires Azure App Registration with SharePoint permissions
- **All six SharePoint parameters are required together.** Passing some but not all raises `ValueError`.
- **Local and SharePoint are mutually exclusive.** In SharePoint mode no local copy is kept — the temp file is uploaded then deleted.

### 3. SharePoint + Dataverse
```python
logger = ExecutionLogger(
    script_name="my_app",
    # SharePoint params (same as above)
    client_id="your-azure-client-id",
    client_secret="your-azure-client-secret", 
    tenant_id="your-azure-tenant-id",
    sharepoint_url="https://company.sharepoint.com/sites/sitename",
    drive_name="Documents",
    folder_path="Logs/MyApp",
    # Dataverse params (often same as SharePoint - see note below)
    dv_client_id="your-dataverse-client-id",  # Usually same as SharePoint
    dv_client_secret="your-dataverse-client-secret",  # Usually same as SharePoint
    dv_tenant_id="your-tenant-id",  # Optional - falls back to tenant_id
    dv_scope="https://yourorg.crm.dynamics.com/.default",
    dv_api_url="https://yourorg.crm.dynamics.com/api/data/v9.2/your_table_name"
)
```
- Uploads logs to SharePoint
- Sends **only errors** to the Dataverse table (`logger.error(...)`, not `info`/`warning`/`debug`)
- Dataverse is switched on by supplying **both** `dv_client_id` and `dv_client_secret`. Supplying only one raises `ValueError`.
- `dv_scope` and `dv_api_url` have **no defaults** — supply both, or the Dataverse post will fail at runtime.
- Dataverse can be combined with local-only storage too; it is independent of the SharePoint choice.

> **💡 Microsoft Tools Integration**: Since SharePoint and Dataverse are both Microsoft tools, you can typically use the **same Azure App Registration** for both services. This means `client_id`, `client_secret`, and `tenant_id` are often identical for SharePoint and Dataverse - just add the appropriate API permissions to one app registration.

## Dataverse Table Setup

### Required Table Structure
Create a Dataverse table with these **exact column names**:

| Column Name | Data Type | Max Length | Description |
|-------------|-----------|------------|-------------|
| `cr672_app` | Text | 1000 | Application name (`script_name`) |
| `cr672_message` | Text | 1000 | Error message (truncated to 1000 chars) |
| `cr672_source` | Text | 100 | Error context (truncated to **98** chars) |
| `cr672_details` | Text | 4000 | Timestamp + full error details (truncated to 4000 chars) |

> **⚠️ `cr672_source` is truncated to 98 characters** to stay inside a 100-char Dataverse column. If your column is wider, the value is still cut at 98 by the client.

### Dataverse Table Creation Steps
1. Go to [Power Apps](https://make.powerapps.com)
2. Select your environment
3. **Data** → **Tables** → **New table**
4. Name: `App Errors` (or your preferred name)
5. Add the 4 columns above with exact names and data types
6. Save and publish

### API URL Format
```
https://yourorg.crm.dynamics.com/api/data/v9.2/cr672_app_errors
```
Replace:
- `yourorg` with your organization name
- `cr672_app_errors` with your table's **plural** name

`v9.0` and `v9.2` both work; use whichever your environment exposes.

## Dataverse Retry Behaviour

Dataverse posts are retried automatically, because throttling and connection timeouts are common on busy environments.

| Outcome | Behaviour |
|---|---|
| `200` / `201` / `204` | Success, logged as info |
| `429` (rate limit) | Retried |
| `500`, `502`, `503`, `504` | Retried |
| Connection / read timeout | Retried |
| Any other status (`400`, `401`, `403`, `404`…) | **Not** retried — warned immediately |

Defaults: **3 attempts**, **15 seconds** between attempts, and a `(10s connect, 60s read)` request timeout.

If Dataverse returns a `Retry-After` header on a throttled request, the longer of that value and `retry_wait` is used.

You can override the defaults when calling the internal poster directly:

```python
logger._log_to_dataverse(
    script_name="my_app",
    error_message="Something failed",
    timestamp=datetime.now(),
    context="load_stage",
    details="stack trace here",
    max_attempts=5,     # default 3
    retry_wait=30       # default 15 seconds
)
```

Normal use needs none of this — `logger.error(...)` posts with the defaults. Note that a fully unreachable Dataverse makes each `logger.error()` call block for up to ~30s (two 15s waits) before giving up. Failure never propagates: it is logged as a warning and execution continues.

## Teams Alerts (Optional)

Critical failures can raise a Microsoft Teams message through a Power Automate flow with an HTTP trigger.

```python
logger.critical(
    "Nightly load failed",
    teams_flow_url="https://prod-00.westeurope.logic.azure.com/workflows/...",
    subject="Nightly Load - FAILED",
    teams_timeout=20          # seconds, default 20
)
```

- The Teams message is sent **only** when `teams_flow_url` is provided — existing `critical(message)` and `critical(message, details)` calls are unaffected.
- The flow receives a JSON body: `{"message": ..., "subject": ...}`
- Failures are swallowed and logged as a warning, so a broken flow never breaks your script.

You can also send a message directly, independent of logging:

```python
sent = logger.send_teams_message(
    flow_url="https://prod-00.westeurope.logic.azure.com/workflows/...",
    message="Batch finished with warnings",
    subject="Batch Status",
    timeout=20
)   # returns True/False
```

### Power Automate Flow Setup
1. Create a flow with the **When an HTTP request is received** trigger
2. Set the request body schema to two string properties: `message` and `subject`
3. Add a **Post message in a chat or channel** action, using the `message` / `subject` values
4. Save, then copy the generated HTTP POST URL into `teams_flow_url`

## Azure App Registration Setup

> **💡 Single App Registration**: Since SharePoint and Dataverse are both Microsoft tools, you can use **one Azure App Registration** for both services. Simply add permissions for both APIs to the same app.

### For SharePoint Access
1. [Azure Portal](https://portal.azure.com) → **App Registrations** → **New registration**
2. Note **Application (client) ID** and **Directory (tenant) ID**
3. **Certificates & secrets** → Create **client secret**
4. **API permissions** → Add **Microsoft Graph** → **Sites.ReadWrite.All**
5. Grant admin consent

### For Dataverse Access (Same App Registration)
1. **Same app registration** → **API permissions** → Add **Dynamics CRM** → **user_impersonation**
2. Grant admin consent
3. In Dataverse, assign appropriate security role to the app

### Credential Reuse
When using the same app registration:
- `client_id` = Same for both SharePoint and Dataverse
- `client_secret` = Same for both SharePoint and Dataverse  
- `tenant_id` = Same for both SharePoint and Dataverse
- Only `dv_scope` and `dv_api_url` are Dataverse-specific

## Logging Methods

```python
logger.info("Information message", "Optional details")
logger.warning("Warning message", "Warning context")
logger.error("Error message", "Error details")  # Only errors go to Dataverse
logger.debug("Debug message", "Debug context")  # Only written when debug=True
logger.critical("Critical message", "Critical details")
```

`error()` captures the active traceback by default (`exc_info=True`) and increments the error count, which drives the `SUCCESS`/`FAILED` status in the finalize summary. Pass `exc_info=False` to skip traceback capture.

### Utility Methods

```python
logger.finalize()                    # Write summary, upload/save, then clean up
logger.upload_to_sharepoint()        # Manual SharePoint upload
logger.save_to_local()               # Manual local save
logger.cleanup()                     # Close handlers, delete temp file
logger.get_log_file_path()           # Final log destination
logger.get_configuration_summary()   # dict of current config + error count
```

`finalize()` should be called in a `finally:` block — it flushes handlers, writes the run summary, stores the log in the configured location, and removes the temp file.

### Legacy Aliases

Kept for backward compatibility; prefer the short names above.

| Legacy | Use instead |
|---|---|
| `log_info()` | `info()` |
| `log_warning()` | `warning()` |
| `log_error()` | `error()` |
| `log_debug()` | `debug()` |

## Parameter Reference

### SharePoint Parameters (All required for SharePoint upload)
- `client_id`: Azure app client ID
- `client_secret`: Azure app client secret  
- `tenant_id`: Azure tenant ID
- `sharepoint_url`: Site URL (e.g., `https://company.sharepoint.com/sites/sitename`)
- `drive_name`: Document library name (e.g., `Documents`, `Logs`)
- `folder_path`: Target folder (e.g., `Logs/MyApp`)

### Dataverse Parameters (Optional)
- `dv_client_id`: Dataverse app client ID (typically same as SharePoint `client_id`) — **required to enable Dataverse**
- `dv_client_secret`: Dataverse app client secret (typically same as SharePoint `client_secret`) — **required to enable Dataverse**
- `dv_tenant_id`: Dataverse tenant ID (defaults to SharePoint `tenant_id` if omitted)
- `dv_scope`: Dataverse scope (e.g., `https://yourorg.crm.dynamics.com/.default`) — no default
- `dv_api_url`: Dataverse API endpoint (e.g., `https://yourorg.crm.dynamics.com/api/data/v9.2/cr672_app_errors`) — no default

> **Note**: Since both SharePoint and Dataverse are Microsoft services, most organizations use the same Azure App Registration for both, meaning the first three parameters are identical.

### Other Parameters
- `local_log_directory`: Custom local directory (defaults to the calling script's directory)
- `debug`: Enable debug logging (default: `False`)

## Error Handling

- **SharePoint upload fails**: Warning logged, execution continues
- **Dataverse post fails**: Retried (see above), then warning logged; execution continues
- **Dataverse authentication fails**: Warning logged and Dataverse is **disabled for the run** — no exception raised
- **Misconfigured parameters**: `ValueError` raised during initialization (partial SharePoint set, or only one of the two Dataverse credentials)
- **Teams flow fails**: Warning logged, `send_teams_message()` returns `False`
- **Local save fails**: Warning logged

The logger is designed never to be the reason a script dies.

## Troubleshooting

### SharePoint Issues
```
❌ sharepoint-uploader module not installed
```
**Fix**: `pip install sharepoint-uploader`

```
❌ Authentication failed
```
**Fix**: Check Azure app permissions and credentials

### Dataverse Issues
```
❌ Failed to post error to Dataverse: 404
```
**Fix**: Verify table exists and the API URL uses the table's **plural** name

```
❌ Failed to post error to Dataverse: 401
```
**Fix**: Check app registration has Dataverse permissions and a security role assigned

```
❌ Dataverse post attempt 1/3 failed (429 - ...). Retrying in 15s...
```
**Info, not an error**: rate limited, the retry usually succeeds. If you see all 3 attempts fail regularly, raise `max_attempts`/`retry_wait` or reduce how many errors you log per run.

```
❌ Failed to post error to Dataverse after 3 attempts: ... timed out
```
**Fix**: Network or Dataverse availability problem. Check outbound HTTPS access to `*.crm.dynamics.com` from the host.

```
❌ Failed to post error to Dataverse: 400
```
**Fix**: Usually a column name or length mismatch — confirm the four `cr672_*` columns exist with the lengths in the table above.

### Teams Issues
```
❌ PA flow URL is empty - cannot send Teams message
```
**Fix**: Pass a valid `teams_flow_url` / `flow_url`

```
❌ Failed to send Teams message
```
**Fix**: Confirm the Power Automate flow is turned on and its trigger schema has `message` and `subject`

### Common URL Formats
- ✅ **Correct SharePoint URL**: `https://company.sharepoint.com/sites/sitename`
- ❌ **Wrong**: `https://company.sharepoint.com/sites/sitename/Shared Documents`
- ✅ **Correct Dataverse URL**: `https://orgname.crm.dynamics.com/api/data/v9.2/tablename`

## Complete Example

```python
import os
from execution_logger import ExecutionLogger

def main():
    # Using same Azure app registration for both SharePoint and Dataverse
    azure_client_id = os.getenv('AZURE_CLIENT_ID')
    azure_client_secret = os.getenv('AZURE_CLIENT_SECRET')
    azure_tenant_id = os.getenv('AZURE_TENANT_ID')
    teams_flow_url = os.getenv('TEAMS_FLOW_URL')
    
    logger = ExecutionLogger(
        script_name="daily_processor",
        # SharePoint
        client_id=azure_client_id,
        client_secret=azure_client_secret,
        tenant_id=azure_tenant_id,
        sharepoint_url="https://company.sharepoint.com/sites/logs",
        drive_name="Documents",
        folder_path="ApplicationLogs",
        # Dataverse (reusing same credentials)
        dv_client_id=azure_client_id,  # Same app registration
        dv_client_secret=azure_client_secret,  # Same app registration
        dv_tenant_id=azure_tenant_id,  # Same app registration
        dv_scope="https://company.crm.dynamics.com/.default",
        dv_api_url="https://company.crm.dynamics.com/api/data/v9.2/cr672_app_errors"
    )
    
    try:
        logger.info("Process started")
        # Your application logic
        process_data()
        logger.info("Process completed successfully")
    except Exception as e:
        # Goes to the log file, SharePoint, and the Dataverse table (with retry)
        logger.error("Process failed", f"Exception: {str(e)}")
        # Also alert the team in Teams
        logger.critical(
            f"daily_processor failed: {e}",
            teams_flow_url=teams_flow_url,
            subject="Daily Processor - FAILED"
        )
    finally:
        logger.finalize()

if __name__ == "__main__":
    main()
```

## Releasing

Packaging metadata lives in `pyproject.toml`. The version has a **single source of truth**: `__version__` in `execution_logger/__init__.py` — `pyproject.toml` reads it dynamically, so there is nothing else to bump.

### One-time PyPI setup (Trusted Publishing)

The workflow authenticates with OIDC, so no API token is stored in the repo. Configure this once on PyPI:

1. [pypi.org](https://pypi.org) → your account → **Publishing** → **Add a new pending publisher**
2. Fill in:
   - **PyPI Project Name**: `execution-logger`
   - **Owner**: `The-Fruit-People`
   - **Repository name**: `Execution-Logger-pckg`
   - **Workflow name**: `python-publish.yml`
   - **Environment name**: `pypi`
3. In GitHub → **Settings** → **Environments** → create an environment named `pypi` (and `testpypi` if you want the dry-run path)
4. Repeat on [test.pypi.org](https://test.pypi.org) with environment `testpypi` for the dry-run target

> **Token fallback**: if you'd rather use the existing `PYPI_FULL_UPLOAD_TOKEN` secret, `python-publish.yml` has the token lines ready to uncomment — comment out the `permissions: id-token: write` block and uncomment the `password:` lines in the same job.

### Cutting a release

Releases are automatic. **Bump the version and push — that's it.**

```bash
# 1. Bump the version (single source of truth)
#    edit execution_logger/__init__.py -> __version__ = "1.2.0"

# 2. Commit and push
git add -A
git commit -m "Release v1.2.0"
git push origin master
```

On every push to `master` the workflow builds the sdist and wheel, runs `twine check`, then asks PyPI whether that version already exists:

| Situation | What happens |
|---|---|
| Version is **new** | Built, checked, and **published to PyPI** |
| Version **unchanged** | Built and checked, publish **skipped** (not a failure) |
| Can't reach PyPI to check | Build **fails** rather than guessing |

So a normal code push without a version bump is safe — it just runs the checks. To release, bump `__version__`.

> **⚠️ There is no review gate.** Any push to `master` carrying a new version goes straight to PyPI, and PyPI versions can never be reused or overwritten. Use the TestPyPI dry run below if you want to inspect an artifact first.

### Tagging (optional)

Pushing a `v*.*.*` tag also triggers a publish, and additionally asserts the tag matches the built version:

```bash
git tag v1.2.0
git push origin v1.2.0
```

**A tag that doesn't match `__version__` fails the build before anything is uploaded.** Tagging after the push is harmless — the version will already be on PyPI, so the publish step skips.

### Dry run to TestPyPI

TestPyPI is **manual only** — it never fires on a push. GitHub → **Actions** → **Publish to PyPI** → **Run workflow** → target `testpypi`. Then verify:

```bash
pip install -i https://test.pypi.org/simple/ execution-logger
```

### Building locally

```bash
pip install build twine
python -m build
twine check dist/*
```

## Key Features

- **3 flexible storage options**: Local, SharePoint, or SharePoint + Dataverse
- **Automatic error tracking**: Only errors sent to Dataverse
- **Resilient Dataverse posting**: Automatic retry on rate limits and timeouts, with bounded request timeouts
- **Optional Teams alerting**: Critical failures pushed to a channel via Power Automate
- **Robust error handling**: Graceful degradation when services unavailable
- **Easy authentication**: Uses proven sharepoint-uploader module
- **Production ready**: Environment variable support and comprehensive logging
