Metadata-Version: 2.5
Name: datasecops-cli
Version: 0.7.0b7
Summary: DataSecOps Framework CLI for Snowflake Native App
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: click>=8.1
Requires-Dist: gitpython>=3.1
Requires-Dist: jinja2>=3.1
Requires-Dist: pydantic>=2.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: snowflake-connector-python>=3.0
Provides-Extra: mcp
Requires-Dist: mcp>=1.0; extra == 'mcp'
Provides-Extra: test
Requires-Dist: pytest-cov>=4.0; extra == 'test'
Requires-Dist: pytest>=7.0; extra == 'test'
Description-Content-Type: text/markdown

# DataSecOps CLI

A command-line interface for the [Monitorial Orbit Native App](https://app.snowflake.com/marketplace) on Snowflake. Streamlines dbt development, source control workflows, and framework configuration management for teams using the DataSecOps Framework.

## What is this?

The DataSecOps CLI is the local developer companion to the **Monitorial Orbit Native App** — a Snowflake Native App that provides governance, configuration management, and standardised development workflows for data teams.

This CLI connects to the native app and gives developers:

- **dbt development commands** — run, build, test, lint, and manage dbt projects via dbt 2.x or dbt Core 1.x
- **Source control operations** — branching, committing, rebasing, and deploying via GitPython with naming conventions enforced by the framework
- **DCM projects** — validate, render, plan and deploy any number of Snowflake DCM projects from the shared DCM model (`config/dcm_model.json`), in dependency order with each project's deploy role
- **Configuration downloads** — pull SQLFluff rules, CI/CD pipelines, dbt packages, and CoCo skills from the native app to your local project
- **MCP server** — expose framework governance rules to AI coding assistants (VS Code, Cursor, CoCo, Claude Code)

## Installation

```bash
pip install datasecops-cli
```

With MCP server support:

```bash
pip install "datasecops-cli[mcp]"
```

Requires Python 3.10 or later.

## Prerequisites

- A Snowflake connection configured in `~/.snowflake/connections.toml`
- The **Data Engineers DataSecOps Native App** installed in your Snowflake account
- A project profile created in the native app

Optional:

- **dbt** — required for dbt commands (`dbt`). Install via `pip install dbt` (dbt 2.x) or `pip install dbt-core dbt-snowflake` (dbt Core 1.x), matching the engine configured in the native app's Project Settings.
- **CoCo** for skill downloads
- **Node.js 18+** for GitHub/Azure DevOps MCP servers

## Quick Start

### 1. Configure your project

```bash
datasecops setup
```

This prompts for your Snowflake connection name and native app database, then writes `.datasecops.yml`.

If you skip this step, running `datasecops` will offer to run setup automatically.

### 2. Run the CLI

```bash
datasecops
```

## Features

| Menu | Capabilities |
|------|-------------|
| **Development** | dbt run, build, test, lint (SQLFluff), deps, seed, compile, snapshot, freshness, docs |
| **Git** | Branch create/checkout/delete, commit & push, rebase, squash, deploy to environment branches, cherry-pick, reset branch from main |
| **DCM Operations** | Plan, deploy, status, download and list the DCM projects in the model |
| **Downloads** | SQLFluff config, CI/CD pipelines (GitHub Actions / Azure DevOps), dbt packages, CoCo skills |

## Non-Interactive Mode (CI/CD)

The `download` subcommand lets you pull framework config in CI/CD pipelines without interactive prompts:

```bash
# Download specific items
datasecops download sqlfluff
datasecops download sqlfluff packages
datasecops download pipelines macros

# Install framework-pinned package versions
datasecops download install-sqlfluff
datasecops download install-dbt

# Download config and install packages together
datasecops download sqlfluff install-sqlfluff

# Download and install everything
datasecops download all
```

Available items: `sqlfluff`, `pipelines`, `packages`, `macros`, `scripts`, `install-sqlfluff`, `install-dbt`, `all`

The pipeline platform (GitHub / Azure DevOps) is auto-detected from the native app's source control configuration.

### Pipeline Setup

Your pipeline needs two things:

1. **A `.datasecops.yml`** in the repo (already committed — contains no secrets):

   ```yaml
   connection_name: "ci"
   app_database: "MONITORIAL_ORBIT"
   ```

2. **A Snowflake connection** in `~/.snowflake/connections.toml` for the CI service account:

   ```yaml
   # GitHub Actions example
   - name: Configure Snowflake connection
     run: |
       mkdir -p ~/.snowflake
       cat > ~/.snowflake/connections.toml << EOF
       [ci]
       account = "${{ vars.SNOWFLAKE_ACCOUNT }}"
       user = "${{ vars.SNOWFLAKE_USER }}"
       authenticator = "snowflake_jwt"
       private_key_file = "/tmp/rsa_key.p8"
       warehouse = "CI_WH"
       role = "CI_ROLE"
       EOF

   - name: Download SQLFluff config
     run: datasecops download sqlfluff
   ```

The exit code is `0` on success, `1` if any download fails.

### Environment settings for the branch

`datasecops pipeline env` reads the dbt Targets from the native app (**Settings › Data › dbt
Targets**) and sets `TARGET`, `SF_ROLE`, `SF_WAREHOUSE` and `TARGET_SCHEMA` for the target mapped to
the branch, so pipelines don't hard-code them:

```bash
datasecops pipeline env --branch "$GITHUB_REF_NAME"         # GitHub Actions: target whose Branch is this branch
datasecops pipeline env --branch "$BUILD_SOURCEBRANCH"      # Azure DevOps (the full ref, not BUILD_SOURCEBRANCHNAME)
datasecops pipeline env --target unit-test                  # a named target
```

Pass the branch through the runner's environment variable, not a `${{ ... }}` / `$(...)` expression:
those are substituted into the script before the shell runs, so a branch name containing `$(...)`
would execute.

- `--branch` matches a target's **Branch**, then a target named like the branch; `refs/heads/` is
  ignored and a pull request's merge ref resolves to its source branch (`$GITHUB_HEAD_REF` on GitHub,
  `$SYSTEM_PULLREQUEST_SOURCEBRANCH` on Azure DevOps). On Azure DevOps pass `$BUILD_SOURCEBRANCH`:
  `$BUILD_SOURCEBRANCHNAME` is only the last part of the ref (`release/prod` -> `prod`).
  Exact names win; a case-insensitive match is used only when it is unambiguous.
- GitHub Actions: the values are appended to `$GITHUB_ENV`. Azure DevOps: they are set with
  `##vso[task.setvariable]`. Elsewhere (or `--format shell`) they are printed as shell-quoted `KEY=value` lines.
- A target with no role, warehouse or schema leaves that variable as it is.
- The command connects with `.datasecops.yml`'s connection, so the CI user needs the `orbit_developer`
  application role. It exits `1` when no target matches.

## DCM projects

```bash
datasecops model pull                       # native app -> config/dcm_model.json (this repo's projects; --full = all)
datasecops model validate
datasecops model render --check             # generated definitions are current
datasecops dcm plan -p security --target PROD --summary dcm-plan.md
datasecops dcm deploy -p security --target PROD
datasecops dcm pipelines                    # the app-defined pipelines assigned to each project
```

Each DCM project usually lives in its own repository, and its pipelines (*DCM PR validate*, *DCM
deploy*) plan and deploy that single project. See [docs/dcm.md](docs/dcm.md) for the model, every
command, the deployment scripts and the pipelines.

## MCP Server

The package includes an MCP (Model Context Protocol) server that exposes your framework's governance configuration to AI coding assistants. Instead of static skill files, the MCP server gives AI tools live access to your native app's current rules.

### Available Tools

| Category | Tools |
|----------|-------|
| **Configuration** | `get_branching_rules`, `get_linting_rules`, `get_dbt_packages`, `get_pipeline_config`, `get_dbt_versions` |
| **Project** | `get_project_settings`, `get_project_profiles`, `get_deployment_targets`, `get_available_skills` |
| **Linting** | `lint_sql`, `fix_sql`, `lint_project` |
| **Source Control** | `validate_branch_name`, `get_deployment_workflow` |

### Setup for AI Tools

**CoCo** — add to `.snowflake/cortex/mcp.json` in your project root:

```json
{
  "mcpServers": {
    "datasecops-framework": {
      "type": "stdio",
      "command": "datasecops-mcp",
      "args": ["--connection-name", "<CONNECTION>", "--app-database", "<APP_DB>"]
    },
    "github": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-github"],
      "env": {
        "GITHUB_PERSONAL_ACCESS_TOKEN": "<your-github-pat>"
      }
    },
    "dbt": {
      "type": "stdio",
      "command": "uvx",
      "args": ["dbt-mcp"]
    }
  }
}
```

For Azure DevOps instead of GitHub:

```json
{
  "mcpServers": {
    "datasecops-framework": {
      "type": "stdio",
      "command": "datasecops-mcp",
      "args": ["--connection-name", "<CONNECTION>", "--app-database", "<APP_DB>"]
    },
    "azure-devops": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "@tiberriver256/mcp-server-azure-devops"],
      "env": {
        "AZURE_DEVOPS_ORG_URL": "https://dev.azure.com/<your-org>",
        "AZURE_DEVOPS_AUTH_METHOD": "pat",
        "AZURE_DEVOPS_PAT": "<your-azure-pat>"
      }
    },
    "dbt": {
      "type": "stdio",
      "command": "uvx",
      "args": ["dbt-mcp"]
    }
  }
}
```

**VS Code / Cursor** — add to `.vscode/mcp.json` or `.cursor/mcp.json`:

```json
{
  "mcpServers": {
    "datasecops-framework": {
      "command": "datasecops-mcp",
      "args": ["--connection-name", "<CONNECTION>", "--app-database", "<APP_DB>"]
    }
  }
}
```

### Recommended MCP Server Stack

For full integration with Snowflake, dbt, and your source control platform, configure these MCP servers alongside the framework server:

| Server | Purpose | Install |
|--------|---------|---------|
| **datasecops-framework** | Governance rules, linting, branch validation | Included (`datasecops-mcp`) |
| **dbt** | Lineage, model discovery, codegen, semantic layer | `pip install dbt-mcp` |
| **GitHub** | PRs, issues, CI checks, releases | `npx -y @modelcontextprotocol/server-github` |
| **Azure DevOps** | PRs, pipelines, work items, boards | `npx -y @tiberriver256/mcp-server-azure-devops` |

See the [Getting Started Guide](docs/getting-started.md) for full configuration instructions for each editor and platform.

## Configuration

The CLI reads from a `.datasecops.yml` file in your project root (created by the setup script):

```yaml
connection_name: "my_connection"
app_database: "MONITORIAL_ORBIT"
```

Project profiles, linting rules, pipeline templates, and deployment targets are all managed centrally in the native app and pulled down by the CLI.

## License

MIT
