Metadata-Version: 2.5
Name: soqlmodel
Version: 0.2.0
Summary: Typed models and a SOQL query builder for Salesforce, generated from your org's schema.
Project-URL: Repository, https://github.com/BetterSaas-engg/soqlmodel
Project-URL: Issues, https://github.com/BetterSaas-engg/soqlmodel/issues
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Code Generators
Classifier: Typing :: Typed
Requires-Python: >=3.11
Provides-Extra: salesforce
Requires-Dist: simple-salesforce>=1.12; extra == 'salesforce'
Description-Content-Type: text/markdown

# soqlmodel

[![CI](https://github.com/BetterSaas-engg/soqlmodel/actions/workflows/ci.yml/badge.svg)](https://github.com/BetterSaas-engg/soqlmodel/actions/workflows/ci.yml)

Typed Python models generated from a Salesforce org's schema, a SOQL builder
that uses them, and a drift check that fails your build when the org moves
underneath you.

## The problem

A pipeline reads `Commission_Rate__c` from Salesforce every night. An admin
retires the field — deactivates the picklist value, drops it from the page
layout, eventually deletes it. Nobody tells the data team, because nobody knows
the data team is reading it.

Here is what does *not* happen: an error. SOQL does not fail loudly the way a
missing SQL column does, and even when it does, it fails in a nightly job whose
output nobody reads until the quarter closes. The realistic outcomes are worse
than a crash:

- A field is deleted. Your `SELECT` fails at 3am. You find out on Thursday.
- A picklist value is renamed. Your `if stage == "Closed Won"` silently matches
  nothing. The report shows zero, and zero looks like a real number.
- A field's type changes from Number to Text. Your sum still runs. It is wrong.
- A query returns 2000 rows because that is Salesforce's batch size, not
  because there are 2000 records. Nothing anywhere says so.

Every one of those is a *plausible answer* where you wanted an error. That is
what this tool exists to prevent — not by being clever, but by writing the org's
schema down, committing it, and diffing it.

## How it works

Four stages, connected by files:

    extract  →  snapshot  →  generate  →  query
    (the org)   schema/*.json  models.py   SOQL

Only extract touches the org. Everything downstream reads the committed
snapshot, never the network — so the snapshot is a seam you can review, diff,
and check in CI.

## Install

```bash
pip install soqlmodel
```

Zero runtime dependencies. Schema extraction shells out to the
[Salesforce CLI](https://developer.salesforce.com/tools/salesforcecli), which
you have already authenticated — or, with `--source credentials`, talks to the
org directly and needs no CLI at all. See
[Extracting without the sf CLI](#extracting-without-the-sf-cli).

To execute queries, or to use `--source credentials`:

```bash
pip install "soqlmodel[salesforce]"
```

## Walkthrough

Everything below is real output from a real org, not illustrative.

### 1. Declare what you depend on

`soqlmodel.toml`, at the project root:

```toml
org = "FULL Sandbox"
api_version = "68.0"

[objects]
Account = ["Name", "AnnualRevenue"]
```

This is the point of the tool. You are not mirroring the org — a snapshot of
*everything* would report drift on hundreds of fields nobody reads, and a check
that cries wolf gets muted. You are declaring a dependency. `["*"]` takes every
field when you want it.

### 2. Snapshot the schema

```console
$ soqlmodel snapshot
wrote schema\Account.json
1 snapshot(s) written.
```

```json
{
  "fields": [
    {
      "calculated": false,
      "custom": false,
      "deprecatedAndHidden": false,
      "filterable": true,
      "label": "Annual Revenue",
      "length": 0,
      "name": "AnnualRevenue",
      "nillable": true,
      "precision": 18,
      "referenceTo": [],
      "restrictedPicklist": false,
      "scale": 0,
      "sortable": true,
      "type": "currency"
    },
    {
      "calculated": false,
      "custom": false,
      "deprecatedAndHidden": false,
      "filterable": true,
      "label": "Account Name",
      "length": 255,
      "name": "Name",
      "nillable": false,
      "precision": 0,
      "referenceTo": [],
      "restrictedPicklist": false,
      "scale": 0,
      "sortable": true,
      "type": "string"
    }
  ],
  "format_version": 1,
  "org": "FULL Sandbox",
  "requested_fields": ["AnnualRevenue", "Name"],
  "sobject": "Account"
}
```

**Commit this file.** It is the artifact the whole tool turns on. Output is
deterministic — sorted keys, no timestamps, LF endings — so re-running against
an unchanged org produces byte-identical bytes and a re-snapshot is an empty
diff. Picklist values are stored too, because a renamed value is the drift most
likely to give you a wrong number instead of an error.

### 3. Generate models

```console
$ soqlmodel generate
wrote models.py
```

```python
# Generated by soqlmodel. Do not edit.
# Org: FULL Sandbox
# Snapshot format: 1

from soqlmodel.fields import Field


class Account:
    AnnualRevenue: Field[float] = Field("AnnualRevenue", "currency")
    Name: Field[str] = Field("Name", "string")
```

Plain classes with explicit annotated attributes. No metaclasses, no
`__getattr__`, nothing dynamic — if mypy and Pylance cannot see every field
statically, the product does not work. Commit this too, or regenerate it in CI.

The output is **stable under `ruff format` and Black**: long field names wrap
with a trailing comma, which formatters leave alone, so regenerating never
fights your formatter. The default assumes a line length of 88 — ruff's and
Black's default. If yours differs, say so and the output matches it:

```toml
line_length = 120
```

### 4. Build queries

```python
from models import Account
from soqlmodel.query import select

q = (
    select(Account, Account.Name, Account.AnnualRevenue)
    .where(Account.AnnualRevenue > 1_000_000)
    .order_by(Account.AnnualRevenue, desc=True)
    .limit(50)
)

print(q.render())
```

```
SELECT Name, AnnualRevenue FROM Account WHERE AnnualRevenue > 1000000 ORDER BY AnnualRevenue DESC LIMIT 50
```

Literals are escaped following Salesforce's documented sequence list:

```python
select(Account, Account.Name).where(Account.Name == "O'Brien & Co").render()
# SELECT Name FROM Account WHERE Name = 'O\'Brien & Co'
```

A field outside your declared scope is not a runtime surprise — it does not
exist:

```python
select(Account, Account.Name).where(Account.Industry == "Retail")
# AttributeError: type object 'Account' has no attribute 'Industry'
```

Your type checker catches that before you run anything. Conditions combine with
`&` and `|`, and **each side must be parenthesized** — Python binds `&` tighter
than `>`, so `.where((Account.Name == "Acme") & (Account.AnnualRevenue > 1000))`
is required. Using `and` / `or` raises rather than silently doing the wrong
thing.

### 5. Check for drift

Run this in CI. Here is a snapshot that still claims a field the org no longer
has, and a `Name` whose length moved:

```console
$ soqlmodel check
CRITICAL  Account.Rating__c: field no longer exists in the org
WARNING   Account.Name: length changed from 120 to 255
$ echo $?
1
```

Against an unchanged org:

```console
$ soqlmodel check
No drift.
$ echo $?
0
```

### 6. Execute (optional)

```python
from simple_salesforce import Salesforce
from soqlmodel.execute import execute, execute_iter

sf = Salesforce(...)  # you construct and own this
rows = execute(q, sf)  # list[dict], every row
```

`execute` **drains the cursor**. A Salesforce query returns the first batch —
2000 rows by default — plus a `nextRecordsUrl`. Code that reads
`response["records"]` and stops has 2000 rows and no indication that is not the
answer. That is the single most common way to get a plausible wrong number out
of this API, and it is the reason this function exists.

For result sets too large to hold in memory, `execute_iter(q, sf)` yields rows
as batches arrive. There is no row cap: `.limit(n)` already caps server-side,
which beats a client-side cap that would either fail a legitimate export or
truncate one silently.

Rows come back as plain `list[dict]`, exactly as the API returned them. Mapping
them onto the generated models is not in v1.

## Severity: what fails your build

| | Meaning | Exit code |
|---|---|---|
| **CRITICAL** | A pipeline is broken, or about to return wrong numbers | 1 |
| **WARNING** | Real information, breaks nothing that already runs | 0 |

CRITICAL: a declared field no longer exists; a field's type changed; a picklist
value was **removed** (a mapping keyed on it is now dead); `filterable` or
`sortable` became false, breaking existing `WHERE` and `ORDER BY` clauses;
`deprecatedAndHidden` became true.

WARNING: a picklist value was **added** (silently unmapped, which is precisely
why values are stored); `nillable` changed; `length`, `precision` or `scale`
changed; a new field appeared.

**Warnings alone do not fail the build.** A value added to a picklist is worth
reading on a Monday morning; it is not a reason to block a deploy at 5pm, and a
check that blocks on things nobody can act on immediately is a check people
learn to skip — at which point the CRITICALs stop being seen either.

Exit codes overall:

| Code | Meaning |
|---|---|
| 0 | Success, or `check` found no CRITICAL drift |
| 1 | `check` found CRITICAL drift |
| 2 | Usage error, bad config, or the org could not be reached |

### In CI

The `sf` CLI is usually not installed on a runner, so use the credential
source. See [Extracting without the sf CLI](#extracting-without-the-sf-cli).

```yaml
- run: pip install "soqlmodel[salesforce]"
- run: soqlmodel check --source credentials
  env:
    SOQLMODEL_SF_USERNAME: ${{ secrets.SOQLMODEL_SF_USERNAME }}
    SOQLMODEL_SF_CONSUMER_KEY: ${{ secrets.SOQLMODEL_SF_CONSUMER_KEY }}
    SOQLMODEL_SF_PRIVATEKEY_FILE: ${{ secrets.SOQLMODEL_SF_PRIVATEKEY_FILE }}
    SOQLMODEL_SF_DOMAIN: ${{ secrets.SOQLMODEL_SF_DOMAIN }}
```

`check` exits 1 on CRITICAL drift, so this fails the build.

## What this is not

- **Not an ORM.** No sessions, no identity map, no lazy loading, no
  relationship traversal. Models are schema descriptors.
- **No writes. Ever.** There is no DML in this package and no code path that
  reaches it. It reads schema and it reads rows.
- **No writes, and no credential storage.** On the default `sf` source this
  package performs no auth and no HTTP at all: extraction shells out to a CLI
  you have already authenticated, and execution is handed to a
  `simple_salesforce` client *you* construct. With `--source credentials` it
  does authenticate, reading four environment variables to do so. Those
  values are read at the moment they are used, never written to disk, never
  logged, and never read from `soqlmodel.toml`. No message soqlmodel raises
  contains a credential; note that an underlying `OSError` can name the key
  file's *path* or the org host, so treat build logs accordingly.
- **Not in-org dependency analysis.** Salesforce's own "Where is this used?"
  and tools like Elements.cloud answer what *in the org* references a field.
  This answers the opposite question: what does **your repository** depend on,
  expressed as a file you can diff in review.
- **Not a migration tool.** It reports drift. Deciding what to do is yours.

## Prior art

**[simple-salesforce](https://pypi.org/project/simple-salesforce/)** — the REST
client everyone uses, and a dependency of this package rather than a
competitor. It does auth, HTTP, DML, bulk, and it has `query_all`. It has no
opinion about your org's schema.

**[soql](https://pypi.org/project/soql/)** (plangrid, v1.2.0, last released
February 2023) — the closest thing. Declarative models and a SOQL generator,
designed to pair with simple-salesforce. It is genuinely good, and it does two
things this package does not: relationship traversal / joins, and loading
response rows back into typed model instances.

The difference is where the models come from. In `soql` you hand-write them:

```python
class Account(Model):
    id = attributes.String("Id")
    name = attributes.String("Name")
    custom_field = attributes.String("CustomField__c", nullable=True)
```

That declaration is your belief about the org. Nothing checks it. If
`CustomField__c` is retired, this code keeps compiling and keeps being wrong —
the exact failure at the top of this README.

In soqlmodel the models are **generated from the org's own describe payload**,
and the snapshot they came from is committed so `check` can diff it against the
org later. That is the whole difference: generation from real schema, plus
drift detection. Worth noting too that `soql`'s documented usage is
`sf.query(str(query))`, which returns the first batch only.

If you want joins and typed row objects today, use `soql`. If you want your
build to fail when the org changes, use this.

## Extracting without the `sf` CLI

`snapshot` and `check` extract through the `sf` CLI by default. Pass
`--source credentials` and they talk to the org over REST instead, which is
what makes them runnable in a container, in CI, or from a scheduler.

```bash
soqlmodel check --source credentials
```

Credentials come from the environment, never from `soqlmodel.toml` — that file
is committed, and a credential in it would be too.

| variable | what it is |
|---|---|
| `SOQLMODEL_SF_USERNAME` | the username the JWT is issued for |
| `SOQLMODEL_SF_CONSUMER_KEY` | the connected app's consumer key |
| `SOQLMODEL_SF_PRIVATEKEY_FILE` | path to the private key, outside the repo |
| `SOQLMODEL_SF_DOMAIN` | e.g. `example--sandbox.my` |

Needs the extra: `pip install "soqlmodel[salesforce]"`. Any missing variable,
or a missing extra, is an error naming exactly what is absent, before any
network call.

**The source is never inferred.** Credentials sitting in your environment do
not change how `soqlmodel` behaves; only the flag does. Presence of a secret is
not consent to use it (D21).

### `api_version` is required

```toml
api_version = "68.0"
```

Both sources are pinned to it, and there is no default. Left to themselves the
`sf` CLI and simple-salesforce negotiate *different* versions, and two describes
at different versions return different field lists — which `check` reports as
`field no longer exists in the org` at CRITICAL. A red build, blamed on the org,
caused by nothing but which client asked.

A default would only pick a winner between the skews while hiding the pin. This
is a config change for existing projects, and it is deliberate: one line now
beats a false CRITICAL you cannot diagnose later.

`generate` never touches an org and does not need it.

## On the word "snapshot"

Salesforce already uses "snapshot" for
[Reporting Snapshots](https://help.salesforce.com/s/articleView?id=sf.reports_definereportingsnapshot.htm),
which periodically write *report results* into a custom object. That is a
different thing at every level: report data rather than schema, inside the org
rather than in your repo, on a schedule rather than on commit.

The name was kept anyway. "Snapshot" is broadly understood outside Salesforce —
VM snapshots, database snapshots, snapshot testing — and the alternatives are
worse: "lock" implies dependency resolution that is not happening here, "freeze"
implies pinning something that could otherwise move, and "schema dump" describes
a mirror rather than the scoped declaration this actually is. The collision is
with a niche reporting feature that a data engineer reading this is unlikely to
have in mind.

Where ambiguity is possible, the docs say **schema snapshot**, and the files
live in `schema/`.

## Known limitations

Stated plainly rather than discovered later.

- **The simple-salesforce execution path is verified by hand, not by CI.** A
  JWT-authenticated `simple_salesforce.Salesforce` has now executed through
  `execute` against a real org: 4786 Contact rows drained in three batches of
  2000 / 2000 / 786, matching `COUNT()` exactly, through simple-salesforce's own
  `query` and `query_more` over HTTP. `execute_iter`'s laziness was checked on
  the same cursor — building the iterator issues no request, and the first row
  costs one batch out of three. That closes the gap this section used to
  describe.

  What it does not close: **CI does not run these tests, and will not.** They
  live in `tests/test_live_org.py`, are skipped unless `SOQLMODEL_LIVE_ORG=1`,
  and need credentials that exist only on a developer machine — a build that can
  go red because a sandbox was refreshed or a cert expired is worse than no
  build. So the wire path is confirmed as of a point in time, against one org,
  one auth flow and one object, rather than continuously. What CI *does* run
  every commit is the transport-stubbed conformance suite, which drives the
  genuine `query` and `query_more` — real signatures, URL building, response
  parsing — without an org.
- **Compound field types** (address, location) fall back to `Any`.
- **No relationship traversal.** Single-object queries only; no parent/child
  joins.
- **Rows are not mapped onto models.** `execute` returns `list[dict]`.
- **Extraction needs either the `sf` CLI or credentials.** `--source
  credentials` removes the CLI requirement, but adds the `[salesforce]` extra
  and four environment variables. There is no third option, and no way to
  extract from a committed file alone — that is what `generate` is for.

## Development

```bash
uv sync
uv run python -m pytest
uv run ruff check .
uv run ruff format --check .
uv run mypy --strict src/soqlmodel
```

That suite is offline. The live-org tests are opt-in and skipped by default;
`SOQLMODEL_LIVE_ORG=1` enables them and turns any subsequent skip into a
failure, so an opted-in run cannot pass by testing nothing. Credentials come
from the environment only — see the docstring in `tests/test_live_org.py` for
the variables, and never put a key or an org name in this repo.

`DECISIONS.md` records why things are the way they are, and is append-only.
`KNOWN_ISSUES.md` records what is wrong or unverified. Read both before
proposing a change that touches a prior decision.

## License

MIT. Copyright (c) 2026 OptimaCore. See [LICENSE](LICENSE).
