Metadata-Version: 2.4
Name: isynth-ai-data-config
Version: 0.8.2
Summary: Generate iSynth data_config test-data definitions from a plain-language request, using an LLM that reads the environment's own metadata.
Author-email: Markus Herrmann <markus.herrmann@itopia.ch>
License-Expression: LicenseRef-Proprietary
Project-URL: Homepage, https://isynth.io
Keywords: isynth,llm,data-config,test-data,synthetic-data
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.13
Description-Content-Type: text/markdown
Requires-Dist: litellm>=1.0
Requires-Dist: fastmcp>=4.0
Requires-Dist: openpyxl>=3.0
Requires-Dist: python-box>=7.0
Requires-Dist: python-dotenv>=1.0
Requires-Dist: httpx>=0.28
Requires-Dist: PyYAML>=6.0
Requires-Dist: starlette>=1.0
Requires-Dist: pytest>=7.0
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"

# isynth-ai-data-config

Generate iSynth `data_config` files from a request written in plain language, instead
of hand-writing YAML against an iSynth model you have to memorise first.

You describe the test data you want — *"a single private client, Canadian, born in
Switzerland, speaking English, with two accounts and a debit card on the first, one
portfolio, two phone numbers and two email addresses"* — and you get back a ready
`data_config` that iSynth can execute:

```yaml
simple_client:
  based_on: banking.individual_client
  number_of_copies: 1
  parameters:
    number_of_accounts: 2
    number_of_portfolios: 1
    number_of_phone_numbers: 2
    number_of_email_addresses: 2
  composers:
    - debit_cards:
        card_holder_label: NP1
        account_label: NP1/AC1
  modify:
    Constellation:
      description: a single private client, Canadian, born in Switzerland, speaking English, with two accounts and a debit card on the first, one portfolio, two phone numbers and two email addresses
    NaturalPerson NP1:
      is_main_contact: true
      language: EN
      nationality: CAN
      country_of_birth: CH
      civil_status: SINGLE
```

An LLM writes that file, but it does not guess the vocabulary. `banking.individual_client`,
`debit_cards` and every other name in the output come from **your** env's iSynth model,
which the LLM reads while it works. The draft is then validated against that same
iSynth model, and anything that does not check out goes back to the LLM for correction
before you ever see it. (The names above are from an example env; yours will differ.)

Two models are involved: The **iSynth model** — your
env's object types, constellation templates, composers, enums and functions — and the
**LLM**, the large language model that writes the `data_config`.

## Requirements

- **Python 3.13** or newer
- **iSynth 4.3.0** or newer
- **An iSynth env** whose iSynth model is generated (**Initialize environment**). Curated examples in `data_configs/ai_training/` are optional, but generation is noticeably worse without them: they are what the LLM orients itself on.
- **A tool-calling LLM**, hosted or local (e.g. LM Studio). It must handle *native*
  function calling over several turns; prompt-engineered tool use is not enough. For a
  local LLM, see [*LLM settings for a local LLM*](#llm-settings-for-a-local-llm).

## Install

Add this to your env's `requirements.txt`:

```
isynth-ai-data-config
```

Then `pip install -r requirements.txt` as usual. The package brings its plugins along
and registers them with iSynth, so every env of that iSynth installation has them.

## Setting up an env

The package is installed once per iSynth installation and serves all of its envs; each
env generates from its own iSynth model. Each env that uses the package needs:

1. **The `models_generated` hook** below — the one file every env needs a copy of.
2. **Its own settings and keys**, if it deviates from the defaults: `settings.yml` and
   `.keys` in the env directory, see [*Configuration*](#configuration).

### The `models_generated` hook

The LLM has to know the order composer entries must be in. iSynth's lifecycle hook
`models_generated` writes it to `__generated__/composer_sequence.txt`, after every
**Generate models** and as part of **Initialize environment**. iSynth looks for the hook
only in the env, so put this file into every env:

```python
# <env>/hooks/models_generated.py
from pathlib import Path

from ai_data_config_plugins.hooks import models_generated

if __name__ == "__main__":
    models_generated(Path(__file__).resolve().parents[1])
```

Then run **Generate models** once. A failing hook shows only in the iSynth log, as
`Lifecycle hook "models_generated" failed with return code N!` — **Generate models**
itself still reports success.

## Configuration

### Settings

Defaults ship with the package. To deviate — to use another LLM, for example — repeat
the key in your **env's** `settings.yml`; the env wins:

```yaml
LLM_MODEL: "openai/gpt-5.6-sol"
```

`${VARIABLE_NAME}` substitution works as in any iSynth `settings.yml`. Only the keys
below are read from the env; an `LLM_*`/`AI_*` key with a typo is reported, by name, as
a warning in the appserver's log. The last column says what `0`, an empty value or
`false` does — it does not always mean "no limit".

**The LLM and its requests**

| Key | Default | What it does | If `0`, empty or `false` |
|---|---|---|---|
| `LLM_MODEL` | `openai/gpt-5.6-sol` | the LLM, as a [LiteLLM](https://www.litellm.ai/) identifier `<provider>/<name>` — local and hosted LLMs are configured the same way | empty: every generation fails — an LLM has to be named |
| `LLM_API_BASE` | — | the LLM server's URL, for local or self-hosted LLMs | empty (default): LiteLLM's default URL for the provider |
| `LLM_REQUEST_TIMEOUT` | `1800` | seconds one request may wait for the LLM's answer — how long the LLM may stay silent, not how long a generation may take. Raise it for slow local LLMs | `0`: no timeout — a request waits as long as the LLM server takes |
| `LLM_NUM_RETRIES` | `0` | how often a failed request is sent again — after a connection error, a timeout, or a passing error of the LLM server such as 429 or 500. Each retry can wait a full `LLM_REQUEST_TIMEOUT` again | `0` (default): no retry — the first failed request ends the generation with its error |
| `LLM_MAX_OUTPUT_TOKENS` | `0` | most tokens the LLM may generate in one answer. An answer cut off at the cap is logged as a warning | `0` (default): no cap — only the LLM server's own limit applies |

**The tool loop**

| Key | Default | What it does | If `0`, empty or `false` |
|---|---|---|---|
| `AI_MAX_TOOL_TURNS` | `100` | most tool-calling turns per generation; after the last one the LLM has to answer without tools | `0`: no tools at all — the LLM answers from the request and the names it is given up front, without looking anything up |
| `AI_MAX_TRAINING_EXAMPLES` | `3` | most curated examples the LLM may fetch per generation. Every example stays in the conversation and is sent again with each turn, so this is the main lever against a growing context | `0`: no limit — it does not switch the examples off |
| `AI_MAX_TOOL_RESULT_CHARS` | `20000` | most characters of one tool result; a longer result is cut and marked as cut. A safety brake: results normally stay well below it | `0`: no limit |
| `AI_INLINE_METADATA_LISTS` | `true` | give the LLM the names of the constellation templates, object types, composers and training examples up front, so it needs no turns to look them up. Enums and functions are always looked up | `false`: the LLM looks the names up itself, which costs turns |

**Validation**

| Key | Default | What it does | If `0`, empty or `false` |
|---|---|---|---|
| `AI_MAX_VALIDATION_ROUNDS` | `2` | correction rounds: a draft that fails validation against the iSynth model goes back to the LLM with the problems, up to this many times. If problems remain after the last round, the generation fails and lists them | `0`: no validation — the LLM's draft goes into the file unchecked, even if it is invalid |
| `AI_MODEL_SELF_VALIDATION` | `false` | let the LLM check its own draft against the iSynth model before it answers. Costs an extra write-out of the `data_config` per generation | `false` (default): the LLM does not check its draft; the result is validated afterwards, as `AI_MAX_VALIDATION_ROUNDS` sets |

**Thinking LLMs** — only sent to local LLM servers (`lm_studio/`, `openai_like/`,
`llamafile/`, `hosted_vllm/`, `ollama/`, `ollama_chat/`), so switching `LLM_MODEL` to a
hosted LLM needs no change here.

| Key | Default | What it does | If `0`, empty or `false` |
|---|---|---|---|
| `LLM_CHAT_TEMPLATE_KWARGS` | `{}` | extra variables for the LLM's chat template, e.g. `enable_thinking` or `reasoning_effort`. llama.cpp and vLLM pass them on; LM Studio accepts the field but ignores it | `{}` (default): nothing is sent |
| `LLM_SUPPRESS_THINKING` | `false` | end each request with an empty think block, so a thinking LLM starts on its answer instead of its reasoning; works through LM Studio too. Test before enabling it: some LLMs stop converging without their reasoning | `false` (default): requests go out unchanged; the LLM thinks as its server is set up |

### API keys

Keys do **not** belong in `settings.yml`. They go in `<env>/.keys`, in dotenv format.
Anything ending in `_API_KEY` or `_API_BASE` is picked up:

```
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
```

A missing file is not an error — an env that sets its keys as real environment
variables does not need one.

### GitHub Copilot

`LLM_MODEL: "github_copilot/<llm>"` runs on a GitHub Copilot subscription instead of
an API key. The login happens once, through GitHub's device flow, and its token is kept
in a directory that has to survive a container rebuild — so set that directory
**first**:

1. **Point the token directory at the env**, in `<env>/.keys`:

   ```
   GITHUB_COPILOT_TOKEN_DIR=/isynth_envs/<env>/.copilot
   ```

   Add `.copilot/` to the env's `.gitignore` — it holds a credential.

2. **Log in once**, in a terminal on the host. It is one line, so it survives copy and
   paste:

   ```bash
   docker exec -it -e ENV_HOME=/isynth_envs/<env> <appserver> sh -c 'cd /isynth_envs/<env> && python3 -c "import ai_data_config.llm_helper, litellm; r = litellm.completion(model=\"github_copilot/<llm>\", messages=[{\"role\": \"user\", \"content\": \"hi\"}]); print(r.choices[0].message.content)"'
   ```

   It prints `Please visit https://github.com/login/device and enter code XXXX-XXXX`.
   Open that page, enter the code, approve; the call then answers. Keep `-it`, or you do
   not see the code.

3. **Check** that the directory now holds `access-token` and `api-key.json`:

   ```bash
   docker exec <appserver> ls /isynth_envs/<env>/.copilot
   ```

4. **Choose the LLM** in the env's `settings.yml` and leave `LLM_API_BASE` empty:

   ```yaml
   LLM_MODEL: "github_copilot/<llm>"
   ```

   Which LLMs you can name is Copilot's list.

Steps 1 and 2 are needed once per env; the token is refreshed by itself until it is
revoked on GitHub. For GitHub Enterprise, `GITHUB_COPILOT_API_BASE`,
`GITHUB_COPILOT_DEVICE_CODE_URL`, `GITHUB_COPILOT_ACCESS_TOKEN_URL` and
`GITHUB_COPILOT_API_KEY_URL` go in `.keys` as well.

## Using it

### Generating a data_config

1. In the iSynth UI, open a `.yml` file in the env — an empty one for a new
   `data_config`, for example under `data_configs/`.
2. Choose **AI Powered Test Data Generation** from the file's context menu.
3. Describe the test data under **Prompt** — *"a private client with one account, no
   portfolio"* — and press **Generate**.

The result replaces the file's content in the editor. A run usually takes minutes.

- **An existing `data_config` is changed, not replaced by a new one.** The file's
  content goes to the LLM with the prompt, so *"add a second account"* works on what is
  already there.
- **Checked before the LLM runs.** An empty prompt, or a file that is not a
  `data_config` (not YAML, or YAML that is not a mapping of cases), is refused at once.
- **A failed generation is an error, not a file.** A result that still fails validation
  after the last correction round is reported as an error listing the problems; the
  file is left as it was.

### The other plugins

- **Validate AI Prompts**, on the env: runs the env's validation cases in
  `ai_config/validation_cases/` and appends the results to `logs/validation_runs.xlsx` —
  to compare LLMs and settings.
- **Bill of Materials**, on a file under `data_configs/`: lists the objects the
  `data_config` produces, by object type.

## Troubleshooting

| Symptom | Cause |
|---|---|
| A plugin is missing in the UI | the package is not installed in the iSynth installation — see [*Install*](#install). The plugins also appear only where they apply: **AI Powered Test Data Generation** in the context menu of a `.yml` file, **Bill of Materials** in that of a file under `data_configs/` |
| `FileNotFoundError: ... source not found: ...__generated__...` | **Initialize environment** has not run in the env yet |
| Composer entries of a generated `data_config` are in the wrong order | the env has no `hooks/models_generated.py`, or it failed — see [*The `models_generated` hook*](#the-models_generated-hook). To see its output, run it by hand: `docker exec <appserver> sh -c 'cd /isynth_envs/<env> && PYTHONPATH=.:/isynth python3 hooks/models_generated.py'` |
| Generation fails with an authentication, model or connection error | the API key is missing in `<env>/.keys`, `LLM_MODEL` names no LLM of that provider, or — for a local LLM — `LLM_API_BASE` is wrong or the LLM server is not running |
| Generation fails with a timeout | the LLM stayed silent for longer than `LLM_REQUEST_TIMEOUT` — raise it for a slow local LLM |
| `LLM returned an empty response` or `No JSON or YAML object found in the LLM response` | the LLM did not answer with a `data_config` — often a thinking LLM that spent its answer on reasoning: switch thinking off (see [*LLM settings for a local LLM*](#llm-settings-for-a-local-llm)), or use a stronger LLM |
| `data_order still failed validation after N correction rounds` | the LLM could not fix the problems listed — make the prompt more precise, add curated examples to `data_configs/ai_training/`, or use a stronger LLM |
| A key in the env's `settings.yml` is ignored | it is not one of the keys under [*Settings*](#settings). A misspelt `LLM_*`/`AI_*` key is reported, by name, as a warning in the appserver's log (`docker logs <appserver>`) |
| A changed `settings.yml` or `.keys`, a new constellation template or composer order has no effect yet | it reaches an env with new env processes, which iSynth starts after **Sync definitions** and **Generate models** — or restart iSynth |
| A key in `.keys` has no effect, even after that | the same variable is set in the appserver's own environment, and that one wins |
| The Copilot login does not land in `<env>/.copilot` | `GITHUB_COPILOT_TOKEN_DIR` is not in `<env>/.keys` — step 1 under [*GitHub Copilot*](#github-copilot) |
| Where are the logs? | generation in `<env>/logs/ai_data_config.llm_helper.log`; a validation run also in `<env>/logs/ai_data_config.validation.config_validator.log`. Neither goes to the appserver's log |

## Appendix

### LLM settings for a local LLM

The package sends **no sampling parameters** like temperature, no top_p, etc. Those come
from the defaults of whatever serves the LLM (LM Studio, vLLM, the hosted API). The one
exception is `LLM_CHAT_TEMPLATE_KWARGS`, which travels with each request. So these are
settings you make on the LLM server, not in the isynth installation. The following table contains recommendations that serve as a good starting point for many models. **The LLM vendor's own
recommendation always takes precedence over the table below.**

| Setting | Suggested | Why |
|---|---|---|
| Reasoning / thinking mode | **off** | Decode dominates the runtime, and thinking tokens are decode. Risk of thinking blocks or empty answers instead of JSON. |
| Tool / function calling | **on, native** | Mandatory — the tools are passed over the API as `tools=[...]`. |
| Temperature | 0.1–0.2 to start | Qwen warns against greedy decoding (`0.0` causes repetition loops, it recommends `0.7`); gpt-oss is specified for `1.0`. Start low, raise if you see loops. |
| Top P | 0.8–0.9 | Keep conservative for structured output. |
| Top K | 20–40 | Qwen recommends 20. |
| Repeat penalty | 1.0–1.05 | Higher values corrupt recurring field names. |
| Context length | 64K | A run including two correction rounds needs ~15–25K tokens. Use 128K only if you need it. |
| Max response tokens | 2048–4096 | Per assistant turn, tool-call arguments included. The final JSON and the largest tool argument stay under 1K. |
| Prompt truncation | **disabled** | Set the overflow policy to stop at the limit. The transcript grows over the correction rounds; "truncate middle" would silently cut away rules or examples. |
| Structured output / JSON schema | use with care | Forcing a schema on every answer can suppress tool calls. The final JSON is validated anyway. |
| Quantization | 6–8 bit where memory allows | 4-bit produces measurably less precise tool arguments. Drop to 4-bit only as a memory fallback. |
| Seed | fixed | Makes a validation run reproducible. |

Running locally, pick an LLM by class rather than by name — a specific ranking dates
quickly. With 16 GB expect to be limited to a small MoE or an aggressively quantized
LLM, and to lose precision in tool arguments; 24–32 GB comfortably runs a mid-size
MoE with few active parameters at 4-bit; from 64 GB a dense LLM in the 27–35B range
at 6–8 bit becomes practical. Mixture-of-experts LLMs with roughly 3–4B active
parameters give the best interactive latency, because decode speed, not the LLM's size,
is what a run waits on.
