# Changelog

## 0.9.0 - unreleased

### Projects were switched on and doing nothing

**A deploy with a project active was written to the user directory, and the
repository kept the file it was created with.** `PythonFlowStorage` overrides
the read and the write, and everything that makes a project a project lives in
the hooks around them: loading the active project, refusing a deploy during an
unresolved merge, committing one when the workflow is `auto`. The overrides
dropped all three. It also settled its path once at start-up, so activating a
project - which is what switches the flow file - moved nothing.

The runtime said so and nobody could see it: the start-up log reported
`Flows file: <project>/flow.json` while the deploy went to
`<userDir>/flows.py`, and `POST /flows` answered with a `rev` either way.

It survived because **the whole test suite ran against the JSON storage**.
`runtime.configure` does not choose the storage module - `__main__` does - so
a test that configures the runtime directly gets a storage no instance runs.
The harness keeps the flows as Python now - see below - and without the fix
four project tests fail.

### A project keeps its flows as Python

Which is what the Python flow file was for: a deploy produces a readable diff,
and the history of a flow is the history of a file. A new project is laid out
with `flows.py`, generated rather than written as a literal, so the first
commit holds exactly what a deploy would write.

A project made before this - or a repository cloned from one - names a `.json`
in its `package.json`. That name is taken as naming the **flows** rather than
the format, the same way a settings file naming one is: the sibling `.py` is
used, the JSON is converted on first read and kept as `.json.migrated`, and
`package.json` is rewritten to name the new file rather than left pointing at
something that is no longer there.

### `flowFile` naming a project worked until the flows became Python

Naming a project there is how an instance comes up on one without anybody
opening the editor. A project is a directory, so the name carries no suffix -
and the rule that turns every suffixless name into `<name>.py` reached it. The
project went unfound and the instance came up on an empty flow file in the
user directory, which from outside is indistinguishable from having lost the
flows.

### A setting inside a block can be named from the environment

`PYFLOWRED_EDITOR_THEME__PROJECTS__ENABLED=true` - a double underscore for
each level. Before this the only way to reach a nested setting was to send the
whole block as JSON, which replaced every sibling in it, so turning projects
on from a container also threw away the editor's title. The layers merge block
by block now for the same reason, and what the environment set is listed once
at start-up under `From the environment:`.

### A deploy leaves one line in the project's status

The flow file's safety copy is written beside it on every deploy, and reading
the flows is an import, which caches. Both turned up as permanently untracked
entries in the version-control sidebar - in the one view that exists to show
what changed.

Bytecode is no longer written for a flow file at all; the cache saved nothing,
since the file is read once at start-up and once per deploy. The backup is
excluded through `.git/info/exclude` rather than `.gitignore`, because it is a
fact about this working copy and not about the project: a repository cloned
from elsewhere is not ours to add a tracked file to.

### Export and import offer the Python file, not only the JSON

The flows have been Python since 0.7 and the editor's two dialogs still only
knew about JSON - so the one thing you could not get out of the editor was the
form the flows are actually kept in.

**Export** has a third view beside the preview and the JSON. Download offers
`flows.py` there and `flows.json` on the JSON view, the compact/formatted
toggle is disabled on it (the file is formatted the one way on purpose, so
that two exports of the same flows are the same bytes), and which view you
left it on is what it opens with.

**Import** takes either, decided on the first character - flow data is a JSON
array or object, so anything else is the file form. The file picker accepts
`.py` as well as `.json`, and a file that will not import is refused with the
line it failed on rather than as broken JSON.

Both go through the runtime: `POST <httpAdminRoot>flows/python` renders,
`POST <httpAdminRoot>flows/parse` reads back. Rendering in the browser would
have meant a second generator in JavaScript, drifting from the file the
runtime writes within a release - and then the export would no longer be the
file you get on disk, which is the only reason to offer it. Reading needs the
runtime outright: **importing a Python flow file runs it**, which is why that
half needs `flows.write` rather than `flows.read`.

### A Function node with no name was renamed by every restart

Found by building the round trip above: export a flow set as Python, import
it, and the unnamed Function nodes came back called `function`.

The reader falls back on the identifier when a node declares no name, so the
generator has to write `name=""` when there is one and it is empty. Every
other node type did. The Function node's decorator tested the name for
*truth* where the rest tested it against the identifier, so an empty one was
dropped - and the flow file on disk is where this landed, not just the
export: a restart renamed every unnamed Function node in the instance.

### The editor calls itself PyFlowRED

It said **Node-RED** in a few hundred places a user could read: the welcome
line in the log, the help sidebar's section for the core nodes, the unknown
node's help, the file node's tip about the working directory, the JSONata
function descriptions, the proxy note on the HTTP Request node - and the same
again in every other language catalogue.

Three kinds of occurrence were deliberately left alone, because renaming them
would have broken something or made a sentence false:

* **the two wire headers**, `Node-RED-API-Version` and
  `Node-RED-Deployment-Type`. They are protocol. Renaming them would stop
  Node-RED's own tooling talking to this runtime, which is the point of
  being wire-compatible;
* **lower-case `node-red`** - CSS classes, element ids, i18n namespaces, the
  module name the core node set is registered under. Identifiers, not prose,
  and about four hundred of them;
* **statements that are about Node-RED**: what the Prometheus node was
  ported from, which upstream release introduced the Complete node's
  behaviour, when JSONata support first required `msg`. Those were renamed by
  the first pass and put back, because "as introduced in PyFlowRED 1.0" is
  not a true sentence.

Two things were wrong rather than just misnamed, and are fixed with it: the
help menu pointed at `nodered.org/docs` under a PyFlowRED label - it now
points at this project and can be overridden with
`editorTheme.menu["menu-item-help"]` - and the unknown node's help told you
to run `npm install`, which this runtime has no use for.

`tests/test_product_name.py` checks every visible English string and names
the three exceptions one by one, so the next catalogue updated from upstream
fails the build rather than the eye.

Measured the way it is read: the editor driven in a browser, scanning the
rendered text of the main window, the menus, the settings, the palette
manager and the node help. **Nothing visible says Node-RED any more** except
the About dialog, which shows this changelog - where it belongs.

### A version in the Setup tab was ignored when the runtime had the package

`pyarrow==20.0.0` imported the runtime's 25.0.1 and said nothing: the import
succeeded, and nothing afterwards looked at which version had answered. The
check for "do I need to install this?" was "does it import?", which is the
wrong question whenever a version was named.

There is one way in now - `external_modules.resolve` - and it compares what
imported against what was asked for. A version the runtime satisfies is taken
as it stands; one it cannot give is refused by name and number, because this
environment is appended to `sys.path` and a second copy beside the runtime's
would be installed and then never used.

The gate on *installing* moved with it: an instance with
`"allowInstall": False` can still import what is already there, which it
could not before.

### The palette manager could not actually install anything

Reported from the editor, where it looks like one fault and is three.

**The search never searched.** The box filters what a catalogue already
handed over, and that catalogue is built by downloading PyPI's full simple
index - 43 MB, 900,000 names - and keeping the ones with a `pyflowred-node-`
prefix, once a day. For a young ecosystem that is empty, and a project
published an hour ago is not in the listing yet even though its own page is
live: installable and unfindable at the same time. `GET
<httpAdminRoot>palette/search?q=` asks the index for the **name** instead -
`lowercase` finds `pyflowred-node-lowercase` - and the box falls back to it
when the catalogue has nothing.

**The refresh button refreshed the browser's cache, not the catalogue.** It
appended a cache-buster and got the same day-old answer from the runtime.
Opening the tab still takes the cached one; the button forces a rebuild.

**And the install never reached the running editor.** `node/added` carried
the *module* where the editor reads `id`, `types` and `version` off each
**node set**. It read `undefined` off all three and threw in its semver
parser, so the new node type appeared only after a reload - which is
indistinguishable from an install that did not work. `node/removed` always
sent node sets; this one never did, since the first commit.

A catalogue failure is also logged at `warn` now and carried in the response.
It was swallowed at `debug`, so the symptom was an empty box with nothing to
explain it.

### Installing a node package was impossible on a development build

The resolution is constrained to what the runtime already provides, which is
what stops a second `pyarrow` being installed and shadowed. But PyFlowRED's
own distribution is in that set, and in a checkout it is an editable install
whose version no index has - so the constraint was unsatisfiable, and since
every node package depends on `pyflowred`, **none of them could be installed
at all**. It only worked while the checkout's version happened to match a
released one.

A distribution installed from a path or a checkout - PEP 610's
`direct_url.json` - is now named in `constraints.txt` without a version. It
is still kept out of the lock; it is just no longer pinned to something that
cannot be fetched.

### The version says what is running

`0.9.0.dev0`. It said `0.8.2` while running everything above, which is the
released version and not what was there - and "is this the current one?" has
no answer if the only thing to go on is wrong.

### The example node package is published

[`pyflowred-node-lowercase` 0.1.0](https://pypi.org/project/pyflowred-node-lowercase/)
is on PyPI - so the palette manager has something real to install, and
"install a node package by name" can be shown rather than described. The
whole cycle was driven against the real index before this was written:
installed by name, declared, locked to `==0.1.0`, used in a flow, and rebuilt
from the lock after the environment was deleted.

It carries no repository links. `pyflowred`'s own project page has five and
**all five answer 404**, because the repository is private; the metadata is
corrected here for the next release, and the 29 further links in the README
that page shows still need deciding rather than editing - there is nothing
public to point at, though the documentation does travel in the source
distribution.

### The palette manager installs into the declared environment

A node package went into the environment PyFlowRED runs in, and nothing was
written down - so a rebuilt container came up without node types its flows
need, which is a flow that stops working for a reason nobody can see.

Installing one now **declares** it: the name goes into
`<userDir>/externalModules/requirements.txt`, uv locks it to an exact version
and installs it beside the flows' own imports. Entry points are found there
because the environment's `site-packages` is on the path. Removing a package
takes it out of the declaration too, and one that installs but advertises no
`pyflowred.nodes` entry point is taken out again with an answer that says so,
rather than being left to be reinstalled at every start.

Installing **from PyPI by name** is the ordinary case - `{"module":
"some-node-package"}` - with an optional version; a local directory or a URL
is written as a direct reference (`name @ file:///...`) so the line still
says which distribution it is. Against an internal index, point uv at it with
`UV_EXTRA_INDEX_URL`.

### The resolution knows what the runtime already has

Installing one node package pulled in **150 distributions**: a second
`pyarrow`, a second `duckdb`, and a pinned `pyflowred` that was not the one
running - because a node package depends on `pyflowred` and the resolver had
no way to know it was already there. All of it then sat on an appended path
being shadowed.

The runtime's own set is written to `constraints.txt` at every lock and
handed to uv twice: as a constraint, and as `--no-emit-package`. The lock
carries only what is genuinely extra - the same install is now one line - and
a distribution that needs a *different* version of something the runtime has
is refused with uv's explanation rather than installed and silently overruled
by `sys.path` order.

### A flow file could not name a node type from a package

The editor builds its palette from the registry; the Python flow file builds a
second one by reading the dialogs directly, because it needs the identifier a
type is reached by before any runtime exists. That second one only ever read
the **core** types. So a node type that was in the editor, in the registry and
working on the canvas could not be written to the flow file - and an instance
restarting onto a flow that used one came up with
`no node type is reached by 'lower_case'` and no flows at all.

The loader tells the flow palette where an installed package's dialogs are,
from the one place both start-up and an install go through.

### The tests run on the storage an instance runs

`runtime.configure` does not pick the storage module - `__main__` does - so
every test that configured the runtime directly got the **JSON** storage while
every instance runs the Python one. The harness keeps the flows as Python now
and one test asserts that it does, because a default is only worth having if
something fails when it changes back. JSON is reached on purpose, where a JSON
file is the subject: migrating one, importing one, reading one from a store
filled before 0.7.

### A flow store written before 0.7 came up empty

Object storage holds a copy of the flow file, and that file has been Python
since 0.7 - the documentation said JSON in three places and was simply wrong.
A bucket filled by an older instance therefore held `flows.json` and no
`flows.py`, so a fresh instance pointed at it pulled **only the credentials**,
started with no flows, and logged `Flows pulled from ...: flows_cred.json`,
which reads like it worked.

The JSON is pulled down under its own name now and converted on first read -
the same takeover an instance does for a `flows.json` it finds locally - and a
store holding no flow file at all says so instead of being silent.

### A flow's imports are declared, locked, and installed apart

What the Function node's *Setup* tab asks for went into the environment
PyFlowRED itself runs in, and nothing was written down. Two consequences, and
the second is the expensive one:

* a flow could move the runtime's own pinned dependencies - a Function node
  asking for a different `pyarrow` was resolving against the interpreter
  serving the editor;
* the set existed only as whatever happened to be installed. A replacement
  instance had to discover it one failing deploy at a time, and a flow that
  stopped working did so for a reason nobody could see.
  `.config.modules.json` was documented as holding "externally installed
  modules" and was never written by anything.

Now `<userDir>/externalModules/` holds `requirements.txt` (the declaration),
`requirements.lock` (exact versions, resolved by uv, with a digest of the
declaration so a stale lock is recognised) and `.venv` (derived from the
lock, and deletable). Its `site-packages` is **appended** to `sys.path`, so
nothing a flow asked for can shadow what the runtime needs.

**It was never per message.** Those imports happen while the node is being
built, so once per deploy at worst - and now once per distribution, because
the declaration survives a restart. Measured: one install, then five
messages, a redeploy and a restart add none.

**It starts without an index.** Once `.venv` exists the start-up sync needs
nothing from the network, which is what makes this usable on a closed
network: build it where there is an index, ship the directory, set
`"allowInstall": False`.

**Removing a line removes the distribution.** `uv pip sync` with
`--allow-empty-requirements` - without the flag uv declines to clear an
environment and says so, and the first attempt at this left everything
installed when the last line was removed, which is the one thing declaring
the set was for.

Smaller, from the same work:

- **A version can be named**: `humanize==4.9.0` in the Setup tab is kept, and
  asking later for the bare name does not widen a pin somebody wrote on
  purpose. The spec is validated before it reaches a command line - the
  install path did not apply the name check the palette path did, so `-r
  /etc/passwd` would have been passed to uv as an option.
- **A declaration that cannot be satisfied no longer takes away the imports
  that worked.** It is the declaration that is broken, not the environment:
  the instance starts, what is installed stays importable, uv's explanation
  goes into the log in full, and the health check reports `degraded` with the
  summary line rather than the last line of the explanation.
- **`GET <httpAdminRoot>modules`** - what is declared, what is installed,
  where, which uv, and the last error unabridged. The question is asked from
  the editor, not from a shell on the host.
- **The `pip` fallback is gone.** A uv-created virtual environment has no pip
  in it, so it answered "No module named pip" - a message about the wrong
  thing entirely. Where uv is missing, the error says what to install and
  what to set instead.

### Projects are documented

The feature had one paragraph, and cloning an existing repository - the way a
set of flows moves between instances - was not mentioned anywhere although it
has been implemented throughout. `docs/operations.md` now covers switching it
on from a settings file or the environment, what is in a project, the three
ways to start one with the API calls for each, bringing an instance up on a
project without the editor, and what the runtime writes that git should not
see.

## 0.8.2 - 2026-09-28

### The flows are Python, and only Python

From 0.8 that was the intent and the default; it is now the only format.
Naming a `.json` in `settings.py` is taken as naming the **flows** rather than
the format: the sibling `.py` is used, the change is logged at start-up, and
an existing JSON file is converted on first start and kept beside it as
`.json.migrated`. Refusing to start would be the alternative, and it would
leave an instance that worked yesterday down for a reason that costs one line
to fix.

**The wire format is unchanged.** The admin API, the editor's import and
export, and a flow file on object storage are all still JSON, which is what
keeps flows exported from Node-RED importable. Only the file on disk became
Python.

### A property with a fixed list of values now says what they are

Found by building a whole IoT pipeline through the agentic interface: the
`aggregate` node reported `window` with a default of `"tumbling"` and nothing
about the other six window shapes, because the `defaults` block in a dialog
records the default and says nothing about what else is allowed. That list
exists in exactly one place - the drop-down in the same dialog - and it is now
read from there.

**42 of the 74 types** gained allowed values this way, configuration nodes
included: their dialog ids carry `node-config-input-` rather than
`node-input-`, and missing that convention had left every broker, store and
TLS setting without its list - which is exactly where guessing is most
expensive.

### The application test behind both

A complete IoT pipeline, built on an empty instance through the MCP tools
alone: two configuration nodes, a source, a ten-second tumbling window per
machine with five aggregation functions, rows into DuckDB, and an HTTP page
rendered from a SQL query across them. Eleven nodes in one batch, and it
processes: three machines windowed, fifteen rows stored, the page rendering.

The tap on the window output answered with `window.start` and `window.end` -
the field this repository's own IoT example had wrong when it was written by
hand, because `windowEnd` is the name it looks like it should have.

## 0.8.1 - 2026-09-28

Found by **using** the agentic interface rather than testing it: building a
working HTTP endpoint on a fresh instance through the MCP tools alone, without
reading the source. Everything here is something that got in the way of doing
that, and the first one is not about the interface at all.

### A fresh instance wrote JSON into a file called `flows.py`

**The most common start, and it was wrong since 0.7.1.** `_use_python_flows`
is handed the *user's* settings; the packaged defaults are merged in later
inside `runtime.configure`. So an instance whose settings file says nothing
about `flowFile` had `None` there, `None` does not end in `.py`, and the JSON
storage was chosen — for a file the runtime then named `flows.py`. `python
flows.py` could not run it and a deploy from the editor did the same thing.

Only instances started with an explicit flow file were unaffected, which is
why it survived: every test and every example passes one.

### Node ids are not stable across a restart, and now it says so

When the flows are kept as Python each node is written as a variable named
after it, so a node added as `route` and named *by kind* comes back as
`agent_demo.by_kind` on the next start — and until that restart the file and
the running instance disagree. That is the design, but nothing said it, and an
agent that remembers an id across a deploy finds it gone. `engine_rules` now
carries it, and `edit_flow` says it in the answer that hands the ids out.

### The tap can be narrowed

An HTTP request carries a dozen header fields and a socket handle, and a shape
of the whole message buried the two fields the question was about — the
context problem this group exists to avoid, in miniature. `tap` takes a
`path` now: `payload`, or `payload.reading`. On the demo flow that is
eighty-two messages in four lines instead of twenty.

### Smaller, all from the same session

- **A mistyped node type now gets a suggestion.** Substring matching missed
  the ordinary case: `funktion` contains no substring of `function`, so an
  agent that misspelled a type was told only that it does not exist, which is
  the one situation a suggestion exists for. Typo-tolerant now, for node types
  and for property names.
- **An object in a sample no longer carries its memory address.** Two samples
  of the same thing looked different on every call, which is a difference
  about nothing and one an agent would try to explain.
- **An example is now found by what is in it, not by its file name.** One
  example is deliberately about more than one node type - the Kafka one needs
  its broker, the Cloud Storage one its configuration node, the Prometheus one
  both of its nodes - so a file name can only ever be right about the first of
  them, and `get_node_example` answered "no example ships for that type" for
  four types that have one. It also listed file names as if they were type
  names, offering `kafka-in` for a type spelled `kafka in`, so the name it
  suggested failed on the next call. Where an example merely *uses* the type
  rather than being about it, the answer says so - every example is driven by
  an Inject, and offering one of them as "the Inject example" would mislead.

## 0.8.0 - 2026-09-28

### An agentic interface, as an MCP server

A coding agent can now work **on a running instance**: see what is there,
learn what the nodes can do, change the flows, find out immediately whether
the change holds, and look at what the data is doing without drowning in it.
Off unless switched on, in the editor under *Settings -> Agent* or in
`settings.py`. Reference: [docs/agent.md](docs/agent.md).

- **Mounted on the instance's own port**, over streamable HTTP, behind the
  same authentication as the admin API. One port, one way in, one set of
  credentials - and whatever already stands in front of the instance stands
  in front of it too.
- **It is an adapter, not a second implementation.** The node shapes come
  from `flow/palette.py`, which reads the edit dialogs; the diagnostics come
  from the language server; what is flowing comes from the runtime's own
  hooks; the state comes from the context stores and RocksDB. A second
  implementation of any of those would drift within a release and the agent
  would be the last to find out.
- **Sixteen tools in five switchable groups**: *inspect* (what is there),
  *palette* (what can be built), *edit* (change it), *data* (what is
  flowing), *code* (evaluate Python inside the instance, off unless asked
  for by name). Diagnostics are not switchable, because an interface an
  agent cannot get error messages out of is one it cannot correct itself on.

### No tool returns an unbounded sequence

One industrial topic at thirty messages a second fills a large context window
in a minute and teaches the agent nothing it could not have learned from
thirty of them. Three answers are allowed instead: a **shape** (every field
path, its types *with their split*, presence, range, distinct values), a
bounded **sample** of whole messages, and an **aggregate**. Every response has
a size budget, and **truncation is never silent** - a cut answer says it was
cut, how much is missing and what to narrow. An agent that cannot tell a
partial answer from a complete one states a conclusion confidently and has no
reason to doubt it.

### The failure is part of the interface

Every failure answers four questions and no others - what was attempted,
which rule, where, and what to try instead - because that text goes straight
into the agent's next attempt. Never a traceback.

`edit_flow` takes a **batch**, validates the whole result, and applies all of
it or none. An agent's change is usually several moves that only make sense
together, and half of one leaves the instance in a state nobody intended.

The linter checks unknown types, unknown properties, out-of-range outputs,
dangling wires, a property naming a node that is missing or of the wrong
type, and a Function body that does not compile. **Not cycles** - a loop is a
normal construction in a message router, and refusing one would teach the
agent something false about this engine.

### Reading the edit dialogs properly

The linter is only worth having if it is right about flows that work, so the
dialog parsing was rebuilt to match brackets and skip strings, comments and
**regular expressions**. The Inject dialog validates with `/^\${[^}]+}$/`,
whose braces are not braces; counting them left the old scan convinced the
defaults block ended a third of the way through, and everything after that
was invisible. `rbe` carries `validate: RED.validators.typedInput({type:
'msg'})` inside a property, which a search that ignores depth reads as "this
property names a node of type msg".

Read carelessly, the linter reported **430 problems in the 821-node
end-to-end set**. It now reports **none**, there and in both example flows,
and that is a test.

- `flow.palette.Type` gained `properties`: every property the dialog
  persists, with its default, whether it is required, and whether it names
  another node - which is what `describe_node_type` hands an agent.
- A key beginning with `_` is an **annotation** rather than a property and is
  not reported. `tools/e2eflows.py` carried its coverage bookkeeping as a
  bare `covers`, which is indistinguishable from a guessed property name;
  it is `_covers` now. The linter also found two keys those flows wrote that
  nothing reads (`topicType` on `mqtt in`, `value` on `global-config`), both
  removed.

### `execute_code`, and what it cannot reach

Python on the runtime's own event loop, with a curated scope: the flow
configurations, the live nodes through a read-only proxy, the context stores.
Credentials are kept away by three means at once - the proxy withholds any
attribute that reads like a secret, the configurations are handed over
stripped, and imports are limited to standard library modules that cannot
reach the secret store.

**It is not a sandbox**, and the module says so. Python in a shared process
never is, and claiming otherwise would be worse than saying so: the guard
rails stop an agent reaching a credential by accident or by asking, and would
not stop one that was trying. What protects the instance is who holds the
token.

### Also

- `fastmcp` is a dependency rather than an extra: an interface that is only
  there after a second install is not there when somebody needs it.
- `mcp` is deliberately **not** in the default settings - that layer beats
  anything the runtime persists, so a default there would make the editor's
  switch silently ineffective. The settings file wins if it says anything;
  otherwise the switch does, and it survives a restart.

## 0.7.4 - 2026-09-28

### A path in a flow is relative to where you started

One rule now, and it holds for every path any node takes: **relative means
relative to the directory the engine was started in**, and an absolute path is
taken exactly as it stands. That is how every other command-line tool behaves,
and it is the only answer that survives the same flow being run from two
places.

- **`stream-state` no longer defaults under the user directory.** With no path
  configured its RocksDB went to `<userDir>/stream-state/<id>`, which on a
  default installation is a home directory. It now goes to
  `stream-state/<id>` beside the command. The user directory is where the
  *application* keeps its own files - the flows, the settings, the
  credentials, the installed nodes - and where a flow's data belongs is a
  different question.
- **`duckdb-store` fixes its `path` and `tempDirectory` when the node is
  configured** rather than leaving them to the operating system at each use,
  so the path a node reports and the file it opens are the same thing.
  `:memory:` is DuckDB's own word for no file and passes through untouched.
- The **file nodes** behave as before when `fileWorkingDirectory` is set, and
  make the path absolute either way.
- `examples/kafka-iot/` now defaults `IOT_ARCHIVE` and `IOT_DB` to relative
  paths. In the container this is the same place as before, because the
  image's working directory is the `/data` volume; run it from a directory
  and the data appears there instead.

**If you have an existing state store under the user directory**, nothing is
moved and nothing is read from there: the node starts empty and says once,
with both paths, that state is sitting at the old location. Move the
directory, or set the path on the node.

## 0.7.3 - 2026-09-28

Everything 0.7.2 brought, plus the item below.

### A Kafka refusal says which name was refused

- **`Not allowed to use consumer group '…'`.** The broker answers
  `GROUP_AUTHORIZATION_FAILED` and names the kind of permission it wanted,
  never the name it wanted it for - and the name is the whole difference
  between a consumer group refused because it falls outside an allowed prefix
  and one refused because the credentials are wrong. Both nodes now fill it
  in, for the group, the topic and the cluster.
- Worth telling apart from the three failures 0.7.2 reports, because reaching
  it means the connection opened, the certificate was accepted and the
  credentials were taken: everything that usually goes wrong is already past,
  and the answer is an access rule on the broker rather than anything set in
  the node. On Confluent Cloud a service-account API key needs the group
  granted separately from the topic, often by prefix.
- It also arrives by a different route - a poll result, not the error callback
  - so it is the one stage 0.7.2's callback does not cover, and 0.7.0 reported
  it identically. Measured against a broker with an authorizer.

## 0.7.2 - 2026-09-28

### Kafka says why it will not connect

- **A connection failure now reaches the node.** No librdkafka `error_cb` was
  registered, so a broker that could not be reached wrote to standard error
  and nothing else happened: the node kept whatever status it last set, the
  flow looked healthy, and the only sign was a line in a terminal nobody is
  watching. Bootstrap failures surface no other way - they are not exceptions
  and they are not poll results.
- **The status carries librdkafka's own reason rather than the word *error*.**
  `… in state CONNECT`, `SSL handshake failed` and `Authentication failed` are
  three different problems with three different answers - the network, the CA,
  the credentials - and a status that says "error" throws that away. The
  reason is cut to fit from the *end*, because the bootstrap URL takes up most
  of the front.

### The CA comes from the machine

- **`trustStore` on the broker node, `host` by default.** librdkafka does not
  read `SSL_CERT_FILE` and does not use Python's trust store; it has its own
  `ssl.ca.location` whose default differs per platform, so a corporate CA
  installed for everything else on the machine did not reach Kafka.
- On **Windows** the certificate stores, widened from librdkafka's default
  `Root` to `Root,CA` so an intermediate the domain put in the `CA` store is
  found too. Elsewhere the first bundle that exists of `PYFLOWRED_CA_BUNDLE`,
  `SSL_CERT_FILE`, Python's default verify path and `certifi`, falling back to
  `probe`.
- **This changes TLS behaviour for an existing `SSL` or `SASL_SSL` broker**:
  before, only the two environment variables were consulted. Set `trustStore`
  to `librdkafka` for the old behaviour. A TLS config node or an explicit
  `ssl.ca.*` property still wins either way.

### Storing data nobody declared

- **`duckdb out` grows the table to fit the block.** It created the table from
  one block's inferred types, so a column that was all-null in the first block
  became an integer one - and the first real string then failed that insert
  and every one after it. It now leaves an all-null column out until a block
  carries a value for it, and adds a column that appears later with the type
  that block reveals.

### An example for a real plant

- **`examples/kafka-iot/`** - one Kafka topic carrying a whole plant, in the
  envelope a real middleware sends. Discovery, measurement, storage and KPIs
  with **nothing in the flows naming a tag, a machine or a line**: the series
  identity is derived from the envelope and the value's type decides the
  treatment.
- Every stream operator has a job: `aggregate` for a window of each numeric
  point and again for the state durations summed per state, which *is* the
  availability; `derive` for a counter's rate; `transition` for how long each
  state was held; `threshold` on a z-score against the point's own window, so
  no limit table is needed; `compress` as a deadband in front of the archive;
  `assemble` for the machine as one record; `pattern` for an anomaly followed
  by a state change.
- Two HTML windows onto the processing, fed by what the operators produced and
  not by a query: `/iot/live` for what the plant is doing and `/iot/pipeline`
  for what the flow is doing - the page to open when something has stopped.
- `simulate.py` speaks the same envelope, so the example runs without the
  middleware and nothing in it is written against a simulator.

## 0.7.1 - 2026-09-28

- **The flow file is Python by default.** 0.7.0 shipped the storage but left
  `flowFile` pointing at `flows.json`, so a fresh instance came up writing
  JSON and the feature looked as though it had not been merged. It had; it
  was only ever reached by asking for it.
- **An instance that already had a `flows.json` is converted on first
  start.** The JSON is read, the Python written beside it, and the original
  renamed to `flows.json.migrated` rather than deleted - a migration nobody
  can undo is not one anybody should have to trust. Credentials do not move:
  they are named after the flow file's stem, which does not change.
- Naming a `.json` in `flowFile` still writes the old format, for anyone who
  wants it.
- **A relative `flowFile` is relative to the user directory**, not to
  wherever the process was started. It was resolved against the working
  directory, which put the file somewhere nobody was looking: the instance
  came up with no flows, an existing `flows.json` beside it was never found,
  and the first deploy would have written an empty file into the current
  directory. Present in 0.7.0 for anyone who set a relative `.py` path.

## 0.7.0 - 2026-09-28

### Flows can be Python files

- **Point `flowFile` at a `.py` and the flows are kept as code** - one file,
  every tab, no JSON. The editor is unchanged and still where flows are
  drawn; only what reaches the disk is different, and the suffix is the whole
  of the configuration.
- It runs on its own: `python flow.py` starts the engine and keeps running,
  taking its user directory from the file's own directory.
  `pyflowred check flow.py` reads and validates it without starting anything.
- Nodes are declared and wires are separate statements, so **a cycle needs no
  special syntax** - `attempt[1] >> waited` is written like any forward wire.
  A flow may be deliberately cyclic, and a form that nested the calls could
  not express that; it would also be claiming something untrue, since the
  variable carries no message at run time.
- Three outputs unpack into three names; one output into two nodes is the
  handle used twice; two outputs into one node is two statements, because a
  node has at most one input and none of the seventy-four types has more.
- Tabs, subflows and Function bodies are functions, which is what they are. A
  tab's locals are its node identifiers, so two tabs may each have a
  `readings`. A Function node becomes an `async def` that ruff and pytest can
  reach into.
- A node's identifier survives a rename: the editor changes `name`, never the
  id. Measured through the editor in a browser on an 821-node set - renaming
  a node changes two lines, moving one changes two, adding one changes
  thirteen and removes none, and a deploy with nothing changed changes
  nothing.
- Measured lossless against that same set: 821 nodes, 69 types, 290
  configuration keys, everything identical bar one documented normalisation -
  a `def` cannot carry trailing blank lines, so a Function body that had them
  loses them once and idempotently.
- Credentials stay JSON, encrypted with the instance key. They are not
  anybody's to read in a diff.
- See [docs/porting.md](docs/porting.md#flows-as-python) and
  [examples/python-flows](examples/python-flows/README.md).

### Blocks, files and a body you can compute over

- **`buffer`** holds messages until an interval, a count, a quiet gap or a
  control message closes the block, then emits one message carrying all of
  them in arrival order. The messages are not changed - no field added,
  renamed or reordered, nothing aggregated - because a buffer that altered
  what it held would make the block something other than what was measured.
- What it holds lives in RocksDB, so a part-filled buffer survives a restart
  or a redeploy. `maxMessages` closes early and says so, because a boundary
  that never arrives would otherwise grow until the process is killed.
- **`parquet`** turns the block into a file on the message. The schema is
  read from the block rather than declared: a field only some records carry
  becomes a column with nulls, and a field whose type varies becomes text and
  is reported - guessing from the first row would write every other row wrong
  in silence.
- **`duckdb-store`, `duckdb out` and `duckdb`** are a body of messages you
  can run SQL across. Twenty million rows are 171 MB on disk here and a
  grouped aggregate over all of them takes 34 ms; the same rows held in
  memory instead would cost 919 MB and still need the computation written by
  hand.
- Writing is buffered through RocksDB and the buffer is switchable, because
  feeding DuckDB one row at a time reaches about 1,100 rows a second - below
  what this runtime itself carries. Blocks of five hundred reach a few
  hundred thousand, and at two thousand messages a second the writing takes
  3% of the time.
- Table, row property, SQL and parameters are typed inputs, so a flow can
  work out which table it writes to and assemble a statement elsewhere. A
  table name is checked rather than trusted, because it is allowed to come
  from the message.
- A query returns JSON records and runs in a thread: DuckDB releases the GIL
  for about 99% of a query, so unlike the same work in a Function node it
  does not hold the event loop.
- pyarrow and duckdb are dependencies rather than extras.

### Measured, and written down

- **An IoT ingest benchmark against Node-RED 5.0.7** - `tools/iotflows.py`
  and `tools/iotload.py` - driving the same flow set into both runtimes over
  MQTT and Kafka, steady and in bursts. Throughput turns out to be one over
  the CPU a message costs and nothing else, the fork costs what its branches
  cost, and the gap between the runtimes is raw speed rather than scheduling:
  3x with no work in the flow, 1.06x with half a millisecond, the other way
  round with two.
- Under a burst PyFlowRED stays answerable where Node-RED does not - 479 ms
  against 10,517 ms at the 95th percentile while draining five thousand
  messages - because the same `call_soon` scheduling that stops branches
  running in parallel is what lets anything else in between.
- **`tools/hopprofile.py`** takes a node hop apart. It is 10 us for a Change
  node, and about 40% of it is the Prometheus instrumentation. Turning
  metrics off lifts throughput 38-46%. That is written down as a price list
  rather than a task list: the instrumentation is most expensive exactly
  where it is most needed, and a counter that goes cheap by going approximate
  is worthless in the one situation it exists for.

### Settings

- **The settings file a fresh instance writes is now the reference it should
  have been.** It was thirty-five lines and named nine settings; the runtime
  reads fifty-five. It is now grouped and explained, with everything commented
  out at its default except a small active core - the server, the flow file,
  logging, the health endpoint and the editor theme.
- It lists what *this* runtime reads rather than what Node-RED offers, so
  nothing in it does nothing. `tests/test_settings_template.py` extracts the
  names from the source and fails when one is missing, and the other way round
  when the file promises something nothing reads.
- Written down there for the first time: that `PYFLOWRED_UI_PORT` is how
  `uiPort` is spelled in the environment, the order in which the four
  configuration sources win, and that **`httpNodeResponseTimeout` is seconds
  here and milliseconds in Node-RED**.

## 0.6.0 - 2026-09-27

- **The Function node's editor knows Python.** Completion, hover, signature
  help and problems as you type, from a `pylsp` process the runtime starts
  when somebody first opens the dialog. Both halves at once: `node.`, `msg.`,
  `flow.`, `global_.` and `env.` from stubs **generated by introspecting the
  runtime's own classes**, and `json.`, `hashlib.sha2` or anything the Setup
  tab installed from the language server proper.
- The stubs carry the real signatures and the real docstrings, so twenty-one
  methods on the node and context proxies gained one — they are what the
  hover shows.
- **The 2 MB of Node.js TypeScript declarations are gone**, along with the
  editor code that fetched them. They described `node.send(msg, clone?:
  Boolean)` and `import('util')` — a JavaScript API this runtime does not
  have — and were never reachable anyway, because every model in the Function
  node is a Python model.
- Only the plugins an editor needs are enabled, and only the `pyflakes` extra
  is installed. The formatters and style checkers in
  `python-lsp-server[all]` would report on the generated preamble rather than
  on anything a person wrote.
- `GET <httpAdminRoot>lsp` says whether it is there and running. Without
  `pylsp` the editor says so once and falls back to a plain code box; nothing
  about how a body *runs* depends on any of this.
- `tools/uilsp.py` checks it in a browser, which is the only place it exists.

## 0.5.0 - 2026-09-27

- **A health check, in the product rather than in the repository's tools.**
  `pyflowred health` asks a running instance whether it is keeping up and
  exits 0 healthy, 1 degraded, 2 failing, 3 unreachable; `GET
  <httpAdminRoot>health` returns the same report as JSON with 200 or 503, for
  a Kubernetes probe or an uptime monitor. It reports CPU as a share of one
  core with how much more would fit, event loop lag against the budget,
  throughput with the CPU cost of a message, node errors and backpressure.
  Only *the flows are not running* and *the loop is a second behind* fail the
  probe - a busy core or a flow raising errors warn, because restarting would
  not fix either.
  Rates need a window and a probe cannot wait one, so the sampler that already
  ticks once a second for the lag histogram now writes a small record per
  tick; five minutes are kept and asking costs a lookup.
- Both Dockerfiles health-check the verdict rather than just whether HTTP
  answers, so an unhealthy container means something a restart may fix.
  Building the released `Dockerfile` with `PYFLOWRED_VERSION` older than
  0.5.0 would check an endpoint that version lacks; it is documented in the
  file.
- **A worked example flow per added node type**, offered by the editor under
  *Import > Examples*: sixteen tabs, one for each node PyFlowRED adds beyond
  the ported set. Each is a header saying what the node is for, then one row
  per feature - an inject that produces the input, the node configured for
  that feature, a debug showing what came out - with a comment block beside
  each row saying what to look for. `tools/exampleflows.py` writes them and
  `tests/test_example_flows.py` deploys eleven of them and drives them exactly
  as their comments say to, so those sentences are assertions rather than
  claims.
- **`examples/e2e-stack`** — the end-to-end integration test as a compose
  stack: the application, both brokers, and a one-shot that deploys the flows
  and runs the checks. `docker compose -f examples/e2e-stack/compose.yml up -d`
  and the editor is on <http://127.0.0.1:1881/pyflowred/> with all eighteen
  tabs in it. It reaches 149 of 157 assertions and 95.5% of node types on a
  machine with nothing set up; the whole difference from a host run is the four
  checks that need a Google service account.
- **The stack builds its image, it does not pull one.** `pyflowred:<version>`
  is not on any registry, and a Compose that resolves `image:` by pulling
  stopped with `pull access denied for pyflowred` - on which `docker login`
  looks like the remedy and is not. The service now says `pull_policy: build`,
  and the one-shot deployer uses a public `python:3.13-slim` rather than the
  local tag, so nothing on the stack asks a registry for something it does not
  have.
- **`.gitattributes`, and Dockerfiles that tolerate CRLF.** A checkout made on
  Windows turned `docker/entrypoint.sh` into CRLF, the shebang then named an
  interpreter called `/bin/sh\r`, and the container failed with
  `exec /usr/local/bin/pyflowred-entrypoint: no such file or directory` for a
  file that was plainly there. The repository is now LF in the working tree on
  every platform, and both Dockerfiles strip carriage returns from the
  entrypoint so an image built from an older checkout works anyway.
- The stack runs `tools/e2eflows.py` from a read-only bind mount rather than
  carrying a checked-in `flows.json`, so there is one test flow set and not
  two. The generator now takes the broker addresses from `E2E_MQTT_HOST`,
  `E2E_MQTT_PORT` and `E2E_KAFKA_BROKERS`, which is the only thing about the
  test an environment changes; with the defaults it produces byte-identical
  output to before.

## 0.4.0 - 2026-09-27

Documentation only; no change to any node's behaviour.

- **A node reference, one page per node package**, in `docs/nodes/`. The
  nineteen node types that have no Node-RED counterpart — stream processing,
  Cloud Storage, Kafka, secrets, Prometheus — now have every setting, every
  option value and every message property they read or write written down,
  together with the reasoning wherever the behaviour is not the obvious one.
  Up to now they were described only in `docs/operations.md`, from the
  operator's side, and in the editor's info sidebar.
- **The README says what the package contains.** It listed the added nodes as
  three lines in a table of porting layers, so a reader on PyPI — where the
  README *is* the project page — could not tell that the distribution ships a
  stream-processing engine and four other node packages. There is now a
  section naming all nineteen types and linking to their pages.
- **README links are absolute.** Relative links resolve on GitHub and not on
  PyPI, so every link into `docs/` and the editor screenshot were dead on the
  project page. They point at `github.com`/`raw.githubusercontent.com` now.
- `docs/porting.md` lists the added types beside the ported ones and gains a
  section on stream processing; `docs/operations.md`, `docs/writing-nodes.md`
  and `docs/porting.md` link to the node reference from their matching
  sections. A `Node reference` URL is on the PyPI sidebar.
- Removed a duplicated block in the README's layout listing, where the
  end-to-end test tooling appeared twice — once with its superseded figures.

## 0.3.0 - 2026-09-27

- **An end-to-end integration test** over the node set and its settings, run
  against the service rather than in-process: `tools/e2eflows.py` deploys
  eighteen tabs with 157 assertions, `tools/coverage.py` measures what they
  reached. **66 of 66 countable node types** carry a message during a run
  (97.1% counting the two set aside), and **183 of 191 reachable dialog
  choices** are covered by a passing assertion (95.8%). Everything set aside
  is named with its reason and both figures are printed, so an exclusion
  cannot flatter the result. `tools/e2e/compose.yml` brings up the broker and
  Kafka it wants.
- Defects it found:
  - **Window closing wedged for good** on a timestamp containing the byte
    used as the index separator - about one millisecond in 250. The key was
    split on that separator, the unpack raised, and because the index is
    walked oldest-first the poisoned entry blocked every window behind it
    from ever closing again. It is sliced by offset now.
  - **The Assemble node shadowed `Node._complete`**, the runtime's own method
    for finishing a message, so every message it emitted raised.
  - `kafka out` took the topic and key from the message in preference to the
    dialog, the opposite of the MQTT node and of what its own placeholder
    says.
  - The stream operators showed raw catalogue keys as their node status.
  - A Global Config node carrying coordinates is treated as a placed node
    and, belonging to no tab, dropped - so its environment variables were
    never read.

- **Stream processing.** Seven operators - `aggregate`, `derive`,
  `threshold`, `transition`, `assemble`, `compress`, `pattern` - on a
  `stream-state` configuration node backed by RocksDB. Windows tumble, hop,
  slide, count, session, and close on a control message, which is what a
  shift, a batch, a production order and a cleaning cycle all are. Event time
  with watermarks and a second output for late messages; quality-aware
  aggregation; a key limit that refuses rather than evicts; and
  `GET <httpAdminRoot>stream-state` to see what is held.
  RocksDB is the state store rather than a fallback: 100,000 keys each
  holding a 300-sample window cost 1,193 MB as Python objects and 39 MB
  resident through it, and its slowest hot path is eighteen times the rate
  the runtime can feed it.

- Google Cloud Storage nodes: a `gcs-config` bucket configuration plus
  `gcs in` and `gcs out`, on obstore - the layer the flow store already uses.
  Every operational value is a typed property, so bucket, path, format,
  character set and content type can each be a literal, a message property, a
  flow or global value, an environment variable or a JSONata expression; left
  blank, each falls back to a documented message property. A flow that works
  out where it is writing needs one node, not one per destination.
  Credentials stay on the configuration node, where `${secret:NAME}` reaches
  Secret Manager, and a path cannot climb out of the bucket's prefix.
- `gcs list` and `gcs delete` alongside them. Listing is shallow (objects and
  child folders at one level) or deep, with a limit and a `truncated` flag.
  Deleting takes one object or a whole prefix, and is idempotent.
- Folders work the same with and without a bucket's hierarchical namespace,
  as far as the object API allows: shallow listing reports a folder as a
  folder in either case, while obstore cannot address a name ending in `/`,
  so a prefix delete leaves folder markers behind and reports them in
  `msg.gcs.remainingFolders` rather than claiming a clean sweep.

## 0.2.0

**Everything is now called `pyflowred`.** 0.1.0 shipped under that name on
PyPI but kept `pyred` everywhere else, which meant the distribution and the
thing you imported disagreed. This release makes them one name, and there is
no compatibility shim - upgrading means changing the names below.

| was | is |
|---|---|
| `import pyred` | `import pyflowred` |
| the `pyred` command | `pyflowred` |
| `PYRED_*` environment variables | `PYFLOWRED_*` |
| `[tool.pyred]` in `pyproject.toml` | `[tool.pyflowred]` |
| the `pyred.nodes` entry point group | `pyflowred.nodes` |
| `pyred_*` Prometheus metrics | `pyflowred_*` |
| the default user directory `~/.pyred` | `~/.pyflowred` |

Flows, credentials and project repositories are untouched: the flow file
format has not changed, so moving the user directory across is all an
existing installation needs. A node package has to re-declare its entry
point group to be found.

Also in this release:

- The Google Secret Manager provider could never have worked. The reply
  object handed to google-auth assigned `status`/`headers`/`data` in
  `__init__`, but they are abstract properties on its base class, so the
  class could not be instantiated and every token refresh raised
  `TypeError`. The path is now covered by an offline test as well as the
  one that needs a reachable Secret Manager.
- `POST /flows` with a bare array under API v2 returned an internal
  attribute error as its message. It answers 400 and names the v1 header.

## 0.1.0

- Initial port of the Node-RED 5.0.7 runtime and editor to Python.
- Palette catalogue served from PyPI, so the palette manager's Install tab
  can search for PyFlowRED node packages.
- Multiplayer: two open editors see each other's presence and cursors.
- Projects: git-backed flow storage, with the SSH key store that goes with
  it. Off unless `editorTheme.projects.enabled` is set and git is available.
- Benchmark flows and a load-test harness (`tools/benchflows.py`,
  `tools/loadtest.py`), plus the three defects they found: the HTTP Response
  node replaying an upstream response's headers, JSONata refusing to evaluate
  against an HTTP message, and the HTTP Request node ignoring its keep-alive
  setting.
- A native Prometheus endpoint at `<httpAdminRoot>metrics`: process, instance,
  per-node-type throughput, and backpressure as event loop lag, in-flight
  messages and the queue depth of the nodes that hold messages back. Flows can
  publish their own metrics through the ported `prometheus-metric-config` and
  `prometheus out` nodes, into the same registry.
- The wheel now ships the built editor. It was being excluded because the
  build output is git-ignored, so the package installed and served a blank
  page; the editor source, which is useless at runtime, was shipped instead.
  A build hook refuses to build a wheel without an editor.
- Flows can be kept on object storage. `flowStore` names a URL - GCS, S3,
  Azure, a directory - and obstore syncs to it: the store wins at start-up,
  the local file is pushed on every deploy, and the editor gains an "Object
  storage" menu to force either direction. Settings can now also come from a
  `[tool.pyred]` table in the working directory's `pyproject.toml` and from
  `PYRED_*` environment variables (renamed in 0.2.0).
- Secrets can be read at runtime from Google Secret Manager, or from a
  directory of files as Kubernetes mounts them. `${secret:NAME}` works in any
  node property - resolved when the node is built, so a broker or a Kafka
  client gets its credential at connect time and the flow file keeps the
  placeholder - and a `secret` node puts one on a message for the per-call
  case. Values are cached for `secrets.ttl`, so a rotation needs no rebuild.
- Kafka: a `kafka-broker` configuration node with `kafka in` and `kafka out`,
  the same shape as the MQTT nodes. On confluent-kafka, with any librdkafka
  setting reachable through a properties table. Values are JSON in both
  directions; a record that is not valid JSON raises an error rather than
  travelling on. `tools/kafka/compose.yml` brings up a broker to develop
  against.
- A container image, built in three stages with `uv`: Node builds the editor,
  uv resolves the environment, and the runtime image carries neither. Proxy,
  internal package index and internal npm registry are build arguments. Your
  own CA can be baked in or mounted, and reaches httpx, obstore, Python's ssl,
  certifi and librdkafka - the last of which ignores `SSL_CERT_FILE` and now
  takes its default `ssl.ca.location` from the container's bundle.
- Published on PyPI as **pyflowred**, because `pyred` there belongs to an
  unrelated project. At this version the import package and the command were
  still `pyred`; 0.2.0 renamed them to match.
- A second image. `Dockerfile` installs the released package with uv and
  carries no build tooling; `Dockerfile.source` is the previous one, which
  builds this checkout. Build arguments no longer leak into the runtime, and
  both images carry uv with a writable environment so the palette manager
  works.
