Metadata-Version: 2.5
Name: kymoscope-collector
Version: 0.5.3
Summary: Site collector for Kymoscope: OPC UA, logger files and custom sources to end-to-end encrypted streams
Project-URL: Homepage, https://kymoscope.com
Project-URL: Documentation, https://kymoscope.com/docs/collector
License: # Kymoscope SDK License Agreement
        
        Copyright © 2026 Kymoscope. All rights reserved.
        
        This is proprietary software. It is licensed, not sold. By installing, copying
        or using the Kymoscope SDK (the "Software") you agree to these terms. If you do not
        agree, do not use the Software.
        
        > Template notice for the owner: this agreement was drafted as a starting point.
        > Have it reviewed by a lawyer for your jurisdiction and fill in the legal
        > entity, governing law and contact details before relying on it.
        
        ## 1. Licenses
        
        **1.1 Evaluation.** You may use the Software without a license key to evaluate
        it and to develop your integration. Unlicensed use displays a "Powered by
        Kymoscope" attribution in the workspace.
        
        **1.2 Production.** Using the Software in a product, service or internal tool
        made available to anyone other than your own developers ("Production Use")
        requires a valid license key issued by Kymoscope for each domain the Software is
        served from, under an order or agreement with Kymoscope.
        
        **1.3 Scope.** A license key is personal to the licensee named in it, for the
        domains listed in it and until its expiry date.
        
        ## 2. Restrictions
        
        You must not, and must not allow others to:
        
        1. make the Software available in Production Use without a valid license key
           for that domain;
        2. remove, hide, obscure or alter the attribution shown by unlicensed use, or
           the copyright notices in the Software;
        3. circumvent, disable or modify the license check, or create, alter or share
           license keys;
        4. copy, sell, sublicense, distribute or publish the Software, or offer it as a
           standalone component library or SDK, except as embedded in your own product
           as permitted above;
        5. reverse engineer, decompile or disassemble the Software, except to the
           extent applicable law expressly permits it despite this restriction;
        6. use the Software to build a competing embeddable analytics or charting
           product.
        
        ## 3. Kymoscope services
        
        Live-stream relaying, watch tickets and other hosted services are provided
        under your Kymoscope account terms and may be rate limited or suspended for abuse.
        
        ## 4. Ownership
        
        Kymoscope and its licensors own all rights in the Software. No rights are granted
        except those expressly stated here.
        
        ## 5. Termination
        
        This license ends automatically if you breach it. On termination you must stop
        using the Software and delete all copies.
        
        ## 6. No warranty and limitation of liability
        
        The Software is provided "as is", without warranties of any kind. To the
        maximum extent permitted by law, Kymoscope is not liable for any indirect,
        incidental or consequential damages, and its total liability is limited to the
        fees you paid for the Software in the twelve months before the claim.
        
        ## 7. Contact
        
        For license keys and commercial terms, contact Kymoscope.
License-File: LICENSE.md
Requires-Python: >=3.10
Requires-Dist: asyncua<3,>=2.1
Requires-Dist: kymoscope>=0.5.1
Requires-Dist: numpy>=1.22
Requires-Dist: pyyaml>=6
Requires-Dist: tzdata; sys_platform == 'win32'
Provides-Extra: ads
Requires-Dist: pyads>=3.4; extra == 'ads'
Provides-Extra: bacnet
Requires-Dist: bacpypes3>=0.0.98; extra == 'bacnet'
Provides-Extra: can
Requires-Dist: cantools>=39; extra == 'can'
Requires-Dist: python-can>=4.3; extra == 'can'
Provides-Extra: cyclers
Requires-Dist: newarenda>=2024; extra == 'cyclers'
Requires-Dist: openpyxl>=3.1; extra == 'cyclers'
Requires-Dist: pyodbc>=5; (sys_platform == 'win32') and extra == 'cyclers'
Provides-Extra: files
Requires-Dist: asammdf>=7.4; extra == 'files'
Requires-Dist: nptdms>=1.7; extra == 'files'
Requires-Dist: pyarrow>=12; extra == 'files'
Provides-Extra: hdf5
Requires-Dist: h5py>=3; extra == 'hdf5'
Provides-Extra: history
Requires-Dist: pyarrow>=12; extra == 'history'
Provides-Extra: influx
Requires-Dist: httpx>=0.27; extra == 'influx'
Provides-Extra: kafka
Requires-Dist: aiokafka<0.15,>=0.10; extra == 'kafka'
Provides-Extra: logix
Requires-Dist: pylogix>=1.0; extra == 'logix'
Provides-Extra: modbus
Requires-Dist: pymodbus<4,>=3.10; extra == 'modbus'
Requires-Dist: pyserial>=3.5; extra == 'modbus'
Provides-Extra: mqtt
Requires-Dist: paho-mqtt<3,>=2; extra == 'mqtt'
Provides-Extra: mtconnect
Requires-Dist: defusedxml>=0.7; extra == 'mtconnect'
Requires-Dist: httpx>=0.27; extra == 'mtconnect'
Provides-Extra: opcda
Requires-Dist: comtypes>=1.4; (sys_platform == 'win32') and extra == 'opcda'
Provides-Extra: rest
Requires-Dist: httpx>=0.27; extra == 'rest'
Provides-Extra: s7
Requires-Dist: python-snap7>=3; extra == 's7'
Provides-Extra: scpi
Requires-Dist: pyvisa-py>=0.7; extra == 'scpi'
Requires-Dist: pyvisa>=1.13; extra == 'scpi'
Provides-Extra: serial
Requires-Dist: pyserial>=3.5; extra == 'serial'
Provides-Extra: sql
Requires-Dist: asyncpg>=0.29; extra == 'sql'
Requires-Dist: oracledb>=2.2; extra == 'sql'
Requires-Dist: pyodbc>=5; extra == 'sql'
Provides-Extra: test
Requires-Dist: pytest-asyncio>=0.23; extra == 'test'
Requires-Dist: pytest-timeout>=2.2; extra == 'test'
Requires-Dist: pytest>=7; extra == 'test'
Provides-Extra: webhook
Requires-Dist: aiohttp>=3.9; extra == 'webhook'
Provides-Extra: windows
Requires-Dist: pywin32>=306; (sys_platform == 'win32') and extra == 'windows'
Description-Content-Type: text/markdown

# kymoscope-collector

Site collector for Kymoscope. It runs on one host in a plant (normally the DMZ), reads industrial protocols, databases, message
brokers, a folder of logger files or your own Python source, encrypts the data and publishes it as Kymoscope streams. Outbound
connections only; the disk buffer holds ciphertext.

## Install

```
pip install kymoscope-collector              # core: OPC UA, CSV files, UDP/TCP, Python sources
pip install "kymoscope-collector[files]"     # + Parquet, TDMS and MF4 files
pip install "kymoscope-collector[windows]"   # + Windows service
pip install "kymoscope-collector[history]"   # + day history (Historian)
pip install "kymoscope-collector[sql]"       # + SQL sources and sinks
```

Requires Python 3.10 to 3.13 (python.org or your distribution; the Microsoft Store Python cannot host the Windows
service and `service install` refuses it).

- **Docker:** `ghcr.io/javijur/kymoscope-collector` (mount a folder with `collector.yaml` at `/data`). The container
  runs as uid 10001, so give it the folder: `sudo chown -R 10001:10001 /srv/kymoscope` (or use `--user`). It stops with
  a clear message when `/data` is not writable.
- **Linux (systemd):** `sudo useradd --system --no-create-home --shell /usr/sbin/nologin kymoscope`, then copy the unit
  file to `/etc/systemd/system/`. It ships in the package at
  `<site-packages>/kymoscope_collector/packaging/kymoscope-collector.service` (and in the repository at
  `packages/collector/packaging/`). Uncomment `ReadWritePaths=` for any folder a files source writes to
  (`ProtectSystem=strict`).
- **Windows service:** runs as LocalSystem by default. For a dedicated account:
  `sc config KymoscopeCollector obj= .\kymoscope password= ...`, plus the "Log on as a service" right (`sc config`
  doesn't grant it), read access to the Python installation and the venv (so not a Python installed under
  `C:\Users`), modify rights on `state_dir` and on every watched folder (its `done` and `failed` folders too), and
  read access to `state_dir\pki\own\key.pem` if it already exists (it is private to the account that created it and
  doesn't inherit: `icacls ...\pki\own\key.pem /grant kymoscope:R`). The service restarts itself after any failure.

### Before installing (for the plant's IT)

- Application control (WDAC, AppLocker, Smart App Control): the service host `pythonservice.exe` from pywin32 is not
  code-signed (`python.exe` is). If unsigned binaries are blocked, allow it by path or hash, use Docker or WSL, or wait
  for the signed installer.
- Outbound TCP 443 to Kymoscope, directly or through an HTTP proxy (below). Nothing inbound.
- Python 3.10 to 3.13, installed for all users if the service runs under a dedicated account. Not the Microsoft Store
  Python.

### Updating

pip can't replace files while the service runs (`WinError 32`), so stop it first:

```
kymoscope-collector service stop --config C:\ProgramData\Kymoscope\collector.yaml
pip install -U "kymoscope-collector[windows]"
kymoscope-collector service start --config C:\ProgramData\Kymoscope\collector.yaml
```

Docker: `docker pull ghcr.io/javijur/kymoscope-collector`, then remove and recreate the container with the same
options. systemd: `systemctl stop kymoscope-collector`, `pip install -U kymoscope-collector` in the unit's environment,
`systemctl start kymoscope-collector`. The buffer and certificates in `state_dir` survive the update.

### Proxy

```yaml
proxy: env:KYMOSCOPE_PROXY      # http://host:port or http://user:password@host:port (percent-encode @ : \)
no_proxy: [historian.plant.local]
```

Both the HTTPS calls and the relay WebSocket tunnel through it (HTTP CONNECT, Basic auth); OPC UA never does. A
Windows service doesn't see a user's Windows proxy settings, so set it here. The log never shows the credentials.

Code-signed installers are coming.

## Quick start (no plant needed)

```
kymoscope-collector simulate                                   # demo OPC UA server on port 4840
kymoscope-collector browse --url opc.tcp://localhost:4840 --out tags.csv
kymoscope-collector run --config collector.yaml
```

A minimal `collector.yaml`:

```yaml
api_key: env:KYMOSCOPE_API_KEY
sources:
  - name: Demo plant
    type: opcua
    url: opc.tcp://localhost:4840
    security: None
    allow_insecure: true      # demo server only
    tags: tags.csv
```

Notes:

- Unknown keys in `collector.yaml` are an error that lists the valid ones; a password with `security: None` is
  allowed but logged as a warning.
- Editing `tags.csv` reconnects the source, so its tags have a short gap.
- One OPC UA source handles about 2,000 tags reporting at distinct millisecond times per second. Split larger tag
  lists into several `type: opcua` sources with different `tags` files.
- The enumeration labels are not sent. Blocks wait in a memory queue bounded by `buffer.memory_mb` (default 256)
  while a stream cannot register; the oldest are dropped and counted in `<source>.queue_dropped`.
- The private key `state_dir/pki/own/key.pem` is created owner-only (0600; on Windows user, SYSTEM and
  Administrators).

Other commands: `cert` (show or renew the collector's certificate), `service install|remove|start|stop`
(Windows), `discover --config collector.yaml --source NAME --out tags.csv` (the tags a configured source offers) and
`backfill --config collector.yaml --source NAME --from 2026-10-01T00:00:00Z --to 2026-10-02T00:00:00Z` (a time range
of a source's history sent into its stream, marked historical, for sources that keep history).

## Day history

With the Historian on your plan and the stream in an end-to-end encrypted project, the collector keeps every value of
its sources for years (`pip install "kymoscope-collector[history]"`):

```yaml
history:
  timezone: Europe/Madrid      # days are cut at the site's midnight
  raw_years: 2                 # raw data kept this many years (your plan may allow fewer); rollups 10 years
  sources: [Plant A DCS]       # omit for every source
  max_local_gb: 50             # cap of each source's local day spool (default 50)
```

Every block a source sends also goes into an encrypted local day spool (`state_dir/history/<source>/days`). About 15
minutes after midnight (`close_after_min`) each day becomes an encrypted raw day file plus 1-minute and 1-hour rollups,
uploaded straight to Kymoscope storage; the local copy goes `keep_local_days` (7) after the upload. Values that arrive
later (a `backfill`, a buffer catching up) make a new version of the day. The day files are sealed with a history key
that only the project's members can open. A stream of an unencrypted project gets no history: the log says so and the
health stream shows `<source>.history_needs_e2ee = 1`.

Disk: the local day spool never fills the disk. Past 90 % of `max_local_gb` the uploaded days go first, oldest first
(before `keep_local_days`); a day not uploaded yet is never deleted for space. When the spool holds `max_local_gb` of
days not uploaded (the Historian off, the stream not registered, a long outage), or the disk would keep less than
`buffer.min_free_mb` free, new blocks are not kept for day history (`<source>.history_spool_refused` counts them; live
data goes on as usual). A day's build waits while the disk has less free than twice its spool
(`<source>.history_disk_low = 1`). Plan for about channels x Hz x 3600 x 12 bytes per day before compression and an
hour of it in memory, 2-3 times over, during a build (builds run one at a time): 1,000 channels at 60 Hz is about
20 GB a day and 2.6 GB per build window, the most one source should carry; split a larger tag list into sources.

Firewall: outbound HTTPS to `<account>.eu.r2.cloudflarestorage.com` (allow `*.eu.r2.cloudflarestorage.com` if your
proxy takes wildcards), besides the usual Kymoscope hosts.

```
kymoscope-collector history status --config collector.yaml
kymoscope-collector history upload --config collector.yaml --source "Plant A DCS" --day 2026-10-20   # now, not at midnight
kymoscope-collector history verify --config collector.yaml --source "Plant A DCS" --day 2026-10-20   # uploaded vs local
```

Own storage (Enterprise): when your organization keeps its day history in its own S3-compatible bucket (set up by
its admin in the app, Organization > Storage), the collector uploads there itself through links signed by Kymoscope,
so allow outbound HTTPS to that bucket instead (https://kymoscope.com/docs/collector-history-storage).

Health channels per source: `history_days_pending`, `history_errors`, `history_needs_e2ee`, `history_spool_errors`,
`history_spool_mb`, `history_spool_refused`, `history_disk_low`, `history_storage_waiting` (1 while new days wait for
the organization's own bucket: not tested yet, or its contract ended), `history_spool_key_resets` (the local spool key could
not be read, e.g. the service reinstalled under another Windows account: it was set aside as `spool.key.bad` and
replaced; days spooled before can't be read locally any more).

## Sinks

Enterprise: every value of your sources can also go to a database you run, in plaintext on your network, besides the
encrypted stream. Each sink keeps a local store-and-forward log, so an unreachable database never slows live data and
catches up later.

```yaml
sinks:
  - name: Plant SQL
    type: sql                        # PostgreSQL/TimescaleDB, SQL Server, MySQL/MariaDB, Oracle (extra `sql`)
    dsn: env:PLANT_SQL_DSN           # postgresql://kymo_writer:password@db.plant.local:5432/plant
    table: kymoscope_values          # created when missing: ts (UTC), source, channel, value
  - name: Plant Influx
    type: influx                     # InfluxDB 2.x (api: v2) or 1.x (api: v1)
    url: https://influx.plant.local:8086
    token: env:INFLUX_TOKEN
    org: plant
    bucket: kymoscope
```

Health channels per sink: `sink_backlog_mb`, `sink_connected`, `sink_rows_written`, `sink_rows_rejected`, `sink_dropped`, `sink_errors`, `sink_not_in_plan`. SQL Server and MySQL/MariaDB need their
ODBC driver on the host; the Docker image has unixODBC but not those drivers. Full reference:
https://kymoscope.com/docs/collector-sinks

## Source types

Each `sources` entry has a `type`. Install the extra for the types you use; extras combine
(`pip install "kymoscope-collector[mqtt,sql]"`).

| `type` | Reads | pip extra |
|---|---|---|
| `opcua` | OPC UA servers (with history backfill) | core |
| `opcda` | OPC DA (Classic) servers, Windows only | `opcda` (+ `windows`) |
| `mqtt` | MQTT brokers, Unified Namespace, Sparkplug B | `mqtt` |
| `modbus` | Modbus TCP / RTU, SunSpec inverters | `modbus` |
| `can` | CAN / CAN FD buses decoded with a DBC | `can` |
| `logix` | Rockwell ControlLogix, CompactLogix, Micro800 | `logix` |
| `s7` | Siemens S7-300/400/1200/1500 | `s7` |
| `ads` | Beckhoff TwinCAT PLCs by symbol name | `ads` |
| `bacnet` | BACnet/IP building automation | `bacnet` |
| `serial` | RS-232/RS-485 or serial-server instruments, line by line | `serial` |
| `scpi` | SCPI/VISA instruments (LAN, GPIB, USB) | `scpi` |
| `udp` | Binary struct packets or JSON/CSV lines over UDP or TCP | core |
| `kafka` | Kafka topics: JSON, raw numbers, Sparkplug B | `kafka` |
| `mtconnect` | CNC machines through an MTConnect agent | `mtconnect` |
| `influx` | InfluxDB 2.x (Flux) or 1.x (InfluxQL) | `influx` |
| `sql` | Databases; AVEVA Historian, IP.21 and Timescale presets (validate on your system) | `sql` |
| `rest` | REST APIs; Canary and Proficy presets (validate on your system) | `rest` |
| `webhook` | Receives pushed JSON | `webhook` |
| `files` | CSV (core), Parquet, TDMS, MF4; battery cyclers (Neware, BioLogic, Arbin, Maccor, Basytec, Digatron) and HDF5 | `files` for non-CSV; `cyclers`, `hdf5` |
| `python` | Your own adapter class | core |

Anything else (IEC 60870-5-104, IEC 61850, DNP3, Mitsubishi MC, Omron FINS...): expose it as OPC UA or MQTT with a
gateway (Kepware, Ignition Edge, Matrikon, Node-RED, Telegraf...).
`discover` and `backfill` (above) work with any source type that supports them.

## Documentation

- Install, `collector.yaml` reference, certificates, security, health, troubleshooting, and "Which source do I use?":
  https://kymoscope.com/docs/collector
- Per source: `/docs/collector-opcda`, `-opcua-history`, `-mqtt`, `-can`, `-modbus`, `-logix`, `-s7`, `-ads`, `-bacnet`,
  `-serial`, `-udp`, `-sql`, `-rest`, `-kafka`, `-mtconnect`, `-influx`, `-cyclers`, and
  `/docs/collector-gateways` for protocols the collector does not read natively.
