Metadata-Version: 2.4
Name: labrat_project
Version: 0.9.1
Summary: A simple framework to log experimental conditions and to control instrumentation
License-File: LICENSE
Author: Mike Parkes
Author-email: michael.parkes@ucl.ac.uk
Requires-Python: >=3.12
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Dist: flask (==3.1.2)
Requires-Dist: flask-wtf (>=1.2.2,<2.0.0)
Requires-Dist: jsonschema (>=4.26.0,<5.0.0)
Requires-Dist: matplotlib (>=3.11.0,<4.0.0)
Requires-Dist: paho-mqtt (==2.1.0)
Requires-Dist: pillow (>=12.2.0,<13.0.0)
Requires-Dist: plotly (>=6.8.0,<7.0.0)
Requires-Dist: pyserial (==3.5)
Requires-Dist: toml (==0.10.2)
Description-Content-Type: text/markdown

# Labrat Laboratory Monitoring

![image](/src/images/bit.png)

[![PyPI version](https://img.shields.io/pypi/v/LabRat-project?pypiBaseUrl=https%3A%2F%2Fpypi.org%2F&style=flat&logo=pypi&logoColor=forest&logoSize=auto&label=PyPi&labelColor=forest&color=forest&cacheSeconds=3600)](https://pypi.org/project/labrat_project/)
[![pipeline status](https://gitlab.com/inator-project/LabRat/badges/master/pipeline.svg)](https://gitlab.com/inator-project/LabRat/-/pipelines)
[![coverage](https://gitlab.com/inator-project/LabRat/badges/master/coverage.svg)](https://gitlab.com/inator-project/LabRat/-/commits/master)

Connect sensors and log data to a database, file, or both.

Display the data from this database in a dashboard

Current connection modes available:

| Connection    | Description                        |
|---------------|------------------------------------|
| `MQTT`        | Subscribe to an MQTT broker        |
| `serial`      | Read from a wired serial port      |
| `http`        | Receive data via a Flask API       |

## Table of Contents

- [Requirements](#requirements)
- [Quick Start](#quick-start)
- [All Options](#all-options)
- [Setup & First Run](#first-run)
  - [1. Create a Secrets File](#1-create-a-secrets-file)
  - [2. Set Up Database Contents](#2-set-up-database-contents)
  - [3. Run](#3-run)
- [Running as a Daemon](#running-as-a-daemon)

---

## Requirements

To run LabRat you need to have python installed. Please install via your preferred method, we advise creating an environment.

For example using conda or python's built in venv. Use of a manager such as poetry is advantagous.

LabRat can be installed using poetry,

```bash
poetry install
```

 pip,

```bash
pip install -r /path/to/requirements.txt
```

 or conda/mamba.

 ```bash
conda env create -f environment.yml
 ```

Download the files from the repo and run the above. LabRat also exists as a [PyPi Package](https://pypi.org/project/labrat_project/)

In the installation folder can be found a conda environment.yml file and a pip requirements.txt file.

## Quick Start

Not sure where to begin?
There are two options:
[Create a Secrets File](#1-create-a-secrets-file).
The minimum secrets file is:

```toml
VERSION = 1.0

[CONFIG]
    "AUTOBROW" = true
    "ALLOW_SETUP" = true

[CONNECTIONS]

   [CONNECTIONS.FLASK]
   "KEY" = "" # this can be filled with anything for this step.
       [CONNECTIONS.FLASK.SETUP]
       "ROUTE" = "/setup"
```

A copy of this is in the installation folder

then run the setup wizard.

```bash
python labrat_log.py -secrets "mypath/secrets.toml" -quick_start
```

or run this command

```bash
 python labrat_log.py -quick_start --config $'VERSION = 1.0\n\n[CONFIG]\nAUTOBROW = true\nALLOW_SETUP = true\n[CONNECTIONS]\n[CONNECTIONS.FLASK]\nKEY = "secret"\nLOCAL_ON = true\n[CONNECTIONS.FLASK.SETUP]\nROUTE = "/setup"'
```

This will open a browser window in which you can give a name for the database and add inator devices.

This can be used to create a database or to add devices to an existing database

When you have finished close the browser window and then turn off the flask server

```bash
ctl+c
```

---

## All Options

```bash
usage: labrat_log.py [-h] [-sql SQL] [-secrets SECRETS] [-quick_start]
```

| Flag                  | Description                                                                 |
|-----------------------|-----------------------------------------------------------------------------|
| `-secrets SECRETS`    | Path to the TOML secrets file (logins, broker details, etc.)                |
| `-quick_start`        | Run the quick start wizard 🧙                                               |

---

## First Run

### 1. Create a Secrets File

Once you have run the quick start to create a database and add a device, then move on to the following:

Create a `.toml` file with your credentials and connection details:
This has the following main sections which have their own sub-sections

```toml
[CONFIG]
[LOGGING] # Options to control logging
[CONNECTIONS] # Options to control the connections
```

These sections are then further broken up

<details>
<summary><strong>Config TOML options</strong></summary>

```toml
[CONFIG]
    "AUTOBROW" = false # Whether to automatically open the browser or not
    "ALLOW_SETUP" = false # Whether to allow database set up and addition of devices or not
    "COMMAND_QUEUE" = false # whether to allow a command COMMAND_QUEUE to run

[LOGGING]
    [LOGGING.SQLITE] # Settings for logging to SQLite databases 
      [LOGGING.SQLITE.NAME] # Details of a SQLITE database NAME must be unique
        "FILE" = "" # The path to sql database file
        "CREATE" = false # Whether to create the DB or not (true for quick set up)

    [LOGGING.FILE]  # Settings for logging to a text file
      [LOGGING.FILE.NAME]  # Details for the text file, NAME must be unique
        "FILE" = "" # Path for the file
        "CREATE" = false # Whether to create the file or append

    [LOGGING.ELN] #Settings for connecting to an Electronic Laboratory notebook
      [LOGGING.ELN.NAME]
        "TYPE" = "RSpace" # Type of the ELN, e.g an RSpace or LabArchives instance
        "USERNAME" = "" # If required the Username to connect to ELN
        "KEY" = ""  # If required a key to logging to the ELN

[CONNECTIONS]
    
    [CONNECTIONS.SOCKET_CLI]  # Socket-based CLI, an alternative to the interactive
    # terminal (TTY) for headless/background runs. Multiple instances allowed,
    # Requires CONFIG.COMMAND_QUEUE = true to actually run.
        [CONNECTIONS.SOCKET_CLI.NAME] # Details of a Socket CLI connection, NAME should be changed and be unique
            "HOST" = "127.0.0.1" # Interface to bind to. Keep as loopback unless you specifically need remote access
            "PORT" =  # Port to listen on
            "TOKEN" = "" # Shared secret clients send as 'AUTH <token>' to authenticate, see below for how to generate*

    [CONNECTIONS.MQTT] # MQTT connection details
        [CONNECTIONS.MQTT.NAME] # Details of a MQTT connection NAME should be changed and be unique
            "VERB" = "false" # If set to true will print whatever is recieved by connection
            "USERNAME" = "" # Any user names needed by broker
            "KEY" = ""  # Key needed to connect to broker
            "BROKER" = "" # Address of the broker
            "PORT" = 1883 # The port number of the broker, 8883 is traditional
            "TOPIC" = ""  # The Topic to subscribe to, adding # is normally useful
            "TLS" = true  # Whether the MQTT broker is using a TLS connection, i.e is encrypted. true or false
    
    [CONNECTIONS.SERIAL]  # Serial connection details
        [CONNECTIONS.SERIAL.NAME] # Details of a Serial connection NAME should be changed and be unique
            "VERB" = "false" # If set to true will print whatever is recieved by connection
            "PORT" = "" # The name of the serial port
            "BAUD" = "" # The baud rate for the port

   [CONNECTIONS.FLASK]  # Flask server connection details
   # Due to its nature Flask is treated differently to other connections, only one server for all routes
   "KEY" = "" # The key to connect to http logging, see below for how to generate*
   "LOCAL_ON" = true  # Whether to run the routes through Flask, only safe on a local network
   "HOST" = ""  # Host address to use
   "PORT" =   # Port number to use
      [CONNECTIONS.FLASK.HTTP] # Route information for the HTTP server
        "ROUTE" = "/datalog" # The route to the HTTP logging server
        "KEY" = "" # The key to connect to this route, if different from main key, see below for how to generate*
      [CONNECTIONS.FLASK.DASH] # Route information for the display dashboard 
        "ROUTE" = "/dash"  # The route to the display dashboard
        "KEY" = "" # The key to connect to this route, if different from main key, see below for how to generate*
      [CONNECTIONS.FLASK.SETUP]  # Route information for the quick set up page
        "ROUTE" = "/setup" # The route to the quick setup page
        "KEY" = "" # The key to connect to this route, if different from main key, see below for how to generate*
      # API routes - no html page
      [CONNECTIONS.FLASK.DEVICES] # Route to the add devices api
        "ROUTE" = "/api/devices"  # The route for the add devices
        "KEY" = "" # The key to connect to this route, if different from main key, see below for how to generate*
      [CONNECTIONS.FLASK.UPDATE_CONNS] # Route information for Connections control api
        "ROUTE" = "api/conns" # The route for the connections control
        "KEY" = "" # The key to connect to this route, if different from main key, see below for how to generate*
      [CONNECTIONS.FLASK.PUBLISH_API]
        "ROUTE" = "api/publish" # The route for publishing via a connection
        "KEY" = "" # The key to connect to this route, if different from main key, see below for how to generate*
```

</details>
These options can be given as a toml string on the cli, any values given on the cli will overwrite those from the config file

All connection methods can run in parallel

\* To generate a key use the following python code:

```python
import secrets

print(secrets.token_urlsafe(32))

```

or directly on the command line

```bash

python -c 'import secrets; print(secrets.token_urlsafe(32))'
```

### 2. Set Up Database Contents

<!-- If logging to SQLite, initialise the database with your device configuration:

```bash
# Create a new DB and load device info from a JSON file
python labrat_log.py -secrets "secrets.toml" -sql "/abs/path/to/data.db" \
    -setupsql --createdb --dev_file "/abs/path/to/devices.json" mqtt
```

> Omit `--createdb` if the database already exists. -->

Add devices to the database.

The device file must conform to the [Inator DB schema](https://inator-project.org/Protocols/pro_database.html).
This can be done using the quick start page

<details>
<summary><strong>Device file JSON schema</strong></summary>

```json
{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "devices": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "device_name":     { "type": "string" },
          "device_guid":     { "type": "string" },
          "num_sensors":     { "type": "integer" },
          "device_info":     { "type": "string" },
          "device_type":     { "type": "string" },
          "device_location": { "type": "string" },
          "device_active":   { "type": "integer" },
          "connection": {
            "type": "string",
            "enum": ["Serial", "MQTT", "Other"]
          },
          "sensors": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "sens_name": { "type": "string" },
                "measures":  { "type": "string" },
                "returns":   { "type": "string" },
                "calib":     { "type": "string" },
                "range":     { "type": "string" },
                "info":      { "type": "string" },
                "comments":  { "type": "string" }
              },
              "required": ["sens_name", "measures", "returns", "calib", "range", "info", "comments"]
            }
          }
        },
        "required": [
          "device_name", "device_guid", "num_sensors", "device_info",
          "device_type", "device_location", "device_active", "connection", "sensors"
        ]
      }
    }
  },
  "required": ["devices"]
}
```

</details>

### 3. Run

To run LabRat put the options in the settings/secrets file for what logging, connections and displays you need.

Then depending on your installationn method run using:

```bash
python labrat_log.py -secrets path/to/secrets.toml
```

or

```bash
poetry run python labrat_log.py -secrets path/to/secrets.toml
```

---

## Data from a device

Once connections are set up and a device is registered in the database then any data sent on the connection will be logged.

The format is simple and extendable. A JSON string is sent (i.e has {} around the contents) in this format

```json
{
  "inator": "name",  # the name of the device, must match name added to database
  "<sensor1>": "value", the value from the sensor, name must match name used in database for the sensor
  "<sensor2>": "value",
  "<sensor3>": "value",
  .
  .
  .
  "<sensorN>": "value",
}
```

As long as the names used match those entered into the database the logging will work

---

## Running as a Daemon

Run the logger as a persistent background service using [Supervisor](http://supervisord.org/).

### 1. Install

```bash
sudo apt update && sudo apt install supervisor
```

### 2. Configure

Edit the provided `supervisord.conf` file, updating these fields:

| Field             | What to set                                                   |
|-------------------|---------------------------------------------------------------|
| `command=`        | Full command with all required paths and flags                |
| `user=`           | System user to run the process as                             |
| `stdout_logfile=` | Absolute path and filename for log output                     |
| `directory=`      | Working directory (home directory for MQTT)                   |

### 3. Start

```bash
supervisord -c /path/to/supervisord.conf
```

### 4. Monitor & Control

Use [`supervisorctl`](https://supervisord.org/running.html#running-supervisorctl) to manage the process:

```bash
supervisorctl status          # check if running
supervisorctl restart labrat  # restart the service
supervisorctl stop labrat     # stop the service
```

### 5. Testing

If you want to run the code tests (which are currently a work in progress) then make sure pytest and pytest-cov are installed (they are in the dev group).

Then run

```bash
poetry run pytest
```

### 6. Running with a WSGI Server

If you would like to run behind a WSGI server, solely for the Flask connections, then do the following

```bash
gunicorn --workers 1 --threads 8 --worker-class gthread -b 0.0.0.0:8090 src.setup.wsgi:app
```

to run as a Daemon (i.e in the background)

```bash
gunicorn --workers 1 --threads 8 --worker-class gthread \
  -b 0.0.0.0:8090 \
  src.setup.wsgi:app \
  --daemon \
  --pid ./gunicorn.pid \
  --access-logfile ./access.log \
  --error-logfile ./error.log
```

To end the Daemon when it is running

```bash
kill -TERM $(cat gunicorn.pid)
```

