Metadata-Version: 2.4
Name: pywis-pubsub
Version: 0.13.0
Summary: pywis-pubsub provides Publish-Subscribe capabilities in support of WIS2 message notification, processing and data download
Author-email: Tom Kralidis <tomkralidis@gmail.com>
Maintainer-email: Tom Kralidis <tomkralidis@gmail.com>
License-Expression: Apache-2.0
Project-URL: homepage, https://github.com/World-Meteorological-Organization/pywis-pubsub
Project-URL: source, https://github.com/World-Meteorological-Organization/pywis-pubsub
Project-URL: documentation, https://github.com/World-Meteorological-Organization/pywis-pubsub
Project-URL: issues, https://github.com/World-Meteorological-Organization/pywis-pubsub/issues
Keywords: WIS2,Pub/Sub,broker,topic
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click
Requires-Dist: jsonschema
Requires-Dist: paho-mqtt
Requires-Dist: pyyaml
Requires-Dist: requests
Requires-Dist: shapely
Provides-Extra: dev
Requires-Dist: flake8; extra == "dev"
Provides-Extra: test
Requires-Dist: requests; extra == "test"
Provides-Extra: release
Requires-Dist: build; extra == "release"
Requires-Dist: twine; extra == "release"
Requires-Dist: wheel; extra == "release"
Provides-Extra: s3
Requires-Dist: boto3; extra == "s3"
Dynamic: license-file

[![flake8](https://github.com/World-Meteorological-Organization/pywis-pubsub/workflows/flake8/badge.svg)](https://github.com/World-Meteorological-Organization/pywis-pubsub/actions)
[![test-publish-subscribe-download](https://github.com/World-Meteorological-Organization/pywis-pubsub/workflows/test-publish-subscribe-download/badge.svg)](https://github.com/World-Meteorological-Organization/pywis-pubsub/actions)

# pywis-pubsub

## Overview

pywis-pubsub provides Publish-Subscribe capabilities in support of WIS2 message notification, processing and data download.

## Installation

The easiest way to install pywis-pubsub is via the Python [pip](https://pip.pypa.io) utility:

```bash
# default install
pip3 install pywis-pubsub

# install with storage spport for S3
pip3 install pywis-pubsub[backend-s3]
```

### Requirements
- Python 3
- [virtualenv](https://virtualenv.pypa.io)

### Dependencies
Dependencies are listed in [pyproject.toml](pyproject.toml). Dependencies
are automatically installed during pywis-pubsub installation.

#### Windows installations
Note that you will need Cython and [Shapely Windows wheels](https://pypi.org/project/shapely/#files) for windows for your architecture prior to installing pywis-pubsub.

### Installing pywis-pubsub

```bash
# setup virtualenv
python3 -m venv --system-site-packages pywis-pubsub
cd pywis-pubsub
source bin/activate

# clone codebase and install
git clone https://github.com/World-Meteorological-Organization/pywis-pubsub.git
cd pywis-pubsub
pip3 install .
```

## Running

First check pywis-pubsub was correctly installed

```bash
pywis-pubsub --version
```

Create configuration

```bash
cp pywis-pubsub-config.yml local.yml
vim local.yml # update accordingly to configure subscribe options
```


### Subscribing


```bash
# sync WIS2 notification and monitoring event schemas
pywis-pubsub bundle sync

# connect, and simply echo messages
pywis-pubsub subscribe --config local.yml

# subscribe, and download data from message
pywis-pubsub subscribe --config local.yml --download

# subscribe, and filter messages by geometry
pywis-pubsub subscribe --config local.yml --bbox=-142,42,-52,84

# subscribe, and filter messages by geometry, adjust debugging verbosity
pywis-pubsub subscribe --config local.yml --bbox=-142,42,-52,84 --verbosity=DEBUG
```

### WIS2 Notification Message (WNM) workflow

#### Validating a message and verifying data

```bash
# validate a message
pywis-pubsub wnm ets validate /path/to/message1.json

# verify data from a message
pywis-pubsub wnm ets verify /path/to/message1.json

# validate WNM against abstract test suite (file on disk)
pywis-pubsub wnm ets validate /path/to/file.json

# validate WNM against abstract test suite (URL)
pywis-pubsub wnm ets validate https://example.org/path/to/file.json

# validate WNM against abstract test suite (URL), but turn JSON Schema validation off
pywis-pubsub wnm ets validate https://example.org/path/to/file.json --no-fail-on-schema-validation

# key performance indicators

# set environment variable for GDC URL
export PYWIS_PUBSUB_GDC_URL=https://api.weather.gc.ca/collections/wis2-discovery-metadata

# all WNM key performance indicators at once
pywis-pubsub wnm kpi validate https://example.org/path/to/file.json --verbosity DEBUG

# all WNM key performance indicators at once, but turn ETS validation off
pywis-pubsub wnm kpi validate https://example.org/path/to/file.json --no-fail-on-ets --verbosity DEBUG

# all WNM key performance indicators at once, in summary
pywis-pubsub wnm kpi validate https://example.org/path/to/file.json --verbosity DEBUG --summary

# selected WNM key performance indicator
pywis-pubsub wnm kpi validate --kpi metadata_id /path/to/file.json -v INFO
```

#### Publishing

```bash
cp pub-config-example.yml pub-local.yml
vim pub-local.yml # update accordingly to configure publishing options

# example publishing a WIS2 notification message with attributes: 
# data-url=http://www.meteo.xx/stationXYZ-20221111085500.bufr4 
# lon,lat,elevation=33.8,11.8,112
# wigos_station_identifier=0-20000-12345
pywis-pubsub wnm publish --topic origin/a/wis2/centre-id/data/core/weather --config pub-local.yml -u https://example.org/stationXYZ-20221111085500.bufr4 -g 33.8,-11.8,8.112 -w 0-20000-12345

# publish a message with a WCMP2 metadata id
pywis-pubsub wnm publish --topic origin/a/wis2/centre-id/data/core/weather --config pub-local.yml -u https://example.org/stationXYZ-20221111085500.bufr4 -g 33.8,-11.8,8.112 -w 0-20000-12345 --metadata-id "x-urn:wmo:md:test-foo:htebmal2001"

# publish a message with a datetime (instant)
pywis-pubsub wnm publish --topic origin/a/wis2/centre-id/data/core/weather --config pub-local.yml -u https://example.org/stationXYZ-20221111085500.bufr4 -g 33.8,-11.8,8.112 -w 0-20000-12345 --metadata-id "x-urn:wmo:md:test-foo:htebmal2001" --datetime 2024-01-08T22:56:23Z

# publish a message with a start and end datetime (extent)
pywis-pubsub wnm publish --topic origin/a/wis2/centre-id/data/core/weather --config pub-local.yml -u https://example.org/stationXYZ-20221111085500.bufr4 -g 33.8,-11.8,8.112 -w 0-20000-12345 --metadata-id "x-urn:wmo:md:test-foo:htebmal2001" --datetime 2024-01-08T20:56:23Z/2024-01-08T22:56:43Z

# publish a message as a data update
pywis-pubsub wnm publish --topic origin/a/wis2/centre-id/data/core/weather --config pub-local.yml -u https://example.org/stationXYZ-20221111085500.bufr4 -g 33.8,-11.8,8.112 -w 0-20000-12345 --metadata-id "x-urn:wmo:md:test-foo:htebmal2001" --operation update

# publish a message as a data deletion
pywis-pubsub wnm publish --topic origin/a/wis2/centre-id/data/core/weather --config pub-local.yml -u https://example.org/stationXYZ-20221111085500.bufr4 -g 33.8,-11.8,8.112 -w 0-20000-12345 --metadata-id "x-urn:wmo:md:test-foo:htebmal2001" --operation delete

# publish a message from file on disk
pywis-pubsub wnm publish --topic origin/a/wis2/centre-id/data/core/weather --config pub-local.yml --wnm my_message.json
```

#### Using the API

Python examples:

Validating a WNM
```python

from pywis_pubsub.wnm.ets import WNMTestSuite

try:
    ts = WNMTestSuite(wnm_dict)
    _ = ts.run_tests(fail_on_schema_validation=True)
except Exception as err:
    print(f'Error: {err}')
```

Subscribing to a WIS2 Global Broker
```python
from pywis_pubsub.mqtt import MQTTPubSubClient

options = {
    'storage': {
        'type': 'fs',
        'basedir': '/tmp'
    },
    'bbox': [-90, -180, 90, 180]
}
topics = [
    'topic1',
    'topic2'
]

m = MQTTPubSubClient('mqtt://localhost:1883', options)
# example with credentials
# m = MQTTPubSubClient('mqtt://username:password@localhost:1883', options)

m.sub(topics)
```

Publishing a WIS2 Notification Message
```python
from datetime import datetime, timezone

from pywis_pubsub.mqtt import MQTTPubSubClient
from pywis_pubsub.wnm.publish import create_message
from pywis_pubsub.wnm.ets import WNMTestSuite, WNMKeyPerformanceIndicators

url_info = get_url_info('http://www.meteo.xx/stationXYZ-20221111085500.bufr4')

message = create_message(
        topic='foo/bar',
        content_type='application/bufr',
        url_info=url_info,
        identifier='stationXYZ-20221111085500', 
        datetime_=datetime.now(timezone.utc),
        geometry=[33.8, -11.8, 123],
        metadata_id='x-urn:wmo:md:test-foo:htebmal2001',
        wigos_station_identifier='0-20000-12345',
        operation='update'
)

m = MQTTPubSubClient('mqtt://localhost:1883')
m.pub(topic, json.dumps(message))
```

Running KPIs
```pycon
>>> ts = WNMTestSuite(message)
>>> ts.run_tests()  # raises ValueError error stack on exception
>>> ts.raise_for_status()  # raises pywis_pubsub.errors.TestSuiteError on exception with list of errors captured in .errors property
>>> # test KPI
>>> import json
>>> from pywis_pubsub.kpi import WNMKeyPerformanceIndicators
>>> with open('/path/to/file.json') as fh:
...     data = json.load(fh) >>> kpis = WNMKeyPerformanceIndicators(data)
>>> results = kpis.evaluate()
>>> results['summary']
```

### WIS2 Monitoring Events (WME) workflow


#### Validating a message

```bash
# validate WMEM against abstract test suite (file on disk)
pywis-pubsub wmem ets validate /path/to/file.json

# validate WMEM against abstract test suite (URL)
pywis-pubsub wmem ets validate https://example.org/path/to/file.json

# validate WNM against abstract test suite (URL), but turn JSON Schema validation off
pywis-pubsub wmem ets validate https://example.org/path/to/file.json --no-fail-on-schema-validation
```

#### Publishing

```bash
# publish a message from file on disk
pywis-pubsub wmem publish --topic monitor/a/wis2/centre-id --config pub-local.yml --wmem my_message.json
```

#### Using the API

```python

from pywis_pubsub.wmem.ets import WMEMTestSuite

try:
    ts = WMEMTestSuite(wmem_dict)
    _ = ts.run_tests(fail_on_schema_validation=True)
except Exception as err:
    print(f'Error: {err}')
```

## Development

### Running Tests

```bash
pip3 install ".[tests]"
python3 tests/run_tests.py
```

## Releasing

```bash
# create release (x.y.z is the release version)
vi pyproject.toml  # update [project]/version
git commit -am 'update release version x.y.z'
git push origin main
git tag -a x.y.z -m 'tagging release version x.y.z'
git push --tags

# upload to PyPI
rm -fr build dist *.egg-info
python3 -m build
twine upload dist/*

# publish release on GitHub (https://github.com/World-Meteorological-Organization/pywis-pubsub/releases/new)

# bump version back to dev
vi pyproject.toml  # update [project]/version
git push origin main
```

### Code Conventions

* [PEP8](https://www.python.org/dev/peps/pep-0008)

### Bugs and Issues

All bugs, enhancements and issues are managed on [GitHub](https://github.com/World-Meteorological-Organization/pywis-pubsub/issues).

## Contact

* [Antje Schremmer](https://github.com/antje-s)
* [Tom Kralidis](https://github.com/tomkralidis)
* [Maaike Limper](https://github.com/maaikelimper)
