Changelog
=========

.. changelog::
  :version: 2.1.1
  :released: Aug 13th, 2024

  .. change::
    :tags: Improvement
    :pullreq: 339

    dstore-dist: Fix issue with storage destination when no batch_buffer_size is configured

  .. change::
    :tags: Improvement
    :pullreq: 340

    Add supervisord to dstore images for control by Cloud Control agent

  .. change::
    :tags: Improvement
    :pullreq: 338

    topn-reporter: Fix logging issues in topn reporter

  .. change::
    :tags: Improvement
    :pullreq: 335

    eventforwarder: Fix elastic wait loop issue and add option to append date to elastic index name

  .. change::
    :tags: Improvement
    :pullreq: 336

    dstore-dist: Detect half-closed connections immediately for pdns destinations

.. changelog::
  :version: 2.1.0
  :released: Jul 8th, 2024

  .. change::
    :tags: Improvement
    :pullreq: 332

    dstore-dist: configurable instance name for kafka destinations to allow source dstore-dist instance to be identified

  .. change::
    :tags: Improvement
    :pullreq: 326

    eventforwarder: new daemon to send filtered events to Elasticsearch for reporting purposes

  .. change::
    :tags: Improvement
    :pullreq: 329

    dstore-dist: sasl auth support for kafka destinations (plain, scram-256, scram-512)

  .. change::
    :tags: Improvement
    :pullreq: 325

    dstore-dist: add new prometheus metrics to count filter hits and accepted/rejected/received rates/sec

  .. change::
    :tags: Improvement
    :pullreq: 324

    dstore-dist: new IPset filter to filter based on source IP/prefix

  .. change::
    :tags: Improvement
    :pullreq: 327

    dstore-dist: new storage destination supporting local files or S3, and a variety of encodings including JSON, protobuf or Bind query logging

  .. change::
    :tags: Improvement
    :pullreq: 323

    security: SLSA Level 3 for images (provenance, SBOM) and image signing

  .. change::
    :tags: Improvement
    :pullreq: 322

    dstore-dist: support different framing options for outbound messages (pdns destinations only)

.. changelog::
  :version: 2.0.0
  :released: Dec 5th, 2023

  .. change::
    :tags: Improvement
    :pullreq: 389

    packaging: Enterprise Linux 9 Support and many python fixes
    packaging: Production image for dstore-dist and top-n-reporter uploaded to OX registry in CI

  .. change::
    :tags: Improvement
    :pullreq: 311

    tests: Add and update dstore-dist tests in CI

 .. change::
    :tags: Refactoring
    :pullreq: 318

    cleanup: Remove legacy dstore code leaving only dstore-dist and related tools

 .. change::
    :tags: Improvement
    :pullreq: 313

    security: Add features outstanding from Nixu code-audit

 .. change::
    :tags: Improvement
    :pullreq: 316

    logging: Add structured logging to dstore-dist

 .. change::
    :tags: Improvement
    :pullreq: 319

    api: Add history API to dstore-dist for debugging

.. changelog::
  :version: 1.5.0
  :released: May 3rd, 2023

  .. change::
    :tags: Improvement
    :pullreq: 384

    python: update ``requirements.txt`` to eliminate setuptools failures

  .. change::
    :tags: Improvement
    :pullreq: 380, 382, 383

    docs: upload docs to docs.powerdns.com as part of build

  .. change::
    :tags: New features
    :pullreq: 378

    dstore-dist: compression support for Kafka

  .. change::
    :tags: Improvement
    :pullreq: 378

    go: Update Go to v1.20.2

  .. change::
    :tags: New features
    :pullreq: 381

    dstore-dist: connect and write timeouts for PDNS destinations

.. changelog::
  :version: 1.4.4
  :released: October 20nd, 2022

  .. change::
    :tags: Improvement
    :pullreq: 371

    dnspb2json:
      * Include the dnspb2json tool as dstore-dist-dnspb2json in the dstore-dist RPM package.
      * -b option to record raw protobuf streams instead of outputting JSON
      * -L option to output one JSON object per line for easy grep and processing

  .. change::
    :tags: Improvement
    :pullreq: 374

    dstore-dist: null Kafka key for better load balancing

  .. change::
    :tags: Improvement
    :pullreq: 370

    ev_aggregator: fix elasticsearch webhook username/password config flag parsing

  .. change::
    :tags: Improvement
    :pullreq: 373

    dstore-dist: add filters and JSON output for new protobuf fields

    New filters include:

      * `tag_prefix`
      * `socket_protocol`
      * `is_newly_observed_domain`
      * `policy_type`
      * `policy_kind`
      * `validation_state` and `is_validation_state_bogus`
      * `trace_event` and `trace_match` (see event-trace-enabled recursor setting)
      * `meta_key`, `meta_key_string` and `meta_key_int`

    The example config has been updated with examples for the new filters.

    dnspb2json now also supports a -d option that will output JSON in the same format
    as dstore-dist uses to write to Kafka queues. This format is different from the default
    format, e.g. it uses snake_case instead of camelCase.

.. changelog::
  :version: 1.4.3
  :released: April 22nd, 2022

.. changelog::
  :version: 1.4.3-rc1
  :released: March 18th, 2022

  .. change::
    :tags: Improvement
    :pullreq: 366

    dstore-dist: Support appending arbitrary tags to messages (route specific)

  .. change::
    :tags: Improvement
    :pullreq: 367

    dstore-dist: Build and test on Oracle Linux 8 instead of Centos 8

  .. change::
    :tags: Improvement
    :pullreq: 365

    dstore-dist: Add TLS support for both inbound and outbound traffic
    topn: Add TLS support for inbount traffic

  .. change::
    :tags: Improvement
    :pullreq: 363

    report: Support reporting not just on a per-user basis, but also per-device.

  .. change::
    :tags: Improvement
    :pullreq: 361

    dstore-ev-aggregator: add support for (now mandatory) "cat:" and "rule:" prefixes while reading tag list

  .. change::
    :tags: Improvement, Dev
    :pullreq: 364, 369

    dstore-dist, topn: Add them configured to the development enviroment

  .. change::
    :tags: Bug fixes, Dev
    :pullreq: 362

    Fix and update regression tests

.. changelog::
  :version: 1.4.2
  :released: October 15, 2021

.. changelog::
  :version: 1.4.2-rc1
  :released: October 1, 2021

  .. change::
    :tags: Improvement
    :pullreq: 359

    top-n: support reporting by source IP

  .. change::
    :tags: Improvement
    :pullreq: 349

    top-n: improve templates and provide example configuration files for dstore-dist, topn-reporter and kibana

  .. change::
    :tags: Improvement
    :pullreq: 343

    ev_aggregator: performance improvements

  .. change::
    :tags: Improvement
    :pullreq: 357

    dstore-dist: add is_incoming_response and is_outgoing_query filters

  .. change::
    :tags: Improvement
    :pullreq: 348

    dstore-dist: exit with non-zero code upon error at startup

  .. change::
    :tags: Improvement
    :pullreq: 353

    golang: upgrade protobuf dependency to a more recent version

  .. change::
    :tags: Documentation
    :pullreq: 352

    egateway: document HTTP API

  .. change::
    :tags: Documentation
    :pullreq: 350

    Add a documentation target to the CI

.. changelog::
  :version: 1.4.1
  :released: March 12, 2021

.. changelog::
  :version: 1.4.1-beta2
  :released: February 23, 2021

.. changelog::
  :version: 1.4.1-beta1
  :released: February 23, 2021

  .. change::
    :tags: New features
    :pullreq: 346

    dstore-dist: add file and syslog destinations support for logging

  .. change::
    :tags: Improvements
    :pullreq: 345

    dstore-dist: improve unit tests for protobuf/kafka integration

.. changelog::
  :version: 1.4.0
  :released: January 14, 2021

.. changelog::
  :version: 1.4.0-beta3
  :released: December 18, 2020

  .. change::
    :tags: Bug Fixes
    :pullreq: 344

    build: update alpine linux base images and centos fixes

.. changelog::
  :version: 1.4.0-beta2
  :released: December 16, 2020

  .. change::
    :tags: New features
    :pullreq: 342

    dstore-dist: add support and tooling to measure and report Top N domains

.. changelog::
  :version: 1.4.0-beta1
  :released: November 27, 2020

  .. change::
    :tags: New features
    :pullreq: 339

    dstore-dist: support writing dns messages as JSON for Kafka destinations

  .. change::
    :tags: Improvements
    :pullreq: 340

    all: support new Device ID format while keeping backward compatibility

.. changelog::
  :version: 1.4.0-alpha2
  :released: November 13, 2020

  .. change::
    :tags: New Features
    :pullreq: 332

    dstore-dist: add TLS support for Kafka destination

  .. change::
    :tags: New Features
    :pullreq: 329

    dstore-dist: add support for blackhole destination

  .. change::
    :tags: Improvements
    :pullreq: 337

    doc: fix pdf generation

  .. change::
    :tags: Bug Fixes
    :pullreq: 327

    python-dist: upgrade Twisted package

  .. change::
    :tags: Improvements
    :pullreq: 333

    dstore-dist: kafka: allow multiple dnsmessage per kafka message

.. changelog::
  :version: 1.4.0-alpha1
  :released: October 27, 2020

  .. change::
    :tags: New Features
    :pullreq:

    dstore-dist: add Kafka support as destination for outgoing messages

  .. change::
    :tags: New Features
    :pullreq: 314

    dstore-dist: add sampling and rate limiting support

  .. change::
    :tags: Improvements
    :pullreq: 323

    dstore-dist: allow filtering for qname and subdomains of qname

.. changelog::
  :version: 1.3.3
  :released: July 27, 2020

  .. change::
    :tags: Improvements
    :pullreq: 324

    Use unique build IDs for debug files.

.. changelog::
  :version: 1.3.2
  :released: July 22, 2020

.. changelog::
  :version: 1.3.2-beta2
  :released: July 17, 2020

  .. change::
    :tags: Improvements

    Also build release packages for CentOS 8.

.. changelog::
  :version: 1.3.2-beta1
  :released: July 17, 2020

  .. change::
    :tags: Improvements
    :pullreq: 322

    Add support for centos 8

.. changelog::
  :version: 1.3.2-alpha1
  :released: May 14, 2020

  .. change::
    :tags: Bug Fixes
    :pullreq: 310, 311

    Fix compilation issues with recent versions of the build chain

  .. change::
    :tags: Improvements
    :pullreq: 316

    Add override file to the dstore-report-api gunicorn config

  .. change::
    :tags: Bug Fixes
    :pullreq: 318

    Update Event Aggregator to handle protobuf for non-filtered DNS queries

.. changelog::
  :version: 1.3.1
  :released: February 11, 2020

  .. change::
    :tags: Improvements
    :pullreq: 306

    Build and use our own python distribution.

  .. change::
    :tags: Improvements
    :pullreq: 305

    Sharding support in dstore-dist and other improvements.

  .. change::
    :tags: Improvements
    :pullreq: 307

    Aggregation feature for ``ev_aggregator``.

.. changelog::
  :version: 1.3.0
  :released: November 19, 2019

.. changelog::
  :version: 1.3.0-beta3
  :released: November 6, 2019

  .. change::
    :tags: Bug Fixes
    :pullreq: 302

    report-api: Make sure a default value PROM_STATS_DIR is defined.

  .. change::
    :tags: Bug Fixes
    :pullreq: 300

    report-api: Fix dstore install on RHEL 7.

.. changelog::
  :version: 1.3.0-beta2
  :released: October 23, 2019

  .. change::
    :tags: Improvements
    :pullreq: 278

    event-aggregator: Add support to Prometheus metrics.

  .. change::
    :tags: Improvements
    :pullreq: 279

    report-api: Add support to Prometheus metrics.

.. changelog::
  :version: 1.3.0-beta1
  :released: July 30, 2019

  .. change::
    :tags: New Features
    :pullreq: 288

    egateway: Allow searching by device name.

  .. change::
    :tags: Improvements
    :pullreq: 264

    text2tcp: Close the connection gracefully in order to avoid issues on the server end.

  .. change::
    :tags: Bug Fixes
    :pullreq: 289

    Fix several issues that came up deploying the dstore-1.3.0 alphas:

    - dstore-ev-aggregator: fix an issue preventing Redis authentication to work correctly when the password is specified in the config file instead of command line,
    - dstore-report-api: handle API queries correctly when usernames (and possibly other fields) can be tokenised,
    - dstore-ev-aggregator: add a retry mechanism to gracefully handle situations where Redis connections are broken.

  .. change::
    :tags: Bug Fixes
    :pullreq: 287

    dstore-report-api: Fix OpenAPI spec error where ``user_id`` was specified instead of ``username``.

.. changelog::
  :version: 1.3.0-alpha6
  :released: June 21, 2019

  .. change::
    :tags: Improvements
    :pullreq: 285

    Fix a couple of nits for event aggregator and dstore_alert

  .. change::
    :tags: Improvements
    :pullreq: 284

    UI: remove `queries without response` stat

  .. change::
    :tags: Improvements
    :pullreq: 265

    Improves code readability

  .. change::
    :tags: Improvements
    :pullreq: 250

    egateway: upgrade HTTP handling to libh2o

.. changelog::
  :version: 1.3.0-alpha5
  :released: June 14, 2019

  .. change::
    :tags: Bug Fixes
    :pullreq: 266

    Fix in protobuf split function and batch pool handling.

  .. change::
    :tags: New Features
    :pullreq: 262

    Web UI: export results as CSV

  .. change::
    :tags: Improvements
    :pullreq: 260

    dgrep: Allow to lookup for outgoing queries.

  .. change::
    :tags: Improvements
    :pullreq: 258

    Show versions in UI.

  .. change::
    :tags: Improvements
    :pullreq: 253

    dnspbgen: Add real-time flag to throttle message generation.

  .. change::
    :tags: Improvements
    :pullreq: 246

    dgrep: Add timestamp range specifier option.

.. changelog::
  :version: 1.3.0-alpha4
  :released: May 29, 2019

  *Note: versions 1.3.0-alpha1, -alpha2 and -alpha3 are internal only.*

  .. change::
    :tags: New Features

    Reporting API

  .. change::
    :tags: New Features

    Event Aggregator

  .. change::
    :tags: Improvements

    Make user grouping interval configurable for `scan_malware`.

.. changelog::
  :version: 1.2.7
  :released: February 8, 2019

  **DEPLOYMENT NOTES**

  **Django Prometheus Metrics**

  To monitor database query metrics, you must replace the ``ENGINE`` property of your database, replacing ``django.db.backends`` with ``django_prometheus.db.backends``:

  ::

    DATABASES = {
        'default': {
            'ENGINE': 'django_prometheus.db.backends.sqlite3',
            'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
        },
    }

  This has already been applied to the default sqlite database.

  **Add API-key Option to Setup API Authentication**

  This PR adds support for ``egateway`` API authentication.

  This is done by configuring ``egateway`` with the desired secret (``api-key`` option). The secret must be provided through the HTTP ``x-api-key`` header.

  To match that need, ``dstore-web`` ``EGATEWAY_URLS`` has been updated to support URIs like ``https://egateway.local:1234#s3cr3t``.

  **Changelog**

  .. change::
    :tags: Improvements

    egateway: limit the number of results scanned by egateway

  .. change::
    :tags: Improvements

    ui: malware scan 'no results found' message

  .. change::
    :tags: Bug Fixes

    tcpdistro: prevent FD leak when opening a corrupted file

  .. change::
    :tags: Bug Fixes

    egateway: fix a nullptr deref on a query w/ only additional filters

  .. change::
    :tags: New Features

    egateway: add api-key option to setup api authentication

  .. change::
    :tags: Improvements

    egateway: allow multiple search terms in query

  .. change::
    :tags: New Features

    dstore-web: django prometheus metrics

  .. change::
    :tags: Bug Fixes, Improvements

    ``malware_scan`` cmd fixes

  .. change::
    :tags: Improvements

    enable hardening measures (SSP, PIE, full RELRO, fortify)

  .. change::
    :tags: Bug Fixes

    crow: properly stop metrics webserver

  .. change::
    :tags: Improvements

    ui: adjust search results fields and add ``tcp`` field

  .. change::
    :tags: Improvements

    dgrep: add support for customer and device query params

  .. change::
    :tags: Bug Fixes

    prometheus: only display metric header for distinct metrics

  .. change::
    :tags: Improvements

    just as for the new dcat utility below, this adds ``--raw`` modifier to dgrep output. Combined with ``--quiet``, you get a stream of raw protobuf messages instead of json output

  .. change::
    :tags: New Features

    this adds a dcat utility that pretty-prints a dstore data file

  .. change::
    :tags: New Features

    tcpdistro: add compression-level parameter

.. changelog::
  :version: 1.2.6
  :released: January 14, 2019

  .. change::
    :tags: Improvements

    tcpdistro: Move bucket list to a LRU list with regular cleaning

.. changelog::
  :version: 1.2.5
  :released: November 14, 2018

  **NOTE**: DStore 1.2.5 needs to be reinstalled but after that upgrading will work again.

  .. change::
    :tags: Improvements

    Update dnsmessage.proto to sync with PDNS

  .. change::
    :tags: Bug Fixes

    dstore-web: Fix accidental removal of /usr/share/dstore-web when upgrading dstore

  .. change::
    :tags: Bug Fixes

    egateway: Add an 'ecs-override-requestor' option, export more values via the API

    - Clean up the protobuf bytes to ComboAddress conversions
    - Export 'ecs', 'serverId', a non-overridden 'from' and 'tcp' via the API
    - Add an 'ecs-override-requestor' option to be able to configure whether an ECS value should override the 'from' value when present (default, existing behaviour)

.. changelog::
  :version: 1.2.4
  :released: October 5, 2018

  .. change::
    :tags: New Features

    Grafana dashboard for dstore

  .. change::
    :tags: Improvements

    dstore-web: change default search range from 12 months to 1 hour

  .. change::
    :tags: Bug Fixes

    dstore-web: fix error on packages upgrade

  .. change::
    :tags: Bug Fixes

    pbscanner: fix the error message when reading from a corrupted file

  .. change::
    :tags: Bug Fixes

    dirwalker: explictly remove copy constructor

.. changelog::
  :version: 1.2.3
  :released: September 27, 2018

  .. change::
    :tags: New Features

    Define systemd SyslogIdentifier

  .. change::
    :tags: New Features

    metrics: Add Prometheus support to tcpdistro and egateway

  .. change::
    :tags: Improvements

    dgrep: Add username and device ID to output

  .. change::
    :tags: Bug Fixes

    pbscanner: Close the directory file descriptor in DirWalker

.. changelog::
  :version: 1.2.2
  :released: June 1, 2018

  .. change::
    :tags: New Features

    Sharding

  .. change::
    :tags: New Features

    Support for outgoing protobuf logs

  .. change::
    :tags: New Features

    `program:`dnspbgen`

  .. change::
    :tags: New Features

    `program:`dnspb2json`

  .. change::
    :tags: Bug Fixes

    Stability improvements

  .. change::
    :tags: Bug Fixes

    Accept all 2xx HTTP codes as positive

.. changelog::
  :version: 1.2.1
  :released: June 1, 2018

  .. change::
    :tags: New Features, API

    Support New Notification Centre API

.. changelog::
  :version: 1.2.0
  :released: May 28, 2018

  .. change::
    :tags: Web

    The :program:`dstore-web` packages are now built against Python 3.6.

  .. change::
    :tags: Internals

    Drop requirement for Protobuf 3.

  .. change::
    :tags: New Features

    Add on-disk compression based on `zstd <https://facebook.github.io/zstd/>`__.
    This is enabled by default.

  .. change::
    :tags: New Features

    Add :program:`dstore-dist`, a service that can duplicate and shard data over multiple :program:`dstore` nodes.

  .. change::
    :tags: New Features, Web

    Add support for storing, searching and extracting per-device information.

