Metadata-Version: 2.1
Name: byteblower-test-framework
Version: 1.1.0
Summary: Test Framework for the ByteBlower Traffic Generator.
Keywords: ByteBlower,Test Framework
Author-email: ByteBlower Development Team <support.byteblower@excentis.com>
Maintainer-email: Tom Ghyselinck <tom.ghyselinck@excentis.com>
Requires-Python: >=3.7
Description-Content-Type: text/x-rst
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Manufacturing
Classifier: Intended Audience :: Other Audience
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Telecommunications Industry
Classifier: License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3)
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Software Development
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Testing :: Traffic Generation
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Dist: byteblowerll >=2.19.2a114
Requires-Dist: scapy >=2.2.0
Requires-Dist: junit_xml >=1.8.0
Requires-Dist: pandas >=0.24.2
Requires-Dist: jinja2 >=2.10.1
Requires-Dist: highcharts-excentis >=0.4.4
Requires-Dist: yapf[pyproject] ; extra == "dev"
Requires-Dist: isort ; extra == "dev"
Requires-Dist: rstcheck[sphinx,toml] ; extra == "dev"
Requires-Dist: doc8 ; extra == "dev"
Requires-Dist: Pygments ; extra == "dev"
Requires-Dist: pydocstyle[toml] ; extra == "dev"
Requires-Dist: toml ~=0.10.2 ; extra == "dev"
Requires-Dist: Sphinx >= 5.0 ; extra == "docs-dev"
Requires-Dist: sphinx-jsonschema >= 1.19 ; extra == "docs-dev"
Requires-Dist: sphinx-rtd-theme >=1.0 ; extra == "docs-dev"
Requires-Dist: importlib-metadata>=4.8.3 ; extra == "docs-dev" and ( python_version<'3.8')
Requires-Dist: pylint[spelling] ; extra == "test"
Requires-Dist: pydocstyle[toml] ; extra == "test"
Requires-Dist: rstcheck[sphinx,toml] ; extra == "test"
Requires-Dist: pytest >=6.0 ; extra == "test"
Requires-Dist: pytest-cov ; extra == "test"
Requires-Dist: pytest-pydocstyle ; extra == "test"
Requires-Dist: toml ~=0.10.2 ; extra == "test"
Project-URL: Documentation, https://api.byteblower.com/test-framework#documentation
Project-URL: Examples, https://api.byteblower.com/test-framework/index.html#examples
Project-URL: Homepage, https://www.byteblower.com
Project-URL: Support Portal, https://support.excentis.com
Provides-Extra: dev
Provides-Extra: docs-dev
Provides-Extra: test

*************************
ByteBlower Test Framework
*************************

  An easy accessible library of basic test flows and reporting engines.

.. footer::
   Copyright |copy| |year| - Excentis N.V.

.. |registered| unicode:: U+00AE .. registered sign
.. |copy| unicode:: U+00A9 .. copyright sign
.. |year| date:: %Y

ByteBlower |registered| is a traffic generator/analyzer system
for TCP/IP networks.

This library provides you with the building blocks for:

#. Generating test traffic
#. Collect statistics
#. Analyze traffic statistics
#. Generate reports

Release notes
=============

What can this version bring to you?
See our exciting new and existing features below!

.. _Command-line interface: https://api.byteblower.com/test-framework/latest/cli/index.html

✨ **New since v1.1.0!** ✨
----------------------------

It is with great pleasure that we announce our
new features of the ByteBlower Test Framework!

- `Command-line interface`_

  - Run traffic tests with nothing more than a JSON configuration file!

- 🚧 **Feature previews** 🚧

  - The HTML and JSON report include **flow runtime error** information.
  - Most building blocks now have an option to explicitly release related
    resources on the ByteBlower system.

Features
--------

- Quick and easy automation of traffic tests using ByteBlower
- Straightforward building blocks for your own test scripts

  - Grouped in self-explaining categories
  - Designed with a small 😉 to the workflow you are used to from the GUI

- Supported ByteBlower endpoint types

  - ByteBlower Port (on a physical interface of a ByteBlower server)

- Endpoint configuration

  - IPv4: DHCP and manual address configuration

    - Automatic handling of endpoints located behind a NA(P)T gateway!

  - IPv6: DHCPv6 and SLAAC address configuration

- Traffic types

  - Stateless: frame blasting (UDP-based)
  - Stateful: HTTP (TCP-based)

- Application simulations

  - Voice over IP (VoIP) using the G.711 codec
  - Video streaming (Netflix, YouTube, ...)

- Standard analysis

  - Frame count: transmitted and received over time, frame loss and byte loss
  - Latency: Minimum, maximum, average and jitter over time
  - Aggregation of results over multiple flows
    (*frame blasting only in this release*)
  - PASS/FAIL criteria can be provided to match your KPIs

- Application-specific analysis

  - Mean Opinion Score (MOS): Specific for Voice flows
  - Video buffer analysis: Specific for Video flows

- Reporting

  - Summary *and* realtime results
  - HTML: Neat to share with your chief, customers, vendors, ...

    - Incorporating our brand new style!
    - Interactive charts
    - Includes overview of all latency CCDF results

  - JSON: Allows for machine post-processing
  - XML JUnit: Useful for integration in automation tools

- Helpers

  - Ease-of-use for configuration and/or post-processing
    of the analyzed results

- `Example scripts`_

.. _Example scripts: https://api.byteblower.com/test-framework/index.html#examples

Changelog
---------

For all details, please have a look at our Changelog_.

.. _Changelog: https://api.byteblower.com/test-framework/latest/changelog.html

Requirements
============

* byteblowerll_ (`ByteBlower API`_): Our lower layer API for client-server
  communication (`API documentation <https://api.byteblower.com/python>`_)
* scapy_: Used for frame generation and parsing
* junit-xml_: Used for Unit test report generation
* pandas_: Used for data collection
* highcharts-excentis_: Used for generating graphs
* jinja2_: User for HTML report templating

.. _ByteBlower API: https://setup.byteblower.com/
.. _byteblowerll: https://pypi.org/project/byteblowerll/
.. _scapy: https://pypi.org/project/scapy/
.. _junit-xml: https://pypi.org/project/junit-xml/
.. _pandas: https://pypi.org/project/pandas/
.. _highcharts-excentis: https://pypi.org/project/highcharts-excentis/
.. _jinja2: https://pypi.org/project/Jinja2/

Supported platforms
-------------------

The ByteBlower Test Framework in general supports Python version 3.7 to 3.11.

.. note::
   **NOTE**: *Python >= 3.12 is not yet supported because the ByteBlower API
   libraries are not yet available for Python 3.12* (`byteblowerll`_).

The framework has been tested for the following operating system platforms
and Python versions:

+------------------+----------------------------+----------------+------------------------+
| OS platform      | Distribution               | Python version | source                 |
+==================+============================+================+========================+
| Windows 10       | up to feature release 21H2 | Python 3.10    | `Official Python`_     |
+------------------+----------------------------+----------------+------------------------+
| Windows 10       | up to feature release 21H2 | Python 3.9     | `Official Python`_     |
+------------------+----------------------------+----------------+------------------------+
| Windows 10       | up to feature release 21H2 | Python 3.8     | `Official Python`_     |
+------------------+----------------------------+----------------+------------------------+
| Windows 10       | up to feature release 21H2 | Python 3.7     | `Official Python`_     |
+------------------+----------------------------+----------------+------------------------+
| Windows 10       | up to feature release 21H2 | Python 3.9     | `Windows Apps`_        |
+------------------+----------------------------+----------------+------------------------+
| Windows 10       | up to feature release 21H2 | Python 3.8     | `Windows Apps`_        |
+------------------+----------------------------+----------------+------------------------+
| Windows 10       | up to feature release 21H2 | Python 3.7     | `Windows Apps`_        |
+------------------+----------------------------+----------------+------------------------+
| macOS            | up to Monterey             | Python 3.9     | `Official Python`_     |
|                  |                            |                | (**Intel-only!**)      |
+------------------+----------------------------+----------------+------------------------+
| macOS            | up to Monterey             | Python 3.8     | `Official Python`_     |
|                  |                            |                | (**Intel-only!**)      |
+------------------+----------------------------+----------------+------------------------+
| Linux            | Debian 11 (bullseye)       | Python 3.9.2   | `Debian packages`_     |
+------------------+----------------------------+----------------+------------------------+
| Linux            | Debian 10 (buster)         | Python 3.7.3   | `Debian packages`_     |
+------------------+----------------------------+----------------+------------------------+
| Linux            | Ubuntu 20.04 (Focal Fossa) | Python 3.8.2   | `Ubuntu packages`_     |
+------------------+----------------------------+----------------+------------------------+
| Linux            | Ubuntu 22.04 (Focal Fossa) | Python 3.10.4  | `Ubuntu packages`_     |
+------------------+----------------------------+----------------+------------------------+
| Docker           | python:3.10-slim-buster    | Python 3.10.11 | `Docker Python`_       |
+------------------+----------------------------+----------------+------------------------+
| Docker           | python:3.9-slim-buster     | Python 3.9.16  | `Docker Python`_       |
+------------------+----------------------------+----------------+------------------------+
| Docker           | python:3.8-slim-buster     | Python 3.8.16  | `Docker Python`_       |
+------------------+----------------------------+----------------+------------------------+
| Docker           | python:3.7-slim-buster     | Python 3.7.13  | `Docker Python`_       |
+------------------+----------------------------+----------------+------------------------+

.. _Official Python: https://www.python.org
.. _Windows Apps: https://apps.microsoft.com/
.. _Debian packages: https://packages.debian.org/search?suite=all&exact=1&searchon=names&keywords=python3
.. _Ubuntu packages: https://packages.ubuntu.com/search?keywords=python3&searchon=names&exact=1&suite=all&section=all
.. _Docker Python: https://hub.docker.com/_/python

Installation
============

Prepare runtime environment
---------------------------

We recommend managing the runtime environment in a Python virtual
environment. This guarantees proper separation of the system-wide
installed Python and pip packages.

Python virtual environment
^^^^^^^^^^^^^^^^^^^^^^^^^^

Prepare Python virtual environment: Create the virtual environment
and install/update ``pip`` and ``build``.

1. On Unix-based systems (Linux, WSL, macOS):

   **Note**:
   *Mind the leading* ``.`` *which means* **sourcing** ``./env/bin/activate``.

   .. code-block:: shell

      python3 -m venv --clear env
      . ./env/bin/activate
      pip install -U pip build

2. On Windows systems using PowerShell:

      **Note**: On Microsoft Windows, it may be required to enable the
      Activate.ps1 script by setting the execution policy for the user.
      You can do this by issuing the following PowerShell command:

      .. code-block:: shell

         PS C:> Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

      See `About Execution Policies`_ for more information.

   .. code-block:: shell

      python3.8.exe -m venv --clear env
      & ".\env\Scripts\activate.ps1"
      python3.8.exe -m pip install -U pip build

.. _About Execution Policies: https://go.microsoft.com/fwlink/?LinkID=135170

Install the ByteBlower Test Framework from PyPI
-----------------------------------------------

First make sure that your *activated* your virtual environment:

1. On Unix-based systems (Linux, WSL, macOS):

   .. code-block:: shell

      . ./env/bin/activate

2. On Windows systems using PowerShell:

   .. code-block:: shell

      & ".\env\Scripts\activate.ps1"

Now install (or update) the ByteBlower Test Framework:

.. code-block:: shell

   pip install -U byteblower-test-framework

Documentation
=============

Online usage documentation: `ByteBlower Test Framework documentation`_

.. _ByteBlower Test Framework documentation: https://api.byteblower.com/test-framework/latest/

The API documentation is also always available in the API:

.. code-block:: python

   help(any_api_object)

Some examples:

For classes (and their members):

.. code-block:: python

   from byteblower_test_framework.host import Server
   from byteblower_test_framework.endpoint import IPv4Port
   from byteblower_test_framework.traffic import FrameBlastingFlow

   help(Server)
   help(Server.start)
   help(Server.info)
   help(IPv4Port)
   help(FrameBlastingFlow)

   from byteblower_test_framework.report import ByteBlowerHtmlReport

   help(ByteBlowerHtmlReport)

For objects (and their members):

.. code-block:: python

   from byteblower_test_framework.host import Server

   my_server = Server('byteblower-39.lab.excentis.com.')

   help(my_server)
   help(my_server.start)

Usage
=====

First make sure that your *activated* your virtual environment:

1. On Unix-based systems (Linux, WSL, macOS):

   .. code-block:: shell

      . ./env/bin/activate

2. On Windows systems using PowerShell:

   .. code-block:: shell

      & ".\env\Scripts\activate.ps1"

Let's give it a test run: Import the test framework and show its
documentation:

.. code-block:: shell

   python

.. code-block:: python

   import byteblower_test_framework
   help(byteblower_test_framework)

This shows you the ByteBlower Test Framework module documentation.

Command-line interface
----------------------

To get help for command line arguments:

#. As a python module:

   .. code-block:: shell

      python -m byteblower_test_framework --help

#. As a command-line script:

   .. code-block:: shell

      byteblower-test-framework --help


For a quick start, you can run a simple test using the JSON
configuration provided below:

.. code-block:: json

   {
       "server": "byteblower.example.com.",
       "ports": [
           {
               "name": "CMTS",
               "port_group": [
                   "source_group"
               ],
               "interface": "nontrunk-1",
               "ipv4": "dhcp"
           },
           {
               "name": "CM-LAN",
               "port_group": [
                   "destination_group"
               ],
               "interface": "trunk-1-1",
               "ipv4": "dhcp",
               "nat": true
           }
       ],
       "flows": [
           {
               "name": "Downstream UDP",
               "source": {
                   "port_group": [
                       "source_group"
                   ]
               },
               "destination": {
                   "port_group": [
                       "destination_group"
                   ]
               },
               "type": "frame_blasting",
               "frame_size": 60,
               "frame_rate": 8500,
               "add_reverse_direction": true,
               "analysis": {
                   "latency": true
               }
           },
           {
               "name": "Upstream HTTP",
               "source": {
                   "port_group": [
                       "destination_group"
                   ]
               },
               "destination": {
                   "port_group": [
                       "source_group"
                   ]
               },
               "add_reverse_direction": true,
               "type": "http"
           }
       ],
       "report": {
           "html": true,
           "json": false,
           "junit_xml": false
       },
       "maximum_run_time": 10.0
   }

Please make sure you change the server and ports configuration according
to the setup you want to run your test on.

Copy this configuration into ``byteblower_test_framework.json``.
Then, run the test in the command line interface using:

.. code-block:: shell

   python -m byteblower_test_framework

The resulting reports will be saved into the current directory.
You can specify a different *config file name* and *report path* using:

.. code-block:: shell

   python -m byteblower_test_framework --config_file path/to/my_test_config.json  --report_path path/to/my_test_reports_directory

You can find more details on how to customize your own configuration file
in `Configuration file`_.

.. _Configuration file: https://api.byteblower.com/test-framework/latest/cli/config_file.html

.. TODO: Provide a quick start guide  (NOT NEEDED ANYMORE ?)
.. .. note::
..    **To-do**: *We will provide a quick start guide in the future.*

Development
===========

Would you like to contribute to this project? You're very welcome! 😊

Please contact us at `ByteBlower Support`_ and we'll be there to guide you.

Support
=======

.. See http://docutils.sourceforge.net/0.4/docs/ref/rst/directives.html#image

If you have any questions or feature request you can contact the ByteBlower
support team using:

|globe|: `Excentis Support Portal`_

|e-mail|: `ByteBlower Support`_

|telephone|: +32 (0) 9 269 22 91

.. e-mail icon:
.. |e-mail| unicode:: U+1F582

.. globe icon:
.. |globe| unicode:: U+1F30D
.. .. |globe| unicode:: U+1F310

.. telephone icon:
.. |telephone| unicode:: U+1F57D

.. ByteBlower logo
.. image:: http://static.excentis.com/byteblower_blue_transparent_background.png
   :width: 400
   :scale: 60
   :align: right
   :alt: ByteBlower
   :target: byteblower_

.. "A product by Excentis" logo
.. image:: http://static.excentis.com/Aproductby.png
   :width: 320
   :scale: 60
   :align: right
   :alt: A product by Excentis
   :target: excentis_

.. _byteblower: https://byteblower.com
.. _excentis: https://www.excentis.com
.. _Excentis Support Portal: https://support.excentis.com
.. _ByteBlower Support: mailto:support.byteblower@excentis.com

