Metadata-Version: 2.1
Name: byteblower-test-framework
Version: 1.1.2
Summary: Test Framework for the ByteBlower Traffic Generator.
Keywords: ByteBlower,Test Framework
Author-email: ByteBlower Development Team <support.byteblower@excentis.com>
Maintainer-email: Tom Ghyselinck <tom.ghyselinck@excentis.com>
Requires-Python: >=3.7
Description-Content-Type: text/x-rst
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Manufacturing
Classifier: Intended Audience :: Other Audience
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Telecommunications Industry
Classifier: License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3)
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Software Development
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Testing :: Traffic Generation
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Dist: byteblowerll >=2.19.2a114
Requires-Dist: scapy >=2.2.0
Requires-Dist: junit_xml >=1.8.0
Requires-Dist: pandas >=0.24.2
Requires-Dist: jinja2 >=2.10.1
Requires-Dist: highcharts-excentis >=0.4.4
Requires-Dist: yapf[pyproject] ; extra == "dev"
Requires-Dist: isort ; extra == "dev"
Requires-Dist: rstcheck[sphinx,toml] ; extra == "dev"
Requires-Dist: doc8 ; extra == "dev"
Requires-Dist: Pygments ; extra == "dev"
Requires-Dist: pydocstyle[toml] ; extra == "dev"
Requires-Dist: toml ~=0.10.2 ; extra == "dev"
Requires-Dist: Sphinx >= 5.0 ; extra == "docs-dev"
Requires-Dist: sphinx-rtd-theme >=1.0 ; extra == "docs-dev"
Requires-Dist: sphinx-tabs >= 3.4 ; extra == "docs-dev"
Requires-Dist: sphinx-jsonschema >= 1.19 ; extra == "docs-dev"
Requires-Dist: importlib-metadata>=4.8.3 ; extra == "docs-dev" and ( python_version<'3.8')
Requires-Dist: pylint[spelling] ; extra == "test"
Requires-Dist: pydocstyle[toml] ; extra == "test"
Requires-Dist: rstcheck[sphinx,toml] ; extra == "test"
Requires-Dist: pytest >=6.0 ; extra == "test"
Requires-Dist: pytest-cov ; extra == "test"
Requires-Dist: pytest-pydocstyle ; extra == "test"
Requires-Dist: toml ~=0.10.2 ; extra == "test"
Project-URL: Documentation, https://api.byteblower.com/test-framework/index.html#documentation
Project-URL: Examples, https://api.byteblower.com/test-framework/index.html#examples
Project-URL: Homepage, https://www.byteblower.com
Project-URL: Support Portal, https://support.excentis.com
Provides-Extra: dev
Provides-Extra: docs-dev
Provides-Extra: test

*************************
ByteBlower Test Framework
*************************

  An easy accessible library of basic test flows and reporting engines.

.. footer::
   Copyright |copy| |year| - Excentis N.V.

.. |registered| unicode:: U+00AE .. registered sign
.. |copy| unicode:: U+00A9 .. copyright sign
.. |year| date:: %Y

ByteBlower |registered| is a traffic generator/analyzer system
for TCP/IP networks.

This library provides you with the building blocks for:

#. Generating test traffic
#. Collect statistics
#. Analyze traffic statistics
#. Generate reports

Release notes
=============

What can this version bring to you?
See our exciting new and existing features below!

.. _Command-line interface: https://api.byteblower.com/test-framework/latest/byteblower-test-framework/cli/index.html

✨ **Test Cases** ✨
---------------------

The ByteBlower Test Framework serves as the base for many of our customer's
test cases.

At Excentis, we also develop common test cases and love to
share them with the community.

- `Test Case: RFC 2544 Throughput`_

  - Run an `RFC 2544`_ network performance test with ease!

- `Test Case: Low Latency`_

  - Run low latency validation tests on your network.

.. _Test Case\: RFC 2544 Throughput: https://api.byteblower.com/test-framework/latest/test-cases/rfc-2544/overview.html
.. _Test Case\: Low Latency: https://api.byteblower.com/test-framework/latest/test-cases/low-latency/overview.html
.. _RFC 2544: https://datatracker.ietf.org/doc/html/rfc2544

📢 **New since v1.1.0!** 📢
----------------------------

It is with great pleasure that we announce our
new features of the ByteBlower Test Framework!

- `Command-line interface`_

  - Run traffic tests with nothing more than a JSON configuration file!

- 🚧 **Feature previews** 🚧

  - The HTML and JSON report include **flow runtime error** information.
  - Most building blocks now have an option to explicitly release related
    resources on the ByteBlower system.

Features
--------

- Quick and easy automation of traffic tests using ByteBlower
- Straightforward building blocks for your own test scripts

  - Grouped in self-explaining categories
  - Designed with a small 😉 to the workflow you are used to from the GUI

- Supported ByteBlower endpoint types

  - ByteBlower Port (on a physical interface of a ByteBlower server)

- Endpoint configuration

  - IPv4: DHCP and manual address configuration

    - Automatic handling of endpoints located behind a NA(P)T gateway!

  - IPv6: DHCPv6 and SLAAC address configuration

- Traffic types

  - Stateless: frame blasting (UDP-based)
  - Stateful: HTTP (TCP-based)

- Application simulations

  - Voice over IP (VoIP) using the G.711 codec
  - Video streaming (Netflix, YouTube, ...)

- Standard analysis

  - Frame count: transmitted and received over time, frame loss and byte loss
  - Latency: Minimum, maximum, average and jitter over time
  - Aggregation of results over multiple flows
    (*frame blasting only in this release*)
  - PASS/FAIL criteria can be provided to match your KPIs

- Application-specific analysis

  - Mean Opinion Score (MOS): Specific for Voice flows
  - Video buffer analysis: Specific for Video flows

- Reporting

  - Summary *and* realtime results
  - HTML: Neat to share with your chief, customers, vendors, ...

    - Incorporating our brand new style!
    - Interactive charts
    - Includes overview of all latency CCDF results

  - JSON: Allows for machine post-processing
  - XML JUnit: Useful for integration in automation tools

- Helpers

  - Ease-of-use for configuration and/or post-processing
    of the analyzed results

- `Example scripts`_

.. _Example scripts: https://api.byteblower.com/test-framework/index.html#examples

Changelog
---------

For all details, please have a look at our Changelog_.

.. _Changelog: https://api.byteblower.com/test-framework/latest/changelog.html

Requirements
============

* byteblowerll_ (`ByteBlower API`_): Our lower layer API for client-server
  communication (`API documentation <https://api.byteblower.com/python>`_)
* scapy_: Used for frame generation and parsing
* junit-xml_: Used for Unit test report generation
* pandas_: Used for data collection
* highcharts-excentis_: Used for generating graphs
* jinja2_: User for HTML report templating

.. _ByteBlower API: https://setup.byteblower.com/
.. _byteblowerll: https://pypi.org/project/byteblowerll/
.. _scapy: https://pypi.org/project/scapy/
.. _junit-xml: https://pypi.org/project/junit-xml/
.. _pandas: https://pypi.org/project/pandas/
.. _highcharts-excentis: https://pypi.org/project/highcharts-excentis/
.. _jinja2: https://pypi.org/project/Jinja2/

Supported platforms
-------------------

The ByteBlower Test Framework in general supports Python version 3.7 to 3.11.

.. note::
   **NOTE**: *Python >= 3.12 is not yet supported because the ByteBlower API
   libraries are not yet available for Python 3.12* (`byteblowerll`_).

The framework has been tested for the following operating system platforms
and Python versions:

+------------------+----------------------------+----------------+------------------------+
| OS platform      | Distribution               | Python version | source                 |
+==================+============================+================+========================+
| Windows 10       | up to feature release 21H2 | Python 3.10    | `Official Python`_     |
+------------------+----------------------------+----------------+------------------------+
| Windows 10       | up to feature release 21H2 | Python 3.9     | `Official Python`_     |
+------------------+----------------------------+----------------+------------------------+
| Windows 10       | up to feature release 21H2 | Python 3.8     | `Official Python`_     |
+------------------+----------------------------+----------------+------------------------+
| Windows 10       | up to feature release 21H2 | Python 3.7     | `Official Python`_     |
+------------------+----------------------------+----------------+------------------------+
| Windows 10       | up to feature release 21H2 | Python 3.9     | `Windows Apps`_        |
+------------------+----------------------------+----------------+------------------------+
| Windows 10       | up to feature release 21H2 | Python 3.8     | `Windows Apps`_        |
+------------------+----------------------------+----------------+------------------------+
| Windows 10       | up to feature release 21H2 | Python 3.7     | `Windows Apps`_        |
+------------------+----------------------------+----------------+------------------------+
| macOS            | up to Monterey             | Python 3.9     | `Official Python`_     |
|                  |                            |                | (**Intel-only!**)      |
+------------------+----------------------------+----------------+------------------------+
| macOS            | up to Monterey             | Python 3.8     | `Official Python`_     |
|                  |                            |                | (**Intel-only!**)      |
+------------------+----------------------------+----------------+------------------------+
| Linux            | Debian 11 (bullseye)       | Python 3.9.2   | `Debian packages`_     |
+------------------+----------------------------+----------------+------------------------+
| Linux            | Debian 10 (buster)         | Python 3.7.3   | `Debian packages`_     |
+------------------+----------------------------+----------------+------------------------+
| Linux            | Ubuntu 20.04 (Focal Fossa) | Python 3.8.2   | `Ubuntu packages`_     |
+------------------+----------------------------+----------------+------------------------+
| Linux            | Ubuntu 22.04 (Focal Fossa) | Python 3.10.4  | `Ubuntu packages`_     |
+------------------+----------------------------+----------------+------------------------+
| Docker           | python:3.10-slim-buster    | Python 3.10.11 | `Docker Python`_       |
+------------------+----------------------------+----------------+------------------------+
| Docker           | python:3.9-slim-buster     | Python 3.9.16  | `Docker Python`_       |
+------------------+----------------------------+----------------+------------------------+
| Docker           | python:3.8-slim-buster     | Python 3.8.16  | `Docker Python`_       |
+------------------+----------------------------+----------------+------------------------+
| Docker           | python:3.7-slim-buster     | Python 3.7.13  | `Docker Python`_       |
+------------------+----------------------------+----------------+------------------------+

.. _Official Python: https://www.python.org
.. _Windows Apps: https://apps.microsoft.com/
.. _Debian packages: https://packages.debian.org/search?suite=all&exact=1&searchon=names&keywords=python3
.. _Ubuntu packages: https://packages.ubuntu.com/search?keywords=python3&searchon=names&exact=1&suite=all&section=all
.. _Docker Python: https://hub.docker.com/_/python

Installation
============

Prepare runtime environment
---------------------------

We recommend managing the runtime environment in a Python virtual
environment. This guarantees proper separation of the system-wide
installed Python and pip packages.

Python virtual environment
^^^^^^^^^^^^^^^^^^^^^^^^^^

Make sure to use the right Python version (>= 3.7, <= 3.11),
list all Python versions installed in your machine by running:

#. On Windows systems using PowerShell:

   .. code-block:: shell

      py --list

If no Python version is in the required range, you can download and install
Python 3.7 or above using your system package manager
or from https://www.python.org/ftp/python.

Prepare Python virtual environment: Create the virtual environment
and install/update ``pip`` and ``build``.

#. On Unix-based systems (Linux, WSL, macOS):

   **Note**:
   *Mind the leading* ``.`` *which means* **sourcing** ``./env/bin/activate``.

   .. code-block:: shell

      python3 -m venv --clear env
      . ./env/bin/activate
      pip install -U pip build

#. On Windows systems using PowerShell:

      **Note**: On Microsoft Windows, it may be required to enable the
      Activate.ps1 script by setting the execution policy for the user.
      You can do this by issuing the following PowerShell command:

      .. code-block:: shell

         PS C:> Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

      See `About Execution Policies`_ for more information.

   .. code-block:: shell

      py -3.8 -m venv --clear env
      & ".\env\Scripts\activate.ps1"
      python -m pip install -U pip build

.. _About Execution Policies: https://go.microsoft.com/fwlink/?LinkID=135170

Install the ByteBlower Test Framework from PyPI
-----------------------------------------------

First make sure that your *activated* your virtual environment:

#. On Unix-based systems (Linux, WSL, macOS):

   .. code-block:: shell

      . ./env/bin/activate

#. On Windows systems using PowerShell:

   .. code-block:: shell

      & ".\env\Scripts\activate.ps1"

Now install (or update) the ByteBlower Test Framework:

.. code-block:: shell

   pip install -U byteblower-test-framework

Documentation
=============

Online usage documentation: `ByteBlower Test Framework documentation`_

.. _ByteBlower Test Framework documentation: https://api.byteblower.com/test-framework/latest/

The API documentation is also always available in the API:

.. code-block:: python

   help(any_api_object)

Some examples:

For classes (and their members):

.. code-block:: python

   from byteblower_test_framework.host import Server
   from byteblower_test_framework.endpoint import IPv4Port
   from byteblower_test_framework.traffic import FrameBlastingFlow

   help(Server)
   help(Server.start)
   help(Server.info)
   help(IPv4Port)
   help(FrameBlastingFlow)

   from byteblower_test_framework.report import ByteBlowerHtmlReport

   help(ByteBlowerHtmlReport)

For objects (and their members):

.. code-block:: python

   from byteblower_test_framework.host import Server

   my_server = Server('byteblower-39.lab.excentis.com.')

   help(my_server)
   help(my_server.start)

Usage
=====

First make sure that your *activated* your virtual environment:

#. On Unix-based systems (Linux, WSL, macOS):

   .. code-block:: shell

      . ./env/bin/activate

#. On Windows systems using PowerShell:

   .. code-block:: shell

      & ".\env\Scripts\activate.ps1"

Let's give it a test run: Import the test framework and show its
documentation:

.. code-block:: shell

   python

.. code-block:: python

   import byteblower_test_framework
   help(byteblower_test_framework)

This shows you the ByteBlower Test Framework module documentation.

Command-line interface
----------------------

To get help for command line arguments:

#. As a python module:

   .. code-block:: shell

      python -m byteblower_test_framework --help

#. As a command-line script:

   .. code-block:: shell

      byteblower-test-framework --help


For a quick start, you can run a simple test using the JSON
configuration provided below:

.. code-block:: json

   {
       "server": "byteblower.example.com.",
       "ports": [
           {
               "name": "CMTS",
               "port_group": [
                   "source_group"
               ],
               "interface": "nontrunk-1",
               "ipv4": "dhcp"
           },
           {
               "name": "CM-LAN",
               "port_group": [
                   "destination_group"
               ],
               "interface": "trunk-1-1",
               "ipv4": "dhcp",
               "nat": true
           }
       ],
       "flows": [
           {
               "name": "Downstream UDP",
               "source": {
                   "port_group": [
                       "source_group"
                   ]
               },
               "destination": {
                   "port_group": [
                       "destination_group"
                   ]
               },
               "type": "frame_blasting",
               "frame_size": 60,
               "frame_rate": 8500,
               "add_reverse_direction": true,
               "analysis": {
                   "latency": true
               }
           },
           {
               "name": "Upstream HTTP",
               "source": {
                   "port_group": [
                       "destination_group"
                   ]
               },
               "destination": {
                   "port_group": [
                       "source_group"
                   ]
               },
               "add_reverse_direction": true,
               "type": "http"
           }
       ],
       "report": {
           "html": true,
           "json": false,
           "junit_xml": false
       },
       "maximum_run_time": 10.0
   }

Please make sure you change the server and ports configuration according
to the setup you want to run your test on.

Copy this configuration into ``byteblower_test_framework.json``.
Then, run the test in the command line interface using:

.. code-block:: shell

   byteblower-test-framework

The resulting reports will be saved into the current directory.
You can specify a different *config file name* and *report path* using:

.. code-block:: shell

   byteblower-test-framework --config-file path/to/my_test_config.json  --report-path path/to/my_test_reports_directory

You can find more details on how to customize your own configuration file
in `Configuration file`_.

.. _Configuration file: https://api.byteblower.com/test-framework/latest/byteblower-test-framework/cli/config_file.html

.. TODO: Provide a quick start guide  (NOT NEEDED ANYMORE ?)
.. note::
   **To-do**: *We will provide a quick start guide in the future.*

Development
===========

Would you like to contribute to this project? You're very welcome! 😊

Please contact us at `ByteBlower Support`_ and we'll be there to guide you.

Support
=======

.. See http://docutils.sourceforge.net/0.4/docs/ref/rst/directives.html#image

If you have any questions or feature request you can contact the ByteBlower
support team using:

|globe|: `Excentis Support Portal`_

|e-mail|: `ByteBlower Support`_

|telephone|: +32 (0) 9 269 22 91

.. e-mail icon:
.. |e-mail| unicode:: U+1F582

.. globe icon:
.. |globe| unicode:: U+1F30D
.. .. |globe| unicode:: U+1F310

.. telephone icon:
.. |telephone| unicode:: U+1F57D

.. ByteBlower logo
.. image:: http://static.excentis.com/byteblower_blue_transparent_background.png
   :width: 400
   :scale: 60
   :align: right
   :alt: ByteBlower
   :target: byteblower_

.. "A product by Excentis" logo
.. image:: http://static.excentis.com/Aproductby.png
   :width: 320
   :scale: 60
   :align: right
   :alt: A product by Excentis
   :target: excentis_

.. _byteblower: https://byteblower.com
.. _excentis: https://www.excentis.com
.. _Excentis Support Portal: https://support.excentis.com
.. _ByteBlower Support: mailto:support.byteblower@excentis.com

