Metadata-Version: 2.0
Name: acumos
Version: 0.8.0
Summary: Acumos client library for building and pushing Python models
Home-page: https://gerrit.acumos.org/r/gitweb?p=acumos-python-client.git
Author: Paul Triantafyllou
Author-email: trianta@research.att.com
License: Apache License 2.0
Keywords: acumos machine learning model modeling artificial intelligence ml ai
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: License :: OSI Approved :: Apache Software License
Requires-Python: >=3.5, <3.7
Requires-Dist: appdirs
Requires-Dist: dill
Requires-Dist: filelock
Requires-Dist: numpy
Requires-Dist: protobuf
Requires-Dist: requests

.. ===============LICENSE_START=======================================================
.. Acumos CC-BY-4.0
.. ===================================================================================
.. Copyright (C) 2017-2018 AT&T Intellectual Property & Tech Mahindra. All rights reserved.
.. ===================================================================================
.. This Acumos documentation file is distributed by AT&T and Tech Mahindra
.. under the Creative Commons Attribution 4.0 International License (the "License");
.. you may not use this file except in compliance with the License.
.. You may obtain a copy of the License at
..
..      http://creativecommons.org/licenses/by/4.0
..
.. This file is distributed on an "AS IS" BASIS,
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.. See the License for the specific language governing permissions and
.. limitations under the License.
.. ===============LICENSE_END=========================================================

===============================
Acumos Python Client User Guide
===============================


|Build Status|

``acumos`` is a client library that allows modelers to push their Python models
to the `Acumos platform <https://www.acumos.org/>`__.

Installation
============

You will need a Python 3.5+ environment in order to install ``acumos``.
You can use `Anaconda <https://www.anaconda.com/download/>`__
(preferred) or `pyenv <https://github.com/pyenv/pyenv>`__ to install and
manage Python environments.

If you’re new to Python and need an IDE to start developing, we
recommend using `Spyder <https://github.com/spyder-ide/spyder>`__ which
can easily be installed with Anaconda.

The ``acumos`` package can be installed with pip:

.. code:: bash

    pip install acumos


Protocol Buffers
----------------

The ``acumos`` package uses protocol buffers and **assumes you have
the protobuf compiler** ``protoc`` **installed**. Please visit the `protobuf
repository <https://github.com/google/protobuf/releases/tag/v3.4.0>`__
and install the appropriate ``protoc`` for your operating system.
Installation is as easy as downloading a binary release and adding it to
your system ``$PATH``. This is a temporary requirement that will be
removed in a future version of ``acumos``.

**Anaconda Users**: You can easily install ``protoc`` from `an Anaconda
package <https://anaconda.org/anaconda/libprotobuf>`__ via:

.. code:: bash

    conda install -c anaconda libprotobuf


.. |Build Status| image:: https://jenkins.acumos.org/buildStatus/icon?job=acumos-python-client-tox-verify-master
   :target: https://jenkins.acumos.org/job/acumos-python-client-tox-verify-master/

.. ===============LICENSE_START=======================================================
.. Acumos CC-BY-4.0
.. ===================================================================================
.. Copyright (C) 2017-2018 AT&T Intellectual Property & Tech Mahindra. All rights reserved.
.. ===================================================================================
.. This Acumos documentation file is distributed by AT&T and Tech Mahindra
.. under the Creative Commons Attribution 4.0 International License (the "License");
.. you may not use this file except in compliance with the License.
.. You may obtain a copy of the License at
..
..      http://creativecommons.org/licenses/by/4.0
..
.. This file is distributed on an "AS IS" BASIS,
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.. See the License for the specific language governing permissions and
.. limitations under the License.
.. ===============LICENSE_END=========================================================

=============================
Acumos Python Client Tutorial
=============================

This tutorial provides a brief overview of ``acumos`` for creating
Acumos models. The tutorial is meant to be followed linearly, and some
code snippets depend on earlier imports and objects. Full examples are
available in the ``examples/`` directory of the `Acumos Python client repository <https://gerrit.acumos.org/r/gitweb?p=acumos-python-client.git;a=summary>`__.

#.  `Importing Acumos`_
#.  `Creating A Session`_
#.  `A Simple Model`_
#.  `Exporting Models`_
#.  `Defining Types`_
#.  `Using DataFrames with scikit-learn`_
#.  `Declaring Requirements`_
#.  `Declaring Options`_
#.  `Keras and TensorFlow`_
#. `Testing Models`_
#. `More Examples`_

Importing Acumos
================

First import the modeling and session packages:

.. code:: python

    from acumos.modeling import Model, List, Dict, create_namedtuple, create_dataframe
    from acumos.session import AcumosSession

Creating A Session
==================

An ``AcumosSession`` allows you to export your models to Acumos. You can
either dump a model to disk locally, so that you can upload it via the
Acumos website, or push the model to Acumos directly.

If you’d like to push directly to Acumos, create a session with the ``push_api`` argument:

.. code:: python

    session = AcumosSession(push_api="https://my.acumos.instance.com/push")

See the onboarding page of your Acumos instance website to find the correct
``push_api`` URL to use.

If you’re only interested in dumping a model to disk, arguments aren’t needed:

.. code:: python

    session = AcumosSession()

A Simple Model
==============

Any Python function can be used to define an Acumos model using `Python
type hints <https://docs.python.org/3/library/typing.html>`__.

Let’s first create a simple model that adds two integers together.
Acumos needs to know what the inputs and outputs of your functions are.
We can use the Python type annotation syntax to specify the function
signature.

Below we define a function ``add_numbers`` with ``int`` type parameters
``x`` and ``y``, and an ``int`` return type. We then build an Acumos
model with an ``add`` method.

**Note:** Function
`docstrings <https://www.python.org/dev/peps/pep-0257/>`__ are included
with your model and used for documentation, so be sure to include one!

.. code:: python

    def add_numbers(x: int, y: int) -> int:
        '''Returns the sum of x and y'''
        return x + y

    model = Model(add=add_numbers)

Exporting Models
================

We can now export our model using the ``AcumosSession`` object created
earlier. The ``push`` and ``dump`` APIs are shown below. The ``dump`` method will
save the model to disk so that it can be onboarded via the Acumos website. The
``push`` method pushes the model directly to Acumos.

.. code:: python

    session.push(model, 'my-model')
    session.dump(model, 'my-model', '~/')  # creates ~/my-model

For more information on how to onboard a dumped model via the Acumos website,
see the `web onboarding guide <https://docs.acumos.org/en/latest/AcumosUser/portal-user/portal/portal-onboarding-web.html>`__.

**Note:** Pushing a model to Acumos will prompt you for an onboarding token if
you have not previously provided one. The interactive prompt can be avoided by
exporting the ``ACUMOS_TOKEN`` environment variable, which corresponds to an
authentication token that can be found in your account settings on the Acumos
website.

Defining Types
==============

In this example, we make a model that can read binary images and output
some metadata about them. This model makes use of a custom type
``ImageShape``.

We first create a ``NamedTuple`` type called ``ImageShape``, which is
like an ordinary ``tuple`` but with field accessors. We can then use
``ImageShape`` as the return type of ``get_shape``. Note how
``ImageShape`` can be instantiated as a new object.

.. code:: python

    import io
    import PIL

    ImageShape = create_namedtuple('ImageShape', [('width', int), ('height', int)])

    def get_format(data: bytes) -> str:
        '''Returns the format of an image'''
        buffer = io.BytesIO(data)
        img = PIL.Image.open(buffer)
        return img.format

    def get_shape(data: bytes) -> ImageShape:
        '''Returns the width and height of an image'''
        buffer = io.BytesIO(data)
        img = PIL.Image.open(buffer)
        shape = ImageShape(width=img.width, height=img.height)
        return shape

    model = Model(get_format=get_format, get_shape=get_shape)

**Note:** Starting in Python 3.6, you can alternatively use this simpler
syntax:

.. code:: python

    from acumos.modeling import NamedTuple

    class ImageShape(NamedTuple):
        '''Type representing the shape of an image'''
        width: int
        height: int

Using DataFrames with scikit-learn
==================================

In this example, we train a ``RandomForestClassifier`` using
``scikit-learn`` and use it to create an Acumos model.

When making machine learning models, it’s common to use a dataframe data
structure to represent data. To make things easier, ``acumos`` can
create ``NamedTuple`` types directly from ``pandas.DataFrame`` objects.

``NamedTuple`` types created from ``pandas.DataFrame`` objects store
columns as named attributes and preserve column order. Because
``NamedTuple`` types are like ordinary ``tuple`` types, the resulting
object can be iterated over. Thus, iterating over a ``NamedTuple``
dataframe object is the same as iterating over the columns of a
``pandas.DataFrame``. As a consequence, note how ``np.column_stack`` can
be used to create a ``numpy.ndarray`` from the input ``df``.

Finally, the model returns a ``numpy.ndarray`` of ``int`` corresponding
to predicted iris classes. The ``classify_iris`` function represents
this as ``List[int]`` in the signature return.

.. code:: python

    import numpy as np
    import pandas as pd
    from sklearn.datasets import load_iris
    from sklearn.ensemble import RandomForestClassifier

    iris = load_iris()
    X = iris.data
    y = iris.target

    clf = RandomForestClassifier(random_state=0)
    clf.fit(X, y)

    # here, an appropriate NamedTuple type is inferred from a pandas DataFrame
    X_df = pd.DataFrame(X, columns=['sepal_length', 'sepal_width', 'petal_length', 'petal_width'])
    IrisDataFrame = create_dataframe('IrisDataFrame', X_df)

    # ==================================================================================
    # # or equivalently:
    #
    # IrisDataFrame = create_namedtuple('IrisDataFrame', [('sepal_length', List[float]),
    #                                                     ('sepal_width', List[float]),
    #                                                     ('petal_length', List[float]),
    #                                                     ('petal_width', List[float])])
    # ==================================================================================

    def classify_iris(df: IrisDataFrame) -> List[int]:
        '''Returns an array of iris classifications'''
        X = np.column_stack(df)
        return clf.predict(X)

    model = Model(classify=classify_iris)

Check out the ``sklearn`` examples in the examples directory for full
runnable scripts.

Declaring Requirements
======================

If your model depends on another Python script or package that you wrote, you can
declare the dependency via the ``acumos.metadata.Requirements`` class:

.. code:: python

    from acumos.metadata import Requirements

Note that only pure Python is supported at this time.

Custom Scripts
--------------

Custom scripts can be included by giving ``Requirements`` a sequence of paths
to Python scripts, or directories containing Python scripts. For example, if the
model defined in ``model.py`` depended on ``helper1.py``:

::

    model_workspace/
    ├── model.py
    ├── helper1.py
    └── helper2.py

this dependency could be declared like so:

.. code:: python

    from helper1 import do_thing

    def transform(x: int) -> int:
        '''Does the thing'''
        return do_thing(x)

    model = Model(transform=transform)

    reqs = Requirements(scripts=['./helper1.py'])

    # using the AcumosSession created earlier:
    session.push(model, 'my-model', reqs)
    session.dump(model, 'my-model', '~/', reqs)  # creates ~/my-model

Alternatively, all Python scripts within ``model_workspace/`` could be included
using:

.. code:: python

    reqs = Requirements(scripts=['.'])

Custom Packages
---------------

Custom packages can be included by giving ``Requirements`` a sequence of paths to
Python packages, i.e. directories with an ``__init__.py`` file. Assuming that the
package ``~/repos/my_pkg`` contains:

::

    my_pkg/
    ├── __init__.py
    ├── bar.py
    └── foo.py

then you can bundle ``my_pkg`` with your model like so:

.. code:: python

    from my_pkg.bar import do_thing

    def transform(x: int) -> int:
        '''Does the thing'''
        return do_thing(x)

    model = Model(transform=transform)

    reqs = Requirements(packages=['~/repos/my_pkg'])

    # using the AcumosSession created earlier:
    session.push(model, 'my-model', reqs)
    session.dump(model, 'my-model', '~/', reqs)  # creates ~/my-model

Requirement Mapping
-------------------

Python packaging and `PyPI <https://pypi.org/>`__ aren’t
perfect, and sometimes the name of the Python package you import in your
code is different than the package name used to install it. One example
of this is the ``PIL`` package, which is commonly installed using `a fork
called pillow <https://pillow.readthedocs.io>`_ (i.e.
``pip install pillow`` will provide the ``PIL`` package).

To address this inconsistency, the ``Requirements``
class allows you to map Python package names to PyPI package names. When
your model is analyzed for dependencies by ``acumos``, this mapping is
used to ensure the correct PyPI packages will be used.

In the example below, the ``req_map`` parameter is used to declare a
requirements mapping from the ``PIL`` Python package to the ``pillow``
PyPI package:

.. code:: python

    reqs = Requirements(req_map={'PIL': 'pillow'})

Declaring Options
=================

The ``acumos.metadata.Options`` class is a collection of options that users may
wish to specify along with their Acumos model. If an ``Options`` instance is not
provided to ``AcumosSession.push``, then default options are applied. See the
class docstring for more details.

Below, we demonstrate how options can be used to include additional model metadata
and influence the behavior of the Acumos platform. For example, a license can be
included with a model via the ``license`` parameter, either by providing a license
string or a path to a license file. Likewise, we can specify whether or not the Acumos
platform should eagerly build the model microservice via the ``create_microservice``
parameter.

.. code:: python

    from acumos.metadata import Options

    opts = Options(license="Apache 2.0",       # "./path/to/license_file" also works
                   create_microservice=False,  # don't build the microservice yet

    session.push(model, 'my-model', options=opts)

Keras and TensorFlow
====================

Check out the Keras and TensorFlow examples in the ``examples/`` directory of
the `Acumos Python client repository <https://gerrit.acumos.org/r/gitweb?p=acumos-python-client.git;a=summary>`__.

Testing Models
==============

The ``acumos.modeling.Model`` class wraps your custom functions and
produces corresponding input and output types. This section shows how to
access those types for the purpose of testing. For simplicity, we’ll
create a model using the ``add_numbers`` function again:

.. code:: python

    def add_numbers(x: int, y: int) -> int:
        '''Returns the sum of x and y'''
        return x + y

    model = Model(add=add_numbers)

The ``model`` object now has an ``add`` attribute, which acts as a
wrapper around ``add_numbers``. The ``add_numbers`` function can be
invoked like so:

.. code:: python

    result = model.add.inner(1, 2)
    print(result)  # 3

The ``model.add`` object also has a corresponding *wrapped* function
that is generated by ``acumos.modeling.Model``. The wrapped function is
the primary way your model will be used within Acumos.

We can access the ``input_type`` and ``output_type`` attributes to test
that the function works as expected:

.. code:: python

    AddIn = model.add.input_type
    AddOut = model.add.output_type

    add_in = AddIn(1, 2)
    print(add_in)  # AddIn(x=1, y=2)

    add_out = AddOut(3)
    print(add_out)  # AddOut(value=3)

    model.add.wrapped(add_in) == add_out  # True

More Examples
=============

Below are some additional function examples. Note how ``numpy`` types
can even be used in type hints, as shown in the ``numpy_sum`` function.

.. code:: python

    from collections import Counter
    import numpy as np

    def list_sum(x: List[int]) -> int:
        '''Computes the sum of a sequence of integers'''
        return sum(x)

    def numpy_sum(x: List[np.int32]) -> np.int32:
        '''Uses numpy to compute a vectorized sum over x'''
        return np.sum(x)

    def count_strings(x: List[str]) -> Dict[str, int]:
        '''Returns a count mapping from a sequence of strings'''
        return Counter(x)

.. ===============LICENSE_START=======================================================
.. Acumos CC-BY-4.0
.. ===================================================================================
.. Copyright (C) 2017-2018 AT&T Intellectual Property & Tech Mahindra. All rights reserved.
.. ===================================================================================
.. This Acumos documentation file is distributed by AT&T and Tech Mahindra
.. under the Creative Commons Attribution 4.0 International License (the "License");
.. you may not use this file except in compliance with the License.
.. You may obtain a copy of the License at
..
..      http://creativecommons.org/licenses/by/4.0
..
.. This file is distributed on an "AS IS" BASIS,
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.. See the License for the specific language governing permissions and
.. limitations under the License.
.. ===============LICENSE_END=========================================================

==================================
Acumos Python Client Release Notes
==================================

v0.8.0
======

-  Enhancements

   - Users may now specify additional options when pushing their Acumos model. See the options section in the tutorial for more information.
   - ``acumos`` now supports Keras models built with ``tensorflow.keras``

-  Support changes

   - ``acumos`` no longer supports Python 3.4


v0.7.2
======

-  Bug fixes

   - The deprecated authentication API is now considered optional
   - A more portable path solution is now used when saving models, to avoid issues with models developed in Windows


v0.7.1
======

-  Authentication

   - Username and password authentication has been deprecated
   - Users are now interactively prompted for an onboarding token, as opposed to a username and password

v0.7.0
======

-  Requirements

   - Python script dependencies can now be specified using a Requirements object
   - Python script dependencies found during the introspection stage are now included with the model

v0.6.5
======

-  Bug fixes

   - Don't attempt to use an empty auth token (avoids blank strings to be set in environment)

v0.6.4
======

-  Bug fixes

   - The normalized path of the system base prefix is now used for identifying stdlib packages

v0.6.3
======

-  Bug fixes

   - Improved dependency inspection when using a virtualenv
   - Removed custom packages from model metadata, as it caused image build failures
   - Fixed Python 3.5.2 ordering bug in wrapped model usage

v0.6.2
======

-  TensorFlow

   - Fixed a serialization issue that occurred when using a frozen graph

v0.6.1
======

-  Model upload

   - The JWT is now cleared immediately after a failed upload
   - Additional HTTP information is now included in the error message

v0.6.0
======

-  Authentication token

   -  A new environment variable ``ACUMOS_TOKEN`` can be used to short-circuit
      the authentication process

-  Extra headers

   -  ``AcumosSession.push`` now accepts an optional ``extra_headers`` argument,
      which will allow users and systems to include additional information when
      pushing models to the onboarding server

v0.5.0
======

-  Modeling

   -  Python 3.6 NamedTuple syntax support now tested
   -  User documentation includes example of new NamedTuple syntax

-  Model wrapper

   -  Model wrapper now has APIs for consuming and producing Python
      dicts and JSON strings

-  Protobuf and protoc

   -  An explicit check for protoc is now made, which raises a more
      informative error message
   -  User documentation is more clear about dependence on protoc, and
      provides an easier way to install protoc via Anaconda

-  Keras

   -  The active keras backend is now included as a tracked module
   -  keras_contrib layers are now supported

v0.4.0
======

-  Replaced library-specific onboarding functions with “new-style”
   models

   -  Support for arbitrary Python functions using type hints
   -  Support for custom user-defined types
   -  Support for TensorFlow models
   -  Improved dependency introspection
   -  Improved object serialization mechanisms

.. ===============LICENSE_START=======================================================
.. Acumos CC-BY-4.0
.. ===================================================================================
.. Copyright (C) 2017-2018 AT&T Intellectual Property & Tech Mahindra. All rights reserved.
.. ===================================================================================
.. This Acumos documentation file is distributed by AT&T and Tech Mahindra
.. under the Creative Commons Attribution 4.0 International License (the "License");
.. you may not use this file except in compliance with the License.
.. You may obtain a copy of the License at
..
..      http://creativecommons.org/licenses/by/4.0
..
.. This file is distributed on an "AS IS" BASIS,
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.. See the License for the specific language governing permissions and
.. limitations under the License.
.. ===============LICENSE_END=========================================================

====================================
Acumos Python Client Developer Guide
====================================

Testing
=======

We use a combination of ``tox``, ``pytest``, and ``flake8`` to test
``acumos``. Code which is not PEP8 compliant (aside from E501) will be
considered a failing test. You can use tools like ``autopep8`` to
“clean” your code as follows:

.. code:: bash

    $ pip install autopep8
    $ cd acumos-python-client
    $ autopep8 -r --in-place --ignore E501 acumos/ testing/ examples/

Run tox directly:

.. code:: bash

    $ cd acumos-python-client
    $ export WORKSPACE=$(pwd)  # env var normally provided by Jenkins
    $ tox

You can also specify certain tox environments to test:

.. code:: bash

    $ tox -e py36  # only test against Python 3.6
    $ tox -e flake8  # only lint code


