Metadata-Version: 2.1
Name: Kqlmagic
Version: 0.1.55
Summary: Kqlmagic: KQL (Kusto Query Language), enable query Azure Monitor data from a Jupyter compatibe notebook using magic
Home-page: https://pypi.python.org/pypi/jupyter-kql-magic
Author: Michael Binshtock
Author-email: michabin@microsoft.com
Maintainer: Michael Binshtock
Maintainer-email: michabin@microsoft.com
License: MIT License
Download-URL: https://pypi.python.org/pypi/jupyter-kql-magic
Keywords: database ipython jupyter jupyterlab jupyter-notebook query language kql kusto loganalytics applicationinsights
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: Intended Audience :: Customer Service
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: Environment :: Console
Classifier: Environment :: Plugins
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: English
Classifier: Topic :: Database
Classifier: Topic :: Database :: Front-Ends
Classifier: Framework :: IPython
Classifier: Framework :: Jupyter
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Requires-Dist: ipython (>=7.0.1)
Requires-Dist: plotly (>=3.3.0)
Requires-Dist: prettytable (>=0.7.2)
Requires-Dist: matplotlib (>=3.0.0)
Requires-Dist: pandas (>=0.23.4)
Requires-Dist: azure-kusto-data (>=0.0.15)
Requires-Dist: azure-kusto-ingest (>=0.0.15)
Requires-Dist: adal (>=1.1.0)
Requires-Dist: Pygments (>=2.2.0)
Requires-Dist: seaborn (>=0.9.0)
Requires-Dist: requests (>=2.19.1)
Requires-Dist: python-dateutil (>=2.7.3)
Requires-Dist: traitlets
Requires-Dist: psutil
Requires-Dist: six (>=1.11.0)
Requires-Dist: setuptools (>=40.4.3)


.. image:: https://mybinder.org/badge.svg :target: https://mybinder.org/v2/gh/mbnshtck/jupyter-kql-magic/master?filepath=notebooks%2FQuickStart.ipynb


.. image:: https://mybinder.org/badge.svg :target: https://mybinder.org/v2/gh/mbnshtck/jupyter-kql-magic/master?filepath=notebooks%2FQuickStartAI.ipynb


jupyter-Kqlmagic
===========

Extension (Magic) to Jupyter notebook and Jupyter lab, that enable notebook experience working with Kusto, ApplicationInsights, and LogAnalytics data. 
===========

:Author: Michael Binshtock, mbnshtck@gmail.com

Introduces a %kql (or %%kql) magic.


Connect to kusto, using a connect strings, then issue KQL
commands within IPython or IPython Notebook.

.. image:: https://github.com/mbnshtck/jupyter-kql-magic/master/examples/writers.png
   :width: 600px
   :alt: screenshot of jupyter-kql-magic in the Notebook

Examples
--------

.. code-block:: python

    In [1]: %load_ext kql

    In [2]: %%kql kusto://username('myname').password('mypassword').cluster('mycluster').database('mydatabase')
       ...: character
       ...: | where abbrev = 'ALICE'
       ...:
    Out[2]: [(u'Alice', u'Alice', u'ALICE', u'a lady attending on Princess Katherine', 22)]

    In [3]: result = _

    In [4]: print(result)
    charid   charname   abbrev                description                 speechcount
    =================================================================================
    Alice    Alice      ALICE    a lady attending on Princess Katherine   22

    In [4]: result.columns_name
    Out[5]: [u'charid', u'charname', u'abbrev', u'description', u'speechcount']

    In [6]: result[0][0]
    Out[6]: u'Alice'

    In [7]: result[0].description
    Out[7]: u'a lady attending on Princess Katherine'

After the first connection, connect info can be omitted::

    In [8]: %kql work | count
    Out[8]: [(43L)]

Connections to multiple databases can be maintained.  You can refer to
an existing connection by database@cluster

.. code-block:: python

    In [9]: %%kql mydatabase1@mycluster
       ...: character
       ...: | where  speechcount = (character | summarize max(speechcount))
       ...: | project charname, speechcount
       ...:
    Out[9]: [(u'Poet', 733)]

    In [10]: print(_)
    charname   speechcount
    ======================
    Poet       733

If no connect string is supplied, ``%kql`` will provide a list of existing connections;
however, if no connections have yet been made and the environment variable ``KQLMAGIC_CONNECTION_STR``
is available, that will be used.

For secure access, you may dynamically access your credentials (e.g. from your system environment or `getpass.getpass`) to avoid storing your password in the notebook itself. Use the `$` before any variable to access it in your `%kql` command.

.. code-block:: python

    In [11]: user = os.getenv('SOME_USER')
       ....: password = os.getenv('SOME_PASSWORD')
       ....: connection_string = "kusto://username('{user}'.password('{password}').cluster('some_cluster').database('some_database')".format(user=user, password=password)
       ....: %kql $connection_string
    Out[11]: u'Connected: some_database@some_cluster'

You may use multiple Kql statements inside a single cell, but you will
only see any query results from the last of them, so this really only
makes sense for statements with no output

.. code-block:: python

    In [11]: %%kql
       ....: work | limit 1
       ....: work | count
       ....:
    Out[11]: [(43L)]


Bind variables (bind parameters) can be used in the "named" (:x) style.
The variable names used should be defined in the local namespace

.. code-block:: python

    In [12]: name = 'Countess'

    In [13]: %kql select description from character where charname = :name
    Out[13]: [(u'mother to Bertram',)]

As a convenience, dict-style access for result sets is supported, with the
leftmost column serving as key, for unique values.

.. code-block:: python

    In [14]: result = %kql work

    In [15]: result['richard2']
    Out[15]: (u'richard2', u'Richard II', u'History of Richard II', 1595, u'h', None, u'Moby', 22411, 628)

Results can also be retrieved as an iterator of dictionaries (``result.dicts_iterator()``)
or a single dictionary with a tuple of scalar values per key (``result.to_dict()``)

Assignment
----------

Ordinary IPython assignment works for single-line `%kql` queries:

.. code-block:: python

    In [16]: works = %kql work | project title, year

The `<<` operator captures query results in a local variable, and
can be used in multi-line ``%%kql``:

.. code-block:: python

    In [17]: %%kql works << work
        ...: | project title, year
        ...:
    Returning data to local variable works

Connecting
----------

Some example connection strings::

    kusto://username('username').password('password').cluster('clustername').database('databasename')
    kusto://username('username').password('password').cluster('clustername')
    kusto://username('username').password('password')
    kusto://cluster('clustername').database('databasename')
    kusto://cluster('clustername')
    kusto://database('databasename')


Configuration
-------------

Query results are loaded as lists, so very large result sets may use up
your system's memory and/or hang your browser.  There is no auto_limit
by default.  However, `auto_limit` (if set) limits the size of the result
set (usually with a `LIMIT` clause in the KQL).  `display_limit` is similar,
but the entire result set is still pulled into memory (for later analysis);
only the screen display is truncated.

.. code-block:: python

    In [2]: %config Kqlmagic
    Kqlmagic options
    --------------
    Kqlmagic.auto_limit=<Int>
        Current: 0
        Automatically limit the size of the returned result sets
    Kqlmagic.auto_dataframe=<Bool>
        Current: False
        Return Pandas DataFrames instead of regular result sets
    Kqlmagic.display_limit=<Int>
        Current: 0
        Automatically limit the number of rows displayed (full result set is still
        stored)
    Kqlmagic.feedback=<Bool>
        Current: True
        Print number of records returned, and assigned variables
    Kqlmagic.short_errors=<Bool>
        Current: True
        Don't display the full traceback on KQL Programming Error
    Kqlmagic.prettytable_style=<Unicode>
        Current: 'DEFAULT'
        Set the table printing style to any of prettytable's defined styles
        (currently DEFAULT, MSWORD_FRIENDLY, PLAIN_COLUMNS, RANDOM)

    In[3]: %config Kqlmagic.feedback = False

Please note: if you have auto_dataframe set to true, the option will not apply. You can set the pandas display limit by using the pandas ``max_rows`` option as described in the `pandas documentation <http://pandas.pydata.org/pandas-docs/version/0.18.1/options.html#frequently-used-options>`_.

Pandas
------

If you have installed ``pandas``, you can use a result set's
``.DataFrame()`` method

.. code-block:: python

    In [3]: result = %kql character | where speechcount > 25

    In [4]: dataframe = result.DataFrame()

.. _Pandas: http://pandas.pydata.org/

Graphing
--------

If you have installed ``matplotlib``, you can use a result set's
``.plot()``, ``.pie()``, and ``.bar()`` methods for quick plotting

.. code-block:: python

    In[5]: result = %kql work | where genretype = 'c' | project title, totalwords

    In[6]: %matplotlib inline

    In[7]: result.pie()

.. image:: https://raw.github.com/catherinedevlin/ipython-sql/master/examples/wordcount.png
   :alt: pie chart of word count of Shakespeare's comedies

Dumping
-------

Result sets come with a ``.csv(filename=None)`` method.  This generates
comma-separated text either as a return value (if ``filename`` is not
specified) or in a file of the given name.

.. code-block:: python

    In[8]: result = %kql work | where genretype = 'c' | project title, totalwords 

    In[9]: result.csv(filename='work.csv')


Installing
----------

Install the lastest release with::

    pip install jupyter-kql-magic

or download from https://github.com/mbnshtck/jupyter-kql-magic and::

    cd jupyter-kql-magic
    sudo python setup.py install

Development
-----------

https://github.com/mbnshtck/jupyter-kql-magic

Credits
-------

- Kql_
- Kusto_
- Distribute_
- Buildout_
- modern-package-template_

.. _Distribute: http://pypi.python.org/pypi/distribute
.. _Buildout: http://www.buildout.org/
.. _modern-package-template: http://pypi.python.org/pypi/modern-package-template
.. _Kql: https://kusto.azurewebsites.net/docs/queryLanguage/query_language.html
.. _Kusto: https://kusto.azurewebsites.net/docs/


News
====

0.1.1
---

*Release date: 06-Sep-2017*

* Initial release

0.1.3
---

*Release date: 01-July-2018*

KQL Interfaces
* Query kusto
  - Multi Tenant autheticate with client/secret or user/pw or code
* Query Application insight
  - Authenticate with appid/appkey
* Query Log Analytics
  - Authenticate with workspace/appkey

ConnectionString
* Explicit connection, that starts with one of the support schems: kusto://, appinsights://, loganalytics://
  i.e. kusto://tenant('authority').username('name').password('pw').clustername('cname').database('dbname')
       kusto://tenant('authority').clientid('cid').clientsecret('secret').clustername('cname').database('dbname')
       kusto://tenant('authority').code().clustername('cname').database('dbname')
       appinsights://appid('id').appkey('key')
       loganalytics://workspace('ws').appkey('key')
* Reference to an already existing connection: database@cluster, appid@appinsights, workspace@loganalytics
* Reference to a section in config file, that contains all the fields of the connection string (DSN style), file name must be set in config.dsn_filename
* Substring of the form $name or ${name}, in windows also %name% if found in env variabes. 
  i.e. $MY_CONNECTION 
       ${MY_CONNECTION}
       %MY_CONNECTION%
* Current connection. If connection string is not specified, current (last) connection is used.
* Partial explicit connection string. If some fields are missing they are set to a default value if exist, or are taken from current (last) connection, or user is prompted.
  i.e. if kusto://... is missing tenant(...) component, 'microsoft.com' is used
       if kusto://... is missing credentials component (username.password or clientid/clientsecret or code()), credentials are taken from current(last) connection
       if kusto://... is missing cluster(...) component, cluster value is taken from current (last) connection
       if credential secret is missing, user is prompted to enter the credential secret

Authentication
* loganalytics://, workspace/appkey. If appkey is missing in connection string, user is prompted to enter appkey
* appinsights://, appid/appkey. If appkey is missing in connection string, user is prompted to enter appkey
* kusto://, username/password. If password is missing in connection string, user is prompted to enter password
* kusto://, clientid/clientsecret. If clientsecret is missing in connection string, user is prompted to enter clientsecret
* kusto://, code. code is displayed, and user is reffered to a the authentication page to enter the code.

ConnectionString validation
* If a single line kql magic (%kql) contains the connectionString part, but without the query part, an implicit validation query is used, to validate the connectionString

UI
* info, warning and error messages - are displayed in colors, blue, green, and brown







TODO
*Support parametrize based on local/global ns


Authetication
- Authenticate loganalytics with additional schemes (current authentication is appKey only which used mainly for demo)
- Solve ambiguity if kusto authentication to same cluster is definned by two differnt schemes, for example code and user/pw
- Make kusto code() authentication more user friendly
- Make user/pw authentication when used from Windows browser, automatic based on windows user/pw
- when missing components in connection string are taken from current connection, they should be taken from the same type of connection, if exist

Ingestion
- support basic ingestion to kusto, appinsight and loganalytics

Performance
- Add retries to kusto ,appinsights and loganalytics for some errors
- Add option to use WebGl in rendering using plotly Scattergl instead of scatter for increased speed (see plotly interface)

UI
- make kql errors that return from server easier to read (open json)
- detect unknown refernces to  connection string of type db@cluster and provide a better result, instead of current error message

Connection Management
- if connection fails to connect either should be marked as such or shouldn't be in list

08/25/2018 - Kusto Connection to a new database on a cluster that already have a database connected, will skip authentication, and all databases on same cluster will use same authenticated client to that cluster.
             The new connection_str can be either kusto://... or database@cluster
08/26/2018 - replaced show_conn_list option with show_conn_info that has 3 states: 'current', 'list' and 'None'
08/26/2018 - added the following methods:
  display_table - will display the result table in the cell
  display_chart - will display the result chart in the cell
  display - will display the result in the cell


