Development setup
=================

Running nose tests with Jupyter is tricky, so there's a
run_tests.sh script for it.

To temporarily insert breakpoints for debugging: `from nose.tools import set_trace; set_trace()`

Tests have requirements not installed by setup.py:

- nose
- pandas

Release HOWTO
=============

To make a release,

  1) Update release date/version in NEWS.txt and setup.py
  2) Run 'python setup.py sdist'
  3) Test the generated source distribution in dist/
  4) Upload to PyPI: 'python setup.py sdist register upload'
  5) Increase version in setup.py (for next release)

  Get kusto_client
  1. link https://kusto.azurewebsites.net/docs/api/kusto_python_client_library.html
  2. launch anaconda prompt (folder "C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Anaconda3 (64-bit)")
  2. pip install http://52.173.187.55/simple/kusto_client-0.4.0-py2.py3-none-any.whl --no-cache-dir --upgrade
  3. pip install prettytable from PyPI --no-cache-dir --upgrade



  Run test
  1. launch anaconda prompt (folder "C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Anaconda3 (64-bit)")
  2. >cd c:\My Projects\KqlMagic\azure
  3. >set TEST_CONNECTION_STR=kusto://username('michabin@microsoft.com').password(<password>).cluster('Oiildc').database('OperationInsights_PFS_PROD')
  4. >set KQLMAGIC_CONNECTION_STR=kusto://username('michabin@microsoft.com').password(<password>).cluster('Oiildc').database('OperationInsights_PFS_PROD')
  5. >set HELP_CONNECTION_STR=kusto://username('michabin@microsoft.com').password(<password>).cluster('help').database('Samples')
  6. >
  3  >ipython -c "import nose; nose.run()"

  Add to Jupyter Notebook
  1. %cd c:\My Projects\KqlMagic\azure
  2. %load_ext kql               (in case of modified: %reload_ext kql)

  Hints:
  F11 toggle Chrome to full screen
  View/Toggle Header 
  View/Toggle Toolbar
  Kernel/Restart & Clear Output

  Visualization Technologies
  PlotLy (PlotLy)
  PixieDust (IBM)

  Notes for Jupyter Notebbok
      conda install -c conda-forge ipywidgets

  Note: modified file C:\Users\michabin\AppData\Local\Continuum\Anaconda3\Lib\site-packages\jupyterlab\staging\webpack.prod.config.js
    parallel: false, (was true) to overcome ['webpack' is not recognized as an internal or external command] error in jupyter lab build


  Notes for jupyter labs environment:
    1. need to install npm (and restart computer), to be able to install extensions
    2. A Plotly - enables use of plotly offline:
        jupyter labextension install @jupyterlab/plotly-extension
        jupyter labextension enable @jupyterlab/plotly-extension
    3. Add jupyter widgets extension:
        jupyter labextension install @jupyter-widgets/jupyterlab-manager@0.27.0
        jupyter labextension enable @jupyter-widgets/jupyterlab-manager
    4. Add qgrid extension:
        jupyter labextension install qgrid-jupyterlab@1.0.0-beta.7
        jupyter labextension enable qgrid-jupyterlab

    3. A JupyterLab extension for rendering JSON as a tree:
        jupyter labextension install @jupyterlab/json-extension

Update jupyter
conda update jupyter

run commandline
activate environment
  C:\Users\michabin\AppData\Local\Continuum\Anaconda3\Scripts\activate
  set TEST_CONNECTION_STR=kusto://username('michabin@microsoft.com').password('<password>').cluster('Oiildc').database('OperationInsights_PFS_PROD')
  // env default connection string
  set KQLMAGIC_CONNECTION_STR=kusto://username('michabin@microsoft.com').password('<password>').cluster('Oiildc').database('OperationInsights_PFS_PROD')

  set HELP_CONNECTION_STR=kusto://username('michabin@microsoft.com').password('<password>').cluster('help').database('Samples')
  set HELP_CONNECTION_STR=kusto://username('michabin@microsoft.com').cluster('help').database('Samples')
  set HELP_CONNECTION_STR=kusto://code().cluster('help').database('Samples')

  cd c:\My Projects
  jupyter lab | jupyter notebook
// set Path=C:\Users\michabin\AppData\Local\Continuum\Anaconda3;%Path%

jupyter-notebook-client - 8430759c-5626-4577-b151-d0755f5355d8       Web app / API   multi-tenant
app access key : BjQr0uQxYoqbo+MlumLSt4qXJVrsNlynutaAAGgmzVE= (secret)
already granted permissions

testws-demo - workspace-id 75ba02bf-531a-43d2-89a7-27c58a610933
aa-weu-demo - workspace-id 903a5133-d37d-4c01-82fa-bd42e2d6f29e

use with clientsecret:
goto log analytics, select your workspace, select Access control (IAM), select +Add (left top), Set Role to LOG ANALYTICS READER, Select jupyter-notebook-client, and press Save

with code, no need for secret


%kql loganalytics://code().workspace('903a5133-d37d-4c01-82fa-bd42e2d6f29e') -!se
%kql loganalytics://tenant('microsoft.com').clientid('8430759c-5626-4577-b151-d0755f5355d8').clientsecret('BjQr0uQxYoqbo+MlumLSt4qXJVrsNlynutaAAGgmzVE=').workspace('75ba02bf-531a-43d2-89a7-27c58a610933')
%kql loganalytics://code().workspace('75ba02bf-531a-43d2-89a7-27c58a610933')

set HELP_CONNECTION_STR=kusto://code().cluster('Oiildc').database('OperationInsights_PFS_PROD')


Sharing notebook via gist
 - create a notebook
   - restart Kernel
   - clear all cells
   - run all
   - save notebook
   - download as ipynb
   - open file in text editor
   - copy the content of the file (CTRL-A, CTRL-C)
   ** Note: local resources such as local images won't be seen, make sure to provide a url instead to a shared store

 - publish the notebook in github gist
    - open with browser url: http://gist.github.com
    - paste the content of the file
    - give it a name with extension .ipynb
    - add description
    - publish by clicking the "Create Public Gist" button 
        ** it will give it a url with anumber 

 - View the notebook with nbviewer
    - open with browser url: http://nbviewer.jupyter.org/
    - provide the gist number or Github username/repo or Github username

   

!pip install git+git://github.com/mbnshtck/jupyter-kql-magic.git --no-cache-dir --upgrade
pip uninstall -y Kqlmagic 

to builf jupyter lab make sure npm registry is set to default:
>> npm config set registry https://registry.npmjs.org/















OLD:
; cli configs
metrics-registry = "https://1essharedassets.pkgs.visualstudio.com/_packaging/Kusto/npm/registry/"
scope = ""
user-agent = "npm/5.10.0 node/v8.11.3 win32 x64"

; userconfig C:\Users\michabin\.npmrc
registry = "https://1essharedassets.pkgs.visualstudio.com/_packaging/Kusto/npm/registry/"

; builtin config undefined
prefix = "C:\\Users\\michabin\\AppData\\Roaming\\npm"

; node bin location = C:\Program Files\nodejs\node.exe
; cwd = C:\Users\michabin
; HOME = C:\Users\michabin
; "npm config ls -l" to show all defaults.
MyBinder:
github location: https://github.com/mbnshtck/jupyter-kql-magic
notebooks/ColorYourCharts.ipynb
Kusto QuickStart
https://mybinder.org/v2/gh/mbnshtck/jupyter-kql-magic/master?filepath=notebooks%2FQuickStart.ipynb
ApplicationInsight
https://mybinder.org/v2/gh/mbnshtck/jupyter-kql-magic/master?filepath=notebooks%2FQuickStartAI.ipynb
ColorYourCharts.ipynb
https://mybinder.org/v2/gh/mbnshtck/jupyter-kql-magic/master?filepath=notebooks%2FColorYourCharts.ipynb

# run before checkin in
black -l 150 KqlMagic\azure\Kqlmagic

C:\Users\michabin\AppData\Local\Continuum\Anaconda3;
C:\Users\michabin\AppData\Local\Continuum\Anaconda3\Library\mingw-w64\bin;
C:\Users\michabin\AppData\Local\Continuum\Anaconda3\Library\usr\bin;
C:\Users\michabin\AppData\Local\Continuum\Anaconda3\Library\bin;
C:\Users\michabin\AppData\Local\Continuum\Anaconda3\Scripts

how to get version of a package from Pypi:
https://pypi.org/simple/<package-name>

cd C:\My Projects\jupyter-Kqlmagic-microsoft
del /Q build
del /Q dist
setup.py sdist bdist_wheel
twine upload -u mbnshtck -p Py112PI@ dist/*
# twine upload -u mbnshtck -p Te112st@ --repository-url https://test.pypi.org/legacy/ dist/*
pip uninstall Kqlmagic
pip install Kqlmagic --no-cache-dir --upgrade
# pip install --index-url https://test.pypi.org/simple/ Kqlmagic --no-cache-dir --upgrade

# https://test.pypi.org login: mbnshtck, Te<112>st@
# https://pypi.org login: mbnshtck, Py<112>PI@












News
====

0.1.1
---

*Release date: 06-Sep-2017*

* Initial release

0.1.3
---

*Release date: 01-July-2018*

KQL Interfaces
* Query kusto
  - Multi Tenant autheticate with client/secret or user/pw or code
* Query Application insight
  - Authenticate with appid/appkey
* Query Log Analytics
  - Authenticate with workspace/appkey

ConnectionString
* Explicit connection, that starts with one of the support schems: kusto://, appinsights://, loganalytics://
  i.e. kusto://tenant('authority').username('name').password('pw').clustername('cname').database('dbname')
       kusto://tenant('authority').clientid('cid').clientsecret('secret').clustername('cname').database('dbname')
       kusto://tenant('authority').code().clustername('cname').database('dbname')
       appinsights://appid('id').appkey('key')
       loganalytics://workspace('ws').appkey('key')
* Reference to an already existing connection: database@cluster, appid@appinsights, workspace@loganalytics
* Reference to a section in config file, that contains all the fields of the connection string (DSN style), file name must be set in config.dsn_filename
* Substring of the form $name or ${name}, in windows also %name% if found in env variabes. 
  i.e. $MY_CONNECTION 
       ${MY_CONNECTION}
       %MY_CONNECTION%
* Current connection. If connection string is not specified, current (last) connection is used.
* Partial explicit connection string. If some fields are missing they are set to a default value if exist, or are taken from current (last) connection, or user is prompted.
  i.e. if kusto://... is missing tenant(...) component, 'microsoft.com' is used
       if kusto://... is missing credentials component (username.password or clientid/clientsecret or code()), credentials are taken from current(last) connection
       if kusto://... is missing cluster(...) component, cluster value is taken from current (last) connection
       if credential secret is missing, user is prompted to enter the credential secret

Authentication
* loganalytics://, workspace/appkey. If appkey is missing in connection string, user is prompted to enter appkey
* appinsights://, appid/appkey. If appkey is missing in connection string, user is prompted to enter appkey
* kusto://, username/password. If password is missing in connection string, user is prompted to enter password
* kusto://, clientid/clientsecret. If clientsecret is missing in connection string, user is prompted to enter clientsecret
* kusto://, code. code is displayed, and user is reffered to a the authentication page to enter the code.

ConnectionString validation
* If a single line kql magic (%kql) contains the connectionString part, but without the query part, an implicit validation query is used, to validate the connectionString

UI
* info, warning and error messages - are displayed in colors, blue, green, and brown







TODO
*Support parametrize based on local/global ns


Authetication
- Authenticate loganalytics with additional schemes (current authentication is appKey only which used mainly for demo)
- Solve ambiguity if kusto authentication to same cluster is definned by two differnt schemes, for example code and user/pw
- Make kusto code() authentication more user friendly
- Make user/pw authentication when used from Windows browser, automatic based on windows user/pw
- when missing components in connection string are taken from current connection, they should be taken from the same type of connection, if exist

Ingestion
- support basic ingestion to kusto, appinsight and loganalytics

Performance
- Add retries to kusto ,appinsights and loganalytics for some errors
- Add option to use WebGl in rendering using plotly Scattergl instead of scatter for increased speed (see plotly interface)

UI
- make kql errors that return from server easier to read (open json)
- detect unknown refernces to  connection string of type db@cluster and provide a better result, instead of current error message

Connection Management
- if connection fails to connect either should be marked as such or shouldn't be in list

08/25/2018 - Kusto Connection to a new database on a cluster that already have a database connected, will skip authentication, and all databases on same cluster will use same authenticated client to that cluster.
             The new connection_str can be either kusto://... or database@cluster
08/26/2018 - replaced show_conn_list option with show_conn_info that has 3 states: 'current', 'list' and 'None'
08/26/2018 - added the following methods:
  display_table - will display the result table in the cell
  display_chart - will display the result chart in the cell
  display - will display the result in the cell
