aiscalator.core package

Submodules

aiscalator.core.config module

Handles configurations files for the application

class aiscalator.core.config.AiscalatorConfig(user_config_override=None, step_config=None, steps_selection=None)

Bases: object

A configuration object for the Aiscalator application.

This object stores:

• global configuration for the whole application
• configuration for a particular context specified in a step configuration file.
• In this case, we might even focus on a particular step.

…

Variables:

• _user_config_override (str) – path to the specified user config
• _app_conf – global configuration object for the application
• _step_config (str) – path to the step configuration (or plain configuration as string)
• _focused_steps (list) – list of selected steps
• _step_it (int) – index of the step being processed
• _step – configuration object for the currently processed step

Methods

app_config()	returns:str – the configuration object for the aiscalator application
app_config_has(field)	Tests if the applicatin config has a configuration value for the field.
container_name()	Return the docker container name to execute this step
extract_parameters()	Returns a list of docker parameters
file_path(string)	Returns absolute path of a file from a field of the currently focused step.
find_user_config_file(filename)	Looks for configuration files in the user configuration folder
has_step_field(field)	Tests if the currently focused step has a configuration value for the field.
next_step()	Iterates to the next configuration step from the list of selected steps
notebook_output_path(notebook)	Generates the name of the output notebook
root_dir()	returns:str – Returns the path to the folder containing the step
setup_app_config()	Setup global application configuration.
step_config_path()	returns:str – Returns the path to the step configuration file.
step_field(field)	Returns the value associated with the field for the currently focused step.
step_name()	Returns the name of the currently focused step
timestamp_now()	Depending on how the timezone is configured, returns the timestamp for this instant.
user_env_file()	Find a list of env files to pass to docker containers
user_id()	returns:str – the user id stored when the application was first setup
validate_config()	Check if all the fields in the reference config are defined in focused steps too.

app_config()

Returns:str – the configuration object for the aiscalator application

app_config_has(field) → bool

Tests if the applicatin config has a configuration value for the field.

container_name() → str

Return the docker container name to execute this step

extract_parameters() → list

Returns a list of docker parameters

file_path(string)

Returns absolute path of a file from a field of the currently focused step.

find_user_config_file(filename) → str

Looks for configuration files in the user configuration folder

Parameters:filename (str) – file to search for Returns:str – path to the filename in the user configuration folder

has_step_field(field) → bool

Tests if the currently focused step has a configuration value for the field.

next_step()

Iterates to the next configuration step from the list of selected steps

Returns:the next configuration object

notebook_output_path(notebook) → str

Generates the name of the output notebook

root_dir()

Returns:str – Returns the path to the folder containing the step configuration file

setup_app_config()

Setup global application configuration. If not found in the default location, this method will generate a brand new one.

step_config_path()

Returns:str – Returns the path to the step configuration file. If it was an URL, it will return the path to the temporary downloaded version of it. If it was a plain string, then returns None

step_field(field)

Returns the value associated with the field for the currently focused step.

step_name()

Returns the name of the currently focused step

timestamp_now() → str

Depending on how the timezone is configured, returns the timestamp for this instant.

user_env_file() → list

Find a list of env files to pass to docker containers

Returns:List – env files

user_id() → str

Returns:str – the user id stored when the application was first setup

validate_config()

Check if all the fields in the reference config are defined in focused steps too. Otherwise raise an Exception (either pyhocon.ConfigMissingException or pyhocon.ConfigWrongTypeException)

aiscalator.core.config.convert_to_format(file: str, output: str, output_format: str)

Converts a HOCON file to another format

Parameters:

• file (str) – hocon file to convert
• output (str) – output file to produce
• output_format (str) – format of the output file

Returns:

str – the output file

aiscalator.core.config.find_tasks(tree: pyhocon.config_tree.ConfigTree, path='')

Find all Tasks objects in the Configuration object and report their paths.

Parameters:

• tree (pyhocon.ConfigTree) – Configuration object
• path (str) – path that was traversed to get to this tree

Returns:

list – list of names of Configuration objects containing a definition of a section ‘task’

aiscalator.core.config.generate_global_config() → str

Generate a standard configuration file for the application in the user’s home folder ~/.aiscalator/config/aiscalator.conf from the template file in aiscalator/config/template/aiscalator.conf

aiscalator.core.config.generate_user_id() → str

Returns:str – Returns a string identifying this user when the setup was run first

aiscalator.core.config.load_logging_conf(file)

Reads and loads the logging configuration file

aiscalator.core.config.parse_step_config(step_config)

Interpret the step_config to produce a step configuration object. It could be provided as: - a path to a local file - a url to a remote file - the plain configuration stored as string

Returns:Step configuration object

aiscalator.core.config.select_steps(step_conf, steps_selection: list) → list

Extract the list of step objects corresponding to the list of names provided.

Parameters:

• step_conf – step configuration object
• steps_selection (list) – list of names of step to extract

Returns:

list – list of tuples of (step_name, step) of selected configuration objects

aiscalator.core.config.setup_logging()

Setup the logging configuration of the application

aiscalator.core.config.validate_configs(test, reference, path, missing_exception=True, type_mismatch_exception=True)

Recursively check two configs if they match

Parameters:

• test – configuration object to test
• reference – reference configuration object
• path (str) – this accumulates the recursive path for details in Exceptions
• missing_exception (bool) – when a missing field is found, raise xception?
• type_mismatch_exception (bool) – when a field has type mismatch, raise xception?

aiscalator.core.log_regex_analyzer module

Class to parse output logs from subprocess and catch particular expressions

class aiscalator.core.log_regex_analyzer.LogRegexAnalyzer(pattern=None)

Bases: object

A regular expression analyzer object to parse logs and extract values from patterns in the logs. …

Variables:

• _artifact (str) – Value of the pattern found in the logs
• _pattern (bytes) – Regular expression to search for in the logs

Methods

artifact()	Returns the artifact extracted from the logs.
grep_logs(pipe)	Reads the logs and extract values defined by the pattern

artifact()

Returns the artifact extracted from the logs.

grep_logs(pipe)

Reads the logs and extract values defined by the pattern

Parameters:pipe – Stream of logs to analyze

aiscalator.core.utils module

Various Utility functions

class aiscalator.core.utils.BackgroundThreadRunner(command, log_function, no_redirect=False)

Bases: object

Worker Thread to run logging output in the background

…

Variables:

• _process – Process object of the command running in the background
• _log_function (function(stream -> bool)) – callback function to log the output of the command
• _no_redirect (bool) – whether the subprocess STDOUT and STDERR should be redirected to logs
• _worker (Thread) – Thread object

Methods

process()	Returns the process object.
run()	Starts the Thread, process the output of the process.

process()

Returns the process object.

run()

Starts the Thread, process the output of the process.

aiscalator.core.utils.copy_replace(src, dst, pattern='', replace_value='')

Copies a file from src to dst replacing pattern by replace_value

Parameters:

• src (string) – Path to the source filename to copy from
• dst (string) – Path to the output filename to copy to
• pattern (string) – Pattern to replace inside the src file
• replace_value – Value to replace by in the dst file

aiscalator.core.utils.data_file(path)

Utility function to find resources data file packaged along with code

Parameters:path (path) – path to the resource file in the package Returns:absolute path to the resource data file

aiscalator.core.utils.find(collection, item, field='name')

Finds an element in a collection which has a field equal to particular item value

Parameters:

• collection (Set) – Collection of objects
• item – value of the item that we are looking for
• field (string) – Name of the field from the object to inspect

Returns:

object – Corresponding element that has a field matching item in the collection

aiscalator.core.utils.log_info(pipe)

Default logging function

aiscalator.core.utils.subprocess_run(command, log_function=<function log_info>, no_redirect=False, wait=True)

Run command in a subprocess while redirecting output to log_function.

The subprocess either runs synchroneoulsy or in the background depending on the wait parameter.

Parameters:

• command (List) – Command to run in the subprocess
• log_function (function) – Callback function to log the output of the subprocess
• no_redirect (bool) – whether the subprocess STDOUT and STDERR should be redirected to logs
• wait (bool) – Whether the subprocess should be run synchroneously or in the background

Returns:

• int – return code of the subprocess
• BackgroundThreadRunner – the thread running in the background

Module contents