Metadata-Version: 2.1
Name: alkymi
Version: 0.0.3
Summary: alkymi - Pythonic task automation
Home-page: https://github.com/MathiasStokholm/alkymi
Author: Mathias Bøgh Stokholm
Author-email: mathias.stokholm@gmail.com
License: MIT
Project-URL: Source, https://github.com/MathiasStokholm/alkymi/
Project-URL: Tracker, https://github.com/MathiasStokholm/alkymi/issues
Description: # alkymi ⚗️
        Pythonic task automation
        
        ![build](https://github.com/MathiasStokholm/alkymi/workflows/build/badge.svg?branch=master)
        
        alkymi uses Python's basic building blocks to describe a directed-acyclic-graph (DAG) of computation, and adds a layer
        of caching to only evaluate functions when inputs have changed.
        
        The key idea behind alkymi is to have your data or validation pipeline defined in the same language as the actual
        pipeline steps, allowing you to use standard Python tools (unit testing, linting, type checkers) to check the
        correctness of your full pipeline. No more `make dataset`!
        
        All alkymi tasks (recipes) are created using references to other alkymi recipes. There's no magic tying together inputs
        and outputs based on file names, regexes, etc. - only function calls where alkymi provides the input arguments based on
        outputs further up the DAG.
        
        NOTE: alkymi is very much in the experimental alpha stage, and probably shouldn't be used for anything critical.
        Performance optimizations are still TODO, and only single-threaded evaluation is possible at this point.
        
        ## Sample Code
        Downloading and parsing the MNIST handwritten character dataset w/ caching (see `examples/mnist` for full code)
        ```python
        import alkymi as alk
        
        @alk.recipe()
        def urls() -> List[str]:
            # Return URLs of various parts of the dataset - alkymi will cache these as a list of strings
            train_images_url = "http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz"
            train_labels_url = "http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz"
            test_images_url = "http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz"
            test_labels_url = "http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz"
            return [train_images_url, train_labels_url, test_images_url, test_labels_url]
        
        
        @alk.foreach(urls)
        def download_gzips(url: str) -> bytes:
            # Download each gzip file as raw bytes - alkymi will cache these to binary files
            # This will run once per URL, and only if the URL has changed since the last evaluation
            return urllib.request.urlopen(url).read()
        
        
        @alk.foreach(download_gzips)
        def parse_gzip_to_arrays(data: bytes) -> np.ndarray:
            # Unzip binary data and parse into numpy arrays - alkymi will cache the numpy arrays
            # This will run once per blob of input data, and only if the binary data has changed since the last evaluation
            with io.BytesIO(data) as f:
                with gzip.open(f) as gzip_file:
                    return parse_idx(gzip_file)  # parse_idx definition left out for brevity (see examples/mnist)
        
        
        # Evaluate 'parse_gzip_to_arrays' and all dependencies
        # On subsequent evaluations, the final numpy arrays will be read from the cache and returned immediately - unless one of
        # the recipes is marked dirty (if inputs have changed, or the recipe function itself has changed) - in that case, alkymi
        # will do the minimum amount of work to bring the pipeline up-to-date, and then return the final numpy arrays 
        train_images, train_labels, test_images, test_labels = parse_gzip_to_arrays.brew()
        ```
        Or, if you need to wrap existing functions, you can simply do:
        ```python
        import alkymi as alk
        
        download_archives = alk.foreach(urls)(download_gzips)
        parse_arrays = alk.foreach(download_archives)(parse_gzip_to_arrays)
        train_images, train_labels, test_images, test_labels = parse_arrays.brew()
        ```
        
        ## Documentation
        TODO(mathias): Add a readthedocs.org page
        
        ## Installation
        Install via pip:
        ```shell script
        pip install --user alkymi
        ```
        
        Or clone and install directly from source
        ```shell script
        git clone https://github.com/MathiasStokholm/alkymi.git
        cd alkymi
        pip install --user .
        ```
        
        Or install using pip and github
        ```shell script
        pip install --user git+https://github.com/MathiasStokholm/alkymi.git
        ```
        
        ### Testing
        After installing, you can run the test suite:
        ```shell script
        python3 lab.py brew test
        ```
        
        ## License
        alkymi is licensed under The MIT License as found in the LICENSE.md file
        
Keywords: automation,pipeline,validation,preprocessing,make,build,task
Platform: UNKNOWN
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: MIT License
Classifier: Environment :: Console
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: System Administrators
Classifier: Topic :: Software Development :: Build Tools
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.5
Description-Content-Type: text/markdown
