Metadata-Version: 2.1
Name: agi-pack
Version: 0.1.2
Summary: Dockerfile generator for AGI -- nothing more, nothing less.
License: MIT License
        
        Copyright (c) 2023 Sudeep Pillai
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Documentation, https://spillai.github.io/agi-pack/
Project-URL: Source Code, https://github.com/spillai/agi-pack
Project-URL: Homepage, https://spillai.github.io/agi-pack/
Classifier: Development Status :: 4 - Beta
Classifier: Programming Language :: Python
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Python: >=3.7
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: jinja2
Requires-Dist: pydantic
Requires-Dist: pyyaml
Requires-Dist: typer[all]
Provides-Extra: dev
Requires-Dist: pre-commit ; extra == 'dev'
Requires-Dist: pytest ; extra == 'dev'
Requires-Dist: build ; extra == 'dev'

<h1 align="center" style="font-size:64px;font-weight: bold;font-color:black;">agi-pack</h1>
<h4 align="center">
<i>A Dockerfile builder for AGI — nothing more, nothing less.</i>
</h4>

<p align="center">
<a href="https://pypi.org/project/agi-pack/">
    <img alt="PyPi Version" src="https://badge.fury.io/py/agi-pack.svg">
</a>
<a href="https://pypi.org/project/agi-pack/">
    <img alt="PyPi Version" src="https://img.shields.io/pypi/pyversions/agi-pack">
</a>
<a href="https://pypi.org/project/agi-pack/">
    <img alt="PyPi Downloads" src="https://img.shields.io/pypi/dm/agi-pack">
</a>

</p>

📦 **`agi-pack`** allows you to define your Docker images using a simple YAML format, and then generate them on-the-fly using Jinja2 templates. It's a simple tool that aims to simplify the process of building Docker images for ML.

🚨 **Disclaimer:** More than **75%** of this initial implementation was generated by GPT-4 and [Github Co-Pilot](https://github.com/features/copilot). See [attribution](#inspiration-and-attribution-🌟) section below for more details.

## Installation 📦

```bash
pip install agi-pack
```

For shell completion, you can install them via:
```bash
agi-pack --install-completion <bash|zsh|fish|powershell|pwsh>
```

## Quickstart 🛠

1. Create a simple YAML configuration file called `agibuild.yaml`. You can use `agi-pack init` to generate a sample configuration file.

    ```bash
    agi-pack init
    ```

2. Edit `agibuild.yaml` to define your custom system and python packages.

    ```yaml
    images:
      base-sklearn:
        base: debian:buster-slim
        system:
        - wget
        - build-essential
        python: 3.8.10
        pip:
        - loguru
        - typer
        - scikit-learn
    ```

    Let's break this down:
    - `base-sklearn`: name of the target you want to build. Usually, these could be variants like `base-*`, `dev-*`, `prod-*`, `test-*` etc.
    - `base`: base image to build from.
    - `system`: system packages to install via `apt-get install`.
    - `python`: specific python version to install via `miniconda`.
    - `pip`: python packages to install via `pip install`.

3. Generate the Dockerfile using `agi-pack generate`

    ```bash
    agi-pack generate -c agibuild.yaml
    ```

    You should see the following output:

    ```bash
    $ agi-pack generate -c agibuild.yaml
    📦 base-sklearn
    └── 🎉 Successfully generated Dockerfile (target=base-sklearn, filename=Dockerfile).
        └── `docker build -f Dockerfile --target base-sklearn .`
    ```

That's it! Use the generated `Dockerfile` to run `docker build` and build the image directly.

## Goals 🎯

- **Simplicity**: Make it easy to define and build docker images for ML.
- **Best-practices**: Bring best-practices to building docker images for ML -- good base images, multi-stage builds, minimal image sizes, etc.
- **Modular, Re-usable, Composable**: Define `base`, `dev` and `prod` targets with multi-stage builds, and re-use them wherever possible.
- **Extensible**: Make the YAML / DSL extensible to support the ML ecosystem, as more libraries, drivers, HW vendors, come into the market.
- **Vendor-agnostic**: `agi-pack` is not intended to be built for any specific vendor -- I need this tool for internal purposes, but I decided to build it in the open and keep it simple.

## Rationale 🤔

Docker has become the standard for building and managing isolated environments for ML. However, any one who has gone down this rabbit-hole knows how broken ML development is, especially when you need to experiment and re-configure your environments constantly. Production is another nightmare -- large docker images (`10GB+`), bloated docker images with model weights that are `~5-10GB` in size, 10+ minute long docker build times, sloppy package management to name just a few.

**What makes Dockerfiles painful?** If you've ever tried to roll your own Dockerfiles with all the best-practices while fully understanding their internals, you'll still find yourself building, and re-building, and re-building these images across a whole host of use-cases. Having to build Dockerfile(s) for `dev`, `prod`, and `test` all turn out to be a nightmare when you add the complexity of hardware targets (CPUs, GPUs, TPUs etc), drivers, python, virtual environments, build and runtime dependencies.

**agi-pack** aims to simplify this by allowing developers to define Dockerfiles in a concise YAML format and then generate them based on your environment needs (i.e. python version, system packages, conda/pip dependencies, GPU drivers etc).

For example, you should be able to easily configure your `dev` environment for local development, and have a separate `prod` environment where you'll only need the runtime dependencies avoiding any bloat.

`agi-pack` hopes to also standardize the base images, so that we can really build on top of giants.

## More Complex Example 📚

Now imagine you want to build a more complex image that has multiple stages, and you want to build a `base` image that has all the basic dependencies, and a `dev` image that has additional build-time dependencies.

```yaml
images:
  base-cpu:
    name: agi
    base: debian:buster-slim
    system:
        - wget
    python: 3.8.10
    pip:
        - scikit-learn
    run:
        - echo "Hello, world!"

  dev-cpu:
    base: base-cpu
    system:
    - build-essential
```

Once you've defined this `agibuild.yaml`, running `agi-pack generate` will generate the following output:

```bash
$ agi-pack generate -c agibuild.yaml
📦 base-cpu
└── 🎉 Successfully generated Dockerfile (target=base-cpu, filename=Dockerfile).
    └── `docker build -f Dockerfile --target base-cpu .`
📦 dev-cpu
└── 🎉 Successfully generated Dockerfile (target=dev-cpu, filename=Dockerfile).
    └── `docker build -f Dockerfile --target dev-cpu .`
```

As you can see, `agi-pack` will generate a **single** Dockerfile for each of the targets defined in the YAML file. You can then build the individual images from the same Dockerfile using docker targets: `docker build -f Dockerfile --target <target> .` where `<target>` is the name of the image target you want to build.

Here's the corresponding [`Dockerfile`](./examples/generated/Dockerfile-multistage-example) that was generated.


## Why the name? 🤷‍♂️
`agi-pack` is very much intended to be tongue-in-cheek -- we are soon going to be living in a world full of quasi-AGI agents orchestrated via ML containers. At the very least, `agi-pack` should provide the building blocks for us to build a more modular, re-usable, and distribution-friendly container format for "AGI".

## Inspiration and Attribution 🌟

> **TL;DR** `agi-pack` was inspired by a combination of [Replicate's `cog`](https://github.com/replicate/cog), [Baseten's `truss`](https://github.com/basetenlabs/truss/), [skaffold](https://skaffold.dev/), and [Docker Compose Services](https://docs.docker.com/compose/compose-file/05-services/). I wanted a standalone project without any added cruft/dependencies of vendors and services.

📦 **agi-pack** is simply a weekend project I hacked together, that started with a conversation with [ChatGPT / GPT-4](#chatgpt-prompt).

🚨 **Disclaimer:** More than **75%** of this initial implementation was generated by GPT-4 and [Github Co-Pilot](https://github.com/features/copilot).

### ChatGPT Prompt
---

> **Prompt:** I'm building a Dockerfile generator and builder to simplify machine learning infrastructure. I'd like for the Dockerfile to be dynamically generated (using Jinja templates) with the following parametrizations:

```yaml
# Sample YAML file
images:
  base-gpu:
    base: nvidia/cuda:11.8.0-base-ubuntu22.04
    system:
    - gnupg2
    - build-essential
    - git
    python: 3.8.10
    pip:
    - torch==2.0.1
```
> I'd like for this yaml file to generate a Dockerfile via `agi-pack generate -c <name>.yaml`. You are an expert in Docker and Python programming, how would I implement this builder in Python. Use Jinja2 templating and miniconda python environments wherever possible. I'd like an elegant and concise implementation that I can share on PyPI.

## Contributing 🤝

Contributions are welcome! Please read the [CONTRIBUTING](CONTRIBUTING.md) guide for more information.

## License 📄

This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.
