Metadata-Version: 2.1
Name: aihandler
Version: 1.8.14
Summary: AI Handler: An engine which wraps certain huggingface models
Home-page: https://github.com/w4ffl35/aihandler
Author: Capsize LLC
Author-email: contact@capsize.gg
License: AGPL-3.0
Keywords: ai,chatbot,chat,ai
Requires-Python: >=3.10.0
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch (==1.13.1)
Requires-Dist: torchvision (==0.14.1)
Requires-Dist: torchaudio (==0.13.1)
Requires-Dist: nvidia-pyindex
Requires-Dist: nvidia-tensorrt (==8.4.3.1)
Requires-Dist: triton (==2.0.0.dev20221202)
Requires-Dist: deepspeed (==0.8.0)
Requires-Dist: lightning (==1.9.0)
Requires-Dist: accelerate (==0.18.0)
Requires-Dist: bitsandbytes (==0.37.0)
Requires-Dist: bitsandbytes-cuda117 (==0.26.0.post2)
Requires-Dist: controlnet-aux
Requires-Dist: huggingface-hub (==0.13.3)
Requires-Dist: numpy (==1.23.5)
Requires-Dist: Pillow (==9.4.0)
Requires-Dist: pip (==23.0.1)
Requires-Dist: PyQt6 (==6.4.2)
Requires-Dist: PyQt6-Qt6 (==6.4.3)
Requires-Dist: PyQt6-sip (==13.4.1)
Requires-Dist: pyqtdarktheme (==2.1.0)
Requires-Dist: pyre-extensions (==0.0.23)
Requires-Dist: pytorch-lightning (==2.0.0)
Requires-Dist: requests (==2.28.2)
Requires-Dist: requests-oauthlib (==1.3.1)
Requires-Dist: safetensors (==0.3.0)
Requires-Dist: scipy (==1.10.1)
Requires-Dist: tensorflow (==2.12.0)
Requires-Dist: tokenizers (==0.13.2)
Requires-Dist: tqdm (==4.65.0)
Requires-Dist: xformers (==0.0.16)
Requires-Dist: charset-normalizer (==3.1.0)
Requires-Dist: opencv-python (==4.7.0.72)
Requires-Dist: setuptools (==65.5.1)
Requires-Dist: sympy (==1.11.1)
Requires-Dist: typing-extensions (==4.5.0)
Requires-Dist: urllib3 (==1.26.15)

# AI Handler
[![Upload Python Package](https://github.com/Capsize-Games/aihandler/actions/workflows/python-publish.yml/badge.svg)](https://github.com/Capsize-Games/aihandler/actions/workflows/python-publish.yml)
[![Discord](https://img.shields.io/discord/839511291466219541?color=5865F2&logo=discord&logoColor=white)](https://discord.gg/PUVDDCJ7gz)
![GitHub](https://img.shields.io/github/license/Capsize-Games/aihandler)
![GitHub last commit](https://img.shields.io/github/last-commit/Capsize-Games/aihandler)
![GitHub issues](https://img.shields.io/github/issues/Capsize-Games/aihandler)
![GitHub closed issues](https://img.shields.io/github/issues-closed/Capsize-Games/aihandler)
![GitHub pull requests](https://img.shields.io/github/issues-pr/Capsize-Games/aihandler)
![GitHub closed pull requests](https://img.shields.io/github/issues-pr-closed/Capsize-Games/aihandler)

This is a simple framework for running AI models. It makes use of the huggingface API
which gives you a queue, threading, a simple API, and the ability to run Stable Diffusion and LLMs seamlessly
from your local hardware.

This is not intended to be used as a standalone application.

It can easily be extended and used to power interfaces or it can be run from the command line.

AI Handler is a work in progress. It powers two projects at the moment, but may not be ready for general use.

## Installation

## Pre-requisites

System requirements

- Python 3.10.8
- pip 23.0.1
- CUDA toolkit 11.7
- CUDNN 8.6.0.163
- Cuda capable GPU
- 16gb+ ram

Create a venv and activate it:

**Ubuntu**

```
python -m venv venv
source venv/bin/activate
```

**Windows**

```
python -m venv venv
venv\Scripts\activate
```

Upgrade pip to latest

`python.exe -m pip install --upgrade pip`

### Diffusers and transformers

Install modified versions of diffusers and transformers from these forks:

```
pip install git+https://github.com/w4ffl35/diffusers.git@ckpt_fix
pip install git+https://github.com/w4ffl35/transformers.git@tensor_fix
```

The author of `aihandler` is also the author of those changes.

### Ubuntu 20.04+

```
pip install aihandler
```

### Windows 10+

First install torch

```
pip install torch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 --index-url https://download.pytorch.org/whl/cu117
```

Now install the rest

Install this repo `pip install aihandler`

---

Currently bitsandbytes on windows is bitsandbroken. Here's how you can hack around it:

1. `git clone https://github.com/DeXtmL/bitsandbytes-win-prebuilt`
2. `git clone https://github.com/james-things/bitsandbytes-prebuilt-all_arch`
3. `copy bitsandbytes-win-prebuilt/*.dll <your_venv_path>/site-packages/bitsandbytes`
4. `copy bitsandbytes-prebuilt-all_arch/*.dll <your_venv_path>/site-packages/bitsandbytes`
5. Edit `main.py` in `<your_venv_path>/site-packages/bitsandbytes/cuda_setup` 
6. Find and replace all `ct.cdll.LoadLibrary(binary_path)` with `ct.cdll.LoadLibrary(str(binary_path))`
7. Find and replace all `if not torch.cuda.is_available(): return 'libsbitsandbytes_cpu.so', None, None, None, None` with `if torch.cuda.is_available(): return 'libbitsandbytes_cudaall.dll', None, None, None, None`

#### Optional

These are optional instructions for installing TensorRT and Deepspeed for Windows

##### Install Tensor RT:

1. Download TensorRT-8.4.3.1.Windows10.x86_64.cuda-11.6.cudnn8.4
2. Git clone TensorRT 8.4.3.1
3. Follow their instructions to build TensorRT-8.4.3.1 python wheel
4. Install TensorRT `pip install tensorrt-*.whl`
 
##### Install Deepspeed:

1. Git clone Deepspeed 0.8.1
2. Follow their instructions to build Deepspeed python wheel
3. Install Deepspeed `pip install deepspeed-*.whl

---

## Environment variables

- `AIRUNNER_ENVIRONMENT` - `dev` or `prod`. Defaults to `dev`. This controls the LOG_LEVEL
- `LOG_LEVEL` - `FATAL` for production, `DEBUG` for development. Override this to force a log level
