Metadata-Version: 2.4
Name: lycoris_lora
Version: 4.0.1.dev20260902072855
Summary: Lora beYond Conventional methods, Other Rank adaptation Implementations for Stable diffusion
Author: Yu-Guan Hsieh, Zhidong Gao
Author-email: "Shih-Ying Yeh(KohakuBlueLeaf)" <kohaku@kblueleaf.net>
License: Apache-2.0
Project-URL: Homepage, https://github.com/KohakuBlueleaf/LyCORIS
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: GPU :: NVIDIA CUDA :: 12
Classifier: Environment :: GPU :: NVIDIA CUDA :: 12 :: 12.6
Classifier: Environment :: GPU :: NVIDIA CUDA :: 13
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: End Users/Desktop
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE.md
Requires-Dist: torch
Requires-Dist: einops
Requires-Dist: toml
Requires-Dist: tqdm
Requires-Dist: numpy
Requires-Dist: safetensors
Provides-Extra: dev
Requires-Dist: coverage; extra == "dev"
Requires-Dist: parameterized; extra == "dev"
Requires-Dist: diffusers>=0.40.0; extra == "dev"
Provides-Extra: kernels
Requires-Dist: triton; sys_platform == "linux" and extra == "kernels"
Requires-Dist: triton-windows; sys_platform == "win32" and extra == "kernels"
Requires-Dist: tilelang; extra == "kernels"
Dynamic: license-file

![pypi](https://img.shields.io/pypi/v/lycoris-lora.svg)
![versions](https://img.shields.io/pypi/pyversions/lycoris-lora.svg)
![PyPI - License](https://img.shields.io/pypi/l/lycoris-lora)
![downloads](https://img.shields.io/pypi/dm/lycoris-lora)
![commits](https://img.shields.io/github/commit-activity/m/KohakuBlueleaf/LyCORIS/dev)
![Discord](https://img.shields.io/discord/1082218577395986452)

# LyCORIS - Lora beYond Conventional methods, Other Rank adaptation Implementations for Stable diffusion. (ICLR'24)

![banner image](docs/images/banner2.png)

A project that implements different parameter-efficient fine-tuning algorithms for Stable Diffusion.

This project originated from LoCon (see archive branch).

**If you are interested in discussing more details, you can join [our Discord server](https://discord.gg/VtTFKrj9gJ)**

[![Discord!](https://i.imgur.com/A8tOvFS.jpg)](https://discord.gg/VtTFKrj9gJ)

**If you want to check more in-depth experiment results and discussions for LyCORIS, you can check our [paper](https://openreview.net/forum?id=wfzXa8e783)**

**Documentation starts at [docs/README.md](docs/README.md)** — it maps every
document and source folder to the question it answers.

## Algorithm Overview

LyCORIS currently contains LoRA (LoCon), LoHa, LoKr, (IA)^3, DyLoRA, Native fine-tuning (aka dreambooth).
GLoRA and GLoKr are coming soon.
Please check [List of Implemented Algorithms](docs/algorithms/README.md) and [Guidelines](docs/algorithms/guidelines.md) for more details.

A simple comparison of some of these methods are provided below (to be taken with a grain of salt)

|                       | Full | LoRA | LoHa | LoKr low factor | LoKr high factor $^+$ |
| --------------------- | ---- | ---- | ---- | --------------- | ---------------------- |
| Fidelity              | ★   | ●   | ▲   | ◉              | ▲                     |
| Flexibility $^*$     | ★   | ●   | ◉   | ▲              | ● $^†$              |
| Diversity             | ▲   | ◉   | ★   | ●              | ★                     |
| Size                  | ▲   | ●   | ●   | ●              | ★                     |
| Training Speed Linear | ★   | ●   | ●   | ★              | ★                     |
| Training Speed Conv   | ●   | ★   | ▲   | ●              | ●                     |

★ > ◉ > ● > ▲
[> means better and smaller size is better]

$^+$ Usually we take `factor <= 0.5 * sqrt(dim)` as low factor and `factor >= sqrt(dim` as high factor. For example, factor<=8 for SD1.x/SD2.x/SDXL can be seen as low factor, and, factor>=16 can be seen as high factor. <br>
$^*$ Flexibility means anything related to generating images not similar to those in the training set, and combination of multiple concepts, whether they are trained together or not <br>
$^†$ It may become more difficult to switch base model or combine multiple concepts in this situation <br>

**The actual performance may vary depending on the datasets, tasks, and hyperparameters used. It is recommended to experiment with different settings to achieve optimal results.**

## Usage

### Image Generation

#### [a1111/sd-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui)

After sd-webui 1.5.0, LyCORIS models are officially supported by the built-in LoRA system. You can put them in either `models/Lora` or `models/LyCORIS` and use the default syntax `<lora:filename:multiplier>` to trigger it.

When we add new model types, we will always make sure they can be used with the newest version of sd-webui.

As for sd-webui with version < 1.5.0 or sd-webui-forge, please check this [extension](https://github.com/KohakuBlueleaf/a1111-sd-webui-lycoris).

#### Others

As far as we are aware, LyCORIS models are also supported in the following interfaces / online generation services (please help us complete the list!)

- [ComfyUI](https://github.com/comfyanonymous/ComfyUI)
- [InvokeAI](https://github.com/invoke-ai/InvokeAI)
- [CivitAI](https://civitai.com/)
- [Tensor.Art](https://tensor.art/)

However, newer model types may not always be supported. If you encounter this issue, consider requesting the developers of the corresponding interface or website to include support for the new type.

### Training

There are three different ways to train LyCORIS models.

- With [kohya-ss/sd-scripts](https://github.com/kohya-ss/sd-scripts) (see a list of compatible graphical interfaces and colabs at the end of the section)
- With [Naifu-Diffusion](https://github.com/Mikubill/naifu)
- With your own script by using LyCORIS as standalone wrappers for **ANY** pytorch modules.

In any case, please install this package in the corresponding virtual environment. You can either install it

- through pip

  ```bash
  pip install lycoris-lora
  ```
- or from source

  ```bash
  git clone https://github.com/KohakuBlueleaf/LyCORIS
  cd LyCORIS
  pip install .
  ```

A detailed description of the network arguments is provided in [docs/usage/network-args.md](docs/usage/network-args.md).

#### kohya script

You can use this package's kohya module to run kohya's training script to train lycoris module for SD models

- with command line arguments

  ```bash
  accelerate launch train_network.py \
    --network_module lycoris.kohya \
    --network_dim "DIM_FOR_LINEAR" --network_alpha "ALPHA_FOR_LINEAR"\
    --network_args "conv_dim=DIM_FOR_CONV" "conv_alpha=ALPHA_FOR_CONV" \
    "dropout=DROPOUT_RATE" "algo=locon" \
  ```
- with `toml` files

  ```bash
  accelerate launch train_network.py \
    --config_file example_configs/training_configs/kohya/loha_config.toml \
    --dataset_config example_configs/training_configs/kohya/dataset_config.toml
  ```

  For your convenience, some example `toml` files for kohya LyCORIS training are provided in [example/training_configs/kohya](example_configs/training_configs/kohya).

#### HCP-Diffusion

**The support for HCP-Diffusion has been dropped on LyCORIS3.0.0, we will wait until HCP side finish the implementation of new wrapper**

You can use this package's hcp module to run HCP-Diffusion's training script to train lycoris module for SD models

```bash
accelerate launch -m hcpdiff.train_ac_single \
  --cfg example_configs/training_configs/hcp/hcp_diag_oft.yaml
```

For your convenience, some example `yaml` files for HCP LyCORIS training are provided in [example/training_configs/hcp](example_configs/training_configs/hcp).

For the moment being the outputs of HCP-Diffusion are not directly compatible with a1111/sdwebui.
You can perform conversion with [tools/batch_hcp_convert.py](tools/batch_hcp_convert.py).

In the case of pivotal tuning, [tools/batch_bundle_convert.py](tools/batch_bundle_convert.py) can be further used to convert to and from bundle formats.
Check [docs/usage/conversion-scripts.md](docs/usage/conversion-scripts.md) for more information.

#### As standalone wrappers

See [example/standalone_example.py](example/standalone_example.py) for full example.

Import `create_lycoris` and `LycorisNetwork` from `lycoris` library, put your preset to `LycorisNetwork` and then use `create_lycoris` to create LyCORIS module for your pytorch module.

For example:

```py
from lycoris import create_lycoris, LycorisNetwork

LycorisNetwork.apply_preset(
    {"target_name": [".*attn.*"]}
)
lycoris_net = create_lycoris(
    your_model, 
    1.0, 
    linear_dim=16, 
    linear_alpha=2.0, 
    algo="lokr"
)
lycoris_net.apply_to()

# after apply_to(), your_model() will run with LyCORIS net
lycoris_param = lycoris_net.parameters()
forward_with_lyco = your_model(x)
```

You can also layer multiple wrappers on top of the same module. Each call to `apply_to()` stacks a new adapter after the previously attached ones, and invoking `restore()` on a wrapper removes only its contribution while keeping earlier wrappers active. See [example/standalone_example.py](example/standalone_example.py) for an end-to-end demonstration.

For a stacking-specific walkthrough (including selectively removing adapters), see `python example/stacked_wrapper_demo.py --help`.

You can check my [HakuPhi](https://github.com/KohakuBlueleaf/HakuPhi) project to see how I utilize LyCORIS to finetune the Phi-1.5 models.

#### Other method

After LyCORIS3.0.0, Parametrize API and Functional API have been added, which provide more different ways on utilizing LyCORIS library.

Check [API reference](docs/api/README.md) for more information.
You can also take the [test suites](test/) as a kind of examples.

### Fused kernels (experimental)

Since 4.0.0 LyCORIS ships hand-written Triton and TileLang kernels for the hot
paths of every algorithm, and picks one per call —
**triton > tilelang > torch.compile > eager** — with an automatic fallback to
the stock PyTorch path for anything out of scope. Your code does not change:
the module, functional and wrapper APIs keep the same signatures.

```bash
pip install triton      # or: pip install tilelang
```

On an RTX 4090 this is **1.4x–7.3x** less device (kernel) time than eager and
**1.2x–6.2x** less than `torch.compile` on the fused paths, with lower peak
VRAM and error at or below the eager path — ΔW and its gradient are never
materialised. Full table, method and caveats in
[docs/kernels/benchmarks.md](docs/kernels/benchmarks.md).

**This is an early experimental implementation.** If a result looks wrong, pin
the stock path and tell us:

```bash
export LYCORIS_KERNEL_BACKEND=torch
```

See [docs/kernels/README.md](docs/kernels/README.md) for what is fused per
algorithm, and [docs/kernels/backends.md](docs/kernels/backends.md) for the
selection rules and environment variables.

#### Bitsandbytes support

See [example/bnb_example.py](example/bnb_example.py) for example. Basically as same as standalone wrapper.

#### Graphical interfaces and Colabs (via kohya trainer)

You can also train LyCORIS with the following graphical interfaces

* [bmaltais/kohya_ss](https://github.com/bmaltais/kohya_ss)
* [derrian-distro/LoRA_Easy_Training_Scripts](https://github.com/derrian-distro/LoRA_Easy_Training_Scripts)
* [Akegarasu/lora-scripts](https://github.com/Akegarasu/lora-scripts)

and colabs (please help us complete the list!)

* [hollowstrawberry/kohya-colab](https://github.com/hollowstrawberry/kohya-colab)
* [Linaqruf/kohya-trainer](https://github.com/Linaqruf/kohya-trainer)

However, they are not guaranteed to be up-to-date. In particular, newer types may not be supported. Consider requesting the developers for support or simply use the original kohya script in this case.

## Utilities

### Extract LoCon

You can extract LoCon from a dreambooth model with its base model.

```bash
python3 extract_locon.py <settings> <base_model> <db_model> <output>
```

Use --help to get more info

```
$ python3 extract_locon.py --help
usage: extract_locon.py [-h] [--is_v2] [--is_sdxl] [--device DEVICE] [--mode MODE] [--safetensors] [--linear_dim LINEAR_DIM]
                        [--conv_dim CONV_DIM] [--linear_threshold LINEAR_THRESHOLD] [--conv_threshold CONV_THRESHOLD]
                        [--linear_ratio LINEAR_RATIO] [--conv_ratio CONV_RATIO] [--linear_quantile LINEAR_QUANTILE]
                        [--conv_quantile CONV_QUANTILE] [--use_sparse_bias] [--sparsity SPARSITY] [--disable_cp]
                        base_model db_model output_name
```

### Merge LyCORIS back to model

You can merge your LyCORIS model back to your checkpoint (base model).

```bash
python3 merge.py <settings> <base_model> <lycoris_model> <output>
```

Use --help to get more info

```
$ python3 merge.py --help
usage: merge.py [-h] [--is_v2] [--is_sdxl] [--device DEVICE] [--dtype DTYPE] [--weight WEIGHT] base_model lycoris_model output_name
```

**Note**: The `merge_to()` method has an opt-in parameter, `precise` that uses more CPU memory to store an original snapshot of the modified weights, allowing us to restore the original weights exactly when calling `restore()`. This is useful when you want to merge multiple LyCORIS models sequentially to the same base model without accumulating numerical errors in a production environment. See this [model merge example](example/high_precision_merge_demo.py) for context.

### Conversion of LoRA, LyCORIS and full models between HCP and sd-webui format

This script allows you to use the LyCORIS models trained with HCP-Diffusion in sd-webui.

```bash
python3 batch_hcp_convert.py \
  --network_path /path/to/ckpts \
  --dst_dir /path/to/stable-diffusion-webui/models/Lora \
  --output_prefix something \
  --auto_scale_alpha --to_webui
```

See [docs/usage/conversion-scripts.md](docs/usage/conversion-scripts.md) for more information.

### Conversion from and to bundle format

This script is particularly useful in the case of pivotal tuning.

```bash
python3 batch_bundle_convert.py \
  --network_path /path/to/sd-webui-ssd/models/Lora  \
  --emb_path /path/to/ckpts \
  --dst_dir /path/to/sd-webui-ssd/models/Lora/bundle \
  --to_bundle --verbose 2 
```

See [docs/usage/conversion-scripts.md](docs/usage/conversion-scripts.md) for more information.

## Change Log

For full log, please see [Change.md](Change.md)

### 2026/09/01 update to 4.0.0

#### New Features

* **Fused kernels (early experimental)** — hand-written Triton and TileLang
  kernels for every algorithm: `lora`/`locon`, `loha`, `lokr`, `oft`, `boft`,
  `dora` (shared by dora/doha/dokr), `ia3`, `glora`, `dylora`, `full` and
  `norm`. Up to four kernels each — merge forward, merge backward, bypass
  forward, bypass backward — with ΔW and its gradient never materialised.
* **Automatic backend selection** per call — triton > tilelang >
  `torch.compile` > eager — with a fallback for anything out of scope. Pin it
  with `LYCORIS_KERNEL_BACKEND`, or per call with the new `backend=` argument
  on the functional API.
* **Mixed dtype support** across x and the module weights: fp16, bf16 or fp32
  independently, 16-bit matmul with fp32 accumulation, gradients returned in
  each leaf's own dtype.
* `weight_decompose` and `add_scaled` in `lycoris.functional.general`.

#### Improvements

* Documentation reorganised into a nested tree, indexed at [docs/README.md](docs/README.md).
* Packaging moved to `pyproject.toml`; CI, nightly and release workflows added.


## Todo list

- [ ] Automatically selecting an algorithm based on the specific rank requirement.
- [ ] More experiments for different task, not only diffusion models.
  - [X] LoKr and LoHa have been proven to be useful for Large Language Model.
- [ ] Explore other low-rank representations or parameter-efficient methods to fine-tune either the entire model or specific parts of it.
- [ ] Documentation for whole library.

## Citation

```bibtex
@inproceedings{
  yeh2024navigating,
  title={Navigating Text-To-Image Customization: From Ly{CORIS} Fine-Tuning to Model Evaluation},
  author={SHIH-YING YEH and Yu-Guan Hsieh and Zhidong Gao and Bernard B W Yang and Giyeong Oh and Yanmin Gong},
  booktitle={The Twelfth International Conference on Learning Representations},
  year={2024},
  url={https://openreview.net/forum?id=wfzXa8e783}
}
```
