Metadata-Version: 2.1
Name: transformer-implementations
Version: 0.0.7
Summary: A bunch of transformer implementations
Home-page: https://github.com/UdbhavPrasad072300/Transformer-Implementations
Author: Udbhav Prasad
Author-email: udbhavprasad072300@gmail.com
License: MIT
Platform: UNKNOWN
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: License :: OSI Approved :: MIT License
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Programming Language :: Python :: 3 :: Only
Description-Content-Type: text/markdown

# Transformer Implementations 

Transformer Implementations and some examples with them

Implemented:
<ul>
  <li>Vanilla Transformer</li>
  <li>ViT - Vision Transformers</li>
  <li>DeiT - Data efficient image Transformers</li>
</ul>

## Installation

<a href="https://pypi.org/project/transformer-implementations/">PyPi Installation</a>

```bash
$ pip install transformer-implementations
```

## Language Translation

from "Attention is All You Need": https://arxiv.org/pdf/1706.03762.pdf

Models trained with Implementation:
<ul>
  <li><a href="https://github.com/UdbhavPrasad072300/Transformer-Implementations/blob/main/notebooks/Multi30k%20-%20Language%20Translation.ipynb">Multi30k - German to English</a></li>
</ul>

## Multi-class Image Classification with Vision Transformers (ViT)

from "An Image is Worth 16x16 words: Transformers for image recognition at scale": https://arxiv.org/pdf/2010.11929v1.pdf

Models trained with Implementation:
<ul>
 <li><a href="https://github.com/UdbhavPrasad072300/Transformer-Implementations/blob/main/notebooks/MNIST%20Classification%20-%20ViT.ipynb">MNIST - Grayscale Images</a></li>
  <li><a href="https://github.com/UdbhavPrasad072300/Transformer-Implementations/blob/main/notebooks/CIFAR10%20Classification%20-%20ViT.ipynb">CIFAR10 - MultiChannel Images</a></li>
</ul>

## Multi-class Image Classification with Data-efficient image Transformers (DeiT)

from "Training data-efficient image transformers & distillation through attention": https://arxiv.org/pdf/2012.12877v1.pdf

Models trained with Implementation:
<ul>
 <li><a href="https://github.com/UdbhavPrasad072300/Transformer-Implementations/blob/main/pre-train/VGG16_CIFAR10.ipynb">Pretraining Teacher model for Distillation</a></li>
 <li><a href="https://github.com/UdbhavPrasad072300/Transformer-Implementations/blob/main/notebooks/CIFAR10%20Classification%20-%20DeiT.ipynb">CIFAR10 - Low Res Images</a></li>
</ul>


