Metadata-Version: 2.1
Name: atc-torch
Version: 0.0.1
Summary: atc-torch - Pytorch
Home-page: https://github.com/kyegomez/AttnWithConvolutions
License: MIT
Keywords: artificial intelligence,deep learning,optimizers,Prompt Engineering
Author: Kye Gomez
Author-email: kye@apac.ai
Requires-Python: >=3.9,<4.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.9
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Dist: einops
Requires-Dist: torch
Project-URL: Repository, https://github.com/kyegomez/AttnWithConvolutions
Description-Content-Type: text/markdown

[![Multi-Modality](agorabanner.png)](https://discord.gg/qUtxnK2NMf)

# Attention + Convolution transformer
This is an experimental architecture leveraging convolution blocks with attention blocks to model both the short and long range dynamics of the input tokens. The flow is the following: x -> convolution block -> attn -> FFN

# Install
``


## Usage
```python
import torch
from attnconv.main import ATCTransformer

model = ATCTransformer(
    dim=512,
    depth=6,
    num_tokens=20000,
    dim_head=64,
    heads=8,
    ff_mult=4,
)

x = torch.randint(0, 20000, (1, 512))
logits = model(x)  # (1, 1024, 20000)
print(logits)

```


# License
MIT




