Metadata-Version: 2.1
Name: ASTAligner
Version: 0.1.0
Summary: ASTAligner is designed to align tokens from source code snippets to Abstract Syntax Tree (AST) nodes using Tree-sitter for AST generation and various HuggingFace tokenizers for language tokenization. The library supports a wide range of programming languages and Fast tokenizers, enabling precise mapping between source code elements and their AST representations.
Home-page: https://github.com/csci-435-fall-2024/csci-435-24_p4_ast
Author: Semeru Lab
Author-email: svelascodimate@wm.edu
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.7
Description-Content-Type: text/markdown
Requires-Dist: Flask==3.0.3
Requires-Dist: Flask-Cors==5.0.0
Requires-Dist: protobuf
Requires-Dist: sentencepiece==0.2.0
Requires-Dist: tokenizers==0.19.1
Requires-Dist: transformers==4.44.2
Requires-Dist: tree-sitter==0.23.0
Requires-Dist: tree-sitter-cpp==0.23.1
Requires-Dist: tree-sitter-java==0.23.2
Requires-Dist: tree-sitter-python==0.23.2
Requires-Dist: tree_sitter_c_sharp==0.23.1
Requires-Dist: tree_sitter_go==0.23.3
Requires-Dist: tree_sitter_haskell==0.23.1
Requires-Dist: tree_sitter_javascript==0.23.1
Requires-Dist: tree_sitter_kotlin==1.0.1
Requires-Dist: tree_sitter_rust==0.23.1
Requires-Dist: tree_sitter_html==0.23.2
Requires-Dist: tree_sitter_c==0.23.2
Requires-Dist: tree-sitter-ruby==0.23.1
Requires-Dist: concurrently

# AST-Alignment Tool

Aligns the tokens from a code snippet to their corresponding nodes in an AST representation.

## Description

A Large Language Model (LLM) is a type of AI model designed to understand
and generate human-like text based on vast amounts of data. Trained on diverse source code datasets, LLMs can automate Software Engineering tasks across various contexts, such as code translation, code summarization, test-case generation, and code completion. A critical component of LLMs is the tokenizer, which breaks down text into smaller units, typically words or subwords, that the model can process. The tokenizer's role is essential because it converts source code into a format the model can understand, ensuring efficient and accurate code processing and generation. In the context of Interpretability for AI, post-hoc techniques such as ASTScore, rely on alienation functions (phi) to match the tokens generated by an LLMâ€™s tokenizer with their corresponding nodes in the AST representation of a snippet.

### Goals

This project has two goals: 

(1) Create a library for aligning the tokens from a code snippet to their corresponding nodes in the AST representation 

(2) Create a tool to visualize the alignment of the tokens with their matching AST. 

### Additional Information

For more information regarding this project's background and dependencies, please refer to these readings:

(1) [Evaluating and Explaining Large Language Models for Code Using
Syntactic Structures](https://arxiv.org/abs/2308.03873)

(2) [Tree-Sitter Programming Language Parser](https://github.com/tree-sitter/tree-sitter)

(3) [Hugging Face Tokenizer](https://huggingface.co/learn/nlp-course/en/chapter2/4)

## Installation

Use the package manager [pip](https://pip.pypa.io/en/stable/) to install all backend dependencies needed for the AST-Alignment Tool. All required packages for the backend can be downloaded using requirements.txt, which can found in the base repository.

```bash
pip install -r /path/to/requirements.txt
```

## Supported Features

* 11 supported languages
    * Python
    * C
    * C++
    * C#
    * Java
    * JavaScript
    * Ruby
    * HTML
    * GO
    * Kotlin
    * Rust
* 6 Tokenizers
    * [Bert-Base-Uncased](https://huggingface.co/google-bert/bert-base-uncased)
    * [CodeLlama](https://huggingface.co/docs/transformers/main/en/model_doc/code_llama)
    * [GPT2](https://huggingface.co/docs/transformers/en/model_doc/gpt2)
    * [DialoGPT](https://huggingface.co/microsoft/DialoGPT-small)
    * [Roberta-Base](https://huggingface.co/FacebookAI/roberta-base)
    * [Qwen](https://huggingface.co/Qwen/Qwen2-7B-Instruct)

## Library Usage

**EXPLAIN HOW TO USE THE PYTHON LIBRARY**

## Contributing

## License

[MIT](https://choosealicense.com/licenses/mit/)
