Metadata-Version: 2.4
Name: source_parser
Version: 2.0.0
Summary: Parsers and tools for extracting method/class-level features from source code
Home-page: https://github.com/microsoft/source_parser
Author: Microsoft
Author-email: vsdatascience@microsoft.com
Keywords: tree_sitter universal-ast codesearchnet method-docstring
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: source_parser/tree_sitter/assets/tree-sitter-bash/LICENSE
License-File: source_parser/tree_sitter/assets/tree-sitter-c/LICENSE
License-File: source_parser/tree_sitter/assets/tree-sitter-cpp/LICENSE
License-File: source_parser/tree_sitter/assets/tree-sitter-csharp/LICENSE
License-File: source_parser/tree_sitter/assets/tree-sitter-css/LICENSE
License-File: source_parser/tree_sitter/assets/tree-sitter-go/LICENSE
License-File: source_parser/tree_sitter/assets/tree-sitter-html/LICENSE
License-File: source_parser/tree_sitter/assets/tree-sitter-java/LICENSE
License-File: source_parser/tree_sitter/assets/tree-sitter-javascript/LICENSE
License-File: source_parser/tree_sitter/assets/tree-sitter-json/LICENSE
License-File: source_parser/tree_sitter/assets/tree-sitter-lua/LICENSE.txt
License-File: source_parser/tree_sitter/assets/tree-sitter-php/LICENSE
License-File: source_parser/tree_sitter/assets/tree-sitter-python/LICENSE
License-File: source_parser/tree_sitter/assets/tree-sitter-regex/LICENSE
License-File: source_parser/tree_sitter/assets/tree-sitter-ruby/LICENSE
License-File: source_parser/tree_sitter/assets/tree-sitter-rust/LICENSE
License-File: source_parser/tree_sitter/assets/tree-sitter-typescript/LICENSE
Requires-Dist: autopep8>=1.4.4
Requires-Dist: lz4>=3.1.0
Requires-Dist: networkx>=2.5.1
Requires-Dist: datasketch<2,>=1.5.3
Requires-Dist: tree-sitter<0.27,>=0.26
Requires-Dist: fissix<25,>=24.4.24
Requires-Dist: ray>=1.0.0
Requires-Dist: psutil>=5.6.3
Requires-Dist: PyYAML>=6.0
Requires-Dist: tqdm>=4.36.1
Requires-Dist: columnize>=0.3.10
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license-file
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# Source Parser

This package contains tools for parsing source code into annotated json data structure: we extracted
import statements, global assignments, top-level methods, classes, class methods and attributes, and annotated
and separate each method and class into signature, docstring, body, and other language-specific attributes
for downstream modeling purposes. The tool provides a high-performance multiprocessing tool called `repo_parse`
which is capable of cloning and performing this parsing for all files in a repository at the rate of thousands
of repositories per minute. See subsequence sections for installation, usage, and at the end a general
description of the structural annotated schema.

## Currently supported languages
 * Python
 * Java
 * Javascript/Typescript
 * C#
 * C++
 * Ruby

## Installation

__NOTE__: this tool is only supported on a **NIX-style OS (Linux, MacOS, FreeBSD, etc)**

Source Parser 2.x supports **Python 3.11-3.14** and **Tree-sitter 0.26.x**
(`tree-sitter>=0.26,<0.27`). Python 3.10 and older and the legacy Tree-sitter
bindings are no longer supported.

Grammar snapshots are pinned in the repository and compiled into native
bindings when the package is built. Installing a compatible wheel requires no
C/C++ compiler or writable Tree-sitter grammar cache. Building from a source distribution
requires a C/C++ toolchain and Python development headers; see
[CONTRIBUTING.md](CONTRIBUTING.md) for checkout builds.

### Migrating from 1.x

The parser schema and pinned grammar snapshots are retained. Supported Python
**runtime** versions do not imply that every new Python syntax feature is
recognized by the pinned Python grammar. Python 2 source conversion remains
available through Fissix instead of the removed standard-library `lib2to3`.

`get_language("python")` and `get_language(LanguageId.PYTHON)` continue to work.
Runtime `build_library()` and the `force_build` argument to `get_language()`
have been removed; rebuild/reinstall the package after changing a grammar.
Custom Tree-sitter parsers should use `Parser(get_language("python"))` or assign
`parser.language`, rather than calling the removed `parser.set_language()`.

### PyPI installation

After following the above instructions, simply invoke

```bash
python -m pip install source-parser
```

## Usage

### Scripting

Simply load the source file contents and hand to a parser, e.g.

```python
from source_parser.parsers import PythonParser
pp = PythonParser(open('source_parser/crawler.py').read())
print(pp.schema)
```

will print the schema extracted from `source_parser/crawler.py`.

### Parsing at scale

The real intention of this tool is to run massively at scale with 100s of thousands
of git repositories. 
Two CLI tools are added upon installation:
 - `repo_parse -h`: semantically parses code using `source_parser`
 - `repo_scrape -h`: just grabs all files matching some patterns

Both commands start a CPU-only Ray runtime (`num_gpus=0`). This avoids GPU
device-name detection, which can fail on some NVIDIA/WSL setups. An already
initialized Ray runtime is reused unchanged.

for example:

```bash
repo_parse <language> <repo_list.json> <outdir> [--tmpdir <temporary_directory>]
```

where `<language>` is one of the supported languages indicated in the help message,
`<repo_list.json>` is a path to a `.json` file containin a list of dictionaries with at
least a `'url'` key for a `git` repository and optionally a `'license'` key. `<outdir>`
is the directory in which to place the saved results as a `lz4` compressed `jsonlines`
file, and `--tmpdir` is an optional place to save temporary data like cloned
repositories.

_Protip: mount a RAMdisk and hand it to `--tmpdir` to remove the IO bottleneck
and double parsing speeds! Further, you can set `outdir` to be in the RAMdisk as well, 
so no disk is necessary (if you have enough memory).

```bash
sudo mount -t tmpfs -o size=<size in Gigabytes>G <name-ramdisk> /path/to/ramdisk`
```


### Reading the data

The default compression algorithm used is `lz4` for its high speed and reasonable
compression ratio. Because the data is highly compressible, DO read the data in streaming
fashion and not saving it all in memory uncompressed at once. The JSON dictionaries are 
highly compressible so you can generally expect the uncompressed data to be 2-3x as large.

To this end there is a nice tool in `source_parser/__init__.py`, importable
via `from source_parser import load_zip_json`, which returns an iterator object
which uncompresses and returns only one file-level schema dictionary at a time.

To use:

```python
from source_parser import load_zip_json
for example in load_zip_json('file_saved_from_repocontext.lz4'):
    process_file_example(example)
```

If you'd like to load it all into memory at once:
```python
from source_parser import load_zip_json
all_data = list(load_zip_json('file_saved_from_repocontext.lz4'))
```

### Deduplicating code

`CodeDeduper` detects near-duplicate code using fingerprints of identifiers and
literals. The package uses `datasketch` 1.x to preserve compatibility with
existing fingerprints and duplicate-detection behavior. When adding a code
string immediately after querying it, use `new_hash=False` to reuse those
fingerprints without parsing and hashing the same code again:

```python
from source_parser.deduper import CodeDeduper

deduper = CodeDeduper(language="python")
unique_code = []
for code in code_strings:
    if not deduper.query(code):
        deduper.add(code, new_hash=False)
        unique_code.append(code)
```

Only use `new_hash=False` when the most recent query was for the same code
string. Otherwise, use `add(code)` to compute fresh fingerprints.

### Data Schema

This is a description of the JSON schema into which `source_parser` will
transform source code files, for use in method and class-level code-natural
language modeling. The data will consist of JSON lines, that is valid JSON
separated by newline characters. Each line of JSON will be the features
extracted from a single source code file. The proposed JSON schema for each
individual file is as follows:

_NOTE: See individual language parsers in `source_parser/parsers` for the langauge-specific method and class attributes._

```json
{
	'file_name': 'name_of_file.extension',

    'file_hash': 'hash of file for literal deduplication',

	'relative_path': 'repo_top_level/path/to/file/name_of_file.extension',

	'repo_name': 'owner/repo-name',

    'commit-hash': 'hash of the commit being analyzed',

	'license': {
        'label': 'label provided by github API or in json list',
        'files': [
            'relative_path': 'path/to/license/file',
            'file_contents': 'license file contents',
        ],
    }

    'original_string': 'origina string of file',

	'file_docstring': 'string containing first docstring for all of file',

	'contexts': [
            'import statement 1',
            'import statement 2',
            'global variable expression 1',
            ...
        ],

	'language_version_details': [
        'e.g. python2 syntax detected', 'another languages idiosyncracies'
        ]

	'methods': [  # list of dictionaries annotating each method
		{
            'original_string': 'verbatim code of whole method',

            'byte_span': (start_byte, end_byte),

            'start_point': (start_line_number, start_column),

            'end_point': (end_line_number, end_column),

            'signature': 'string corresponding to definition, name, arguments of method',

            'name': 'name of method',

            'docstring': 'verbatim docstring corresponding to this method',

            'body': 'verbatim code body',

            'original_string_normed': 'code of whole method with string-literal, numeral normalization',

            'signature_normed': 'signature with string-literals/numerals normalized',

            'body_normed': 'code of body with string-literals/numerals normalized',

            'default_arguments': ['arg1': 'default value 1', ...],

            'syntax_pass': 'True/False whether the method is syntactically correct',

            'attributes': [
            	'language_specific_keys': 'language_specific_values',
                'decorators': ['@wrap', '@abstractmethod'],
                ...
            ],
            ...
        },
        ...
	]

	'classes': [
        {
		'original_string': 'verbatim code of class',

        'byte_span': (start_byte, end_byte),

        'start_point': (start_line_number, start_column),

        'end_point': (end_line_number, end_column),

        'name': 'class name',

        'definition': 'class definition statement',

		'class_docstring': 'docstring corresponding to to-level class definition,

		'attributes': {  # language specific keys and values, e.g.
                'expression_statements': [
                    {
                      'expression': 'attribute = 1',
                      'comment': 'comment associated'
                    },
                'classes': [  # classes defined within classes
                    {
                        # same structure as classes
                    }
                ]
                ...
                ]
		    },

		'methods': [
            '# list of class methods of the same form as top-level methods',
            ...
            ]
	    }
    ...
    ]
]
```

## Contributing

We welcome contributions. Please follow [this guideline](CONTRIBUTING.md).

## Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft 
trademarks or logos is subject to and must follow 
[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
Any use of third-party trademarks or logos are subject to those third-party's policies.
