Metadata-Version: 2.0
Name: bigcode-tokenizer
Version: 0.1.0
Summary: Tool to tokenize source code
Home-page: https://github.com/tuvistavie/bigcode-tools/tree/master/bigcode-tokenizer
Author: Daniel Perez
Author-email: tuvistavie@gmail.com
License: UNKNOWN
Download-URL: https://github.com/tuvistavie/bigcode-tools/archive/master.zip
Description-Content-Type: UNKNOWN
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Requires-Dist: pygments
Provides-Extra: test
Requires-Dist: nose; extra == 'test'

# bigcode-tokenizer

Tool to tokenize source code.


