Tokenizers Package
==================

:mod:`Tokenizers` Package
-------------------------

.. automodule:: Ity.Tokenizers
    :members:
    :undoc-members:
    :show-inheritance:

:mod:`RegexTokenizer` Module
----------------------------

.. automodule:: Ity.Tokenizers.RegexTokenizer
    :members:
    :undoc-members:
    :show-inheritance:

:mod:`Tokenizer` Module
-----------------------

.. automodule:: Ity.Tokenizers.Tokenizer
    :members:
    :undoc-members:
    :show-inheritance:

Subpackages
-----------

.. toctree::

    Ity.Tokenizers.WordTokenizer

