megatron.tokenizer.tokenizer
============================

.. rubric:: Description

.. automodule:: megatron.tokenizer.tokenizer

.. currentmodule:: megatron.tokenizer.tokenizer


.. rubric:: Classes

.. autosummary::
    :toctree: .
    
    AbstractTokenizer
    




.. rubric:: Functions

.. autosummary::
    :toctree: .
    
    build_tokenizer