megatron.tokenizer.gpt2\_tokenization
=====================================

.. rubric:: Description

.. automodule:: megatron.tokenizer.gpt2_tokenization

.. currentmodule:: megatron.tokenizer.gpt2_tokenization


.. rubric:: Classes

.. autosummary::
    :toctree: .
    
    GPT2Tokenizer
    




.. rubric:: Functions

.. autosummary::
    :toctree: .
    
    bytes_to_unicode
    
    get_pairs