megatron.text\_generation.tokenization
======================================

.. rubric:: Description

.. automodule:: megatron.text_generation.tokenization

.. currentmodule:: megatron.text_generation.tokenization




.. rubric:: Functions

.. autosummary::
    :toctree: .
    
    detokenize_generations
    
    tokenize_prompts