megatron.text_generation.tokenization#
Description
Tokenization utilities.
Functions
|
Detokenize the generated tokens. |
|
Tokenize prompts and make them avaiable on all ranks. |
Description
Tokenization utilities.
Functions
|
Detokenize the generated tokens. |
|
Tokenize prompts and make them avaiable on all ranks. |