megatron.text_generation.tokenization

Description

Tokenization utilities.

Functions

detokenize_generations(tokens_gpu_tensor, ...)	Detokenize the generated tokens.
tokenize_prompts([prompts, ...])	Tokenize prompts and make them avaiable on all ranks.