megatron.text_generation.tokenization.tokenize_prompts#

megatron.text_generation.tokenization.tokenize_prompts(prompts=None, tokens_to_generate=None, add_BOS=None, rank=0)#

Tokenize prompts and make them avaiable on all ranks.