megatron.text_generation.tokenization.tokenize_prompts# megatron.text_generation.tokenization.tokenize_prompts(prompts=None, tokens_to_generate=None, add_BOS=None, rank=0)# Tokenize prompts and make them avaiable on all ranks.