megatron.tokenizer.bert_tokenization.convert_to_unicode

megatron.tokenizer.bert_tokenization.convert_to_unicode(text)

Converts text to Unicode (if it’s not already), assuming utf-8 input.