megatron.tokenizer.bert_tokenization.whitespace_tokenize#

megatron.tokenizer.bert_tokenization.whitespace_tokenize(text)#

Runs basic whitespace cleaning and splitting on a piece of text.