megatron.tokenizer.bert_tokenization.BasicTokenizer#

class megatron.tokenizer.bert_tokenization.BasicTokenizer(do_lower_case=True)#

Bases: object

Runs basic tokenization (punctuation splitting, lower casing, etc.).

tokenize(text)#

Tokenizes a piece of text.