megatron.tokenizer.bert_tokenization.FullTokenizer

class megatron.tokenizer.bert_tokenization.FullTokenizer(vocab_file, do_lower_case=True)

Bases: object

Runs end-to-end tokenziation.

static convert_tokens_to_string(tokens, clean_up_tokenization_spaces=True)

Converts a sequence of tokens (string) in a single string.