megatron.tokenizer.tokenizer.AbstractTokenizer

class megatron.tokenizer.tokenizer.AbstractTokenizer(name)

Bases: ABC

Abstract class for tokenizer.

abstract property inv_vocab

Dictionary from vocab id token to text token.

abstract property vocab

Dictionary from vocab text token to id token.