megatron.tokenizer.tokenizer# Description Megatron tokenizers. Classes AbstractTokenizer(name) Abstract class for tokenizer. Functions build_tokenizer(args) Initialize tokenizer.