megatron.tokenizer.tokenizer#

Description

Megatron tokenizers.

Classes

AbstractTokenizer(name)

Abstract class for tokenizer.

Functions

build_tokenizer(args)

Initialize tokenizer.