megatron.tokenizer.tokenizer#
Description
Megatron tokenizers.
Classes
|
Abstract class for tokenizer. |
Functions
|
Initialize tokenizer. |
Description
Megatron tokenizers.
Classes
|
Abstract class for tokenizer. |
Functions
|
Initialize tokenizer. |