megatron.tokenizer.tokenizer.AbstractTokenizer
==============================================

.. currentmodule:: megatron.tokenizer.tokenizer

.. autoclass:: AbstractTokenizer
   :members:
   :special-members:
   :show-inheritance:
   :exclude-members: __weakref__, __init__