megatron.model.language_model

Description

Transformer based language model.

Classes

Embedding(hidden_size, vocab_size, ...[, ...])	Language model embeddings.
Pooler(hidden_size, init_method, args)	Pool hidden states of a specific token (for example start of the sequence) and add a linear transformation followed by a tanh.
TransformerLanguageModel(init_method, ...[, ...])	Transformer language model.

Functions

get_language_model(num_tokentypes, ...[, ...])
parallel_lm_logits(input_, ...[, bias])	LM logits using word embedding weights.