megatron.model.utils

Description

Utilities for models.

Functions

attention_mask_func(attention_scores, ...)
get_linear_layer(rows, columns, init_method, ...)	Simple linear layer with weight initialization.
init_method_normal(sigma)	Init method based on N(0, sigma).
scaled_init_method_normal(sigma, num_layers)	Init method based on N(0, sigma/sqrt(2*num_layers).