megatron.data.dataset_utils.truncate_segments#

megatron.data.dataset_utils.truncate_segments(tokens_a, tokens_b, len_a, len_b, max_num_tokens, np_rng)#

Truncates a pair of sequences to a maximum sequence length.