megatron.data.indexed_dataset

Description

Classes

IndexedCachedDataset(path)
IndexedDataset(path)	Loader for IndexedDataset
IndexedDatasetBuilder(out_file[, dtype])
MMapIndexedDataset(path[, skip_warmup])
MMapIndexedDatasetBuilder(out_file[, dtype])

Functions

code(dtype)
create_doc_idx(sizes)
data_file_path(prefix_path)
dataset_exists(path, impl)
get_available_dataset_impl()
index_file_path(prefix_path)
infer_dataset_impl(path)
make_builder(out_file, impl[, vocab_size])
make_dataset(path, impl[, skip_warmup])
read_longs(f, n)
write_longs(f, a)