|
first_shard = ( |
|
dataset.take(1) |
|
.flat_map( |
|
lambda x: tf.data.TFRecordDataset(x, compression_type=compression_type) |
|
) |
|
.map(map_func=parse_fn, num_parallel_calls=num_parallel_calls) |
|
) |
|
block_length = len([0 for _ in first_shard]) |
If the number of volumes in the shard is too large, this snippet of code can be time-consuming. Alternatives are
- use a combination of
n_volumes and number of files with file_pattern to calculate len(first_shard)
- provide metadata (number of volumes in the shard) as well as total number of volumes in the dataset
nobrainer/nobrainer/dataset.py
Lines 115 to 122 in 976691d
If the number of volumes in the shard is too large, this snippet of code can be time-consuming. Alternatives are
n_volumesand number of files withfile_patternto calculatelen(first_shard)