Skip to content

refactor code to calculate records per shard using n_volumes and number of shards #328

Description

@hvgazula

first_shard = (
dataset.take(1)
.flat_map(
lambda x: tf.data.TFRecordDataset(x, compression_type=compression_type)
)
.map(map_func=parse_fn, num_parallel_calls=num_parallel_calls)
)
block_length = len([0 for _ in first_shard])

If the number of volumes in the shard is too large, this snippet of code can be time-consuming. Alternatives are

  • use a combination of n_volumes and number of files with file_pattern to calculate len(first_shard)
  • provide metadata (number of volumes in the shard) as well as total number of volumes in the dataset

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions