Metadata.csv explodes to thousands of rows after view-aligned processing pipeline

[Moyu139]

Hi, thanks for releasing the Pixal3D data toolkit.

I ran into a metadata growth issue when using the view-aligned data processing pipeline from data_toolkit/README.md.

After running the pipeline on a small custom dataset with 72 assets and 2 rendered views, the top-level metadata.csv unexpectedly grew from one row per asset to thousands of rows per asset.

Example output:

Found 72 unique custom models.
Saved merged metadata with 409472 entries and columns: ['sha256', 'cond_rendered', 'dual_grid_view00_converted_1024', 'dual_grid_view00_converted_256', 'dual_grid_view00_converted_512', 'dual_grid_view00_scale_1024', 'dual_grid_view00_scale_256', 'dual_grid_view00_scale_512', 'dual_grid_view00_size_1024', 'dual_grid_view00_size_256', 'dual_grid_view00_size_512', 'dual_grid_view01_converted_1024', 'dual_grid_view01_converted_256', 'dual_grid_view01_converted_512', 'dual_grid_view01_scale_1024', 'dual_grid_view01_scale_256', 'dual_grid_view01_scale_512', 'dual_grid_view01_size_1024', 'dual_grid_view01_size_256', 'dual_grid_view01_size_512', 'file_identifier', 'local_path', 'mesh_dumped', 'num_pbr_voxels_view_fix00_1024', 'num_pbr_voxels_view_fix00_256', 'num_pbr_voxels_view_fix00_512', 'num_pbr_voxels_view_fix01_1024', 'num_pbr_voxels_view_fix01_256', 'num_pbr_voxels_view_fix01_512', 'pbr_dumped', 'pbr_latent_view00_encoded', 'pbr_latent_view00_tokens', 'pbr_latent_view01_encoded', 'pbr_latent_view01_tokens', 'pbr_voxel_scale_view_fix00_1024', 'pbr_voxel_scale_view_fix00_256', 'pbr_voxel_scale_view_fix00_512', 'pbr_voxel_scale_view_fix01_1024', 'pbr_voxel_scale_view_fix01_256', 'pbr_voxel_scale_view_fix01_512', 'pbr_voxelized_view_fix00_1024', 'pbr_voxelized_view_fix00_256', 'pbr_voxelized_view_fix00_512', 'pbr_voxelized_view_fix01_1024', 'pbr_voxelized_view_fix01_256', 'pbr_voxelized_view_fix01_512', 'shape_latent_view00_encoded', 'shape_latent_view00_tokens', 'shape_latent_view01_encoded', 'shape_latent_view01_tokens', 'ss_latent_view00_encoded', 'ss_latent_view01_encoded']
Statistics:
  - Number of assets: 409472
  - Number of assets downloaded: 0
  - Number of assets with render conditions: 72
  - Number of assets with mesh dumped: 72
  - Number of assets with PBR dumped: 72
  - Number of assets with dual grid view:
    - 1024: 72
    - 256: 72
    - 512: 72
  - Number of assets with PBR voxelization view:
    - pbr_voxels_view_fix_256: 72
    - pbr_voxels_view_fix_512: 72
    - pbr_voxels_view_fix_1024: 72
  - Number of assets with sparse structure latents:
    - ss_enc_conv3d_16l8_fp16_64_view: 3231
  - Number of assets with shape latents view:
    - shape_enc_next_dc_f16c32_fp16_1024_view: 72
    - shape_enc_next_dc_f16c32_fp16_256_view: 72
    - shape_enc_next_dc_f16c32_fp16_512_view: 72
  - Number of assets with PBR latents view:
    - tex_enc_next_dc_f16c32_fp16_1024_view_fix: 72
    - tex_enc_next_dc_f16c32_fp16_256_view_fix: 72
    - tex_enc_next_dc_f16c32_fp16_512_view_fix: 72

Since the dataset only has 72 unique assets, I expected the top-level metadata to stay at 72 rows. Am I doing something wrong or is this the intended behavior of the Pixal3D metadata design?

[Moyu139]

I noticed there were many duplicated rows in the metadata, so I removed them. After doing that, I got:

Statistics:
  - Number of assets: 5184 (=72x72)