Skip to content

Preserve tensor feature padding_value during serialization#92

Open
this-AR wants to merge 2 commits into
sb-ai-lab:mainfrom
this-AR:fix/preserve-padding-value
Open

Preserve tensor feature padding_value during serialization#92
this-AR wants to merge 2 commits into
sb-ai-lab:mainfrom
this-AR:fix/preserve-padding-value

Conversation

@this-AR

@this-AR this-AR commented Jun 21, 2026

Copy link
Copy Markdown

Summary

This change ensures the padding_value of tensor features is included during schema serialization and restored when loading. This fixes losing custom padding values (e.g. -1) when saving and loading sequential datasets or tokenizers.

Files changed

  • schema.py — include padding_value in _get_object_args and ensure _create_object_by_args passes it to TensorFeatureInfo.
  • padding_value_serialization_fix.md — describe the bug and fix.
  • test_schema.py — add/verify test_tensor_schema_serialization_preserves_padding_value.
  • test_sequential_dataset.py — verify save/load preserves feature padding values.

Motivation

Previously custom padding values were not saved with tensor schema metadata and were reset to 0 on load. This breaks workflows relying on non-zero padding sentinels.

Backward compatibility

Older saved metadata without padding_value remains compatible because TensorFeatureInfo defaults padding_value to 0.

Tests

pytest -q tests/data/nn/test_schema.py::test_tensor_schema_serialization_preserves_padding_value \
       tests/data/nn/test_sequential_dataset.py::test_save_and_load_sequential_dataset

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant