Add file size tracking to snapshot metrics#1187
Open
fm100 wants to merge 1 commit into
Open
Conversation
Tishj
changed the title
Tishj
reviewed
Jul 17, 2026
| @@ -163,7 +193,6 @@ void IcebergAddSnapshot::CreateUpdate(DatabaseInstance &db, ClientContext &conte | |||
|
|
|||
| //! Create a new manifest list, populate it with the content of the old manifest list (altered if necessary) | |||
Member
There was a problem hiding this comment.
Please move the rest of this down as well, you've broken this comment by separating the steps
Tishj
reviewed
Jul 17, 2026
| commit_state.manifests.clear(); | ||
| } | ||
|
|
||
| static void AddLiveFileSize(int64_t &total_files_size, const IcebergDataFile &data_file) { |
Member
There was a problem hiding this comment.
Duplicate of AddFileSizeChecked ?
Tishj
reviewed
Jul 17, 2026
| idx_t deleted_rows_count = 0; | ||
| //! Size in bytes of content files added by this manifest. This is derived from | ||
| //! manifest entries and is not serialized in the manifest list. | ||
| int64_t added_files_size = 0; |
Member
There was a problem hiding this comment.
No, this is meant to mirror the manifest_file spec of https://iceberg.apache.org/spec/#manifest-lists, this information shouldn't live here.
Tishj
reviewed
Jul 17, 2026
| manifest_file.added_rows_count = 0; | ||
| manifest_file.existing_rows_count = 0; | ||
| manifest_file.deleted_rows_count = 0; | ||
| manifest_file.added_files_size = 0; |
Member
There was a problem hiding this comment.
On top of that this shouldn't use manifest_file to carry this information, this is also the wrong place.
CreateFromEntries is used in many contexts, not just new manifests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add Iceberg snapshot summary metrics:
Sizes are derived from manifest entries. For legacy snapshots without total-files-size, the value is reconstructed from live manifest entries.
Testing
Added coverage for appends, positional deletes, and data-file rewrites. Build, formatting, and targeted tests pass.