feat: add scalar index V3 diagnostic scanner - #514
Open
congqixia wants to merge 1 commit into
Open
Conversation
Collaborator
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: congqixia The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Tick the box to add this pull request to the merge queue (same as
|
There was a problem hiding this comment.
Pull request overview
Adds a new scan-v3-index diagnostic command and supporting V3 packed scalar index parsing utilities so Birdwatcher can range-read .v3 objects from object storage, decode/validate directory + __meta__, and correlate on-disk index structure with Milvus SegmentIndex/collection schema metadata.
Changes:
- Introduces a V3 index inspector/parser (
inspectV3Index) with directory + meta decoding, CRC verification, encryption detection, and physical index type classification. - Adds
ScanV3IndexCommandto enumerate.v3objects inindex_files/andindex_v1/, correlate with etcd metadata, and emit findings in table/plain/JSON formats. - Extends object-store listing/stat metadata (
ETag,LastModified,VersionID) to detect mid-scan object changes.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| states/v3_index_parser.go | Implements V3 index range-read inspection, directory/meta decoding, and classification helpers. |
| states/scan_v3_index.go | Adds the scan-v3-index command, object discovery, metadata correlation, and diagnostics. |
| states/scan_v3_index_test.go | Adds unit coverage for V3 parsing/classification and scan helpers. |
| oss/object_store.go | Enriches listed/stat’d object metadata (ETag/LastModified/VersionID). |
| models/fs_stat.go | Extends FsStat to carry object metadata returned by Stat. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+517
to
+535
| unique := make(map[string]struct{}) | ||
| matching := make([]*indexpb.SegmentIndex, 0, len(candidates)) | ||
| for _, candidate := range candidates { | ||
| signature := fmt.Sprintf("%d/%d/%d/%d/%d", candidate.GetCollectionID(), candidate.GetPartitionID(), candidate.GetSegmentID(), candidate.GetIndexID(), candidate.GetIndexVersion()) | ||
| unique[signature] = struct{}{} | ||
| matches := candidate.GetPartitionID() == objectPath.partitionID && | ||
| candidate.GetSegmentID() == objectPath.segmentID && | ||
| candidate.GetIndexVersion() == objectPath.indexVersion | ||
| if objectPath.layout == "COLLECTION_ROOTED" { | ||
| matches = matches && candidate.GetCollectionID() == objectPath.collectionID | ||
| } | ||
| if matches { | ||
| matching = append(matching, candidate) | ||
| } | ||
| } | ||
| if len(matching) == 1 { | ||
| return matching[0], len(unique) > 1 | ||
| } | ||
| return candidates[0], len(unique) > 1 |
Comment on lines
+230
to
+232
| jobs := make(chan oss.ObjectInfo) | ||
| results := make(chan scanResult, len(objects)) | ||
| var wg sync.WaitGroup |
Comment on lines
+122
to
+131
| inspection.FormatVersion = binary.LittleEndian.Uint16(footer[0:2]) | ||
| metaSize := int64(binary.LittleEndian.Uint32(footer[24:28])) | ||
| directorySize := int64(binary.LittleEndian.Uint32(footer[28:32])) | ||
| if inspection.FormatVersion != 3 { | ||
| return inspection, []V3IndexFinding{{ | ||
| Rule: "V3_FOOTER_INVALID", | ||
| Severity: v3SeverityCritical, | ||
| Message: fmt.Sprintf("unsupported V3 footer format version %d", inspection.FormatVersion), | ||
| }} | ||
| } |
Add scan-v3-index to inspect packed scalar index V3 objects with range reads and correlate them with SegmentIndex metadata and collection schemas. - detect loader type mismatches, invalid paths, malformed metadata, CRC failures, row-count inconsistencies, and encrypted indexes - support build-rooted and collection-rooted storage layouts - resolve effective JSON types from cast metadata and validate JSON path loader formats - detect string and numeric STLSORT mismatches with JSON-specific diagnostics - add parser and scanner coverage plus object-store range-read support Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
congqixia
force-pushed
the
feat/scan_v3_index
branch
from
August 11, 2026 06:39
e0ab61a to
71730f5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add scan-v3-index to inspect packed scalar index V3 objects with range reads and correlate them with SegmentIndex metadata and collection schemas.