feat: add collection-level table format (MS1)

[tedxu]

Implement Iceberg-inspired collection-level table format library
with manifest lists, versioned metadata, schema/index evolution,
snapshot rollback, and OCC transaction commit.

New files under cpp/{include,src,test}/table_format/:

• types.h/types_codec.h: POD structs + Avro codec_traits
• layout.h/cpp: path conventions, version discovery
• manifest_list.h/cpp: ManifestList with shared filesystem I/O
• metadata.h/cpp: Metadata with monotonic snapshot ID counter
• action.h/cpp: ActionBuilder for atomic mutations
• collection_transaction.h/cpp: OCC read/commit with retry
• 7 test files (29 table_format tests, 66 total, 0 failures)

Test plan:

• All 66 tests pass (make test with ASAN)
• Integration tests cover end-to-end, partitions, rollback by
  ID, and rollback by timestamp

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tedxu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [tedxu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tedxu]

Implementation Reference

Collection-level table format on top of existing segment manifests. Enables segment enumeration, schema evolution, time travel, and snapshot rollback without etcd.

Tech: C++17, Apache Avro (DataFile API), Apache Arrow (filesystem + Status/Result), Google Test.

Namespace: milvus_storage::api::table_format

File Layout

cpp/include/milvus-storage/table_format/
├── types.h              POD structs (FieldSchema, SnapshotEntry, SegmentInfo, etc.)
├── types_codec.h        avro::codec_traits<T> for all types (header-only)
├── layout.h             Path conventions, version discovery, unique ID generation
├── manifest_list.h      ManifestList class + shared filesystem I/O helpers
├── metadata.h           Metadata class (root state: schemas, snapshots, indexes)
├── action.h             Action interface + ActionBuilder (mutation API)
└── collection_transaction.h  OCC read/commit lifecycle

cpp/src/table_format/
├── layout.cpp
├── manifest_list.cpp
├── metadata.cpp
├── action.cpp
└── collection_transaction.cpp

cpp/test/table_format/
├── types_test.cpp
├── layout_test.cpp
├── manifest_list_test.cpp
├── metadata_test.cpp
├── action_test.cpp
├── collection_transaction_test.cpp
└── integration_test.cpp

Architecture

CollectionTransaction    — open/read/commit with OCC retry
  ├── Metadata           — root state object (Avro-serialized per version)
  │     ├── CollectionInfo, SchemaInfo[], IndexSpec[]
  │     ├── SnapshotEntry[]  (each references ManifestListInfo[])
  │     └── next_snapshot_id (monotonic counter, like Iceberg's last-sequence-number)
  ├── Action/ActionBuilder — mutations applied atomically to Metadata
  └── ManifestList        — partition→segments mapping (separate Avro file per snapshot)

No separate CollectionReader — read methods live on CollectionTransaction.

Key Design Decisions

Iceberg-inspired patterns:

• All valid snapshots in latest metadata file (enables time travel without reading old versions)
• Optimistic concurrency via CAS file writes with retry
• Monotonic next_snapshot_id counter (not O(n) max scan)
• Snapshot rollback creates a new snapshot referencing historical state (append-only history)
• parent_snapshot_id tracks lineage

Divergences from original plan:

• PartitionSpec removed — partitions tracked directly in manifest list entries
• CollectionResolver removed — OCC retry re-applies the Action on latest metadata
• Read path merged into CollectionTransaction instead of separate CollectionReader
• Codec traits in header (types_codec.h) instead of .cpp — required by Avro DataFile template instantiation
• ActionBuilder pattern replaces direct segment mutation methods on transaction

Mutation via Action/ActionBuilder

All metadata changes go through ActionBuilder → Action::Apply(Metadata&):

auto action = ActionBuilder::Create(fs, base_path)
    .SetCollectionInfo({...})
    .SetSchema(schema)
    .AddSegment("_default", {.segment_id = 1001, .manifest = "..."})
    .AddColumn(new_field)
    .AddIndex(idx)
    .Build();
txn->Commit(action);

Supported operations: SetCollectionInfo, SetSchema, AddColumn, DropColumn, AddPartition, DropPartition, AddSegment, RemoveSegments, AddIndex, DropIndex, SetCurrentSnapshot (rollback by ID), SetCurrentSnapshotByTimestamp (rollback by timestamp).

Mutual exclusivity enforced: rollback by ID and by timestamp cannot coexist in one action.

Internal structure: ActionBuilder uses PIMPL. ActionBuilder::Impl inherits from file-local ActionParams. Build() slices the params into ActionImpl (anonymous namespace). No fragile field-by-field copy.

Commit Protocol

Commit(action):
  loop (up to retry_limit):
    1. Read latest metadata version from filesystem
    2. If version changed since open, re-read latest metadata as base
    3. Apply action to base metadata
    4. Assert action produced >= 1 new snapshot
    5. Squash intermediate snapshots (keep base + final only)
    6. Write metadata file via CAS (conditional write or check-then-write)
    7. If AlreadyExists → retry; else return new version

Serialization

All serialization uses Avro DataFile API (DataFileWriter<T> / DataFileReader<T>). Avro JSON schemas defined as static const char* in .cpp files. Metadata schema includes next_snapshot_id with "default": 1 for backward compatibility; decode repairs the counter if old data lacks it.

On-Disk Layout

<base_path>/
  _metadata/
    v1.metadata.avro      # Metadata (single Avro record)
    v2.metadata.avro
  _manifests/
    <uuid>.avro           # ManifestList (array of ManifestListEntry records)

Test Coverage (66 tests total, 0 failures)

• ActionTest (14): schema evolution, index ops, partition ops, rollback mutual exclusivity, monotonic ID counter
• CollectionTransactionTest (11): commit lifecycle, concurrent commit retry, segment enumeration
• TableFormatIntegrationTest (4): end-to-end, partition management, rollback by ID, rollback by timestamp
• ManifestListTest (4), MetadataTest (1), LayoutTest (6), TypesTest (2)

[codecov]

Codecov Report

❌ Patch coverage is 88.68159% with 91 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.87%. Comparing base (7e550c9) to head (4ae436b).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
cpp/src/table_format/collection_transaction.cpp	69.04%	52 Missing ⚠️
cpp/src/table_format/action.cpp	95.09%	13 Missing ⚠️
cpp/src/table_format/metadata.cpp	81.25%	9 Missing ⚠️
.../include/milvus-storage/table_format/types_codec.h	96.61%	7 Missing ⚠️
cpp/src/table_format/manifest_list.cpp	86.00%	7 Missing ⚠️
cpp/src/table_format/layout.cpp	92.50%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #451      +/-   ##
==========================================
+ Coverage   75.88%   76.87%   +0.98%     
==========================================
  Files         108      117       +9     
  Lines        9450    10256     +806     
  Branches     1316     1429     +113     
==========================================
+ Hits         7171     7884     +713     
- Misses       2279     2372      +93     

Flag	Coverage Δ
cpp	81.40% <88.68%> (+0.69%)	⬆️
python	45.52% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[shaoting-huang]

this seems not thread safe

[shaoting-huang]

should we add deduplication by segment id? in case both transactions want to add the same segment

[shaoting-huang]

should we write a version hint file into object storage to avoid LIST operations? The sync operations from Milvus have commit actions frequently

[tedxu]

/hold