Preserve structured, retry-aware error codes from storage/knowhere across the segcore → Go boundary

[czs007]

Problem

Errors from milvus-storage, knowhere, and native segcore all converge on the shared milvus-common::ErrorCode (2001–2099) and cross cgo as CStatus{error_code}. At several boundaries the precise code is lost: most third-party errors flatten to UnexpectedError 2001, and a few coarse buckets carry the wrong retry default. So the runtime cannot tell input (caller's fault) from transient (retriable) from permanent (non-retriable): a transient OOM/IO failure is never retried, and a malformed request is blamed as a system error.

Design (v2)

The three producers each link milvus-common and own their →ErrorCode mapping ("who produces the code classifies it"). milvus-common::ErrorCode is the single shared convergence point; each producer maps its own internal code into it once, and the Go layer derives the retry/ownership decision.

flowchart TB
  S["milvus-storage<br/>ExtendStatusCode → ToSegcoreError"] --> SC
  K["knowhere<br/>knowhere::Status (+ StatusCategoryOf)"] --> SC
  N["native segcore<br/>arrow / aws / gcp / folly / marisa / simdjson ..."] --> SC
  SC["shared milvus-common::ErrorCode 2001-2099<br/>(cgo CStatus.error_code)"] --> GO{"Go classifySegcoreError"}
  GO --> IN["input · non-retri<br/>InvalidParameter 2042, ExprInvalid 2028 ..."]
  GO --> TR["transient system · retriable<br/>FileReadFailed 2014, MemAllocateFailed 2034, StorageTransientError 2045 ..."]
  GO --> PE["permanent system · non-retri<br/>DataFormatBroken 2024, StorageError 2044, ObjectNotExist 2017 ..."]

Loading

Two orthogonal axes (segcoreClass{inputError, retriable})

class	retriable	whose fault
input	no	caller
transient system	yes	server
permanent system	no	server

Invariant: input ⇒ non-retriable. A single bool retriable is insufficient — input and permanent-system are both non-retriable but different.

Paired generic storage fallbacks

Each fallback carries exactly one retry verdict and they must never be conflated:

code	meaning	retry
StorageError 2044	permanent / internal storage fallback	non-retriable
StorageTransientError 2045 (new)	transient object-store IO / throttle / timeout	retriable

A transient storage IO failure must map to 2045, never collapse into 2044.

Anti-drift: two seams (the core of the design)

• Seam ① internal code → ErrorCode (each producer's own mapper): a switch with no default + a post-switch fallback, under -Werror=switch, so a newly added internal code fails the build until it is classified.
• Seam ② ErrorCode → Go class (-Wswitch cannot cross languages): the Go code list is generated from milvus-common's enum ErrorCode and the classification is an exhaustive switch guarded by the exhaustive linter — the near-compile-time analog of -Werror=switch across the C++→Go seam. Plus a runtime backstop: an unmapped code degrades to non-retriable (never panics) and increments a metric.

Implementation status

milvus-common

• Add StorageTransientError = 2045, paired with StorageError = 2044 (retriable storage transient fallback).

milvus-storage (builds on #572)

• Packed ExtendStatusCode (PackedInvalidArgs / PackedStorageIO / PackedMetadataCorrupted / PackedFileCorrupted / PackedArrowError / PackedUnexpected) + WrapExtendError.
• Producer-owned ToSegcoreErrorCode / ToSegcoreError — no-default switch + -Werror=switch: PackedStorageIO → StorageTransientError(2045), PackedMetadataCorrupted/PackedFileCorrupted → DataFormatBroken, PackedArrowError/Unexpected/AWS/Txn → StorageError, PackedInvalidArgs → InvalidParameter.

knowhere (follows #1675, already merged)

• Harden StatusCategoryOf to a no-default switch + -Werror=switch so a new knowhere::Status cannot silently fall into inner_error (surfaced cardinal_inner_error, now explicit).

milvus (#50768)

• T1 — register StorageError 2044 (non-retriable) and StorageTransientError 2045 (retriable) in pkg/util/merr/segcore.go.
• T2 — KnowhereStatusToErrorCode → no-default switch + -Werror=switch; add build-path variant KnowhereBuildStatusToErrorCode (build-time malloc_error/disk_file_error stay retriable instead of collapsing to a permanent IndexBuildError; generic inner → IndexBuildError).
• T3/T4 — ArrowStatusToErrorCode delegates to the producer's milvus_storage::ToSegcoreError (retires milvus's duplicate mapper); audited and routed 25 storage arrow-status sites that were collapsing to 2001 (PayloadReader/Writer, Remote{Input,Output}Stream, FileManager, StorageV2/TEXT readers) through the single mapper, extracted to storage/StatusToErrorCode.h. Sub-code is preserved in the message (status.ToString()).
• T5 — unmapped-code observability: UnmappedSegcoreCodeTotal{code} counter + rate-limited WARN via an observer hook (merr is a leaf package and cannot import metrics/mlog); registered on QueryNode and DataNode. Unknown code degrades to non-retriable, never panics.
• T6 — codegen + compile-time enforcement: a generated SegcoreCode type (from EasyAssert.h) + an exhaustive classForCode switch marked //exhaustive:enforce, with the exhaustive golangci-lint enabled opt-in. A new C++ code that isn't classified fails lint (seam ②).
• §3 B-tier — classify marisa (StringIndexMarisa: IO/FORMAT/SIZE/MEMORY) and simdjson (Json parse: MEMALLOC/IO/malformed) instead of collapsing to 2001; sub-code carried in the message. simdjson optional-access (NO_SUCH_FIELD/INCORRECT_TYPE) is deliberately left a benign skip; the loon_ffi boundary is untouched.

Deferred (design §4.7)

• FFI / LOON path — LOON_* enum-ization + a category byte on LoonFFIResult (retry travels as a closed 3-value category; specific code best-effort). Non-C++ bindings must never see milvus::ErrorCode. Kept separate from the C++ direct-link main path; the milvus branch dropped the earlier loon-typing to avoid coupling. Follow-up.

Related: #50768 (milvus consumer side), #572 (milvus-storage packed codes), zilliztech/knowhere#1675 (knowhere facade guard). A full visual design doc (v2, the complete 2001–2099 map across the three producers + the anti-drift design) is available.