feat: slim down zvec dynamic libraries by chinaux · Pull Request #627 · alibaba/zvec

chinaux · 2026-07-28T11:33:27Z

Summary

This PR substantially reduces the size of zvec's prebuilt dynamic libraries with
zero functional loss and no public API change. Core search code stays at
-O3, so query performance is unaffected.

Measured on macOS arm64, Release + strip, both built from the same base commit
(d15a37e):

Library	main baseline	this PR	Reduction
`libzvec_c_api.dylib` (C API)	31.11 MB	19.19 MB	-11.92 MB (-38.3%)
`libzvec.dylib` (C++ SDK)	30.88 MB	20.44 MB	-10.44 MB (-33.8%)

It also removes the protobuf/protoc build dependency entirely while keeping
the on-disk manifest format byte-for-byte compatible.

Motivation

Both dynamic libraries statically bundle heavy third-party dependencies (Arrow,
RocksDB, protobuf, ANTLR4, glog). By default they export all of those
symbols and carry code paths zvec never uses. That inflates binary size, slows
down linking, and pollutes the global symbol namespace for downstream consumers.

protobuf was a particularly poor trade: zvec only used it to persist the
collection manifest, yet it pulled in a code generator (protoc), a submodule,
and a runtime library into every build.

Approach

1. Symbol visibility & dead-code elimination

Compile with -ffunction-sections -fdata-sections so unreachable code can be
pruned at link time.
macOS: -exported_symbols_list + -dead_strip
Linux: --version-script + --gc-sections
libzvec_c_api exports only the zvec_* C API.
libzvec exports only zvec's own public C++ API (namespace zvec, including
the nested zvec::ailego / zvec::core / zvec::turbo / zvec::reranker),
hiding ~14.5k third-party symbols (exported symbols: 23,102 → 8,609).

The public C++ headers never expose Arrow/RocksDB/Parquet types, so hiding
those symbols is safe for external C++ consumers.

Two follow-up fixes were needed to make the Linux script correct:

Export the mangled vtable / typeinfo / VTT / guard-variable patterns
(_ZTVN4zvec*, _ZTIN4zvec*, ...). The demangled zvec::* glob does not
match them, so consumers previously failed with
undefined reference to vtable for zvec::core_interface::HNSWIndexParam.
Drop the quoted "zvec::*" entry: in version scripts a quoted name is a
literal match, not a glob. GNU ld silently ignores the never-matching entry,
but lld (Android NDK) fails under --no-undefined-version.

2. Drop the protobuf / protoc dependency

The manifest is now encoded by a self-contained codec instead of generated
protobuf code:

pb_wire.h — a minimal protobuf wire-format reader/writer (varint,
fixed32/64, length-delimited). Unknown fields are consumed and ignored to
preserve proto3 forward compatibility; malformed input (bad varint,
out-of-range length, groups, field number 0) is reported rather than crashing.
manifest_codec.{h,cc} — converts all 16 messages directly between zvec's C++
types and bytes, with no intermediate message object.
manifest_enum.h — the on-disk enum values, preserved exactly.

The thirdparty/protobuf submodule, src/db/proto/zvec.proto and the generated
proto_converter are removed.

Format compatibility is enforced by two test suites:

While libprotobuf was still in the tree (commit 9a27877), a temporary
manifest_codec_test.cc cross-checked both implementations: every message had
to encode to identical bytes, and each side had to parse the other's output.
This covered the subtle cases — always-present base / quantizer_param
sub-messages, the absent quantizer_param of HNSW_RABITQ, Vamana's
fixed32 alpha, empty strings. That test was removed together with the
dependency in d8cd772, since it cannot compile without libprotobuf.
manifest_codec_golden_test.cc (kept) pins golden byte arrays that were
produced by the old protobuf implementation. It has no libprotobuf
dependency, so it continues to guard on-disk format compatibility now that
the dependency is gone.

Existing collections therefore remain readable, and manifests written by this
build remain readable by older releases.

3. Third-party feature trimming

Arrow: ARROW_DATASET=OFF; unused Compute kernels trimmed via
thirdparty/arrow/arrow.slim_compute.patch (aggregates, hash-aggregates,
temporal, round, random, rank, replace, select_k, statistics,
cumulative_ops, pivot).
RocksDB: trace / iostats / perf contexts disabled.

Note on the Arrow patch: an earlier revision over-trimmed and broke production
paths, because several kernels are reached indirectly through Acero nodes and
are invisible to a source grep. sort_indices (needed by the order_by node
that sorts vector scores and FTS BM25 results), make_struct (the backing
implementation of compute::project()), match_like (SQL LIKE) and
list_value_length (SQL array_length()) are all restored, and each retained
kernel now carries a comment stating why it is required.

4. Size-optimized third-party builds

GCC/Clang: -Os for RocksDB / glog / ANTLR4, MinSizeRel for Arrow.
MSVC: /O1 /Ob1 (favor small code) instead of /O2 /Ob2, including Arrow's
Release configuration — mirroring the -Os / MinSizeRel behavior above.

zvec's own code is untouched and keeps its default optimization level.

Where the remaining size goes

For reference, a link-map attribution of the resulting C API library:

Component	Size	Share
Arrow stack (core + compute + acero + parquet + bundled)	9.37 MB	49.6%
zvec's own code (core + db + ailego + turbo + c_api)	5.44 MB	28.8%
RocksDB	2.93 MB	15.5%
Other third-party (snowball / ANTLR4 / lz4 / glog / roaring / FastPFOR)	1.14 MB	6.0%

Compatibility

No public API changes: headers under src/include/zvec/ keep their
existing declarations; only symbol visibility and unused internals changed.
On-disk format unchanged: the manifest is still protobuf wire format,
pinned by golden-byte tests.
Third-party sources are not modified in place: Arrow trimming is applied
as a patch at build time, keeping submodules pristine.
Existing C / C++ / Python examples and tests pass unchanged.

Testing

Full C++ unit test suite, Python tests, and the C/C++ examples pass on
macOS (arm64), Linux (x64/arm64), Windows (x64), plus Android and iOS builds.
Manifest format verified by byte-level cross-checks against libprotobuf and by
golden-byte tests.
Exported symbol tables inspected with nm / objdump to confirm only the
intended public symbols remain, and that no third-party symbols leak.
Library sizes verified on each platform.

Reduce the FAT C API dylib size by ~37% through a 5-layer strategy with zero functional loss; core search code stays at -O3 for performance: - Compiler dead-code sectioning: -ffunction-sections -fdata-sections - Linker pruning: macOS -dead_strip + exported symbol list; Linux --gc-sections + version script, keeping only zvec_* C API symbols - Third-party feature trimming: Arrow disables Dataset & trims Compute kernels; RocksDB disables trace/iostats/perf contexts - Protobuf lite: proto uses LITE_RUNTIME, link libprotobuf-lite - Size-optimized third-party builds: -Os for RocksDB/glog/antlr4, MinSizeRel for Arrow

Extend the size-reduction work to the C++ all-in-one shared library and the Windows build: - libzvec C++ SDK (macOS/Linux): restrict exported symbols to zvec's own public API (namespace zvec, incl. nested zvec::ailego/core/turbo) via -exported_symbols_list / --version-script + -dead_strip / --gc-sections. Hides ~14.5k third-party symbols; libzvec.dylib 24.41MB -> 19.05MB (-21.9%), total -34.1% vs main baseline. Public C++ headers never expose third-party types, so external C++ consumers are unaffected. - Third-party libs (MSVC): favor small code (/O1 /Ob1) over speed (/O2) to mirror the -Os / MinSizeRel size optimization used on GCC/Clang, including Arrow's Release config.

…rom tests The slim compute patch removed kernels that production code paths rely on: - match_like (LIKE queries) from scalar_string_ascii/utf8 - sort_indices (ORDER BY) from vector_sort/vector_array_sort - make_struct / list_value_length from scalar_nested Restore these kernels in arrow.slim_compute.patch while keeping the rest (aggregates, temporal, round, random, etc.) removed. Arrow Dataset stays disabled: sql_expr_validator_test's dataset member was dead code and is removed; sql_expr_parser_test's scan test is rewritten with arrow::compute::ExecuteScalarExpression, preserving the original intent (verify a parsed expression evaluates correctly on real data) without depending on arrow::dataset. All 153 C++ tests pass locally (write_recovery_test flake passes in isolation).

…script The version script only exported zvec::* function/data symbols. vtable, typeinfo, VTT and guard variable symbols demangle to 'vtable for zvec::...' etc., which the zvec::* pattern does not match, so they were localized by 'local: *'. Consumers linking libzvec.so then failed with 'undefined reference to vtable for zvec::core_interface::HNSWIndexParam'. Add the mangled-name patterns (_ZTVN4zvec*, _ZTIN4zvec*, _ZTSN4zvec*, _ZTTN4zvec*, _ZGVN4zvec*), mirroring what exported_symbols_cpp.txt already does on macOS.

…ibility In linker version scripts, a quoted "zvec::*" entry is a literal name match (not a glob). GNU ld silently tolerates the never-matching entry, but lld (used by the Android NDK) enforces --no-undefined-version and fails with: ld.lld: error: version script assignment of 'global' to symbol 'zvec::*' failed: symbol not defined Keep only the unquoted glob pattern, which matches all demangled zvec:: symbols on both GNU ld and lld.

Introduces a self-contained implementation of the manifest on-disk format, in preparation for dropping the libprotobuf/protoc dependency: - pb_wire.h: minimal protobuf wire-format reader/writer (varint, fixed32/64, length-delimited). Unknown fields are consumed and ignored, preserving proto3 forward compatibility; malformed input (bad varint, out-of-range length, groups, field number 0) is reported instead of crashing. - manifest_enum.h: the on-disk enums, mirroring src/db/proto/zvec.proto with identical numeric values. The CodeBooks in type_helper.h now speak these instead of the generated protobuf enums. - manifest_codec.{h,cc}: encodes/decodes all 16 messages directly between zvec's C++ types and bytes, without an intermediate message object. Format compatibility is established by two test suites: - manifest_codec_test.cc cross-checks against libprotobuf while it is still available: every message encoded by both implementations must produce identical bytes, and each side must parse the other's output. This covers the subtle cases - always-present base/quantizer_param sub-messages, the absent quantizer_param of HNSW_RABITQ, Vamana's fixed32 alpha, empty repeated string elements and oneof "last branch wins". - manifest_codec_golden_test.cc pins protobuf's real output as embedded byte arrays and does not depend on libprotobuf, so it keeps guarding the format after the dependency is removed. IVF_RABITQ landed on main while this work was in flight, so the codec carries it too. It reuses the field numbers of the proto definition (IndexParams.ivf_rabitq = 9; base = 1, nlist = 2, total_bits = 3, sample_count = 4) and, like HNSW_RABITQ, leaves the quantizer_param sub-message absent. libprotobuf can no longer produce a reference encoding for it, so its golden bytes are derived from those field numbers by hand rather than cross-checked. src/db/proto/zvec.proto is retained as the authoritative documentation of the format. No behaviour change yet: Version::Save/Load still use protobuf.

Version::Load/Save no longer build an intermediate proto::Manifest; they read and write the manifest bytes with ManifestCodec instead. The on-disk format is unchanged - manifest_codec_test.cc verifies byte-for-byte equality with the protobuf implementation, and manifest_codec_golden_test.cc pins the format independently. Manifests are a few kilobytes, so the file is read/written in one go rather than streamed. ProtoConverter is intentionally kept for now: it is what the cross-check tests compare against. It is removed together with the protobuf dependency.

The manifest is now read and written entirely by ManifestCodec (added and cross-checked against protobuf in the previous commits), so libprotobuf and protoc are no longer needed. Removed: - thirdparty/protobuf submodule and its build integration - src/db/proto compilation (cc_proto_library, zvec.pb.cc, libprotobuf-lite link and zvec_proto target/deps across src and tests) - ProtoConverter and the cross-check test that depended on libprotobuf - the "build host protoc" stage from the Android and iOS CI workflows and the build_android.sh / build_ios.sh scripts, along with the now-unused GLOBAL_CC_PROTOBUF_PROTOC option. Cross-compiling no longer needs a host protoc, simplifying those pipelines noticeably. Kept: - src/db/proto/zvec.proto as the authoritative documentation of the on-disk manifest format - cc_proto_library in cmake/bazel.cmake as a generic helper (no longer used by zvec itself) The manifest on-disk format is unchanged; manifest_codec_golden_test.cc keeps guarding it. Full C++ test suite (156 tests) passes locally.

zvec.proto was retained as documentation when the protobuf dependency was dropped, but the manifest format is now fully defined by manifest_codec.{h,cc} (field numbers) and manifest_enum.h (enum values). Delete the file and the src/db/proto directory, and repoint the doc comments that referenced it to the codec instead. Also drop the stale clang-tidy header cache entries for build/src/db/proto and the src/db/proto/*.proto hashFiles input, which no longer exist.

meta.h uses std::numeric_limits but never included <limits>; it compiled only because type_helper.h transitively included the generated zvec.pb.h, which brought <limits> along. With protobuf gone the omission surfaced on libstdc++/GCC: meta.h:280: error: 'numeric_limits' is not a member of 'std' libc++ happens to provide it transitively, which is why only the Linux jobs failed. Also make the files touched by the codec change include what they use: <memory>/<string>/<string_view>/<vector>/<cstdint> in manifest_codec.cc and <memory> in version_manager.cc.

The export whitelists are explicit, so the C API functions IVF_RABITQ added on main were hidden and c_api_test failed to link against libzvec_c_api with 17 undefined references. Add them to both lists, which now cover exactly the 371 functions declared in zvec/c_api.h with no entry on either side that the header does not declare.

The Windows code-page tests added by "fix: handle UTF-8 paths correctly on Windows" still list zvec_proto, which this branch no longer builds, so MSVC failed with LNK1181: cannot open input file 'zvec_proto.lib'. No source under tests/db/utf8_acp uses protobuf.

feihongxu0824 · 2026-08-13T06:19:52Z

+
+//! IVF_RABITQ index params, encoded by hand from the wire format.
+//!
+//! Unlike the arrays above this one was not produced by the protobuf


这段注释想表达什么意思？为啥“there is nothing left to cross-check”?

feihongxu0824 · 2026-08-13T06:29:25Z

+
+using namespace zvec;
+
+namespace {


这部分兼容验证一定要100%完备啊

EmptyManifest

代码 139–146 行
不完备之处：
名字叫 Empty，但实际包含一个 schema，其中 max_doc_count_per_segment=10000000，测试没有断言该值。
没有明确断言 version、两个 suffix、persisted/writing segment 等默认状态。
没有测试 protobuf 的真正空消息、schema 缺失、空但存在的子消息等情况。
只有一组默认组合，没有验证默认字段省略和 message presence 的差异。

SimpleManifest

代码 148–177 行
不完备之处：
FieldSchema 没有检查 nullable。

HNSW 只检查 metric、m、ef_construction，缺少：
quantize type
quantizer param
use_contiguous_memory
空但存在的 quantizer_param 子消息语义

segment 只检查 ID 和 block 数量，没有检查：
block ID 和 BlockType
min/max doc ID、doc_count
columns
indexed vector fields
writing block 是否不存在

没有明确检查 version 和两个 suffix。

只覆盖单 segment、单 block、单 column，没有 repeated 的多元素组合。

AllIndexTypes

代码 180–225 行
这是缺口最大的一项：
名称叫 AllIndexTypes，但完整 manifest 中没有 IVF_RABITQ；它只在另一个独立参数测试中出现。
对八种索引主要只检查 type()，没有逐字段验证参数。
Vamana 只抽查 alpha 和三个 bool，缺少 metric、quantize、quantizer param、degree、search list。
FTS 没有检查 extra_params。
INVERT、FLAT、IVF、HNSW_RABITQ、DiskANN 基本没有检查具体参数。
没有检查 schema 的名称和 max_doc_count_per_segment。
九个 FieldSchema 没有完整检查 name、DataType、dimension、nullable。
没有检查 enable_mmap=true。
persisted segment 只检查数量，没有检查每个 segment 的 ID、block 内容和 indexed fields。
writing segment 只检查 ID 和 writing block 存在，没有检查 block 的具体内容。
BlockType 实际基本只覆盖 SCALAR，没有覆盖其他类型的 wire 解码。
缺少每种索引的全默认值、false、空消息和边界值组合。

egolearner

rocksdb/arrow做了裁减，并改用Os编译，做下micro benchmark看下对倒排/fts检索和正排filter性能看下有哪些影响吧

egolearner · 2026-08-13T09:37:13Z

+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "db/index/common/manifest_codec.h"


做一定的封装，实现细节搞一个单独的目录，common下面可以仅保留public api。目前不好区分哪些是public api，哪些是内部实现。然后写个README注明下兼容性，是否允许删除字段，新增字段需要做哪些修改。

我觉得保留proto使用protobuf-lite可维护性更好一点，为了精简库大小自己实现也可以接受。

egolearner · 2026-08-13T09:39:43Z

+  //! Advances to the next field. Returns false at end of buffer or on error;
+  //! use ok() to distinguish the two.
+  bool Next() {
+    if (!ok_ || pos_ >= size_) {


长实现放cc文件吧

egolearner · 2026-08-13T09:47:35Z

-  SRCS proto/*.proto
-  PROTOROOT ./
-)
+# NOTE: the manifest on-disk format is read and written by


这个注释去掉吧

egolearner · 2026-08-13T09:57:09Z

rocksdb/arrow做了裁减，并改用Os编译，做下micro benchmark看下对倒排/fts检索和正排filter性能看下有哪些影响吧

另外也做下端到端的性能测试吧

Replace the manifest golden tests with 18 cases whose byte arrays were verified byte-for-byte against two independent protobuf implementations: the legacy libprotobuf-based serializer and a fresh implementation compiled from the archived zvec.proto. The round-trip coverage exposed two decode bugs: manifests without max_doc_count_per_segment decoded to the C++ default instead of zero, unlike the old protobuf reader. Both decode paths now mirror that behaviour.

…anifest The flag was silently dropped on save ever since the old proto schema omitted it. Encode it as field 2 of InvertIndexParams; old manifests decode to the default (false) and old readers skip unknown fields, so the change is compatible in both directions. Covered by the extended VersionLoadSave test.

- db_type_helper_test: rename Proto*/Cpp* tests to Wire*/Cpp* and cover HNSW_RABITQ/IVF_RABITQ/VAMANA/DISKANN/FTS, sparse FP16, MIPSL2 and the AsString overloads. - version_manager_test: VersionLoadSave now round-trips schema fields with index params, enable_mmap, persisted blocks, indexed vector fields and the writing segment. - New pb_wire_test for the wire-format reader/writer (varint edges, proto3 default skipping, malformed input). - Comment/variable naming fixes in type_helper.h, segment.cc and src/CMakeLists.txt; trailing newline in sql_expr_parser_test.cc.

common/ now keeps only the public API (manifest_codec.h, manifest_enum.h). The codec implementation, the pb_wire reader/writer and a new README documenting the wire format and its compatibility rules (how to add fields, why retired field numbers must never be reused) live in common/manifest/. Also split the long Reader::Next()/ReadVarintRaw() implementations out of pb_wire.h into pb_wire.cc, keeping only declarations and short inline methods in the header.

- python binding: the musl static-link rationale referenced protobuf's protodesc_cold section, which no longer applies. - golden test header: drop the reference to the deleted manifest_codec_test.cc. - manifest_codec.h / pb_wire.h: point at the new manifest/ implementation location. - meta.h: remove a leftover instruction comment; gcov.sh: drop the '*/proto/*' coverage filter.

upstream/main still modified the protobuf files this branch deletes (zvec.proto, proto_converter.cc, db_proto_converter_test.cc); keep the deletions. Port the one feature that landed through them — VamanaIndexParams.two_pass_build (proto field 8, upstream alibaba#634) — into ManifestCodec encode/decode, with a round-trip test.

… library upstream alibaba#634 added zvec_index_params_{get,set}_vamana_two_pass_build to the C API and its tests; the slimmed libzvec_c_api hides unlisted symbols, so c_api_test failed to link on every platform. Add both symbols to exported_symbols.lds and exported_symbols.txt.

…arch paths Reviewer asked for micro benchmarks to measure the impact of the rocksdb/arrow slimming and the -Os thirdparty build on inverted/FTS search and forward filter performance. Add perf_bench_test covering: forward filters (int range, string, double predicates), inverted search (term/range/string/or), FTS search (rare/medium/common terms, multi-term AND/OR, FTS+inverted filter), and insert throughput reporting. The suite reuses the sqlengine recall_base fixtures and is gated by ZVEC_RUN_PERF_BENCH=1 (all tests GTEST_SKIP otherwise, so CI is unaffected); ZVEC_PERF_DOC_COUNT (default 100000) and ZVEC_PERF_ITERATIONS (default 30) control scale. Each case prints median/mean/p90/p99/min latency and QPS for A/B comparison between builds.

chinaux requested review from Cuiyus, egolearner, feihongxu0824 and zhourrr as code owners July 28, 2026 11:33

github-actions Bot assigned chinaux Jul 28, 2026

chinaux force-pushed the feat/slim-c-api-library branch 2 times, most recently from b91f370 to 3462df8 Compare July 29, 2026 12:04

chinaux force-pushed the feat/slim-c-api-library branch 3 times, most recently from 287ed09 to 323fba7 Compare August 11, 2026 09:00

chinaux added 13 commits August 11, 2026 19:04

style: apply clang-format to segment.cc and store_helper.h

a53db1a

chinaux force-pushed the feat/slim-c-api-library branch from f6a2758 to 4d6e295 Compare August 11, 2026 11:13

feihongxu0824 requested changes Aug 13, 2026

View reviewed changes

egolearner reviewed Aug 13, 2026

View reviewed changes

chinaux added 2 commits August 13, 2026 20:44

chinaux added 6 commits August 13, 2026 20:48

Navigation Menu

Uh oh!

feat: slim down zvec dynamic libraries - #627

feat: slim down zvec dynamic libraries#627
chinaux wants to merge 21 commits into
alibaba:mainfrom
chinaux:feat/slim-c-api-library

chinaux commented Jul 28, 2026 •

edited

Loading

Uh oh!

feihongxu0824 Aug 13, 2026

Uh oh!

feihongxu0824 Aug 13, 2026

Uh oh!

egolearner left a comment

Uh oh!

egolearner Aug 13, 2026

Uh oh!

egolearner Aug 13, 2026

Uh oh!

egolearner Aug 13, 2026

Uh oh!

egolearner commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

chinaux commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Approach

1. Symbol visibility & dead-code elimination

2. Drop the protobuf / protoc dependency

3. Third-party feature trimming

4. Size-optimized third-party builds

Where the remaining size goes

Compatibility

Testing

Uh oh!

feihongxu0824 Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

feihongxu0824 Aug 13, 2026

Choose a reason for hiding this comment

EmptyManifest

SimpleManifest

AllIndexTypes

Uh oh!

egolearner left a comment

Choose a reason for hiding this comment

Uh oh!

egolearner Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

egolearner Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

egolearner Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

egolearner commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

chinaux commented Jul 28, 2026 •

edited

Loading