Retain and expose schema field annotations from Avro, JSON Schema, and Protobuf codecs

[jfallows]

Describe the desired outcome from the user's perspective

As a Zilla operator, I want the schema codecs to retain and expose whatever field annotations my schemas already carry, so that any consumer of a compiled schema — a model extension, a registry-mode catalog integration, or similar — can read them off the field object without the codec silently discarding whatever it doesn't recognize as a standard keyword.

Acceptance criteria

1. Retain unrecognised field annotations

The codecs discard them today.

• common-avro — AvroSchemaCompiler keeps only logicalType, precision, scale, aliases, name, size; AvroField exposes only name(), type(), aliases(), defaultValue()
• common-json — equivalent gap for JSON Schema annotations
• common-protobuf — ProtobufField retains no options at all (its fields are number, name, jsonName, type, repeated, required, packed, proto3Optional, typeName, oneofName, defaultValue, message, enumeration). ProtobufSourceCompiler.applyOptions consumes the known ones and discards the rest

2. Annotations are read from the field, not the type

Read from the field object only, for every format. A tag on the field's type is ignored, named type references are not followed, and record-level and root-level tags are not inherited. This is a deliberate scope boundary, not an oversight: supporting type-level or inherited tags would require walking type references and merging scopes across them, machinery this issue does not need — a schema author who wants a type's every usage tagged can tag each field directly.

So:

• Avro — field object only.
• JSON Schema — the property's schema, necessarily, since there is no separate field object
• Protobuf — field options only

3. Per-format exposure and per-format evaluation

No shared evaluator. A model is one format at a time, so there is no polymorphic call site to serve, and forcing a common intermediate representation costs more than it saves:

• Avro and JSON Schema annotations are already JsonValue — Avro schemas are JSON, JSON Schema annotations likewise
• Protobuf options come from an ANTLR parse tree of text-format constants, an entirely different source. ProtobufSourceCompiler captures option.optionName().getText() and option.constant().getText() with no descriptor resolution

So each codec exposes annotations in the form natural to it:

// common-avro, common-json
JsonValue attribute(String name);      // null when absent

For common-protobuf, expose the option constant from the parse tree rather than converting it. Do not route through ProtobufJson — that bridge converts protobuf wire data to JSON, whereas these are text-format source constants, which is a different problem. Walking the grammar's constant subtree is the right approach.

Two protobuf details that follow:

• Options need no pre-declaration. Because parsing is source-text with our own grammar and never resolves a descriptor, an arbitrary [(acme.classification) = "PII"] is readable even though nothing declares that extension. Document that the producer's protoc toolchain will still reject an undeclared option at their build time
• No proto3 JSON int64→string quirk. Reading from source text, [(acme.limit) = 123] is an integer constant, so it compares as a number. That quirk only applied under the ProtobufJson route
• String constants come back from getText() with their quotes — unquote before comparing

Also needed alongside field-level retention, since a consumer walking a schema for tagged fields needs to discover which paths carry annotations without re-parsing: a predicate-driven path-discovery method per format (AvroType.matchingPaths(Predicate), JsonSchema.matchingPaths(Predicate), ProtobufMessage.matchingPaths(Predicate)), returning RFC 6901-style pointers for every node a caller-supplied predicate matches.

Coverage

• round-trip: a schema declaring annotations of each shape compiles, exposes them, and survives the existing conformance suites
• protobuf: an option not declared by any imported .proto is still retained and readable
• protobuf: quoted string constants unquote correctly; message-valued options expose their structure rather than flattened text

Why configurable rather than a name we pick. Field tagging has no clear winner — Snowplow's bare pii: true, Adobe XDM's xdm: namespace, assorted x-pii/x-data-classification customs, and Apicurio #7879 building field-level classification with authorization-scoped views. Hardcoding one couples generic code to a vendor; minting our own forks the space and forces adopters to re-tag. This issue only retains and exposes whatever a schema already carries — it declares no vocabulary of its own, and no matching/selection semantics on top of it. What a consumer does with an exposed annotation is out of scope here.

Additional context

Scope note (narrowed from the original issue text): this issue originally also proposed, as §§4–9, a shared tagged:/named: field-matcher config DSL with a normative matcher matrix, an overlay: interaction, and a root-level (schema-root) attribute() accessor — all evaluated generically across model-avro/model-json/model-protobuf. None of that materialized as a shared oss capability, and none currently has a driving consumer:

• The three SPI issues that cited this issue as a blocker — Declare AvroModelExtFactorySpi and wire provider-supplied pipeline stages in avro model #1681, Declare JsonModelExtFactorySpi and wire provider-supplied pipeline stages in json model #1682, Declare ProtobufModelExtFactorySpi and wire provider-supplied pipeline stages in protobuf model #1683 (all closed/completed) — each narrowed their own dependency on this issue to just the retention/exposure accessors above (attribute()/option()/matchingPaths()), and explicitly scoped tag-based field selection as "entirely its own design" for whatever extension installs it — this issue "declares no opinion on what any installed extension is for."
• The one concrete consumer that needed tag-based field selection, zilla-plus's field disclosure feature (aklivity/zilla-plus#1146, Support asyncapi authorization in http kafka and sse kafka #1150, Provide feedback on AsyncAPI v3 security scheme at operation vs server #1152), built its own per-format matcher config (AvroModelExtFieldMatcherConfig/*SelectConfig/*SelectType and the json/protobuf equivalents) directly inside the commercial ext modules, rather than depending on a shared oss primitive.
• The root-level (schema-root) attribute() accessor has no consumer anywhere in zilla or zilla-plus today.

Retitled and re-scoped to reflect the part that actually shipped, is actually consumed, and stays business-agnostic per this repo's own rule against building shared-component surface for a hypothetical future consumer. Fixed by #2368.

[jfallows]

Pushed a first phase of this to claude/zilla-issue-2302-7wu9ht: codec-level annotation retention/exposure (§1–§3), covering all three formats.

Scope note: this issue bundles two very differently-sized pieces of work — (a) making the codecs stop discarding unrecognised field annotations and expose them in a format-native shape, and (b) the tagged:/named: matcher config + per-model evaluation (§4–§8). I confirmed via the codebase that (b) has no existing precedent to build from — no fields:/matcher config, no disclosure:/action: scaffolding, no Avro path-pointer walker equivalent to JsonSchema.retainedPaths() — it's greenfield design in its own right, and rushing it alongside (a) risked shipping a shallow implementation of the harder, more architecturally significant half. So this pass delivers (a) as a complete, tested, mergeable unit, and leaves (b) for focused follow-up work.

What's included (test-first, full unit test + checkstyle + JaCoCo pass on all three modules):

• common-avro: AvroField.attribute(String) reads an arbitrary key from the field's own JSON object (never from the field's type), returned as JsonValue.
• common-json: added JsonSchema.property(String) (there's no separate "field object" in JSON Schema, so this navigates to a property's compiled sub-schema) plus JsonSchema.attribute(String) on any schema node, reading only that schema's own declared keys — not inherited through $ref.
• common-protobuf: ProtobufField.option(String) exposes a field option's value as a new ProtobufConstant sealed type (Identifier / IntegerValue / FloatValue / TextValue / BooleanValue / MessageValue), walked directly from the ANTLR parse tree (not routed through ProtobufJson, per the issue's guidance). Options with no declaring .proto import are retained; message-valued options expose their structure rather than flattened text; string constants are unquoted.

Left for a follow-up PR: the tagged:/named: YAML schema, the matcher-matrix semantics (§6), path-pattern matching (§5), and the overlay-compatibility behavior (§7) across model-avro/model-json/model-protobuf.

---

Generated by Claude Code