rfc: add RFC 0051, JSON output via Arrow

[azimafroozeh]

What & why

TL;DR: To turn Arrow batches into JSON we kept our own conversion layer: two hand-written encoders that copy every value into a serde_json::Value before serializing it. Arrow already ships that conversion as arrow-json, typed at each cell's own width. Using it deletes both encoders and the copy: less code, faster output, and the narrow-float bug class cannot come back. What OmniGraph still owns is small: the base64: blob substitution, a date-range check, and the export line envelope. Exact types come from Arrow IPC on request. Three implementation PRs follow this one.

This PR adds RFC 0051, JSON output via Arrow: every JSON body OmniGraph writes for graph data becomes the arrow-json rendering of the result batches with the crate's default options, replacing the two hand-written encoders; a client that needs exact types requests Arrow IPC instead.

• Motivation: the query-result encoder and the entity-fetch/export encoder disagree on the spelling of several types, and both widen every cell through serde_json::Value; bug: F32 values come back with widened digits #618 is that shape made visible, since the F32 fix has to land in both encoders.
• The RFC fixes the contract as a per-type spelling table plus a complete per-phase list of the spelling changes a JSON consumer can observe; arrow-json is pinned by the lock file and an upgrade that changes a spelling is treated as a wire change.
• OmniGraph keeps three pieces of its own: the base64: blob substitution on export and entity fetch, a date-range check before the writer, and the export line envelope.
• Three phases, one PR each, independently revertible: query results; entity fetch and export; Arrow IPC selected by Accept on the query routes.
• Registry: one row added; the next available number moves from 0050 to 0052 (0050 is held by the rfc/0050-engine-crate-topology branch, 0047 and 0048 by rfc: search contracts and retrieval algebra #606).

Backing issue / RFC

• Is an RFC PR: docs/rfcs/0051-json-output-via-arrow.md (no issue; maintainer-track RFC)

Checklist

• Change is focused (one RFC file plus its registry row and next-number bump)
• Tests added/updated for behavior changes (N/A, docs-only; the RFC names the gates each implementation phase carries: a differential test over every Arrow type, the RFC 0045 corpus, the export round trip)
• Public docs updated if user-facing surface changed (N/A; nothing user-facing changes in this PR, the RFC's Rollout names the OpenAPI and user-doc updates for phase 3)
• Reviewed against docs/dev/invariants.md — no Hard Invariant weakened, no deny-list item hit (the RFC's Invariants section covers 8, 9, 10, 12, 13; the "swallowed errors" deny-list entry is the shape the RFC removes)

Local verification

• python3 scripts/check-docs.py: OK, 125 Markdown files checked (the CI "Verify documentation structure and RFC registry" step)
• git diff --check upstream/main: clean
• cargo check --workspace: not run, docs-only change, no Rust file touched

Notes for reviewers

• Writer defaults everywhere: no date format strings, null keys omitted from row objects, non-finite floats written as null. Each default is a spelling change listed in the phase table; the alternative of per-type overrides was rejected because every override is OmniGraph deciding a spelling, which is the thing being removed.
• Two inputs that succeed silently today become typed errors: a query whose return names collide (the old encoder collapsed them to the last value; the writer would emit both keys) and a Date or DateTime count outside the formatter's range (arrow-json writes an ERROR: string with Ok for those, so OmniGraph checks the range itself).
• ReadOutput.rows and LegacyReadOutput.rows change type to RawValue in phase 1: the OpenAPI schema is unchanged, Rust consumers of omnigraph-api-types see the type change.
• The spelling table and the phase rows describe arrow-json 58.3.0's actual output, not a reading of its source; the differential test in Evidence pins them.
• Phase 1 assumes the bug: F32 values come back with widened digits #618 fix has landed; if it has not, phase 1 carries the engine-encoder half of that fix.
• Number: 0051 assumes 0050 stays reserved by the rfc/0050-engine-crate-topology branch; if that branch opens no PR, this RFC takes 0050 and the registry line moves to 0051.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.