Skip to content

fix: Embed JSON-aliased VARCHAR columns as nested JSON (#38) - #39

Merged
jrosskopf merged 2 commits into
mainfrom
claude/fix-flapi-issue-38-8gA8c
May 20, 2026
Merged

jrosskopf merged 2 commits into
mainfrom
claude/fix-flapi-issue-38-8gA8c

Conversation

@jrosskopf

Copy link
Copy Markdown
Contributor

Summary

Closes #38.

When a query returns a column whose DuckDB logical type is JSON, flAPI emitted the raw text as a JSON-escaped string in the REST response, forcing consumers to JSON.parse the value a second time. The reporter (BigQuery tree with recursive children arrays typed as JSON) correctly identified the dispatch site: QueryResult::convertVectorEntryToJson switches on duckdb_get_type_id(type), which returns the physical type id. DuckDB's JSON is a logical-type alias over VARCHAR, so the cell fell into the DUCKDB_TYPE_VARCHAR branch and got wrapped as a crow::json::wvalue(str).

Fix

  • Inspect the logical type's alias via duckdb_logical_type_get_alias before dispatching VARCHAR. When it equals the literal "JSON" that DuckDB itself sets on LogicalType::JSON(), route to a new convertVectorJsonToJson helper that parses the bytes with crow::json::load and embeds the parsed value via the wvalue(const rvalue&) constructor. Nested objects and arrays now travel through the response unchanged. Malformed JSON degrades to the raw string rather than nulling the row.
  • Destroy the logical type allocated at the top of convertVectorEntryToJson on every exit path. The dead duckdb_destroy_logical_type below the switch was unreachable, leaking one logical-type allocation per emitted cell (~24 bytes each, visible under ASan: removed 13/15 leak allocations the suite reports).

Both changes are scoped to src/query_executor.cpp and src/include/query_executor.hpp. The Arrow output path has the same physical-id dispatch in arrow_serializer.hpp but Arrow conventionally represents JSON via metadata/extension types, so it is intentionally out of scope for this PR.

Tests

Built TDD-style — failing tests landed first, then the fix.

Unit tests (test/cpp/query_executor_test.cpp, new [json] tag):

  • Nested JSON object is embedded as Object (reproduces the issue exactly).
  • JSON array column is embedded as List.
  • CAST(NULL AS JSON) stays Null.
  • Plain VARCHAR continues to render as a JSON string (regression guard).

Integration test (test/integration/test_json_column.tavern.yaml + supporting sqls/json_demo.{yaml,sql}):

End-to-end verification with a debug-instrumented server confirmed the alias path fires (type_id=17 alias=JSON is_json=1) and the response body now matches the issue's expected output:

{"data":[{"id":1,"payload":{"a":1,"b":[10,20],"c":{"nested":true}}}],"total_count":1,"next":""}

Test plan

  • New unit cases pass (flapi_tests "QueryExecutor JSON column" — 23/23 assertions).
  • Full [query_executor] suite green (1092/1092 assertions, 7 cases).
  • Full ctest suite — 586/587 pass; the lone failure (CachingFileProvider thread safety_test) is pre-existing and reproduces on stash-popped pre-change code.
  • End-to-end response of GET /json-demo/ matches the issue's expected JSON.
  • CI matrix (Linux x86/ARM64, macOS ARM64, Windows x64, flapii) — will be exercised by this PR.

Generated by Claude Code

claude added 2 commits May 20, 2026 17:27
- Unit test in query_executor_test.cpp exercises a column whose DuckDB
  type is the JSON logical alias over VARCHAR. Asserts the response is a
  nested Object (not a JSON-escaped String), covers nested struct/array
  values, JSON arrays, NULL, and a plain-VARCHAR regression guard.
- Integration test boots the issue's reproduction endpoint and verifies
  the same shape end-to-end through Crow's response serialiser.

These tests are expected to fail before the production change lands —
the existing convertVectorVarcharToJson wraps the raw text as a string
and Crow escapes it on output.

Closes #38 (test side)
DuckDB's JSON type is a logical-type alias over VARCHAR, so
duckdb_get_type_id() returns DUCKDB_TYPE_VARCHAR for it. The previous
switch in QueryResult::convertVectorEntryToJson therefore dispatched
JSON cells to convertVectorVarcharToJson, which wraps the raw text as
a crow::json::wvalue(str). Crow then escaped the string at
serialisation time, forcing API consumers to JSON.parse the value a
second time.

Inspect the logical type's alias before dispatching. When it equals
the literal "JSON" that DuckDB itself sets on LogicalType::JSON(),
route to a new convertVectorJsonToJson helper that parses the bytes
with crow::json::load and embeds the parsed value via the
wvalue(const rvalue&) constructor, so nested objects and arrays
travel through the response unchanged. Malformed JSON in a source
cell degrades to the raw string rather than nulling the row.

Also destroy the logical type allocated at the top of
convertVectorEntryToJson on every exit path. The dead
duckdb_destroy_logical_type below the switch was unreachable, leaking
one logical-type allocation per emitted cell (~24 bytes each, visible
under ASAN). Eliminating that leak unblocks future leak-clean runs of
the suite.

Closes #38
@jrosskopf
jrosskopf merged commit 96806ac into main May 20, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JSON-aliased VARCHAR columns are returned as escaped strings instead of nested JSON in API responses

2 participants