Skip to content

Give every API field one name across JSON and CSV - #5274

Merged
misaugstad merged 3 commits into
developfrom
4320-canonical-csv-keys
Sep 10, 2026
Merged

Give every API field one name across JSON and CSV#5274
misaugstad merged 3 commits into
developfrom
4320-canonical-csv-keys

Conversation

@misaugstad

@misaugstad misaugstad commented Sep 10, 2026

Copy link
Copy Markdown
Member

resolves #4320

#4320 was that overallStats derived its CSV keys from Title-Case display labels while its JSON
used explicit snake_case, so the two drifted. Fixing only those three field names would have left
the mechanism that produced the drift in place, and the same mechanism had already produced drift
on nearly every other endpoint. So this replaces the mechanism.

What changed

Each model now declares one ordered field list (app/models/api/ApiFields.scala), and its JSON
keys, CSV header, and CSV cells are all derived from it. A field cannot be named one thing in one
format and something else in the other, because there is only one place to name it.

The two endpoints whose response is a single object rather than a list of records (overallStats,
aggregateStats) build their vertical CSV by flattening their own JSON, keying each row by its
dotted path — labels.CurbRamp.count in the CSV is exactly what you would read as
labels.CurbRamp.count from the JSON.

Net 830 insertions, 1124 deletions: each field used to be written three times (JSON key, header
entry, row cell), and in one case four.

Bugs this surfaced

  • outputCSV never separated the header from the first row. Every streaming header string had
    to smuggle in a trailing \n to compensate. userStats used that same string with println,
    so its CSV has been emitting a blank line after the header.
  • aggregateStatsByDay wrote its field order a fourth time, as an interpolated string in
    StatsApiController.
  • labels.count, labels.count_with_severity and total_validations serialized as floats
    (183427.0) because they were built with .toDouble; the docs promised integers.
  • The labelClusters CSV docs sample was missing the intersection_id column.

Breaking changes (v3 preview surface, plus /adminapi cvMetadata)

Field names:

Endpoint Was Now
accessScore* n_curb_ramp, score_curb_ramp, n_curb_ramp_sev1, tag_adj_curb_ramp cluster_counts.CurbRamp, sub_scores.CurbRamp, severity_counts.CurbRamp.1, tag_adjustments.CurbRamp
accessScoreRegions avg_n_curb_ramp avg_cluster_counts.CurbRamp
userStats 53 Title-Case columns snake_case, with stats_by_label_type.CurbRamp.labels
userStats JSON curb_ramp, marked_crosswalk, cant_see_sidewalk, pedestrian_signal CurbRamp, Crosswalk, Occlusion, Signal
/adminapi/labels/cvMetadata Label ID,Panorama ID,… label_id,pano_id,…
overallStats average_label_timestamp, km_explored_without_overlap, total_label_count, curb_ramp_severity_sd labels.avg_label_timestamp, km_explored_no_overlap, labels.count, labels.CurbRamp.severity_stddev
overallStats ai_stats ai_yes_human_concurs ai_yes_maj_vote_concurs
aggregateStats number_of_cities, number_of_countries, number_of_languages num_cities, num_countries, num_languages
aggregateStats curb_ramp_labels, no_sidewalk_labels_validated_disagree by_label_type.CurbRamp.labels, by_label_type.NoSidewalk.labels_validated_disagree

userStats keys stats_by_label_type by the canonical names from /v3/api/labelTypes — it was the
one endpoint whose label types couldn't be joined against the others. The ai_stats leaves say
maj_vote because the key one level up already names whose majority vote it is, and human there
is a specific sibling (every non-AI validator), so admin_majority_vote.ai_yes_human_concurs read
as a contradiction.

Values:

  • A missing value is an empty cell, not NA (this matched most v3 models already).
  • rawLabels CSV time_created becomes an ISO-8601 string. Its JSON already used one; only the CSV cell was epoch
    millis, so unifying the two picks the JSON's format. Anyone parsing that column as a number will need to change.
    (The docs claimed epoch millis for both formats, which was already wrong; fixed here.)
  • aggregateStats no longer carries "status": "OK". It sat inside the stats object rather than wrapping a named
    data key, so flattening put a protocol field at the top of a metric,value table. Its one reader checked it
    immediately before validating that six named fields are numbers, which is the stronger check.
  • Optional fields are written as explicit nulls rather than dropped, so a response's key set doesn't
    vary with the data — which is also what keeps it aligned with the CSV's fixed columns.
  • String arrays render as real JSON (["narrow","steep"], not [narrow,steep], which no parser
    could read).
  • overallStats CSV gains the metric,value header row it was missing; aggregateStats had one.

sidewalk-panorama-tools reads /adminapi/labels/cvMetadata, so it has a heads-up at
ProjectSidewalk/sidewalk-panorama-tools#123. Its CSV reader already expects snake_case, so this
moves toward what it wants; its JSON path is unaffected.

Not included

GeoPackage still builds these column names its own way (n_curb_ramp, and street_id where JSON
and CSV say street_edge_id). That needs its own change — dotted names may not be usable as SQLite
column names, and its attribute writes are positionally coupled to the type spec. Filed as #5273.

Testing

Full backend suite (1424), JS suite (1789), HTMLHint over all api-docs views, locale and asset-path
checks, scalafmt, and a warning-clean compile under -Xfatal-warnings.

One unrelated failure on this branch and on the base: RouteBuilderControllerSpec's soft-deleted
route case, which is the known dev-DB route-accumulation flake (790 routes accumulated locally).
This branch touches no route-builder code.

The api-docs CSV samples were regenerated from the real csvHeader values rather than written by
hand.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LYLhKjbUkFnnpiPrCdHvE9

Each endpoint's JSON keys, CSV header, and CSV cells now come from a single
ordered field list per model (models.api.ApiFields), so a field cannot be named
one thing in one format and something else in the other. The two endpoints whose
response is a single object (overallStats, aggregateStats) build their vertical
CSV by flattening their own JSON, keying each row by its dotted path.

Fixes found while making the formats agree:

- outputCSV put no separator between the header and the first row, so every
  streaming header carried a trailing newline of its own; userStats used that
  same string with println and emitted a blank line after its header.
- aggregateStatsByDay wrote its field order a fourth time, as an interpolated
  string in StatsApiController.
- labels.count, labels.count_with_severity and total_validations serialized as
  floats, having been built with .toDouble.
- The labelClusters CSV docs sample was missing the intersection_id column.

Field names that changed, all on the v3 preview surface:

- accessScore per-type columns take their JSON path: cluster_counts.CurbRamp,
  sub_scores.CurbRamp, severity_counts.CurbRamp.1, tag_adjustments.CurbRamp,
  avg_cluster_counts.CurbRamp.
- userStats and /adminapi/labels/cvMetadata move off their Title-Case headers,
  and userStats keys stats_by_label_type by the canonical label type names from
  /v3/api/labelTypes rather than its own spellings.
- overallStats: labels.count, labels.count_with_severity,
  labels.avg_timestamp_last_100_labels, severity_stddev, and ai_stats leaves
  reading ai_yes_maj_vote_concurs, since the key above already names whose
  majority vote it is.

Values line up too: a missing value is an empty cell rather than NA, optional
fields are written as explicit nulls instead of being dropped, and string arrays
render as real JSON.

GeoPackage still names these fields its own way; that is #5273.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LYLhKjbUkFnnpiPrCdHvE9
Comment thread app/models/api/OverallStatsApiModels.scala
Comment thread app/models/api/OverallStatsApiModels.scala
Comment thread app/models/api/UserStatsApiModels.scala
Comment thread app/models/api/AggregateStatsApiModels.scala
Comment thread app/models/api/AggregateStatsApiModels.scala
Comment thread app/models/api/LabelApiModels.scala
Comment thread app/models/api/ApiModelUtils.scala Outdated
Comment thread app/models/api/LabelClustersApiModels.scala
Comment thread .claude/rules/api.md
Comment thread .claude/rules/api.md
Comment thread app/models/api/AccessScoreApiModels.scala
Comment thread app/models/api/ApiFields.scala Outdated
Comment thread app/models/api/DailyStatsApiModels.scala
Comment thread app/models/api/OverallStatsApiModels.scala Outdated
misaugstad and others added 2 commits September 9, 2026 18:39
… the field lists

Three frontend readers of renamed keys were missed, each failing quietly:

- The cities map popup read labels.label_count, so every city would have shown
  0 labels ("|| 0" swallowing the undefined). It calls other deployments' APIs,
  which upgrade on their own schedule, so it now accepts either name.
- The shared-label spotlight band read the same key, and drops a stat entirely
  when its value is undefined.
- userStatsPreview translated label types through its own table before looking
  up colors, so every chart series would have fallen back to neutral gray with
  a raw key for a legend. The table is gone; it keys off the canonical names.

aggregateStats no longer carries a hardcoded "status": "OK". It sat inside the
stats object rather than wrapping a data key, so flattening put a protocol field
at the top of a metric,value table. Its one reader checked it immediately before
validating six named fields are numbers, which is the stronger check.

ApiFields now resolves its field names into a shape once rather than per record,
and rejects a list it cannot serialize faithfully: a duplicate name, or a name
used as both a value and an object, either of which would drop a field from the
JSON while the CSV kept its column. A new spec covers both, and checks that the
three types whose JSON was macro-derived still have a field per constructor
parameter.

Also: rawLabels CSV time_created is an ISO string, matching the JSON (the docs
claimed epoch millis for both, which was already wrong); the dead perTypeJson
and perTypeBucketJson go with the code they served; RawLabelInClusterDataForApi
no longer exposes a csvHeader one column short of the file it writes; and
"metric,value" is named once, beside the helper that produces those rows.

docs/architecture.md and .claude/rules/api.md describe the field list, and note
GeoPackage as the format still on its own names (#5273).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LYLhKjbUkFnnpiPrCdHvE9
Prod deployments all go out together, so the popup has no window in which a
city would still be answering with the old key. It reads labels.count.

Dropped comments that argue for a naming choice or restate the line under them,
and cut the ScalaDoc on the field-list infrastructure to its contract. The
geometry note lives once on csvOnlyFields rather than at each of its four uses.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LYLhKjbUkFnnpiPrCdHvE9
@misaugstad
misaugstad merged commit 45fa638 into develop Sep 10, 2026
10 checks passed
@misaugstad
misaugstad deleted the 4320-canonical-csv-keys branch September 10, 2026 02:22
This was referenced Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

overallStats CSV uses average_* / Title-derived keys that don't match the JSON snake_case keys (#3871)

1 participant