Give every API field one name across JSON and CSV - #5274
Merged
Conversation
Each endpoint's JSON keys, CSV header, and CSV cells now come from a single ordered field list per model (models.api.ApiFields), so a field cannot be named one thing in one format and something else in the other. The two endpoints whose response is a single object (overallStats, aggregateStats) build their vertical CSV by flattening their own JSON, keying each row by its dotted path. Fixes found while making the formats agree: - outputCSV put no separator between the header and the first row, so every streaming header carried a trailing newline of its own; userStats used that same string with println and emitted a blank line after its header. - aggregateStatsByDay wrote its field order a fourth time, as an interpolated string in StatsApiController. - labels.count, labels.count_with_severity and total_validations serialized as floats, having been built with .toDouble. - The labelClusters CSV docs sample was missing the intersection_id column. Field names that changed, all on the v3 preview surface: - accessScore per-type columns take their JSON path: cluster_counts.CurbRamp, sub_scores.CurbRamp, severity_counts.CurbRamp.1, tag_adjustments.CurbRamp, avg_cluster_counts.CurbRamp. - userStats and /adminapi/labels/cvMetadata move off their Title-Case headers, and userStats keys stats_by_label_type by the canonical label type names from /v3/api/labelTypes rather than its own spellings. - overallStats: labels.count, labels.count_with_severity, labels.avg_timestamp_last_100_labels, severity_stddev, and ai_stats leaves reading ai_yes_maj_vote_concurs, since the key above already names whose majority vote it is. Values line up too: a missing value is an empty cell rather than NA, optional fields are written as explicit nulls instead of being dropped, and string arrays render as real JSON. GeoPackage still names these fields its own way; that is #5273. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYLhKjbUkFnnpiPrCdHvE9
misaugstad
commented
Sep 10, 2026
misaugstad
commented
Sep 10, 2026
misaugstad
commented
Sep 10, 2026
misaugstad
commented
Sep 10, 2026
misaugstad
commented
Sep 10, 2026
misaugstad
commented
Sep 10, 2026
misaugstad
commented
Sep 10, 2026
misaugstad
commented
Sep 10, 2026
misaugstad
commented
Sep 10, 2026
misaugstad
commented
Sep 10, 2026
misaugstad
commented
Sep 10, 2026
misaugstad
commented
Sep 10, 2026
misaugstad
commented
Sep 10, 2026
misaugstad
commented
Sep 10, 2026
… the field lists
Three frontend readers of renamed keys were missed, each failing quietly:
- The cities map popup read labels.label_count, so every city would have shown
0 labels ("|| 0" swallowing the undefined). It calls other deployments' APIs,
which upgrade on their own schedule, so it now accepts either name.
- The shared-label spotlight band read the same key, and drops a stat entirely
when its value is undefined.
- userStatsPreview translated label types through its own table before looking
up colors, so every chart series would have fallen back to neutral gray with
a raw key for a legend. The table is gone; it keys off the canonical names.
aggregateStats no longer carries a hardcoded "status": "OK". It sat inside the
stats object rather than wrapping a data key, so flattening put a protocol field
at the top of a metric,value table. Its one reader checked it immediately before
validating six named fields are numbers, which is the stronger check.
ApiFields now resolves its field names into a shape once rather than per record,
and rejects a list it cannot serialize faithfully: a duplicate name, or a name
used as both a value and an object, either of which would drop a field from the
JSON while the CSV kept its column. A new spec covers both, and checks that the
three types whose JSON was macro-derived still have a field per constructor
parameter.
Also: rawLabels CSV time_created is an ISO string, matching the JSON (the docs
claimed epoch millis for both, which was already wrong); the dead perTypeJson
and perTypeBucketJson go with the code they served; RawLabelInClusterDataForApi
no longer exposes a csvHeader one column short of the file it writes; and
"metric,value" is named once, beside the helper that produces those rows.
docs/architecture.md and .claude/rules/api.md describe the field list, and note
GeoPackage as the format still on its own names (#5273).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LYLhKjbUkFnnpiPrCdHvE9
Prod deployments all go out together, so the popup has no window in which a city would still be answering with the old key. It reads labels.count. Dropped comments that argue for a naming choice or restate the line under them, and cut the ScalaDoc on the field-list infrastructure to its contract. The geometry note lives once on csvOnlyFields rather than at each of its four uses. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYLhKjbUkFnnpiPrCdHvE9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
resolves #4320
#4320 was that
overallStatsderived its CSV keys from Title-Case display labels while its JSONused explicit snake_case, so the two drifted. Fixing only those three field names would have left
the mechanism that produced the drift in place, and the same mechanism had already produced drift
on nearly every other endpoint. So this replaces the mechanism.
What changed
Each model now declares one ordered field list (
app/models/api/ApiFields.scala), and its JSONkeys, CSV header, and CSV cells are all derived from it. A field cannot be named one thing in one
format and something else in the other, because there is only one place to name it.
The two endpoints whose response is a single object rather than a list of records (
overallStats,aggregateStats) build their vertical CSV by flattening their own JSON, keying each row by itsdotted path —
labels.CurbRamp.countin the CSV is exactly what you would read aslabels.CurbRamp.countfrom the JSON.Net 830 insertions, 1124 deletions: each field used to be written three times (JSON key, header
entry, row cell), and in one case four.
Bugs this surfaced
outputCSVnever separated the header from the first row. Every streaming header string hadto smuggle in a trailing
\nto compensate.userStatsused that same string withprintln,so its CSV has been emitting a blank line after the header.
aggregateStatsByDaywrote its field order a fourth time, as an interpolated string inStatsApiController.labels.count,labels.count_with_severityandtotal_validationsserialized as floats(
183427.0) because they were built with.toDouble; the docs promised integers.labelClustersCSV docs sample was missing theintersection_idcolumn.Breaking changes (v3 preview surface, plus /adminapi cvMetadata)
Field names:
accessScore*n_curb_ramp,score_curb_ramp,n_curb_ramp_sev1,tag_adj_curb_rampcluster_counts.CurbRamp,sub_scores.CurbRamp,severity_counts.CurbRamp.1,tag_adjustments.CurbRampaccessScoreRegionsavg_n_curb_rampavg_cluster_counts.CurbRampuserStatsstats_by_label_type.CurbRamp.labelsuserStatsJSONcurb_ramp,marked_crosswalk,cant_see_sidewalk,pedestrian_signalCurbRamp,Crosswalk,Occlusion,Signal/adminapi/labels/cvMetadataLabel ID,Panorama ID,…label_id,pano_id,…overallStatsaverage_label_timestamp,km_explored_without_overlap,total_label_count,curb_ramp_severity_sdlabels.avg_label_timestamp,km_explored_no_overlap,labels.count,labels.CurbRamp.severity_stddevoverallStatsai_statsai_yes_human_concursai_yes_maj_vote_concursaggregateStatsnumber_of_cities,number_of_countries,number_of_languagesnum_cities,num_countries,num_languagesaggregateStatscurb_ramp_labels,no_sidewalk_labels_validated_disagreeby_label_type.CurbRamp.labels,by_label_type.NoSidewalk.labels_validated_disagreeuserStatskeysstats_by_label_typeby the canonical names from/v3/api/labelTypes— it was theone endpoint whose label types couldn't be joined against the others. The
ai_statsleaves saymaj_votebecause the key one level up already names whose majority vote it is, andhumanthereis a specific sibling (every non-AI validator), so
admin_majority_vote.ai_yes_human_concursreadas a contradiction.
Values:
NA(this matched most v3 models already).rawLabelsCSVtime_createdbecomes an ISO-8601 string. Its JSON already used one; only the CSV cell was epochmillis, so unifying the two picks the JSON's format. Anyone parsing that column as a number will need to change.
(The docs claimed epoch millis for both formats, which was already wrong; fixed here.)
aggregateStatsno longer carries"status": "OK". It sat inside the stats object rather than wrapping a nameddata key, so flattening put a protocol field at the top of a
metric,valuetable. Its one reader checked itimmediately before validating that six named fields are numbers, which is the stronger check.
vary with the data — which is also what keeps it aligned with the CSV's fixed columns.
["narrow","steep"], not[narrow,steep], which no parsercould read).
overallStatsCSV gains themetric,valueheader row it was missing;aggregateStatshad one.sidewalk-panorama-toolsreads/adminapi/labels/cvMetadata, so it has a heads-up atProjectSidewalk/sidewalk-panorama-tools#123. Its CSV reader already expects snake_case, so this
moves toward what it wants; its JSON path is unaffected.
Not included
GeoPackage still builds these column names its own way (
n_curb_ramp, andstreet_idwhere JSONand CSV say
street_edge_id). That needs its own change — dotted names may not be usable as SQLitecolumn names, and its attribute writes are positionally coupled to the type spec. Filed as #5273.
Testing
Full backend suite (1424), JS suite (1789), HTMLHint over all api-docs views, locale and asset-path
checks, scalafmt, and a warning-clean compile under
-Xfatal-warnings.One unrelated failure on this branch and on the base:
RouteBuilderControllerSpec's soft-deletedroute case, which is the known dev-DB route-accumulation flake (790 routes accumulated locally).
This branch touches no route-builder code.
The api-docs CSV samples were regenerated from the real
csvHeadervalues rather than written byhand.
🤖 Generated with Claude Code
https://claude.ai/code/session_01LYLhKjbUkFnnpiPrCdHvE9