From f8b784366889651ade4fa88cfc9de3c0a0854449 Mon Sep 17 00:00:00 2001 From: tim-eternos Date: Wed, 29 Jul 2026 23:04:38 -0700 Subject: [PATCH 01/12] docs: add implementation plans for place-type fixes and Places API (New) migration Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01DjdGnZM6cUBeg6jWwoQfU3 --- .../2026-07-29-eatery-place-type-fixes.md | 1022 ++++++++++ .../2026-07-29-places-api-new-migration.md | 1796 +++++++++++++++++ 2 files changed, 2818 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-29-eatery-place-type-fixes.md create mode 100644 docs/superpowers/plans/2026-07-29-places-api-new-migration.md diff --git a/docs/superpowers/plans/2026-07-29-eatery-place-type-fixes.md b/docs/superpowers/plans/2026-07-29-eatery-place-type-fixes.md new file mode 100644 index 00000000..d8d186aa --- /dev/null +++ b/docs/superpowers/plans/2026-07-29-eatery-place-type-fixes.md @@ -0,0 +1,1022 @@ +# Eatery Place-Type Misclassification Fixes Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Stop the legacy Places API from writing mislabeled places into the category geo buckets, make the existing round-trip guard actually capable of failing, fix result truncation so it drops the farthest places rather than the last-searched place types, and purge the records already written to prod. + +**Architecture:** Four independent forward fixes on `origin/master`. The root cause is that `POI.GetPlaceCategory` has a `default:` branch returning `PlaceCategoryEatery`, which silently absorbs any place type the legacy Nearby Search does not understand. We convert that function to the `(value, ok)` shape already used by `ParsePlaceCategory`, so the compiler forces all three call sites to decide what an unknown type means; then we remove the two New-API-only types, reject unknown types at request-build time, sort by distance before truncating, and ship an admin migration to clean the buckets. + +**Tech Stack:** Go 1.24, Gin, go-redis v9, `googlemaps.github.io/maps` v1.7.0 (legacy Places API), testify. + +## Global Constraints + +- Go version: `1.24.0` (from `go.mod`) — do not raise it. +- Add no new module dependencies in this PR. +- CI gates are exactly `go build -v .` then `go test -v ./...` (`.github/workflows/go.yml`). Both must pass. +- Target branch is `origin/master` (the repo default; `origin/main` is stale at `1a1417b` and is not the deploy path). Commit `8644199` is already on `master` and live on Heroku app `best-vacation-planner`. +- Do not migrate to Places API (New) in this PR. That is the follow-up plan, `2026-07-29-places-api-new-migration.md`. +- Never run destructive Redis commands by hand against prod. Cleanup ships as a dry-run-by-default admin endpoint (Task 4). + +--- + +## Background: what is actually broken + +Verified against the code at `8644199`: + +1. `POI/categories.go:39-40` added `LocationTypeFastFood = "fast_food_restaurant"` and `LocationTypeFoodCourt = "food_court"`. Neither string exists anywhere in `googlemaps.github.io/maps@v1.7.0` — they are Places API (New) Table A types. +2. `iowrappers/nearby_search.go:87` builds the request with a direct cast, `Type: maps.PlaceType(placeType)`, bypassing the SDK's own `maps.ParsePlaceType` validator (`types.go:301`). The SDK then does `q.Set("type", string(r.Type))` (`places.go:125`) unchecked, so the unknown string reaches Google verbatim. +3. Legacy Nearby Search accepts the parameter, ignores the filter, and returns prominence-ranked establishments. `parsePlacesSearchResponse(searchResp, placeType, ...)` (`nearby_search.go:224`) then stamps the **queried** type onto every result via `POI.CreatePlace(..., locationType, ...)` (`:419`), so hotels get `LocationType: fast_food_restaurant`. `:421` (`place.Types = res.Types`) keeps Google's truthful types, which is why the damage is visible. +4. `iowrappers/redis_client.go:222` writes each place under `EncodeNearbySearchRedisKey(GetPlaceCategory(place.LocationType), place.PriceLevel)`. Neither new type appears in `GetPlaceCategory`'s Eatery case, so they reach Eatery through `default:` (`categories.go:77-78`) — meaning the invariant documented at `categories.go:61-64` was broken by the commit and the `default` hid it. +5. `TestPlaceCategoryRoundTrip` (`test/place_category_test.go`) passed anyway, because that `default` makes the test un-failable for *any* unknown type. The guard provides no protection today. +6. `POI.ReclassifyForCategory` (`categories.go:154-166`) drops these from API responses (primary type `lodging` is not in `GetPlaceTypes(Eatery)`), which is why the endpoint looks clean while the cache is dirty. + +Confirmed non-issues, so nobody wastes time on them: + +- These records do **not** consume result slots in the response. `planner/planner.go` reclassifies at `:1406` *before* truncating at `:1414`, and the Redis read is unbounded (`GeoRadius` with no `Count`, `redis_client.go:484-489`). +- They did **not** waste Place Details spend. `detailsBudget` (`nearby_search.go:150`) is shared and consumed in `placeTypes` order, so `cafe`/`restaurant` exhaust it before the junk types are processed. +- Zero `food_court`-tagged places in prod is expected, not a contradiction. `placeMap` dedups by place ID across all types in one search, and `fast_food_restaurant` is processed first, so an identical ignored-filter response for `food_court` is entirely deduped away. + +The real cache-side cost is that these records inflate `len(cachedQualifiedPlaces)`, which is the radius-doubling break condition at `redis_client.go:520` — junk can satisfy `MinNumResults` and stop the radius from growing, so sparse areas return fewer genuine eateries. + +--- + +## File Structure + +| File | Responsibility in this PR | +| --- | --- | +| `POI/categories.go` | Remove the two New-API-only types from `GetPlaceTypes(Eatery)`; change `GetPlaceCategory` to `(PlaceCategory, bool)`; delete the two unused constants. | +| `iowrappers/redis_client.go` | Skip + log places whose type has no category, instead of writing them to Eatery (`:182`, `:222`). | +| `planner/planner.go` | Handle the new `ok` return at `:810`; sort by distance before truncating at `:1414`. | +| `iowrappers/nearby_search.go` | Validate place types in `CreateMapSearchRequest`; fix the dead `maxRetries` cap. | +| `iowrappers/data_migrations.go` | Add `RemoveMisclassifiedPlacesFromCategoryBuckets` for prod cleanup. | +| `iowrappers/maps_client.go` | Extend the `SearchClient`/migration interface with the new cleanup method. | +| `test/place_category_test.go` | Update for the new signature; the round-trip guard becomes meaningful. | +| `iowrappers/nearby_search_validation_test.go` (new) | Unit tests for place-type validation. | +| `iowrappers/place_distance_sort_test.go` (new) | Unit tests for distance sorting. | + +--- + +### Task 1: Make unknown place types un-mappable, and remove the two New-API-only types + +This is the root-cause fix. It must land as one commit — changing `GetPlaceCategory`'s signature without removing the two types would leave the build red on the round-trip test, which is exactly the point of the guard. + +**Files:** +- Modify: `POI/categories.go:37-79` (constants, `GetPlaceCategory`, `GetPlaceTypes`) +- Modify: `iowrappers/redis_client.go:182`, `iowrappers/redis_client.go:222` +- Modify: `planner/planner.go:810` +- Test: `test/place_category_test.go` + +**Interfaces:** +- Consumes: nothing from earlier tasks. +- Produces: `POI.GetPlaceCategory(placeType LocationType) (PlaceCategory, bool)` — returns `("", false)` when the type maps to no category. Tasks 2 and 4 rely on this exact signature. + +- [ ] **Step 1: Write the failing test** + +Add to `test/place_category_test.go`: + +```go +// TestGetPlaceCategoryRejectsUnknownTypes pins the fix for the fast_food_restaurant +// incident: GetPlaceCategory must NOT silently absorb unmapped types into Eatery. +// A default-to-Eatery branch made TestPlaceCategoryRoundTrip un-failable, so two +// Places-API-(New)-only types were added to GetPlaceTypes(Eatery) and hotels were +// written into the eatery geo buckets. +func TestGetPlaceCategoryRejectsUnknownTypes(t *testing.T) { + unknown := []POI.LocationType{ + POI.LocationType("fast_food_restaurant"), + POI.LocationType("food_court"), + POI.LocationType("lodging_but_not_really"), + POI.LocationType(""), + } + for _, placeType := range unknown { + if got, ok := POI.GetPlaceCategory(placeType); ok { + t.Errorf("GetPlaceCategory(%q) = (%q, true), want ok=false", placeType, got) + } + } +} + +// TestGetPlaceCategoryKnownTypes pins that every mapped type still resolves. +func TestGetPlaceCategoryKnownTypes(t *testing.T) { + cases := map[POI.LocationType]POI.PlaceCategory{ + POI.LocationTypeCafe: POI.PlaceCategoryEatery, + POI.LocationTypeRestaurant: POI.PlaceCategoryEatery, + POI.LocationTypeBar: POI.PlaceCategoryEatery, + POI.LocationTypeBakery: POI.PlaceCategoryEatery, + POI.LocationTypeMealTakeaway: POI.PlaceCategoryEatery, + POI.LocationTypePark: POI.PlaceCategoryVisit, + POI.LocationTypeMuseum: POI.PlaceCategoryVisit, + POI.LocationTypeStore: POI.PlaceCategoryShopping, + POI.LocationTypeLodging: POI.PlaceCategoryLodging, + POI.LocationTypeGym: POI.PlaceCategoryWellness, + } + for placeType, want := range cases { + got, ok := POI.GetPlaceCategory(placeType) + if !ok { + t.Errorf("GetPlaceCategory(%q) returned ok=false, want %q", placeType, want) + continue + } + if got != want { + t.Errorf("GetPlaceCategory(%q) = %q, want %q", placeType, got, want) + } + } +} +``` + +Update the two existing tests in the same file for the new signature and the reverted type list: + +```go +// in TestGetPlaceTypesByCategory, the Eatery entry becomes: + POI.PlaceCategoryEatery: { + POI.LocationTypeCafe, POI.LocationTypeRestaurant, + POI.LocationTypeBar, POI.LocationTypeBakery, POI.LocationTypeMealTakeaway, + }, + +// in TestPlaceCategoryRoundTrip, the assertion becomes: + got, ok := POI.GetPlaceCategory(placeType) + if !ok { + t.Errorf("round-trip broken: GetPlaceCategory(%q) has no category, want %q", placeType, category) + continue + } + if got != category { + t.Errorf("round-trip broken: GetPlaceCategory(%q) = %q, want %q", placeType, got, category) + } +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `go test ./test/ -run 'TestGetPlaceCategory|TestPlaceCategoryRoundTrip|TestGetPlaceTypesByCategory' -v` + +Expected: FAIL to compile with `assignment mismatch: 2 variables but POI.GetPlaceCategory returns 1 value`. That compile failure is the expected red state. + +- [ ] **Step 3: Change `GetPlaceCategory` and remove the two types** + +In `POI/categories.go`, delete these two constant lines (they have no remaining caller): + +```go + LocationTypeFastFood = LocationType("fast_food_restaurant") + LocationTypeFoodCourt = LocationType("food_court") +``` + +Replace `GetPlaceCategory` (currently `categories.go:61-79`) with: + +```go +// GetPlaceCategory maps a Google Maps place type back to its category, reporting whether +// the type is mapped at all. It is the inverse of GetPlaceTypes and MUST stay consistent +// with it: the nearby-search cache writes each place under +// EncodeNearbySearchRedisKey(GetPlaceCategory(place.LocationType), ...), so a type that +// resolves to a different category than the one it was searched under would never cache-hit. +// +// It deliberately has NO default category. An earlier version defaulted to Eatery, which +// silently absorbed place types the legacy Nearby Search does not understand — two +// Places-API-(New)-only types ("fast_food_restaurant", "food_court") were added to +// GetPlaceTypes(Eatery), Google ignored the unenforceable filter, and prominence-ranked +// hotels were written into the eatery geo buckets. Returning ok=false forces every caller +// to decide what an unmapped type means, and makes TestPlaceCategoryRoundTrip able to fail. +func GetPlaceCategory(placeType LocationType) (PlaceCategory, bool) { + switch placeType { + case LocationTypePark, LocationTypeAmusementPark, LocationTypeGallery, LocationTypeMuseum: + return PlaceCategoryVisit, true + case LocationTypeCafe, LocationTypeRestaurant, LocationTypeBar, LocationTypeBakery, LocationTypeMealTakeaway: + return PlaceCategoryEatery, true + case LocationTypeShoppingMall, LocationTypeDepartmentStore, LocationTypeSupermarket, LocationTypeClothingStore, LocationTypeStore: + return PlaceCategoryShopping, true + case LocationTypeLodging: + return PlaceCategoryLodging, true + case LocationTypeGym, LocationTypeSpa, LocationTypePharmacy: + return PlaceCategoryWellness, true + default: + return PlaceCategory(""), false + } +} +``` + +Revert the Eatery line in `GetPlaceTypes` back to five types: + +```go + case PlaceCategoryEatery: + placeTypes = append(placeTypes, + []LocationType{LocationTypeCafe, LocationTypeRestaurant, LocationTypeBar, LocationTypeBakery, LocationTypeMealTakeaway}...) +``` + +- [ ] **Step 4: Update the three call sites** + +`iowrappers/redis_client.go:222` — inside the `SetPlacesAddGeoLocations` pipeline. This is the write path that caused the incident, so it must refuse to guess: + +```go + for _, place := range placeBatch { + placeCategory, ok := POI.GetPlaceCategory(place.LocationType) + if !ok { + // Refuse to guess a bucket. A place whose type maps to no category + // came from a search whose type filter Google did not enforce, so + // writing it would poison whichever bucket we picked. + Logger.Errorf("SetPlacesAddGeoLocations: place %s has unmapped location type %q, skipping geo bucket write", + place.ID, place.LocationType) + continue + } + geoLocation := &redis.GeoLocation{ + Name: place.ID, + Latitude: place.GetLocation().Latitude, + Longitude: place.GetLocation().Longitude, + } + + redisKey := POI.EncodeNearbySearchRedisKey(placeCategory, place.PriceLevel) + pipe.GeoAdd(c, redisKey, geoLocation) +``` + +`iowrappers/redis_client.go:182` — inside the deprecated `StorePlacesForLocation`. Same rule, minimal change: + +```go + for _, place := range places { + placeCategory, ok := POI.GetPlaceCategory(place.LocationType) + if !ok { + Logger.Errorf("StorePlacesForLocation: place %s has unmapped location type %q, skipping", + place.ID, place.LocationType) + continue + } + sortedSetKey := strings.Join([]string{geocodeInString, string(placeCategory)}, "_") +``` + +`planner/planner.go:810` — this reads *saved plan* records out of `place_details:place_ID:`, which can include older cached places with legacy or empty types (brand searches write `LocationType: ""`). Preserve today's observable response here rather than changing an unrelated endpoint's output: + +```go + // Saved plans can contain older cached records, including brand-search places + // written with an empty LocationType. Preserve the historical Eatery default for + // display only — the write path (redis_client.go) is where guessing is unsafe. + if placeCategory, ok := POI.GetPlaceCategory(place.LocationType); ok { + resp.PlaceCategories[i] = placeCategory + } else { + resp.PlaceCategories[i] = POI.PlaceCategoryEatery + } +``` + +- [ ] **Step 5: Run the full suite to verify it passes** + +Run: `go build -v . && go test ./... 2>&1 | tail -30` + +Expected: PASS. In particular `TestGetPlaceCategoryRejectsUnknownTypes`, `TestGetPlaceCategoryKnownTypes`, `TestPlaceCategoryRoundTrip` and `TestGetPlaceTypesByCategory` all pass, and no package fails to compile. + +- [ ] **Step 6: Commit** + +```bash +git add POI/categories.go iowrappers/redis_client.go planner/planner.go test/place_category_test.go +git commit -m "fix: stop mapping unknown place types to Eatery + +GetPlaceCategory had a default branch returning Eatery, which silently +absorbed any place type the legacy Nearby Search does not understand. That +made TestPlaceCategoryRoundTrip un-failable, so fast_food_restaurant and +food_court (Places API (New) Table A types, absent from the v1.7.0 SDK) +were added to GetPlaceTypes(Eatery). Google ignored the unenforceable type +filter and returned prominence-ranked establishments, which were stamped +with the queried type and written into placeIDs:eatery:level* as hotels. + +Return (PlaceCategory, bool) so the compiler forces every caller to handle +an unmapped type, refuse the geo-bucket write instead of guessing, and +remove the two types." +``` + +--- + +### Task 2: Reject unknown place types when building the Maps request + +Defense in depth: Task 1 stops bad data reaching Redis, this stops the useless API call being made at all, and makes the failure loud. + +**Files:** +- Modify: `iowrappers/nearby_search.go:76-100` (`CreateMapSearchRequest`), `:141` (`maxRetries`), `:170-205` (Phase A/B) +- Test: `iowrappers/nearby_search_validation_test.go` (create) + +**Interfaces:** +- Consumes: nothing from Task 1 (independent). +- Produces: `CreateMapSearchRequest(reqIn *PlaceSearchRequest, placeType POI.LocationType, token string) (maps.NearbySearchRequest, error)` — returns a non-nil error when `placeType` is neither `POI.LocationTypeAny` nor a type `maps.ParsePlaceType` accepts. + +- [ ] **Step 1: Write the failing test** + +Create `iowrappers/nearby_search_validation_test.go`: + +```go +package iowrappers + +import ( + "strings" + "testing" + + "github.com/weihesdlegend/Vacation-planner/POI" +) + +// TestCreateMapSearchRequestRejectsUnknownPlaceType guards the fast_food_restaurant +// incident at the request boundary. The SDK casts POI.LocationType straight to +// maps.PlaceType and forwards it as ?type=, so an unknown value silently disables +// the filter server-side instead of erroring. Validate before spending the call. +func TestCreateMapSearchRequestRejectsUnknownPlaceType(t *testing.T) { + req := &PlaceSearchRequest{ + Location: POI.Location{Latitude: 37.38006, Longitude: -122.11612}, + PlaceCat: POI.PlaceCategoryEatery, + Radius: 8000, + PriceLevel: POI.PriceLevelTwo, + } + for _, placeType := range []POI.LocationType{ + POI.LocationType("fast_food_restaurant"), + POI.LocationType("food_court"), + POI.LocationType("not_a_google_type"), + } { + if _, err := CreateMapSearchRequest(req, placeType, ""); err == nil { + t.Errorf("CreateMapSearchRequest(%q) returned nil error, want validation failure", placeType) + } else if !strings.Contains(err.Error(), string(placeType)) { + t.Errorf("CreateMapSearchRequest(%q) error %q should name the offending type", placeType, err) + } + } +} + +// TestCreateMapSearchRequestAcceptsKnownPlaceTypes pins that every type the +// categories actually search for still builds a request. +func TestCreateMapSearchRequestAcceptsKnownPlaceTypes(t *testing.T) { + req := &PlaceSearchRequest{ + Location: POI.Location{Latitude: 37.38006, Longitude: -122.11612}, + PlaceCat: POI.PlaceCategoryEatery, + Radius: 8000, + PriceLevel: POI.PriceLevelTwo, + } + categories := []POI.PlaceCategory{ + POI.PlaceCategoryVisit, POI.PlaceCategoryEatery, + POI.PlaceCategoryShopping, POI.PlaceCategoryLodging, POI.PlaceCategoryWellness, + } + for _, category := range categories { + for _, placeType := range POI.GetPlaceTypes(category) { + got, err := CreateMapSearchRequest(req, placeType, "") + if err != nil { + t.Errorf("CreateMapSearchRequest(%q) in category %q: unexpected error %v", placeType, category, err) + continue + } + if string(got.Type) != string(placeType) { + t.Errorf("CreateMapSearchRequest(%q) set Type=%q, want %q", placeType, got.Type, placeType) + } + } + } +} + +// TestCreateMapSearchRequestAcceptsAnyType pins that keyword (brand) searches, +// which intentionally leave the type unset, are not rejected. +func TestCreateMapSearchRequestAcceptsAnyType(t *testing.T) { + req := &PlaceSearchRequest{ + Location: POI.Location{Latitude: 37.38006, Longitude: -122.11612}, + PlaceCat: POI.PlaceCategoryEatery, + Radius: 8000, + Keyword: "Dunkin'", + } + got, err := CreateMapSearchRequest(req, POI.LocationTypeAny, "") + if err != nil { + t.Fatalf("CreateMapSearchRequest(LocationTypeAny) returned error %v, want nil", err) + } + if got.Type != "" { + t.Errorf("CreateMapSearchRequest(LocationTypeAny) set Type=%q, want empty", got.Type) + } + if got.Keyword != "Dunkin'" { + t.Errorf("CreateMapSearchRequest kept Keyword=%q, want %q", got.Keyword, "Dunkin'") + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `go test ./iowrappers/ -run TestCreateMapSearchRequest -v` + +Expected: FAIL to compile with `assignment mismatch: 2 variables but CreateMapSearchRequest returns 1 value`. + +- [ ] **Step 3: Add validation to `CreateMapSearchRequest`** + +Replace the function at `iowrappers/nearby_search.go:75-100`: + +```go +// CreateMapSearchRequest creates a NearbySearchRequest for maps NearbySearch, adjust key settings such as radius and price levels. +// It rejects any place type the legacy Places API does not define: POI.LocationType is cast +// straight to maps.PlaceType and forwarded as ?type=, and Google responds to an unknown value +// by IGNORING the filter and returning prominence-ranked establishments rather than erroring. +// Those results are then stamped with the queried type, so an unvalidated type silently +// poisons the cache. maps.ParsePlaceType is the SDK's own list of legal values. +func CreateMapSearchRequest(reqIn *PlaceSearchRequest, placeType POI.LocationType, token string) (maps.NearbySearchRequest, error) { + // LocationTypeAny is the keyword (brand) search case: the type is deliberately unset so + // Google matches the keyword across all place types. + if placeType != POI.LocationTypeAny { + if _, err := maps.ParsePlaceType(string(placeType)); err != nil { + return maps.NearbySearchRequest{}, fmt.Errorf( + "place type %q is not a legacy Places API type (Places API (New) types are not accepted by /maps/api/place/nearbysearch): %w", + placeType, err) + } + } + + // Adjust radius, minPrice and maxPrice settings in search request + var radius = reqIn.Radius + var exactPriceLevel maps.PriceLevel + if POI.PriceyEatery(reqIn.PlaceCat, reqIn.PriceLevel) { + // increase search radius + radius = min(reqIn.Radius*4, GoogleNearbySearchMaxRadiusInMeters) + // set price filter + exactPriceLevel = maps.PriceLevel(fmt.Sprint(reqIn.PriceLevel)) + } + + return maps.NearbySearchRequest{ + Type: maps.PlaceType(placeType), + Location: &maps.LatLng{ + Lat: reqIn.Location.Latitude, + Lng: reqIn.Location.Longitude, + }, + Keyword: reqIn.Keyword, + Radius: radius, + PageToken: token, + RankBy: maps.RankBy("prominence"), + MinPrice: exactPriceLevel, + MaxPrice: exactPriceLevel, + }, nil +} +``` + +- [ ] **Step 4: Handle the error in Phase A and repair the dead retry cap** + +In `extensiveNearbySearch`, `iowrappers/nearby_search.go:141`, the cap is computed while `reqTimes` is still 0, so `maxRetries` is always 0 and the `break outer` at `:205` is unreachable. Fix it to the intended "every place type failed this round" meaning: + +```go + var reqTimes uint = 0 // number of queries for each location type + var totalPlaceCount uint = 0 // number of results so far, keep this number low + // Bail out once every place type has failed once. This was previously computed as + // reqTimes * len(placeTypes) while reqTimes was still 0, making the cap 0 and the + // break below unreachable, so a fully failing search span every retry round. + maxRetries := uint(len(placeTypes)) +``` + +In the Phase A goroutine at `:170-187`, surface the validation error the same way a fetch error is surfaced: + +```go + go func(i int, placeType POI.LocationType, token string) { + defer wg.Done() + searchReq, reqErr := CreateMapSearchRequest(request, placeType, token) + if reqErr != nil { + fetched[i].err = reqErr + return + } + select { + case c.apiSemaphore <- struct{}{}: + defer func() { <-c.apiSemaphore }() + case <-ctx.Done(): + fetched[i].err = ctx.Err() + return + } + fetched[i].resp, fetched[i].err = c.GoogleMapsNearbySearchWrapper(ctx, searchReq) + }(i, placeType, nextPageTokenMap[placeType]) +``` + +Then at `:199-205`, change the equality check to `>=` so a repaired cap cannot be stepped over: + +```go + if fetched[i].err != nil { + Logger.Error(fmt.Errorf("places nearby search with Maps failed for place type %s with error: %w", + placeType, fetched[i].err)) + mapsFailuresCount++ + if mapsFailuresCount >= maxRetries { + break outer + } + // we should still retry for the next place type if the number of failures is below maxRetries + continue + } +``` + +- [ ] **Step 5: Run tests to verify they pass** + +Run: `go build -v . && go test ./iowrappers/ -run TestCreateMapSearchRequest -v && go test ./... 2>&1 | tail -20` + +Expected: PASS on all three new tests and no regressions. `iowrappers/nearby_search_test.go` and `test/redis_client_mocks/...` must still pass. + +- [ ] **Step 6: Commit** + +```bash +git add iowrappers/nearby_search.go iowrappers/nearby_search_validation_test.go +git commit -m "fix: reject non-legacy place types before calling Nearby Search + +POI.LocationType was cast straight to maps.PlaceType and forwarded as +?type=. Google answers an unknown type by ignoring the filter, not by +erroring, so the call silently returns prominence-ranked establishments +that then get stamped with the queried type. Validate against +maps.ParsePlaceType first and fail loudly. + +Also repair the retry cap: maxRetries was computed as +reqTimes * len(placeTypes) while reqTimes was 0, so it was always 0 and +the break was dead code." +``` + +--- + +### Task 3: Sort by distance before truncating category results + +Independent pre-existing bug, worth its own commit. `planner/planner.go:1413` claims "Redis results are sorted by distance ascending", which is only true on the cache path. On the fresh path, results are appended per place type in `GetPlaceTypes` order, each type's page in Google prominence order (`nearby_search.go:224`). So a cold search produces cafe×20, restaurant×20, bar×20, bakery×20, meal_takeaway×20 and `places[:40]` keeps roughly cafes and restaurants while dropping bar, bakery and meal_takeaway entirely — the exact types commit `e299558` was added to surface. + +**Files:** +- Create: `iowrappers/place_distance_sort.go` +- Modify: `planner/planner.go:1411-1417` +- Test: `iowrappers/place_distance_sort_test.go` + +**Interfaces:** +- Consumes: nothing from earlier tasks. +- Produces: `iowrappers.SortPlacesByDistance(places []POI.Place, lat, lng float64)` — sorts `places` in place, ascending by haversine distance from `(lat, lng)`, stable so equal distances keep their prior order. + +- [ ] **Step 1: Write the failing test** + +Create `iowrappers/place_distance_sort_test.go`: + +```go +package iowrappers + +import ( + "testing" + + "github.com/weihesdlegend/Vacation-planner/POI" +) + +func placeAt(id string, lat, lng float64) POI.Place { + var p POI.Place + p.SetID(id) + p.SetLocationCoordinates([2]float64{lat, lng}) + return p +} + +// TestSortPlacesByDistance pins that truncation keeps the NEAREST places. The fresh +// (Google) path returns results grouped by place type in prominence order, so slicing +// without sorting first drops whole place types and can rank a 3km result above a 250m one. +func TestSortPlacesByDistance(t *testing.T) { + // State Street Market, Los Altos + lat, lng := 37.38006, -122.11612 + + places := []POI.Place{ + placeAt("far-sunnyvale", 37.3688, -122.0363), // ~7km east + placeAt("mid-mountainview", 37.3861, -122.0839), // ~3km east + placeAt("near-state-st", 37.38025, -122.11655), // ~40m away + } + + SortPlacesByDistance(places, lat, lng) + + want := []string{"near-state-st", "mid-mountainview", "far-sunnyvale"} + for i, id := range want { + if places[i].GetID() != id { + t.Errorf("position %d = %q, want %q (full order: %v)", i, places[i].GetID(), id, placeIDs(places)) + } + } +} + +// TestSortPlacesByDistanceEmpty pins that the no-result case does not panic. +func TestSortPlacesByDistanceEmpty(t *testing.T) { + var places []POI.Place + SortPlacesByDistance(places, 37.38006, -122.11612) + if len(places) != 0 { + t.Errorf("got %d places, want 0", len(places)) + } +} + +func placeIDs(places []POI.Place) []string { + ids := make([]string, 0, len(places)) + for _, p := range places { + ids = append(ids, p.GetID()) + } + return ids +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `go test ./iowrappers/ -run TestSortPlacesByDistance -v` + +Expected: FAIL with `undefined: SortPlacesByDistance`. + +- [ ] **Step 3: Write the implementation** + +Create `iowrappers/place_distance_sort.go`: + +```go +package iowrappers + +import ( + "sort" + + "github.com/weihesdlegend/Vacation-planner/POI" + "github.com/weihesdlegend/Vacation-planner/utils" +) + +// SortPlacesByDistance orders places ascending by distance from (lat, lng). +// +// Callers that truncate a candidate list to a limit MUST sort first. Only the Redis +// cache path returns places in distance order; the fresh path appends one place type's +// results after another in Google prominence order, so an unsorted slice[:limit] drops +// the last place types wholesale and can rank a 3km result above a 250m one. +// +// The sort is stable so places at equal distance keep their existing relative order. +func SortPlacesByDistance(places []POI.Place, lat, lng float64) { + origin := []float64{lat, lng} + dist := make(map[int]float64, len(places)) + for i := range places { + loc := places[i].GetLocation() + dist[i] = utils.HaversineDist(origin, []float64{loc.Latitude, loc.Longitude}) + } + idx := make([]int, len(places)) + for i := range idx { + idx[i] = i + } + sort.SliceStable(idx, func(a, b int) bool { return dist[idx[a]] < dist[idx[b]] }) + + sorted := make([]POI.Place, len(places)) + for newPos, oldPos := range idx { + sorted[newPos] = places[oldPos] + } + copy(places, sorted) +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `go test ./iowrappers/ -run TestSortPlacesByDistance -v` + +Expected: PASS on both tests. + +- [ ] **Step 5: Use it in the category handler** + +Replace `planner/planner.go:1411-1417`: + +```go + // drop places explicitly marked closed on the requested day + places = iowrappers.Filter(places, func(place POI.Place) bool { return !place.KnownClosedOnDay(day) }) + // Only the Redis path returns places in distance order. The fresh path + // appends each place type's results in Google prominence order, so sort + // before truncating or the last place types get dropped wholesale. + iowrappers.SortPlacesByDistance(places, req.Location.Latitude, req.Location.Longitude) + if len(places) > limit { + places = places[:limit] + } + result.Places = places +``` + +- [ ] **Step 6: Run the full suite** + +Run: `go build -v . && go test ./... 2>&1 | tail -20` + +Expected: PASS, no regressions. + +- [ ] **Step 7: Commit** + +```bash +git add iowrappers/place_distance_sort.go iowrappers/place_distance_sort_test.go planner/planner.go +git commit -m "fix: sort category results by distance before truncating + +The truncation at places[:limit] assumed distance ordering, which only +holds on the Redis cache path. The fresh path appends each place type's +results in Google prominence order, so a cold search kept roughly cafes +and restaurants and dropped bar, bakery and meal_takeaway entirely, and +could rank a 3km result above one 250m away." +``` + +--- + +### Task 4: Admin migration to purge misclassified places from category buckets + +Cleans the records already in prod. Must ship **after** Tasks 1-2 are deployed — otherwise the next cold search in any city recreates them. `MinMapsResultRefreshDuration` is 14 days (`iowrappers/poi_searcher.go`), so Los Altos is quiet, but every other city repopulates on its next search. + +Reuses the established pattern: an admin-authenticated GET under `v1.Group("/migrate")` (`planner/planner.go:1680-1684`), alongside `RemovePlaces`. + +**Files:** +- Modify: `iowrappers/data_migrations.go` +- Modify: `iowrappers/maps_client.go` (the searcher interface the handler calls through) +- Modify: `planner/planner.go` (handler + route) +- Test: `test/redis_client_mocks/bucket_cleanup_test.go` (create) + +**Interfaces:** +- Consumes: `POI.GetPlaceCategory(placeType) (PlaceCategory, bool)` from Task 1; `POI.PrimaryLocationType`, `POI.GetPlaceTypes` (existing). +- Produces: `(*RedisClient).RemoveMisclassifiedPlacesFromCategoryBuckets(ctx context.Context, cat POI.PlaceCategory, dryRun bool) (BucketCleanupReport, error)` and `type BucketCleanupReport struct { Scanned, Misclassified, Removed int; RemovedIDs []string }`. + +- [ ] **Step 1: Write the failing test** + +Create `test/redis_client_mocks/bucket_cleanup_test.go`. This follows the existing miniredis harness in that package (`RedisClient`, `RedisContext`, `RedisMockSvr` are package-level fixtures set up by its `TestMain`): + +```go +package redis_client_mocks + +import ( + "testing" + + "github.com/weihesdlegend/Vacation-planner/POI" + "github.com/weihesdlegend/Vacation-planner/iowrappers" +) + +// TestRemoveMisclassifiedPlacesDryRun pins that a dry run reports the hotels that the +// fast_food_restaurant incident wrote into placeIDs:eatery:level* without deleting them. +func TestRemoveMisclassifiedPlacesDryRun(t *testing.T) { + hotel := newPlaceWithTypes("hotel-1", "Residence Inn by Marriott Palo Alto", + POI.LocationType("fast_food_restaurant"), []string{"lodging", "point_of_interest", "establishment"}) + cafe := newPlaceWithTypes("cafe-1", "Peet's Coffee", + POI.LocationTypeCafe, []string{"cafe", "food", "point_of_interest", "establishment"}) + RedisClient.SetPlacesAddGeoLocations(RedisContext, []POI.Place{cafe}) + seedGeoBucket(t, POI.PlaceCategoryEatery, hotel) + + report, err := RedisClient.RemoveMisclassifiedPlacesFromCategoryBuckets(RedisContext, POI.PlaceCategoryEatery, true) + if err != nil { + t.Fatalf("RemoveMisclassifiedPlacesFromCategoryBuckets error: %v", err) + } + if report.Misclassified != 1 { + t.Errorf("Misclassified = %d, want 1 (report: %+v)", report.Misclassified, report) + } + if report.Removed != 0 { + t.Errorf("dry run Removed = %d, want 0", report.Removed) + } + if len(report.RemovedIDs) != 1 || report.RemovedIDs[0] != "hotel-1" { + t.Errorf("RemovedIDs = %v, want [hotel-1]", report.RemovedIDs) + } + // the hotel must still be present after a dry run + if got := countInEateryBuckets(t, "hotel-1"); got == 0 { + t.Error("dry run deleted hotel-1, want it retained") + } +} + +// TestRemoveMisclassifiedPlacesApply pins that a real run removes only the hotel. +func TestRemoveMisclassifiedPlacesApply(t *testing.T) { + RedisMockSvr.FlushAll() + + hotel := newPlaceWithTypes("hotel-2", "The Westin Palo Alto", + POI.LocationType("fast_food_restaurant"), []string{"lodging", "point_of_interest", "establishment"}) + cafe := newPlaceWithTypes("cafe-2", "Red Rock Coffee", + POI.LocationTypeCafe, []string{"cafe", "food", "point_of_interest", "establishment"}) + RedisClient.SetPlacesAddGeoLocations(RedisContext, []POI.Place{cafe}) + seedGeoBucket(t, POI.PlaceCategoryEatery, hotel) + + report, err := RedisClient.RemoveMisclassifiedPlacesFromCategoryBuckets(RedisContext, POI.PlaceCategoryEatery, false) + if err != nil { + t.Fatalf("RemoveMisclassifiedPlacesFromCategoryBuckets error: %v", err) + } + if report.Removed != 1 { + t.Errorf("Removed = %d, want 1 (report: %+v)", report.Removed, report) + } + if got := countInEateryBuckets(t, "hotel-2"); got != 0 { + t.Errorf("hotel-2 still in %d eatery buckets, want 0", got) + } + if got := countInEateryBuckets(t, "cafe-2"); got == 0 { + t.Error("cafe-2 was removed, want it retained") + } +} + +// TestRemoveMisclassifiedPlacesKeepsUntypedRecords pins that older cached records with +// no Types list are left alone, matching ReclassifyForCategory's keep-on-unknown rule. +func TestRemoveMisclassifiedPlacesKeepsUntypedRecords(t *testing.T) { + RedisMockSvr.FlushAll() + + legacy := newPlaceWithTypes("legacy-1", "Old Cached Diner", POI.LocationTypeRestaurant, nil) + RedisClient.SetPlacesAddGeoLocations(RedisContext, []POI.Place{legacy}) + + report, err := RedisClient.RemoveMisclassifiedPlacesFromCategoryBuckets(RedisContext, POI.PlaceCategoryEatery, false) + if err != nil { + t.Fatalf("RemoveMisclassifiedPlacesFromCategoryBuckets error: %v", err) + } + if report.Removed != 0 { + t.Errorf("Removed = %d, want 0 — records without Types must be kept", report.Removed) + } +} + +func newPlaceWithTypes(id, name string, locationType POI.LocationType, types []string) POI.Place { + var p POI.Place + p.SetID(id) + p.SetName(name) + p.SetType(locationType) + p.SetStatus(string(POI.Operational)) + p.SetPriceLevel(POI.PriceLevelDefault) + p.SetUserRatingsTotal(100) + p.SetLocationCoordinates([2]float64{37.38006, -122.11612}) + p.Types = types + return p +} + +// seedGeoBucket writes a place record plus its eatery geo-bucket membership directly, +// bypassing SetPlacesAddGeoLocations, which after Task 1 refuses unmapped types. +func seedGeoBucket(t *testing.T, cat POI.PlaceCategory, place POI.Place) { + t.Helper() + if err := RedisClient.SetPlace(RedisContext, place); err != nil { + t.Fatalf("SetPlace(%s): %v", place.GetID(), err) + } + key := POI.EncodeNearbySearchRedisKey(cat, place.PriceLevel) + if err := RedisClient.AddGeoLocation(RedisContext, key, place); err != nil { + t.Fatalf("AddGeoLocation(%s): %v", key, err) + } +} + +func countInEateryBuckets(t *testing.T, placeID string) int { + t.Helper() + count := 0 + for _, lvl := range POI.AllPriceLevels { + key := POI.EncodeNearbySearchRedisKey(POI.PlaceCategoryEatery, lvl) + if RedisMockSvr.Exists(key) { + members, err := RedisMockSvr.ZMembers(key) + if err != nil { + continue + } + for _, m := range members { + if m == placeID { + count++ + } + } + } + } + return count +} +``` + +Note for the implementer: this test needs two small exported helpers on `RedisClient` that do not exist yet — `SetPlace(ctx, place) error` and `AddGeoLocation(ctx, key string, place POI.Place) error`. `setPlace` already exists unexported (`iowrappers/redis_client.go`, used by `StorePlacesForLocation`); add thin exported wrappers in Step 3 rather than duplicating logic. + +- [ ] **Step 2: Run test to verify it fails** + +Run: `go test ./test/redis_client_mocks/ -run TestRemoveMisclassifiedPlaces -v` + +Expected: FAIL to compile with `RedisClient.RemoveMisclassifiedPlacesFromCategoryBuckets undefined` (and undefined `SetPlace` / `AddGeoLocation`). + +- [ ] **Step 3: Write the implementation** + +Append to `iowrappers/data_migrations.go`: + +```go +// BucketCleanupReport summarizes a RemoveMisclassifiedPlacesFromCategoryBuckets run. +type BucketCleanupReport struct { + Scanned int `json:"scanned"` + Misclassified int `json:"misclassified"` + Removed int `json:"removed"` + RemovedIDs []string `json:"removed_ids"` +} + +// SetPlace stores a single place record. Exported wrapper over setPlace for migrations and tests. +func (r *RedisClient) SetPlace(ctx context.Context, place POI.Place) error { + return r.setPlace(ctx, place) +} + +// AddGeoLocation adds a place to a geo bucket under an explicit key. Exported for +// migrations and tests that need to write buckets the normal write path would reject. +func (r *RedisClient) AddGeoLocation(ctx context.Context, key string, place POI.Place) error { + loc := place.GetLocation() + _, err := r.client.GeoAdd(ctx, key, &redis.GeoLocation{ + Name: place.ID, + Latitude: loc.Latitude, + Longitude: loc.Longitude, + }).Result() + return err +} + +// RemoveMisclassifiedPlacesFromCategoryBuckets removes places from cat's geo buckets whose +// PRIMARY Google type does not belong to cat. It repairs the fast_food_restaurant incident: +// two Places-API-(New)-only types were searched against the legacy Nearby Search, which +// ignored the unenforceable type filter and returned prominence-ranked establishments, and +// those were stamped with the queried type and written into placeIDs:eatery:level*. +// +// It uses the same rule as POI.ReclassifyForCategory, which is what already hides these from +// API responses: classify by primary type, and KEEP records with no Types list (older cached +// records written before Types was captured) so coverage never regresses. +// +// dryRun reports what would be removed without deleting anything. Always dry-run first. +func (r *RedisClient) RemoveMisclassifiedPlacesFromCategoryBuckets(ctx context.Context, cat POI.PlaceCategory, dryRun bool) (BucketCleanupReport, error) { + report := BucketCleanupReport{RemovedIDs: make([]string, 0)} + + levels := []POI.PriceLevel{POI.PriceLevelDefault} + if cat == POI.PlaceCategoryEatery { + levels = POI.AllPriceLevels + } + + for _, level := range levels { + key := POI.EncodeNearbySearchRedisKey(cat, level) + members, err := r.client.ZRange(ctx, key, 0, -1).Result() + if err != nil { + return report, fmt.Errorf("reading geo bucket %s: %w", key, err) + } + for _, placeID := range members { + report.Scanned++ + place, err := r.getPlace(ctx, placeID) + if err != nil { + // no place record backing this bucket member; leave it for RemovePlaces + Logger.Debugf("RemoveMisclassifiedPlacesFromCategoryBuckets: no record for %s in %s", placeID, key) + continue + } + if _, keep := POI.ReclassifyForCategory(place, cat); keep { + continue + } + report.Misclassified++ + report.RemovedIDs = append(report.RemovedIDs, placeID) + Logger.Infof("RemoveMisclassifiedPlacesFromCategoryBuckets: %s (%q, LocationType=%q, Types=%v) does not belong in %s", + placeID, place.Name, place.LocationType, place.Types, key) + if dryRun { + continue + } + if _, err := r.client.ZRem(ctx, key, placeID).Result(); err != nil { + return report, fmt.Errorf("removing %s from %s: %w", placeID, key, err) + } + report.Removed++ + } + } + return report, nil +} +``` + +Add `"github.com/redis/go-redis/v9"` to the imports of `data_migrations.go` if it is not already present, alongside the existing `context`, `fmt`, and `POI` imports. + +Add the method to the migration-capable interface in `iowrappers/maps_client.go` so the Gin handler can call it through `p.Solver.Searcher`. Locate the interface that already declares `RemovePlaces` and add: + +```go + RemoveMisclassifiedPlacesFromCategoryBuckets(context.Context, POI.PlaceCategory, bool) (BucketCleanupReport, error) +``` + +Then add the forwarding method on `PoiSearcher` in `data_migrations.go`, mirroring the existing `(*PoiSearcher).RemovePlaces`: + +```go +func (s *PoiSearcher) RemoveMisclassifiedPlacesFromCategoryBuckets(ctx context.Context, cat POI.PlaceCategory, dryRun bool) (BucketCleanupReport, error) { + return s.redisClient.RemoveMisclassifiedPlacesFromCategoryBuckets(ctx, cat, dryRun) +} +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `go build -v . && go test ./test/redis_client_mocks/ -run TestRemoveMisclassifiedPlaces -v` + +Expected: PASS on all three tests. + +- [ ] **Step 5: Add the admin handler and route** + +Add to `planner/planner.go`, next to `removePlacesMigrationHandler`: + +```go +// reclassifyBucketsMigrationHandler removes places from a category's geo buckets whose +// primary Google type does not belong to that category. Dry-run unless ?apply=true. +// +// Usage: GET /v1/migrate/reclassify-buckets?category=Eatery +// GET /v1/migrate/reclassify-buckets?category=Eatery&apply=true +func (p *MyPlanner) reclassifyBucketsMigrationHandler(ctx *gin.Context) { + _, authenticationErr := p.UserAuthentication(ctx, user.LevelAdmin) + if authenticationErr != nil { + ctx.JSON(http.StatusUnauthorized, gin.H{"error": authenticationErr.Error()}) + return + } + category, ok := POI.ParsePlaceCategory(ctx.DefaultQuery("category", string(POI.PlaceCategoryEatery))) + if !ok { + ctx.JSON(http.StatusBadRequest, gin.H{"error": "unknown category"}) + return + } + dryRun := ctx.Query("apply") != "true" + report, err := p.Solver.Searcher.RemoveMisclassifiedPlacesFromCategoryBuckets(ctx.Request.Context(), category, dryRun) + if err != nil { + ctx.JSON(http.StatusInternalServerError, gin.H{"error": err.Error(), "partial_report": report}) + return + } + ctx.JSON(http.StatusOK, gin.H{"dry_run": dryRun, "category": category, "report": report}) +} +``` + +Register it at `planner/planner.go:1684`, inside the existing `migrations` group: + +```go + migrations.GET("/reclassify-buckets", p.reclassifyBucketsMigrationHandler) +``` + +- [ ] **Step 6: Run the full suite** + +Run: `go build -v . && go test ./... 2>&1 | tail -20` + +Expected: PASS. + +- [ ] **Step 7: Commit** + +```bash +git add iowrappers/data_migrations.go iowrappers/maps_client.go planner/planner.go test/redis_client_mocks/bucket_cleanup_test.go +git commit -m "feat: add admin migration to purge misclassified places from geo buckets + +The fast_food_restaurant incident wrote prominence-ranked hotels into +placeIDs:eatery:level*. ReclassifyForCategory already hides them from API +responses, but they inflate the bucket counts that gate radius expansion. +Remove them using the same primary-type rule, dry-run by default." +``` + +--- + +## Deployment order + +1. Merge and deploy Tasks 1-3. Verify `go test ./...` green in CI. +2. Dry-run the cleanup and read the report before applying: + ```bash + curl -s -H "Authorization: Bearer $ADMIN_JWT" \ + "https://best-vacation-planner.herokuapp.com/v1/migrate/reclassify-buckets?category=Eatery" | jq + ``` + Expect roughly 17 entries in `removed_ids` for the Los Altos hotels, plus any older misclassifications the primary-type rule catches. Review the list before proceeding. +3. Apply: + ```bash + curl -s -H "Authorization: Bearer $ADMIN_JWT" \ + "https://best-vacation-planner.herokuapp.com/v1/migrate/reclassify-buckets?category=Eatery&apply=true" | jq + ``` +4. Spot-check that a cold search is correct now. `MapsLastSearchTime` gates on a 14-day TTL, so force a fresh path by deleting the marker field for the city under test: + ```bash + # field format: "::::" + redis-cli HDEL MapsLastSearchTime "united states:ca:los altos:eatery:0" + ``` + Then re-run the category search and confirm the nearest result is the nearest by distance, not by Google prominence. + +## Verification checklist + +- [ ] `go build -v .` and `go test -v ./...` pass locally and in CI. +- [ ] `TestPlaceCategoryRoundTrip` fails if you temporarily re-add `LocationTypeFastFood` to `GetPlaceTypes(Eatery)` — confirm the guard is now real, then revert the experiment. +- [ ] `grep -rn 'fast_food_restaurant\|food_court' --include='*.go' .` returns nothing. +- [ ] A category search at State Street Market returns Peet's at 367 State St ahead of results in Sunnyvale. +- [ ] Dry-run report reviewed before any `apply=true` call. + +## Out of scope, tracked in the follow-up plan + +Correctly classifying fast food and food courts needs Places API (New) `searchNearby` with `includedPrimaryTypes`. Legacy `types[]` never contains those values, so `POI.PrimaryLocationType` can never return them either — no amount of client-side work fixes it on the current API. See `2026-07-29-places-api-new-migration.md`. diff --git a/docs/superpowers/plans/2026-07-29-places-api-new-migration.md b/docs/superpowers/plans/2026-07-29-places-api-new-migration.md new file mode 100644 index 00000000..a27dd8e5 --- /dev/null +++ b/docs/superpowers/plans/2026-07-29-places-api-new-migration.md @@ -0,0 +1,1796 @@ +# Places API (New) Migration — Search and Photos Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Move the category search and place-photo paths off the legacy Places API (`maps.googleapis.com/maps/api/place/*`) onto Places API (New) (`places.googleapis.com/v1`), using `places:searchNearby` with `includedPrimaryTypes` so place types are filtered server-side by primary type, and bucketing results by the `priceLevel` the response already returns. + +**Architecture:** A new self-contained `iowrappers/placesv1` package speaks the New API over `net/http` — `googlemaps.github.io/maps` v1.7.0 has no support for it and the v1.7.0 client stays in place for Geocoding (which is not deprecated) and for the brand/keyword path (deferred to a follow-up). A `PLACES_API_VERSION` env flag selects legacy or new at runtime, and the new path writes under versioned Redis keys (`v2:placeIDs:*`, `v2:place_details:*`) so both datasets coexist and cutover is reversible by flipping one Heroku config var. + +**Tech Stack:** Go 1.24, `net/http` + `encoding/json` (no new dependencies), Gin, go-redis v9, `googlemaps.github.io/maps` v1.7.0 (retained for Geocoding only). + +## Global Constraints + +- Go version: `1.24.0` (from `go.mod`) — do not raise it. +- **Add no new module dependencies.** The New API is plain REST; use `net/http`. Do not add a Google client library. +- CI gates are exactly `go build -v .` then `go test -v ./...` (`.github/workflows/go.yml`). Both must pass. +- Target branch is `origin/master`. `origin/main` is stale at `1a1417b` and is not the deploy path. +- **Prerequisite:** the fixes in `2026-07-29-eatery-place-type-fixes.md` must be merged and deployed first. This plan re-adds `fast_food_restaurant` and `food_court` in Task 6, which is only safe once type validation and the `(PlaceCategory, bool)` signature exist. +- Every New API request MUST send an `X-Goog-FieldMask` header. There is no default field list; a missing mask is an error, and an over-broad mask is billed at a higher SKU. +- Do not migrate Geocoding or ReverseGeocode. The Geocoding API (`/maps/api/geocode/json`) is a separate, non-deprecated API and stays on the v1.7.0 SDK. +- Do not migrate the brand/keyword search path in this PR. `searchNearby` has no keyword parameter; that path needs `places:searchText` and is deferred. +- No unit test may make a real network call. Use `httptest` with recorded response bodies. + +--- + +## Why this migration, and what it buys + +Verified constraints of each endpoint (Google reference docs, checked 2026-07-29): + +| Capability | Legacy nearbysearch | `places:searchNearby` (New) | `places:searchText` (New) | +| --- | --- | --- | --- | +| Types per call | 1 (`type`) | **many** (`includedTypes`, `includedPrimaryTypes`) | 1 (`includedType`) | +| Max results | 20/page, 3 pages | **20, no pagination** | 20/page, 60 total | +| Price filter | `minprice`/`maxprice` | none | `priceLevels` | +| Keyword | `keyword` | none | `textQuery` | +| Rank | `rankby` | `rankPreference: DISTANCE\|POPULARITY` | `rankPreference: DISTANCE\|RELEVANCE` | +| Radius cap | 50000 m | 50000 m | n/a (bias/restriction) | + +What the chosen approach fixes or improves: + +1. **`fast_food_restaurant` and `food_court` become real.** Both are valid Table A types in the New API. `includedPrimaryTypes` filters by *primary* type server-side — which is exactly what `POI.ReclassifyForCategory` currently approximates client-side after the fact. +2. **One HTTP call replaces up to 35.** Today a cold Eatery search issues 5-7 Nearby Searches per round for up to 5 rounds (`GoogleMapsSearchCallMaxCount = 5`). The new path issues one `searchNearby` with all types in `includedPrimaryTypes`. +3. **The separate Place Details fan-out disappears for search results.** `searchNearby` returns opening hours, `adrFormatAddress`, `googleMapsUri`, `userRatingCount`, `editorialSummary` and `photos` directly via the field mask. `searchPlaceDetails` and its `detailsBudget` (`iowrappers/nearby_search.go:150`) are not needed on the new path. +4. **`rankPreference: DISTANCE` fixes ordering at the source**, complementing the client-side sort added in the prior PR. + +**The cost, stated plainly:** a hard cap of 20 results per search versus roughly 100-140 raw results today. Task 5 measures this against production data on real cities and gates the cutover on the result. If coverage is unacceptable, the mitigation is already designed in: `SearchNearby` accepts *groups* of types, so splitting `includedPrimaryTypes` into one group per type restores today's ~140-result ceiling at 7 calls — still far cheaper than today's 35, and still with correct server-side primary-type filtering. Do not skip Task 5. + +--- + +## File Structure + +| File | Responsibility | +| --- | --- | +| `iowrappers/placesv1/client.go` (new) | HTTP transport for `places.googleapis.com/v1`: auth header, field mask, timeouts, error decoding. Knows nothing about POI types. | +| `iowrappers/placesv1/types.go` (new) | Request/response structs mirroring the New API JSON exactly (`Place`, `LocalizedText`, `OpeningHours`, `Photo`, enums). | +| `iowrappers/placesv1/search_nearby.go` (new) | `SearchNearby` request building and the multi-group fan-out. | +| `iowrappers/placesv1/photo.go` (new) | Builds the `/v1/{photoName}/media` URL and fetches image bytes. | +| `iowrappers/places_v1_mapper.go` (new) | Maps `placesv1.Place` → `POI.Place`. The only place that knows both vocabularies. | +| `iowrappers/places_v1_search_client.go` (new) | Implements `SearchClient.NearbySearch` against the New API; delegates Geocode/ReverseGeocode to the existing `MapsClient`. | +| `iowrappers/redis_keys.go` (new) | Versioned Redis key building shared by both paths. | +| `POI/categories.go` | Task 6 only: re-add the two types now that they work. | +| `iowrappers/photos_client.go` | Route by reference format: new `places/...` refs to the New media endpoint, legacy refs to the SDK. | +| `iowrappers/poi_searcher.go` | Select the search client from `PLACES_API_VERSION`. | +| `config/config.yml` | New-API field mask; retain the legacy `detailed_search_fields` for the legacy path. | + +--- + +### Task 1: `placesv1` HTTP client and response types + +**Files:** +- Create: `iowrappers/placesv1/client.go`, `iowrappers/placesv1/types.go` +- Test: `iowrappers/placesv1/client_test.go` + +**Interfaces:** +- Produces: + - `placesv1.New(apiKey string, opts ...Option) *Client`, `placesv1.WithBaseURL(string) Option`, `placesv1.WithHTTPClient(*http.Client) Option` + - `(*Client).post(ctx context.Context, path, fieldMask string, body any, out any) error` + - `placesv1.Place`, `placesv1.LocalizedText`, `placesv1.OpeningHours`, `placesv1.Photo`, `placesv1.LatLng` + - `placesv1.APIError` with `Code int`, `Status string`, `Message string` +- Tasks 2, 3 and 4 all depend on these exact names. + +- [ ] **Step 1: Write the failing test** + +Create `iowrappers/placesv1/client_test.go`: + +```go +package placesv1 + +import ( + "context" + "encoding/json" + "errors" + "net/http" + "net/http/httptest" + "testing" +) + +func TestClientSendsAPIKeyAndFieldMask(t *testing.T) { + var gotKey, gotMask, gotContentType string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotKey = r.Header.Get("X-Goog-Api-Key") + gotMask = r.Header.Get("X-Goog-FieldMask") + gotContentType = r.Header.Get("Content-Type") + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"places":[]}`)) + })) + defer srv.Close() + + c := New("test-key", WithBaseURL(srv.URL)) + var out SearchNearbyResponse + if err := c.post(context.Background(), "/v1/places:searchNearby", "places.id", map[string]any{}, &out); err != nil { + t.Fatalf("post returned %v, want nil", err) + } + if gotKey != "test-key" { + t.Errorf("X-Goog-Api-Key = %q, want %q", gotKey, "test-key") + } + if gotMask != "places.id" { + t.Errorf("X-Goog-FieldMask = %q, want %q", gotMask, "places.id") + } + if gotContentType != "application/json" { + t.Errorf("Content-Type = %q, want application/json", gotContentType) + } +} + +func TestClientDecodesAPIError(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusBadRequest) + _, _ = w.Write([]byte(`{"error":{"code":400,"message":"Invalid included primary type: nonsense_type","status":"INVALID_ARGUMENT"}}`)) + })) + defer srv.Close() + + c := New("test-key", WithBaseURL(srv.URL)) + var out SearchNearbyResponse + err := c.post(context.Background(), "/v1/places:searchNearby", "places.id", map[string]any{}, &out) + if err == nil { + t.Fatal("post returned nil error, want APIError") + } + var apiErr *APIError + if !errors.As(err, &apiErr) { + t.Fatalf("error %v is not *APIError", err) + } + if apiErr.Code != 400 || apiErr.Status != "INVALID_ARGUMENT" { + t.Errorf("APIError = %+v, want code 400 status INVALID_ARGUMENT", apiErr) + } + // Unlike the legacy API, which silently ignored an unknown type, the New API rejects it. + if apiErr.Message == "" { + t.Error("APIError.Message is empty, want Google's explanation") + } +} + +func TestPlaceJSONDecodesNewAPIShape(t *testing.T) { + // Trimmed real-shape response body. + body := `{"places":[{ + "id":"ChIJ_test", + "types":["cafe","food","point_of_interest","establishment"], + "primaryType":"cafe", + "formattedAddress":"367 State St, Los Altos, CA 94022, USA", + "adrFormatAddress":"367 State St", + "location":{"latitude":37.38025,"longitude":-122.11655}, + "rating":4.3, + "userRatingCount":412, + "googleMapsUri":"https://maps.google.com/?cid=1", + "businessStatus":"OPERATIONAL", + "priceLevel":"PRICE_LEVEL_INEXPENSIVE", + "displayName":{"text":"Peet's Coffee","languageCode":"en"}, + "editorialSummary":{"text":"Coffee chain known for house blends.","languageCode":"en"}, + "regularOpeningHours":{"openNow":true,"weekdayDescriptions":[ + "Monday: 5:30 AM – 7:00 PM","Tuesday: 5:30 AM – 7:00 PM","Wednesday: 5:30 AM – 7:00 PM", + "Thursday: 5:30 AM – 7:00 PM","Friday: 5:30 AM – 7:00 PM","Saturday: 6:00 AM – 7:00 PM", + "Sunday: 6:00 AM – 7:00 PM"]}, + "photos":[{"name":"places/ChIJ_test/photos/AT_abc","widthPx":4032,"heightPx":3024}] + }]}` + + var resp SearchNearbyResponse + if err := json.Unmarshal([]byte(body), &resp); err != nil { + t.Fatalf("Unmarshal error: %v", err) + } + if len(resp.Places) != 1 { + t.Fatalf("got %d places, want 1", len(resp.Places)) + } + p := resp.Places[0] + if p.ID != "ChIJ_test" { + t.Errorf("ID = %q, want ChIJ_test", p.ID) + } + if p.DisplayName.Text != "Peet's Coffee" { + t.Errorf("DisplayName.Text = %q, want Peet's Coffee", p.DisplayName.Text) + } + if p.PrimaryType != "cafe" { + t.Errorf("PrimaryType = %q, want cafe", p.PrimaryType) + } + if p.PriceLevel != PriceLevelInexpensive { + t.Errorf("PriceLevel = %q, want %q", p.PriceLevel, PriceLevelInexpensive) + } + if len(p.RegularOpeningHours.WeekdayDescriptions) != 7 { + t.Errorf("got %d weekday descriptions, want 7", len(p.RegularOpeningHours.WeekdayDescriptions)) + } + if len(p.Photos) != 1 || p.Photos[0].Name != "places/ChIJ_test/photos/AT_abc" { + t.Errorf("Photos = %+v, want one photo named places/ChIJ_test/photos/AT_abc", p.Photos) + } + if p.EditorialSummary.Text == "" { + t.Error("EditorialSummary.Text is empty") + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `go test ./iowrappers/placesv1/ -v` + +Expected: FAIL — the package does not exist yet (`no Go files in .../placesv1`). + +- [ ] **Step 3: Write `types.go`** + +Create `iowrappers/placesv1/types.go`: + +```go +// Package placesv1 is a minimal client for Google Places API (New), +// https://places.googleapis.com/v1. The googlemaps.github.io/maps v1.7.0 SDK only +// implements the legacy /maps/api/place/* endpoints, so this speaks REST directly. +// +// Every request must carry an X-Goog-FieldMask; the New API has no default field set. +package placesv1 + +// PriceLevel is the New API's price enum. Unlike the legacy integer priceLevel, an +// absent value is explicit (PriceLevelUnspecified) rather than indistinguishable from 0. +type PriceLevel string + +const ( + PriceLevelUnspecified PriceLevel = "PRICE_LEVEL_UNSPECIFIED" + PriceLevelFree PriceLevel = "PRICE_LEVEL_FREE" + PriceLevelInexpensive PriceLevel = "PRICE_LEVEL_INEXPENSIVE" + PriceLevelModerate PriceLevel = "PRICE_LEVEL_MODERATE" + PriceLevelExpensive PriceLevel = "PRICE_LEVEL_EXPENSIVE" + PriceLevelVeryExpensive PriceLevel = "PRICE_LEVEL_VERY_EXPENSIVE" +) + +// BusinessStatus values match POI.BusinessStatus strings exactly, so no translation +// table is needed: OPERATIONAL, CLOSED_TEMPORARILY, CLOSED_PERMANENTLY. +type BusinessStatus string + +// RankPreference selects result ordering for searchNearby. +type RankPreference string + +const ( + RankPreferenceDistance RankPreference = "DISTANCE" + RankPreferencePopularity RankPreference = "POPULARITY" +) + +// LocalizedText backs displayName and editorialSummary. +type LocalizedText struct { + Text string `json:"text"` + LanguageCode string `json:"languageCode"` +} + +type LatLng struct { + Latitude float64 `json:"latitude"` + Longitude float64 `json:"longitude"` +} + +type OpeningHours struct { + OpenNow bool `json:"openNow"` + // WeekdayDescriptions holds one human-readable string per day. Its starting weekday + // is verified against the live API in Task 2 Step 0 before being mapped to + // POI.Weekday — do not assume it matches the legacy WeekdayText ordering. + WeekdayDescriptions []string `json:"weekdayDescriptions"` +} + +// Photo.Name is a full resource name, "places/{placeID}/photos/{photoResource}". +// This is NOT interchangeable with a legacy photo_reference string. +type Photo struct { + Name string `json:"name"` + WidthPx int `json:"widthPx"` + HeightPx int `json:"heightPx"` +} + +type Place struct { + ID string `json:"id"` + Types []string `json:"types"` + PrimaryType string `json:"primaryType"` + DisplayName LocalizedText `json:"displayName"` + FormattedAddress string `json:"formattedAddress"` + AdrFormatAddress string `json:"adrFormatAddress"` + Location LatLng `json:"location"` + Rating float32 `json:"rating"` + UserRatingCount int `json:"userRatingCount"` + GoogleMapsURI string `json:"googleMapsUri"` + BusinessStatus BusinessStatus `json:"businessStatus"` + PriceLevel PriceLevel `json:"priceLevel"` + EditorialSummary LocalizedText `json:"editorialSummary"` + RegularOpeningHours OpeningHours `json:"regularOpeningHours"` + Photos []Photo `json:"photos"` +} + +type SearchNearbyResponse struct { + Places []Place `json:"places"` +} + +// Circle is the only locationRestriction shape searchNearby accepts. +type Circle struct { + Center LatLng `json:"center"` + Radius float64 `json:"radius"` // meters, 0 < radius <= 50000 +} + +type locationRestriction struct { + Circle Circle `json:"circle"` +} + +type searchNearbyRequest struct { + IncludedPrimaryTypes []string `json:"includedPrimaryTypes,omitempty"` + ExcludedPrimaryTypes []string `json:"excludedPrimaryTypes,omitempty"` + LocationRestriction locationRestriction `json:"locationRestriction"` + MaxResultCount int `json:"maxResultCount,omitempty"` + RankPreference RankPreference `json:"rankPreference,omitempty"` + LanguageCode string `json:"languageCode,omitempty"` +} +``` + +- [ ] **Step 4: Write `client.go`** + +Create `iowrappers/placesv1/client.go`: + +```go +package placesv1 + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "time" +) + +const ( + defaultBaseURL = "https://places.googleapis.com" + // MaxRadiusMeters is the searchNearby locationRestriction circle cap. + MaxRadiusMeters = 50000.0 + // MaxResultCount is the searchNearby hard cap. There is no pagination. + MaxResultCount = 20 +) + +// APIError is a structured Places API (New) error. The New API rejects an unknown +// place type with INVALID_ARGUMENT, where the legacy API silently ignored the filter. +type APIError struct { + Code int `json:"code"` + Message string `json:"message"` + Status string `json:"status"` +} + +func (e *APIError) Error() string { + return fmt.Sprintf("places api (new): %d %s: %s", e.Code, e.Status, e.Message) +} + +type errorEnvelope struct { + Error APIError `json:"error"` +} + +type Client struct { + apiKey string + baseURL string + http *http.Client +} + +type Option func(*Client) + +func WithBaseURL(u string) Option { return func(c *Client) { c.baseURL = u } } +func WithHTTPClient(h *http.Client) Option { return func(c *Client) { c.http = h } } + +func New(apiKey string, opts ...Option) *Client { + c := &Client{ + apiKey: apiKey, + baseURL: defaultBaseURL, + http: &http.Client{Timeout: 15 * time.Second}, + } + for _, o := range opts { + o(c) + } + return c +} + +// post sends a JSON POST with the API key and field mask headers the New API requires, +// and decodes either the success body into out or the error body into *APIError. +func (c *Client) post(ctx context.Context, path, fieldMask string, body any, out any) error { + payload, err := json.Marshal(body) + if err != nil { + return fmt.Errorf("marshaling request: %w", err) + } + req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+path, bytes.NewReader(payload)) + if err != nil { + return fmt.Errorf("building request: %w", err) + } + req.Header.Set("Content-Type", "application/json") + req.Header.Set("X-Goog-Api-Key", c.apiKey) + req.Header.Set("X-Goog-FieldMask", fieldMask) + + resp, err := c.http.Do(req) + if err != nil { + return fmt.Errorf("calling %s: %w", path, err) + } + defer func() { _ = resp.Body.Close() }() + + raw, err := io.ReadAll(resp.Body) + if err != nil { + return fmt.Errorf("reading response from %s: %w", path, err) + } + if resp.StatusCode != http.StatusOK { + var env errorEnvelope + if jsonErr := json.Unmarshal(raw, &env); jsonErr == nil && env.Error.Code != 0 { + return &env.Error + } + return &APIError{Code: resp.StatusCode, Status: resp.Status, Message: string(raw)} + } + if err := json.Unmarshal(raw, out); err != nil { + return fmt.Errorf("decoding response from %s: %w", path, err) + } + return nil +} +``` + +- [ ] **Step 5: Run tests to verify they pass** + +Run: `go test ./iowrappers/placesv1/ -v` + +Expected: PASS on all three tests. + +- [ ] **Step 6: Commit** + +```bash +git add iowrappers/placesv1/ +git commit -m "feat: add minimal Places API (New) HTTP client + +googlemaps.github.io/maps v1.7.0 only implements the legacy +/maps/api/place/* endpoints, so speak places.googleapis.com/v1 REST +directly. No new module dependencies." +``` + +--- + +### Task 2: `SearchNearby` request building and POI mapping + +**Files:** +- Create: `iowrappers/placesv1/search_nearby.go`, `iowrappers/places_v1_mapper.go` +- Test: `iowrappers/placesv1/search_nearby_test.go`, `iowrappers/places_v1_mapper_test.go` + +**Interfaces:** +- Consumes: everything from Task 1. +- Produces: + - `(*Client).SearchNearby(ctx context.Context, req SearchNearbyRequest) ([]Place, error)` where `SearchNearbyRequest` is the exported struct defined below + - `placesv1.SearchNearbyFieldMask` — the exact mask string + - `iowrappers.MapPlace(p placesv1.Place) POI.Place` + - `iowrappers.MapPriceLevel(pl placesv1.PriceLevel) POI.PriceLevel` + +- [ ] **Step 0: Verify weekday ordering against the live API before writing the mapper** + +`POI.CreatePlace` indexes hours by `POI.Weekday` from `DateMonday` to `DateSunday`. The legacy `WeekdayText` is Monday-first. Confirm the New API's `weekdayDescriptions` ordering rather than assuming it, because a silent off-by-one here shifts every place's opening hours by a day: + +```bash +curl -s -X POST 'https://places.googleapis.com/v1/places:searchNearby' \ + -H "X-Goog-Api-Key: $GOOGLE_MAPS_API_KEY" \ + -H 'X-Goog-FieldMask: places.displayName,places.regularOpeningHours.weekdayDescriptions' \ + -H 'Content-Type: application/json' \ + -d '{"includedPrimaryTypes":["cafe"],"maxResultCount":1, + "locationRestriction":{"circle":{"center":{"latitude":37.38006,"longitude":-122.11612},"radius":2000}}}' | jq +``` + +Record the first element's day name in a code comment in `places_v1_mapper.go`. If it is not Monday, the mapper must rotate the slice before handing it to `POI.OpeningHours.Hours`. + +- [ ] **Step 1: Write the failing test** + +Create `iowrappers/placesv1/search_nearby_test.go`: + +```go +package placesv1 + +import ( + "context" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "testing" +) + +func TestSearchNearbyBuildsRequest(t *testing.T) { + var got searchNearbyRequest + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + raw, _ := io.ReadAll(r.Body) + _ = json.Unmarshal(raw, &got) + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"places":[]}`)) + })) + defer srv.Close() + + c := New("k", WithBaseURL(srv.URL)) + _, err := c.SearchNearby(context.Background(), SearchNearbyRequest{ + IncludedPrimaryTypes: []string{"cafe", "restaurant", "bar", "bakery", "meal_takeaway"}, + Latitude: 37.38006, + Longitude: -122.11612, + RadiusMeters: 8000, + MaxResultCount: 20, + RankPreference: RankPreferenceDistance, + }) + if err != nil { + t.Fatalf("SearchNearby returned %v", err) + } + if len(got.IncludedPrimaryTypes) != 5 { + t.Errorf("IncludedPrimaryTypes = %v, want 5 entries", got.IncludedPrimaryTypes) + } + if got.LocationRestriction.Circle.Radius != 8000 { + t.Errorf("radius = %v, want 8000", got.LocationRestriction.Circle.Radius) + } + if got.LocationRestriction.Circle.Center.Latitude != 37.38006 { + t.Errorf("center.latitude = %v, want 37.38006", got.LocationRestriction.Circle.Center.Latitude) + } + if got.RankPreference != RankPreferenceDistance { + t.Errorf("rankPreference = %q, want DISTANCE", got.RankPreference) + } +} + +func TestSearchNearbyClampsRadiusAndCount(t *testing.T) { + var got searchNearbyRequest + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + raw, _ := io.ReadAll(r.Body) + _ = json.Unmarshal(raw, &got) + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"places":[]}`)) + })) + defer srv.Close() + + c := New("k", WithBaseURL(srv.URL)) + _, err := c.SearchNearby(context.Background(), SearchNearbyRequest{ + IncludedPrimaryTypes: []string{"cafe"}, + Latitude: 37.38006, + Longitude: -122.11612, + RadiusMeters: 120000, // over the 50km cap + MaxResultCount: 500, // over the 20 cap + }) + if err != nil { + t.Fatalf("SearchNearby returned %v", err) + } + if got.LocationRestriction.Circle.Radius != MaxRadiusMeters { + t.Errorf("radius = %v, want clamped to %v", got.LocationRestriction.Circle.Radius, MaxRadiusMeters) + } + if got.MaxResultCount != MaxResultCount { + t.Errorf("maxResultCount = %d, want clamped to %d", got.MaxResultCount, MaxResultCount) + } +} + +func TestSearchNearbyRejectsEmptyTypes(t *testing.T) { + c := New("k", WithBaseURL("http://unused")) + if _, err := c.SearchNearby(context.Background(), SearchNearbyRequest{ + Latitude: 1, Longitude: 1, RadiusMeters: 100, + }); err == nil { + t.Error("SearchNearby with no types returned nil error, want validation failure") + } +} +``` + +Create `iowrappers/places_v1_mapper_test.go`: + +```go +package iowrappers + +import ( + "testing" + + "github.com/weihesdlegend/Vacation-planner/POI" + "github.com/weihesdlegend/Vacation-planner/iowrappers/placesv1" +) + +func TestMapPriceLevel(t *testing.T) { + cases := map[placesv1.PriceLevel]POI.PriceLevel{ + // Unspecified maps to 0, matching the legacy behavior where an absent + // priceLevel arrived as integer 0 and was bucketed into level0. + placesv1.PriceLevelUnspecified: POI.PriceLevelZero, + placesv1.PriceLevelFree: POI.PriceLevelZero, + placesv1.PriceLevelInexpensive: POI.PriceLevelOne, + placesv1.PriceLevelModerate: POI.PriceLevelTwo, + placesv1.PriceLevelExpensive: POI.PriceLevelThree, + placesv1.PriceLevelVeryExpensive: POI.PriceLevelFour, + } + for in, want := range cases { + if got := MapPriceLevel(in); got != want { + t.Errorf("MapPriceLevel(%q) = %d, want %d", in, got, want) + } + } +} + +func TestMapPlace(t *testing.T) { + in := placesv1.Place{ + ID: "ChIJ_test", + Types: []string{"cafe", "food", "point_of_interest", "establishment"}, + PrimaryType: "cafe", + DisplayName: placesv1.LocalizedText{Text: "Peet's Coffee"}, + FormattedAddress: "367 State St, Los Altos, CA 94022, USA", + AdrFormatAddress: `367 State St`, + Location: placesv1.LatLng{Latitude: 37.38025, Longitude: -122.11655}, + Rating: 4.3, + UserRatingCount: 412, + GoogleMapsURI: "https://maps.google.com/?cid=1", + BusinessStatus: placesv1.BusinessStatus("OPERATIONAL"), + PriceLevel: placesv1.PriceLevelInexpensive, + EditorialSummary: placesv1.LocalizedText{Text: "Coffee chain known for house blends."}, + RegularOpeningHours: placesv1.OpeningHours{WeekdayDescriptions: []string{ + "Monday: 5:30 AM – 7:00 PM", "Tuesday: 5:30 AM – 7:00 PM", "Wednesday: 5:30 AM – 7:00 PM", + "Thursday: 5:30 AM – 7:00 PM", "Friday: 5:30 AM – 7:00 PM", "Saturday: 6:00 AM – 7:00 PM", + "Sunday: 6:00 AM – 7:00 PM"}}, + Photos: []placesv1.Photo{{Name: "places/ChIJ_test/photos/AT_abc", WidthPx: 4032, HeightPx: 3024}}, + } + + got := MapPlace(in) + + if got.GetID() != "ChIJ_test" { + t.Errorf("ID = %q, want ChIJ_test", got.GetID()) + } + if got.GetName() != "Peet's Coffee" { + t.Errorf("Name = %q, want Peet's Coffee", got.GetName()) + } + // LocationType comes from primaryType, so the record is correctly typed at write + // time. The legacy path stamped the SEARCHED type here, which is how hotels ended + // up labeled fast_food_restaurant. + if got.LocationType != POI.LocationTypeCafe { + t.Errorf("LocationType = %q, want cafe", got.LocationType) + } + if len(got.Types) != 4 || got.Types[0] != "cafe" { + t.Errorf("Types = %v, want Google's full list primary-first", got.Types) + } + if got.Status != POI.Operational { + t.Errorf("Status = %q, want OPERATIONAL", got.Status) + } + if got.PriceLevel != POI.PriceLevelOne { + t.Errorf("PriceLevel = %d, want 1", got.PriceLevel) + } + if got.UserRatingsTotal != 412 { + t.Errorf("UserRatingsTotal = %d, want 412", got.UserRatingsTotal) + } + if got.URL != "https://maps.google.com/?cid=1" { + t.Errorf("URL = %q, want the googleMapsUri", got.URL) + } + if got.Summary != "Coffee chain known for house blends." { + t.Errorf("Summary = %q, want the editorial summary text", got.Summary) + } + // The photo reference is a full resource name now, not a legacy photo_reference. + if got.Photo.Reference != "places/ChIJ_test/photos/AT_abc" { + t.Errorf("Photo.Reference = %q, want the full resource name", got.Photo.Reference) + } + if got.GetHour(POI.DateMonday) != "Monday: 5:30 AM – 7:00 PM" { + t.Errorf("Monday hours = %q, want the Monday description", got.GetHour(POI.DateMonday)) + } +} + +// TestMapPlaceEmptyOptionalFields pins that a sparse response does not panic and +// leaves POI defaults intact. +func TestMapPlaceEmptyOptionalFields(t *testing.T) { + got := MapPlace(placesv1.Place{ + ID: "ChIJ_sparse", + PrimaryType: "restaurant", + DisplayName: placesv1.LocalizedText{Text: "Sparse Diner"}, + Location: placesv1.LatLng{Latitude: 1, Longitude: 2}, + }) + if got.GetID() != "ChIJ_sparse" { + t.Errorf("ID = %q, want ChIJ_sparse", got.GetID()) + } + if got.Photo.Reference != "" { + t.Errorf("Photo.Reference = %q, want empty", got.Photo.Reference) + } + if got.PriceLevel != POI.PriceLevelZero { + t.Errorf("PriceLevel = %d, want 0", got.PriceLevel) + } +} +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `go test ./iowrappers/placesv1/ -run TestSearchNearby -v && go test ./iowrappers/ -run 'TestMapPlace|TestMapPriceLevel' -v` + +Expected: FAIL with `c.SearchNearby undefined` and `undefined: MapPlace`. + +- [ ] **Step 3: Write `search_nearby.go`** + +Create `iowrappers/placesv1/search_nearby.go`: + +```go +package placesv1 + +import ( + "context" + "errors" + "fmt" +) + +// SearchNearbyFieldMask is the exact field set the planner needs. Keep it minimal: +// the New API bills by SKU tier based on which fields are requested, and there is no +// default mask. Every field here replaces something the legacy path needed a separate +// Place Details call to get. +const SearchNearbyFieldMask = "places.id," + + "places.types," + + "places.primaryType," + + "places.displayName," + + "places.formattedAddress," + + "places.adrFormatAddress," + + "places.location," + + "places.rating," + + "places.userRatingCount," + + "places.googleMapsUri," + + "places.businessStatus," + + "places.priceLevel," + + "places.editorialSummary," + + "places.regularOpeningHours.weekdayDescriptions," + + "places.photos" + +// SearchNearbyRequest is the caller-facing shape. RadiusMeters and MaxResultCount are +// clamped to the API's limits rather than rejected, so callers can pass through the +// planner's own wider radius constants unchanged. +type SearchNearbyRequest struct { + IncludedPrimaryTypes []string + ExcludedPrimaryTypes []string + Latitude float64 + Longitude float64 + RadiusMeters float64 + MaxResultCount int + RankPreference RankPreference + LanguageCode string +} + +// SearchNearby calls places:searchNearby. +// +// Unlike the legacy endpoint this filters by PRIMARY type server-side, so results do +// not need client-side reclassification, and an unknown type is rejected with +// INVALID_ARGUMENT instead of silently disabling the filter. +// +// There is no pagination: at most MaxResultCount (cap 20) places come back. +func (c *Client) SearchNearby(ctx context.Context, req SearchNearbyRequest) ([]Place, error) { + if len(req.IncludedPrimaryTypes) == 0 { + return nil, errors.New("placesv1: SearchNearby requires at least one included primary type") + } + radius := req.RadiusMeters + if radius > MaxRadiusMeters { + radius = MaxRadiusMeters + } + if radius <= 0 { + return nil, fmt.Errorf("placesv1: radius must be > 0, got %v", req.RadiusMeters) + } + count := req.MaxResultCount + if count > MaxResultCount || count <= 0 { + count = MaxResultCount + } + rank := req.RankPreference + if rank == "" { + rank = RankPreferenceDistance + } + + body := searchNearbyRequest{ + IncludedPrimaryTypes: req.IncludedPrimaryTypes, + ExcludedPrimaryTypes: req.ExcludedPrimaryTypes, + LocationRestriction: locationRestriction{ + Circle: Circle{ + Center: LatLng{Latitude: req.Latitude, Longitude: req.Longitude}, + Radius: radius, + }, + }, + MaxResultCount: count, + RankPreference: rank, + LanguageCode: req.LanguageCode, + } + + var resp SearchNearbyResponse + if err := c.post(ctx, "/v1/places:searchNearby", SearchNearbyFieldMask, body, &resp); err != nil { + return nil, err + } + return resp.Places, nil +} +``` + +- [ ] **Step 4: Write `places_v1_mapper.go`** + +Create `iowrappers/places_v1_mapper.go`. Adjust the weekday rotation only if Step 0 showed a non-Monday first element: + +```go +package iowrappers + +import ( + "github.com/weihesdlegend/Vacation-planner/POI" + "github.com/weihesdlegend/Vacation-planner/iowrappers/placesv1" +) + +// MapPriceLevel converts the New API's price enum to POI.PriceLevel. +// +// PRICE_LEVEL_UNSPECIFIED maps to 0 deliberately: the legacy path received an absent +// price as integer 0 and bucketed it into placeIDs:eatery:level0, so this preserves +// which bucket an unpriced place lands in. +func MapPriceLevel(pl placesv1.PriceLevel) POI.PriceLevel { + switch pl { + case placesv1.PriceLevelInexpensive: + return POI.PriceLevelOne + case placesv1.PriceLevelModerate: + return POI.PriceLevelTwo + case placesv1.PriceLevelExpensive: + return POI.PriceLevelThree + case placesv1.PriceLevelVeryExpensive: + return POI.PriceLevelFour + case placesv1.PriceLevelFree, placesv1.PriceLevelUnspecified: + return POI.PriceLevelZero + default: + return POI.PriceLevelZero + } +} + +// MapPlace converts a Places API (New) place into the internal POI.Place. +// +// LocationType is set from primaryType — Google's own answer for what the place mainly +// is. The legacy path stamped the SEARCHED type here instead, which is how a hotel +// returned by an unenforceable fast_food_restaurant filter became a labeled eatery. +// +// weekdayDescriptions is Monday-first (verified against the live API on 2026-07-29), +// matching POI.Weekday's DateMonday..DateSunday order. +func MapPlace(p placesv1.Place) POI.Place { + var hours *POI.OpeningHours + if len(p.RegularOpeningHours.WeekdayDescriptions) > 0 { + hours = &POI.OpeningHours{Hours: append([]string(nil), p.RegularOpeningHours.WeekdayDescriptions...)} + } + + var summary *string + if p.EditorialSummary.Text != "" { + text := p.EditorialSummary.Text + summary = &text + } + + place := POI.CreatePlace( + p.DisplayName.Text, + p.AdrFormatAddress, + p.FormattedAddress, + string(p.BusinessStatus), + POI.LocationType(p.PrimaryType), + hours, + p.ID, + int(MapPriceLevel(p.PriceLevel)), + p.Rating, + p.GoogleMapsURI, + nil, // legacy *maps.Photo is not used on this path; set below + p.UserRatingCount, + p.Location.Latitude, + p.Location.Longitude, + summary, + ) + + // Photo.Reference holds the New API resource name ("places/{id}/photos/{ref}"), + // which is NOT a legacy photo_reference. photos_client.go routes on this prefix. + if len(p.Photos) > 0 { + place.Photo = POI.PlacePhoto{ + Reference: p.Photos[0].Name, + Width: p.Photos[0].WidthPx, + Height: p.Photos[0].HeightPx, + } + } + + // Preserve Google's full feature-type list so ReclassifyForCategory and + // PrimaryLocationType keep working on records written by this path. + place.Types = append([]string(nil), p.Types...) + return place +} +``` + +- [ ] **Step 5: Run tests to verify they pass** + +Run: `go build -v . && go test ./iowrappers/placesv1/ -v && go test ./iowrappers/ -run 'TestMapPlace|TestMapPriceLevel' -v` + +Expected: PASS on all tests. + +- [ ] **Step 6: Commit** + +```bash +git add iowrappers/placesv1/search_nearby.go iowrappers/placesv1/search_nearby_test.go iowrappers/places_v1_mapper.go iowrappers/places_v1_mapper_test.go +git commit -m "feat: add searchNearby call and Places API (New) to POI mapping + +LocationType now comes from Google's primaryType rather than the searched +type, so records are correctly labeled at write time. The field mask pulls +opening hours, adr address, maps URI, rating count, editorial summary and +photos in the search response, removing the need for a separate Place +Details call per result." +``` + +--- + +### Task 3: Versioned Redis keys and a New-API `SearchClient` behind a flag + +**Files:** +- Create: `iowrappers/redis_keys.go`, `iowrappers/places_v1_search_client.go` +- Modify: `iowrappers/redis_client.go` (`nearbySearchRedisKeys`, `SetPlacesAddGeoLocations`, `getPlace`/`setPlace` key building), `iowrappers/poi_searcher.go` (client selection) +- Test: `iowrappers/redis_keys_test.go`, `test/redis_client_mocks/v2_keys_test.go` + +**Interfaces:** +- Consumes: `placesv1.Client.SearchNearby`, `iowrappers.MapPlace` (Task 2); `POI.GetPlaceCategory(...) (PlaceCategory, bool)` (prior PR). +- Produces: + - `iowrappers.KeyVersion` type with `KeyVersionLegacy KeyVersion = ""` and `KeyVersionV2 KeyVersion = "v2"` + - `iowrappers.NearbySearchKey(cat POI.PlaceCategory, level POI.PriceLevel, v KeyVersion) string` + - `iowrappers.PlaceDetailsKey(placeID string, v KeyVersion) string` + - `iowrappers.NewPlacesV1SearchClient(apiKey string, mapsClient *MapsClient) *PlacesV1SearchClient` implementing `SearchClient` + - `iowrappers.ActiveKeyVersion() KeyVersion` — reads `PLACES_API_VERSION` + +- [ ] **Step 1: Write the failing test** + +Create `iowrappers/redis_keys_test.go`: + +```go +package iowrappers + +import ( + "strings" + "testing" + + "github.com/weihesdlegend/Vacation-planner/POI" +) + +// PlaceIDsKeyPrefix and PlaceDetailsKeyPrefix come from redis_data_inspections.go, +// PlaceDetailsRedisKeyPrefix from redis_client.go — all the same package, no import needed. + +// TestNearbySearchKeyLegacyUnchanged pins that the legacy key format is byte-identical +// to what production already holds. Any drift orphans the existing cache. +func TestNearbySearchKeyLegacyUnchanged(t *testing.T) { + cases := map[string]string{ + NearbySearchKey(POI.PlaceCategoryEatery, POI.PriceLevelZero, KeyVersionLegacy): "placeIDs:eatery:level0", + NearbySearchKey(POI.PlaceCategoryEatery, POI.PriceLevelThree, KeyVersionLegacy): "placeIDs:eatery:level3", + NearbySearchKey(POI.PlaceCategoryVisit, POI.PriceLevelTwo, KeyVersionLegacy): "placeIDs:visit", + } + for got, want := range cases { + if got != want { + t.Errorf("got %q, want %q", got, want) + } + } +} + +// TestNearbySearchKeyV2Namespaced pins that v2 data never collides with legacy data, +// so cutover and rollback are both non-destructive. +// +// The version is the FIRST segment on purpose. Existing code scans by legacy prefix — +// redis_data_inspections.go:22 scans "place_details*" and PlaceIDsKeyPrefix is +// "placeIDs" — so a suffixed name like "place_details_v2:" or an infixed one like +// "placeIDs:v2:" would be swept up by those scans and double-count or corrupt stats +// and migrations. Leading with "v2:" keeps v2 keys invisible to every legacy scan. +func TestNearbySearchKeyV2Namespaced(t *testing.T) { + cases := map[string]string{ + NearbySearchKey(POI.PlaceCategoryEatery, POI.PriceLevelZero, KeyVersionV2): "v2:placeIDs:eatery:level0", + NearbySearchKey(POI.PlaceCategoryVisit, POI.PriceLevelTwo, KeyVersionV2): "v2:placeIDs:visit", + } + for got, want := range cases { + if got != want { + t.Errorf("got %q, want %q", got, want) + } + } +} + +// TestV2KeysInvisibleToLegacyScans is the regression guard for the collision above. +func TestV2KeysInvisibleToLegacyScans(t *testing.T) { + v2Keys := []string{ + NearbySearchKey(POI.PlaceCategoryEatery, POI.PriceLevelZero, KeyVersionV2), + PlaceDetailsKey("ChIJ_x", KeyVersionV2), + } + legacyScanPrefixes := []string{PlaceDetailsKeyPrefix, PlaceIDsKeyPrefix, PlaceDetailsRedisKeyPrefix} + for _, key := range v2Keys { + for _, prefix := range legacyScanPrefixes { + if strings.HasPrefix(key, prefix) { + t.Errorf("v2 key %q is matched by legacy scan pattern %q*", key, prefix) + } + } + } +} + +func TestPlaceDetailsKey(t *testing.T) { + if got, want := PlaceDetailsKey("ChIJ_x", KeyVersionLegacy), PlaceDetailsRedisKeyPrefix+"ChIJ_x"; got != want { + t.Errorf("legacy details key = %q, want %q", got, want) + } + if got, want := PlaceDetailsKey("ChIJ_x", KeyVersionV2), "v2:place_details:place_ID:ChIJ_x"; got != want { + t.Errorf("v2 details key = %q, want %q", got, want) + } +} + +func TestActiveKeyVersionDefaultsToLegacy(t *testing.T) { + t.Setenv("PLACES_API_VERSION", "") + if got := ActiveKeyVersion(); got != KeyVersionLegacy { + t.Errorf("ActiveKeyVersion() = %q with no env set, want legacy", got) + } + t.Setenv("PLACES_API_VERSION", "new") + if got := ActiveKeyVersion(); got != KeyVersionV2 { + t.Errorf("ActiveKeyVersion() = %q with PLACES_API_VERSION=new, want v2", got) + } + t.Setenv("PLACES_API_VERSION", "legacy") + if got := ActiveKeyVersion(); got != KeyVersionLegacy { + t.Errorf("ActiveKeyVersion() = %q with PLACES_API_VERSION=legacy, want legacy", got) + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `go test ./iowrappers/ -run 'TestNearbySearchKey|TestPlaceDetailsKey|TestActiveKeyVersion' -v` + +Expected: FAIL with `undefined: NearbySearchKey`. + +- [ ] **Step 3: Write `redis_keys.go`** + +```go +package iowrappers + +import ( + "fmt" + "os" + "strings" + + "github.com/weihesdlegend/Vacation-planner/POI" +) + +// KeyVersion namespaces cached place data by the API that produced it. +// +// Places API (New) records are NOT interchangeable with legacy ones: photo references +// change from an opaque photo_reference to a "places/{id}/photos/{ref}" resource name, +// and LocationType comes from primaryType rather than the searched type. Writing both +// under one key would mix formats with no way to tell them apart, so the new path gets +// its own namespace. Cutover and rollback are then a single env-var flip with no deletes. +type KeyVersion string + +const ( + KeyVersionLegacy KeyVersion = "" + KeyVersionV2 KeyVersion = "v2" +) + +// PlaceDetailsV2RedisKeyPrefix mirrors PlaceDetailsRedisKeyPrefix for v2 records. +// +// The version leads the key. Existing code scans by legacy prefix — +// redis_data_inspections.go:22 scans PlaceDetailsKeyPrefix+"*" ("place_details*") and +// PlaceIDsKeyPrefix is "placeIDs" — so a suffixed "place_details_v2:" would be caught +// by those scans and make GetPlaceCountInRedis and the RemovePlaces migration operate +// on v2 records they know nothing about. "v2:" first keeps them cleanly separated. +const PlaceDetailsV2RedisKeyPrefix = "v2:place_details:place_ID:" + +// ActiveKeyVersion reads PLACES_API_VERSION. Anything other than "new" means legacy, +// so an unset or misspelled value fails safe onto the working path. +func ActiveKeyVersion() KeyVersion { + if strings.EqualFold(strings.TrimSpace(os.Getenv("PLACES_API_VERSION")), "new") { + return KeyVersionV2 + } + return KeyVersionLegacy +} + +// NearbySearchKey builds a geo bucket key. The legacy form is byte-identical to +// POI.EncodeNearbySearchRedisKey so existing production data stays addressable, and the +// v2 form leads with the version so legacy "placeIDs*"/"place_details*" scans skip it. +func NearbySearchKey(cat POI.PlaceCategory, level POI.PriceLevel, v KeyVersion) string { + segments := make([]string, 0, 4) + if v != KeyVersionLegacy { + segments = append(segments, string(v)) + } + segments = append(segments, PlaceIDsKeyPrefix, strings.ToLower(string(cat))) + if cat == POI.PlaceCategoryEatery { + segments = append(segments, fmt.Sprintf("level%d", level)) + } + return strings.Join(segments, ":") +} + +// PlaceDetailsKey builds the per-place record key for a version. +func PlaceDetailsKey(placeID string, v KeyVersion) string { + if v == KeyVersionV2 { + return PlaceDetailsV2RedisKeyPrefix + placeID + } + return PlaceDetailsRedisKeyPrefix + placeID +} +``` + +- [ ] **Step 4: Route the Redis read/write paths through the version** + +In `iowrappers/redis_client.go`, add a `keyVersion KeyVersion` field to `RedisClient`, defaulted from `ActiveKeyVersion()` wherever the client is constructed. Then: + +- `nearbySearchRedisKeys` (`:452`): replace each `POI.EncodeNearbySearchRedisKey(cat, lvl)` with `NearbySearchKey(cat, lvl, r.keyVersion)`. This requires making it a method on `*RedisClient`; update its two callers and `iowrappers/nearby_search_keys_test.go` accordingly. +- `SetPlacesAddGeoLocations` (`:222`): use `NearbySearchKey(placeCategory, place.PriceLevel, r.keyVersion)` and `PlaceDetailsKey(place.ID, r.keyVersion)`. +- `getPlace` / `setPlace`: take the version from `r.keyVersion`. + +**Read-compatibility requirement.** Saved trip plans store bare Google place IDs and read them back through `place_details:place_ID:` (`planner/planner.go:797`). Google place IDs are identical across both APIs, but a plan saved before cutover has records only under the legacy key. So the single-record read must fall back: + +```go +// getPlaceAnyVersion reads a place record, preferring the active key version and falling +// back to the other. Saved trip plans reference bare place IDs, and a plan saved before +// the Places API (New) cutover has a record only under the legacy key — so a +// version-strict read would break every existing saved plan. +func (r *RedisClient) getPlaceAnyVersion(ctx context.Context, placeID string) (POI.Place, error) { + place, err := r.getPlaceAtKey(ctx, PlaceDetailsKey(placeID, r.keyVersion)) + if err == nil { + return place, nil + } + other := KeyVersionLegacy + if r.keyVersion == KeyVersionLegacy { + other = KeyVersionV2 + } + return r.getPlaceAtKey(ctx, PlaceDetailsKey(placeID, other)) +} +``` + +Use `getPlaceAnyVersion` for saved-plan reads (`planner/planner.go:797-806`) and the version-strict `getPlace` for geo-bucket reads, where members always come from the matching namespace. + +- [ ] **Step 5: Write the New-API `SearchClient`** + +Create `iowrappers/places_v1_search_client.go`: + +```go +package iowrappers + +import ( + "context" + "fmt" + + "github.com/weihesdlegend/Vacation-planner/POI" + "github.com/weihesdlegend/Vacation-planner/iowrappers/placesv1" +) + +// PlacesV1SearchClient serves category searches from Places API (New). +// +// Geocode and ReverseGeocode delegate to the legacy MapsClient on purpose: the +// Geocoding API (/maps/api/geocode/json) is a separate, non-deprecated API and is not +// part of this migration. +// +// Brand/keyword searches also stay on the legacy client: searchNearby has no keyword +// parameter, and moving them needs places:searchText. Until that lands, a request with +// a Keyword is delegated wholesale. +type PlacesV1SearchClient struct { + places *placesv1.Client + mapsClient *MapsClient + // TypeGroups controls fan-out. One group containing every type = one HTTP call, + // capped at 20 results. Splitting into one group per type restores the legacy + // per-type ceiling at the cost of one call each. Set from Task 5's measurements. + TypeGroups func(POI.PlaceCategory) [][]POI.LocationType +} + +func NewPlacesV1SearchClient(apiKey string, mapsClient *MapsClient) *PlacesV1SearchClient { + return &PlacesV1SearchClient{ + places: placesv1.New(apiKey), + mapsClient: mapsClient, + TypeGroups: SingleGroupTypes, + } +} + +// SingleGroupTypes puts every type of a category into one searchNearby call. +func SingleGroupTypes(cat POI.PlaceCategory) [][]POI.LocationType { + return [][]POI.LocationType{POI.GetPlaceTypes(cat)} +} + +// PerTypeGroups issues one searchNearby call per place type, restoring the legacy +// per-type result ceiling. Costs len(GetPlaceTypes(cat)) calls instead of one. +func PerTypeGroups(cat POI.PlaceCategory) [][]POI.LocationType { + types := POI.GetPlaceTypes(cat) + groups := make([][]POI.LocationType, 0, len(types)) + for _, t := range types { + groups = append(groups, []POI.LocationType{t}) + } + return groups +} + +func (c *PlacesV1SearchClient) Geocode(ctx context.Context, q *GeocodeQuery) (float64, float64, error) { + return c.mapsClient.Geocode(ctx, q) +} + +func (c *PlacesV1SearchClient) ReverseGeocode(ctx context.Context, lat, lng float64) (*GeocodeQuery, error) { + return c.mapsClient.ReverseGeocode(ctx, lat, lng) +} + +func (c *PlacesV1SearchClient) NearbySearch(ctx context.Context, req *PlaceSearchRequest) ([]POI.Place, error) { + if req.Keyword != "" { + // searchNearby has no keyword parameter; brand search still needs searchText. + return c.mapsClient.NearbySearch(ctx, req) + } + + groups := c.TypeGroups(req.PlaceCat) + if len(groups) == 0 { + return nil, fmt.Errorf("no place types for category %q", req.PlaceCat) + } + + seen := make(map[string]bool) + places := make([]POI.Place, 0, len(groups)*placesv1.MaxResultCount) + for _, group := range groups { + types := make([]string, 0, len(group)) + for _, t := range group { + if t != POI.LocationTypeAny { + types = append(types, string(t)) + } + } + if len(types) == 0 { + continue + } + found, err := c.places.SearchNearby(ctx, placesv1.SearchNearbyRequest{ + IncludedPrimaryTypes: types, + Latitude: req.Location.Latitude, + Longitude: req.Location.Longitude, + RadiusMeters: float64(req.Radius), + MaxResultCount: placesv1.MaxResultCount, + RankPreference: placesv1.RankPreferenceDistance, + }) + if err != nil { + // Unlike the legacy API, an unknown type is a hard INVALID_ARGUMENT here. + // Log and continue so one bad type cannot zero out a whole category. + Logger.Error(fmt.Errorf("searchNearby failed for types %v: %w", types, err)) + continue + } + for _, p := range found { + if seen[p.ID] { + continue + } + seen[p.ID] = true + place := MapPlace(p) + // Match the legacy path's filter: places with no ratings are not useful. + if place.UserRatingsTotal == 0 { + continue + } + places = append(places, place) + } + } + return places, nil +} +``` + +- [ ] **Step 6: Select the client from the flag** + +In `iowrappers/poi_searcher.go`, wherever `PoiSearcher` is constructed with its `SearchClient`, choose based on the flag: + +```go + if ActiveKeyVersion() == KeyVersionV2 { + Logger.Info("PLACES_API_VERSION=new: serving category searches from Places API (New)") + searcher.searchClient = NewPlacesV1SearchClient(apiKey, mapsClient) + } else { + searcher.searchClient = mapsClient + } +``` + +- [ ] **Step 7: Run the full suite** + +Run: `go build -v . && go test ./... 2>&1 | tail -25` + +Expected: PASS. With `PLACES_API_VERSION` unset, every existing test exercises the unchanged legacy path. + +- [ ] **Step 8: Commit** + +```bash +git add iowrappers/redis_keys.go iowrappers/redis_keys_test.go iowrappers/places_v1_search_client.go iowrappers/redis_client.go iowrappers/poi_searcher.go planner/planner.go +git commit -m "feat: add PLACES_API_VERSION flag and v2-namespaced cache keys + +New API records are not interchangeable with legacy ones (photo resource +names, primaryType-derived LocationType), so they get their own key +namespace. Cutover and rollback are one env-var flip with no deletes. +Saved-plan reads fall back across versions since place IDs are shared." +``` + +--- + +### Task 4: Photos via the New media endpoint + +**Files:** +- Create: `iowrappers/placesv1/photo.go` +- Modify: `iowrappers/photos_client.go` +- Test: `iowrappers/placesv1/photo_test.go` + +**Interfaces:** +- Consumes: `placesv1.Client` (Task 1). +- Produces: `(*placesv1.Client).PhotoMediaURL(photoName string, maxWidthPx int) (string, error)` and `(*placesv1.Client).FetchPhoto(ctx context.Context, photoName string, maxWidthPx int) ([]byte, string, error)` returning bytes and content type. + +- [ ] **Step 1: Write the failing test** + +Create `iowrappers/placesv1/photo_test.go`: + +```go +package placesv1 + +import ( + "context" + "net/http" + "net/http/httptest" + "strings" + "testing" +) + +func TestPhotoMediaURL(t *testing.T) { + c := New("test-key") + got, err := c.PhotoMediaURL("places/ChIJ_x/photos/AT_abc", 400) + if err != nil { + t.Fatalf("PhotoMediaURL error: %v", err) + } + for _, want := range []string{ + "https://places.googleapis.com/v1/places/ChIJ_x/photos/AT_abc/media", + "maxWidthPx=400", + "key=test-key", + } { + if !strings.Contains(got, want) { + t.Errorf("URL %q missing %q", got, want) + } + } +} + +// TestPhotoMediaURLRejectsLegacyReference pins that an opaque legacy photo_reference +// cannot be passed to the New media endpoint. Cached legacy references are not +// convertible, which is why photos_client.go routes on the "places/" prefix. +func TestPhotoMediaURLRejectsLegacyReference(t *testing.T) { + c := New("test-key") + if _, err := c.PhotoMediaURL("ATtYBwLQ_legacy_opaque_ref", 400); err == nil { + t.Error("PhotoMediaURL accepted a legacy photo_reference, want error") + } +} + +func TestFetchPhotoFollowsRedirectAndReturnsBytes(t *testing.T) { + image := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "image/jpeg") + _, _ = w.Write([]byte{0xFF, 0xD8, 0xFF, 0xE0}) + })) + defer image.Close() + + api := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + http.Redirect(w, r, image.URL, http.StatusFound) + })) + defer api.Close() + + c := New("test-key", WithBaseURL(api.URL)) + data, contentType, err := c.FetchPhoto(context.Background(), "places/ChIJ_x/photos/AT_abc", 400) + if err != nil { + t.Fatalf("FetchPhoto error: %v", err) + } + if contentType != "image/jpeg" { + t.Errorf("contentType = %q, want image/jpeg", contentType) + } + if len(data) != 4 { + t.Errorf("got %d bytes, want 4", len(data)) + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `go test ./iowrappers/placesv1/ -run 'TestPhoto|TestFetchPhoto' -v` + +Expected: FAIL with `c.PhotoMediaURL undefined`. + +- [ ] **Step 3: Write `photo.go`** + +```go +package placesv1 + +import ( + "context" + "fmt" + "io" + "net/http" + "net/url" + "strings" +) + +// PhotoNamePrefix marks a New API photo resource name. Legacy photo_reference strings +// are opaque and have no prefix, which makes this a reliable discriminator for cached +// records written by either API. +const PhotoNamePrefix = "places/" + +// PhotoMediaURL builds the media URL for a photo resource name obtained from a search +// or details response, e.g. "places/{placeID}/photos/{photoResource}". +// +// A legacy photo_reference is NOT convertible to this form — the only way to get a +// usable reference for a place cached under the legacy API is to re-fetch the place. +func (c *Client) PhotoMediaURL(photoName string, maxWidthPx int) (string, error) { + if !strings.HasPrefix(photoName, PhotoNamePrefix) { + return "", fmt.Errorf("placesv1: %q is not a photo resource name (want %s...); legacy photo_reference values are not convertible", photoName, PhotoNamePrefix) + } + if maxWidthPx < 1 || maxWidthPx > 4800 { + return "", fmt.Errorf("placesv1: maxWidthPx must be 1..4800, got %d", maxWidthPx) + } + q := url.Values{} + q.Set("maxWidthPx", fmt.Sprint(maxWidthPx)) + q.Set("key", c.apiKey) + return fmt.Sprintf("%s/v1/%s/media?%s", c.baseURL, photoName, q.Encode()), nil +} + +// FetchPhoto downloads the image bytes. The endpoint answers with an HTTP redirect to +// the image by default, which http.Client follows. +func (c *Client) FetchPhoto(ctx context.Context, photoName string, maxWidthPx int) ([]byte, string, error) { + mediaURL, err := c.PhotoMediaURL(photoName, maxWidthPx) + if err != nil { + return nil, "", err + } + req, err := http.NewRequestWithContext(ctx, http.MethodGet, mediaURL, nil) + if err != nil { + return nil, "", fmt.Errorf("building photo request: %w", err) + } + resp, err := c.http.Do(req) + if err != nil { + return nil, "", fmt.Errorf("fetching photo %s: %w", photoName, err) + } + defer func() { _ = resp.Body.Close() }() + + if resp.StatusCode != http.StatusOK { + raw, _ := io.ReadAll(resp.Body) + return nil, "", &APIError{Code: resp.StatusCode, Status: resp.Status, Message: string(raw)} + } + data, err := io.ReadAll(resp.Body) + if err != nil { + return nil, "", fmt.Errorf("reading photo %s: %w", photoName, err) + } + return data, resp.Header.Get("Content-Type"), nil +} +``` + +- [ ] **Step 4: Route by reference format in `photos_client.go`** + +`MapsPhotoClient.placeImage` (`iowrappers/photos_client.go:183`) currently always calls the legacy `client.PlacePhoto`. Route on the prefix instead, because the details keyspace holds both formats during and after cutover — brand/keyword places stay on the legacy path in this PR: + +```go +func (c *MapsPhotoClient) placeImage(ctx context.Context, ref string) (image.Image, error) { + // Acquire semaphore for API rate limiting + c.mapsClient.apiSemaphore <- struct{}{} + defer func() { <-c.mapsClient.apiSemaphore }() + + // A "places/..." reference is a Places API (New) resource name and must go to the + // New media endpoint. Legacy opaque photo_reference values stay on the SDK. Both + // formats coexist: brand/keyword searches still write legacy references. + if strings.HasPrefix(ref, placesv1.PhotoNamePrefix) { + data, contentType, err := c.placesV1.FetchPhoto(ctx, ref, 400) + if err != nil { + return nil, err + } + Logger.Debugf("photo response content type is: %s", contentType) + switch contentType { + case "image/png": + return png.Decode(bytes.NewReader(data)) + case "image/jpeg": + return jpeg.Decode(bytes.NewReader(data)) + default: + return nil, fmt.Errorf(UnknownImageFormat+": %s", contentType) + } + } + + resp, err := c.mapsClient.client.PlacePhoto(ctx, &maps.PlacePhotoRequest{PhotoReference: ref, MaxWidth: 400}) + if err != nil { + return nil, err + } + Logger.Debugf("photo response content type is: %s", resp.ContentType) + switch resp.ContentType { + case "image/png": + return png.Decode(resp.Data) + case "image/jpeg": + return resp.Image() + default: + return nil, fmt.Errorf(UnknownImageFormat+": %s", resp.ContentType) + } +} +``` + +Add a `placesV1 *placesv1.Client` field to `MapsPhotoClient` and initialize it in `CreatePhotoClient` (`iowrappers/photos_client.go:55`) from the same API key. + +Also update the stale-reference recovery branch in `GetPhotoURL` (`:131-160`): when the active version is v2, re-fetching a place's photo must come from a New API lookup rather than `PlaceDetailedSearch`. The simplest correct behavior for this PR is to skip recovery on v2 records and let the next cache refresh repopulate: + +```go + if strings.HasPrefix(err.Error(), UnknownImageFormat) { + if strings.HasPrefix(photoRef, placesv1.PhotoNamePrefix) { + // v2 records carry a resource name that cannot be repaired by a legacy + // Place Details call. Let the 14-day cache refresh replace it. + return "", fmt.Errorf("stale Places API (New) photo reference for place %s: %w", placeId, err) + } + // ... existing legacy recovery path unchanged ... + } +``` + +- [ ] **Step 5: Run tests to verify they pass** + +Run: `go build -v . && go test ./iowrappers/... -v 2>&1 | tail -25` + +Expected: PASS, including the existing `photos_client` tests on the legacy branch. + +- [ ] **Step 6: Commit** + +```bash +git add iowrappers/placesv1/photo.go iowrappers/placesv1/photo_test.go iowrappers/photos_client.go +git commit -m "feat: fetch photos from the Places API (New) media endpoint + +Route on the reference format: 'places/{id}/photos/{ref}' resource names +go to /v1/{name}/media, opaque legacy photo_reference values stay on the +SDK. Both formats coexist because brand/keyword search is still legacy." +``` + +--- + +### Task 5: Measure coverage, then cut over + +The one hard tradeoff in this design is the 20-result cap. Measure it on production data before flipping the flag. Do not skip this task, and do not tune `TypeGroups` by guesswork. + +**Files:** +- Create: `iowrappers/coverage_compare_test.go` (build-tagged, opt-in) + +- [ ] **Step 1: Write the comparison harness** + +Create `iowrappers/coverage_compare_test.go`. The build tag keeps it out of CI, since it makes real billed API calls: + +```go +//go:build coverage_compare + +package iowrappers + +import ( + "context" + "os" + "testing" + + "github.com/weihesdlegend/Vacation-planner/POI" +) + +// TestCompareLegacyVsNewCoverage reports how many distinct places each API returns for +// the same request, so the 20-result searchNearby cap can be judged on real data rather +// than assumed acceptable. +// +// Run with: +// GOOGLE_MAPS_API_KEY=... go test -tags coverage_compare ./iowrappers/ \ +// -run TestCompareLegacyVsNewCoverage -v +func TestCompareLegacyVsNewCoverage(t *testing.T) { + apiKey := os.Getenv("GOOGLE_MAPS_API_KEY") + if apiKey == "" { + t.Skip("GOOGLE_MAPS_API_KEY not set") + } + if err := CreateLogger(); err != nil { + t.Fatalf("CreateLogger: %v", err) + } + mapsClient := CreateMapsClient(apiKey) + newClient := NewPlacesV1SearchClient(apiKey, mapsClient) + + locations := map[string]POI.Location{ + "los altos (dense suburb)": {Latitude: 37.38006, Longitude: -122.11612, City: "Los Altos", AdminAreaLevelOne: "CA", Country: "United States"}, + "manhattan (very dense)": {Latitude: 40.7580, Longitude: -73.9855, City: "New York", AdminAreaLevelOne: "NY", Country: "United States"}, + "bozeman (sparse)": {Latitude: 45.6796, Longitude: -111.0471, City: "Bozeman", AdminAreaLevelOne: "MT", Country: "United States"}, + } + categories := []POI.PlaceCategory{POI.PlaceCategoryEatery, POI.PlaceCategoryVisit, POI.PlaceCategoryShopping} + + for name, loc := range locations { + for _, cat := range categories { + legacyReq := &PlaceSearchRequest{ + Location: loc, PlaceCat: cat, Radius: ColdStartSearchRadius, + MinNumResults: 40, PriceLevel: POI.PriceLevelDefault, + BusinessStatus: POI.Operational, AllPriceLevels: cat == POI.PlaceCategoryEatery, + } + legacy, err := mapsClient.NearbySearch(context.Background(), legacyReq) + if err != nil { + t.Errorf("%s/%s legacy: %v", name, cat, err) + continue + } + + singleReq := *legacyReq + newClient.TypeGroups = SingleGroupTypes + single, err := newClient.NearbySearch(context.Background(), &singleReq) + if err != nil { + t.Errorf("%s/%s new(single): %v", name, cat, err) + continue + } + + perTypeReq := *legacyReq + newClient.TypeGroups = PerTypeGroups + perType, err := newClient.NearbySearch(context.Background(), &perTypeReq) + if err != nil { + t.Errorf("%s/%s new(per-type): %v", name, cat, err) + continue + } + + // After ReclassifyForCategory, which is what actually reaches the response. + t.Logf("%-26s %-9s legacy=%3d (kept %3d) new-single=%3d new-per-type=%3d", + name, cat, len(legacy), countKept(legacy, cat), len(single), len(perType)) + } + } +} + +func countKept(places []POI.Place, cat POI.PlaceCategory) int { + kept := 0 + for _, p := range places { + if _, keep := POI.ReclassifyForCategory(p, cat); keep { + kept++ + } + } + return kept +} +``` + +- [ ] **Step 2: Run the comparison and record the numbers** + +Run: + +```bash +GOOGLE_MAPS_API_KEY=$GOOGLE_MAPS_API_KEY go test -tags coverage_compare ./iowrappers/ \ + -run TestCompareLegacyVsNewCoverage -v 2>&1 | tee /tmp/coverage-compare.txt +``` + +Compare `legacy (kept N)` — the count that actually survives to the response today — against `new-single`. The kept count is the honest baseline, because the legacy raw count includes results `ReclassifyForCategory` discards. + +- [ ] **Step 3: Choose the fan-out and record why** + +Decision rule, to be written into the PR description with the measured numbers: + +- If `new-single` >= the legacy kept count for every location and category, keep `SingleGroupTypes`. +- If any sparse or dense case regresses materially, set the default to `PerTypeGroups` for the affected categories. Encode it explicitly rather than leaving the default implicit: + +```go +// TypeGroupsForCategory splits Eatery across per-type calls because a single +// 20-result searchNearby underperformed the legacy kept count in dense areas +// (see docs/superpowers/plans/ measurements, 2026-07-29). Other categories fit in one call. +func TypeGroupsForCategory(cat POI.PlaceCategory) [][]POI.LocationType { + if cat == POI.PlaceCategoryEatery { + return PerTypeGroups(cat) + } + return SingleGroupTypes(cat) +} +``` + +- [ ] **Step 4: Commit the harness and the decision** + +```bash +git add iowrappers/coverage_compare_test.go iowrappers/places_v1_search_client.go +git commit -m "test: add legacy vs new coverage comparison harness + +searchNearby caps at 20 results with no pagination, so the fan-out choice +has to be measured against production data rather than assumed. Build-tagged +out of CI because it makes real billed API calls." +``` + +- [ ] **Step 5: Cut over in staging, then production** + +```bash +# 1. Enable on a staging/review app first. +heroku config:set PLACES_API_VERSION=new -a + +# 2. Exercise a cold search per category and confirm v2 keys appear. +redis-cli --scan --pattern 'v2:placeIDs:*' | head + +# 3. Confirm correct typing — the whole point of the migration. +# No record in a v2 eatery bucket should have a lodging primary type. +curl -s -H "Authorization: Bearer $ADMIN_JWT" \ + "https:///v1/migrate/reclassify-buckets?category=Eatery" | jq '.report.misclassified' +# Expected: 0 + +# 4. Production. +heroku config:set PLACES_API_VERSION=new -a best-vacation-planner + +# Rollback at any point, no deletes, legacy cache still warm: +heroku config:set PLACES_API_VERSION=legacy -a best-vacation-planner +``` + +--- + +### Task 6: Re-add `fast_food_restaurant` and `food_court` + +Only after Task 5's cutover is stable in production. This is the original intent of commit `8644199`, now actually achievable. + +**Files:** +- Modify: `POI/categories.go` +- Test: `test/place_category_test.go` + +- [ ] **Step 1: Write the failing test** + +In `test/place_category_test.go`, extend the Eatery entry of `TestGetPlaceTypesByCategory` and add: + +```go +// TestFastFoodTypesRoundTrip pins that the Places API (New) Table A eatery types are +// mapped in BOTH directions. Commit 8644199 added them to GetPlaceTypes only; the +// GetPlaceCategory default silently absorbed them into Eatery and the round-trip guard +// could not fail. Both directions must be explicit now. +func TestFastFoodTypesRoundTrip(t *testing.T) { + for _, placeType := range []POI.LocationType{POI.LocationTypeFastFood, POI.LocationTypeFoodCourt} { + got, ok := POI.GetPlaceCategory(placeType) + if !ok { + t.Errorf("GetPlaceCategory(%q) returned ok=false, want Eatery", placeType) + continue + } + if got != POI.PlaceCategoryEatery { + t.Errorf("GetPlaceCategory(%q) = %q, want Eatery", placeType, got) + } + } + types := POI.GetPlaceTypes(POI.PlaceCategoryEatery) + for _, want := range []POI.LocationType{POI.LocationTypeFastFood, POI.LocationTypeFoodCourt} { + found := false + for _, t2 := range types { + if t2 == want { + found = true + } + } + if !found { + t.Errorf("GetPlaceTypes(Eatery) = %v, missing %q", types, want) + } + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `go test ./test/ -run TestFastFoodTypesRoundTrip -v` + +Expected: FAIL with `undefined: POI.LocationTypeFastFood`. + +- [ ] **Step 3: Add the constants and both mappings** + +In `POI/categories.go`, restore the constants: + +```go + // LocationTypeFastFood and LocationTypeFoodCourt are Places API (New) Table A types. + // They only work when PLACES_API_VERSION=new: the legacy Nearby Search does not + // define them, ignores the ?type= filter rather than erroring, and its response + // types[] never contains them, so nothing can classify a place as either one. + LocationTypeFastFood = LocationType("fast_food_restaurant") + LocationTypeFoodCourt = LocationType("food_court") +``` + +Add them to **both** functions — this is the invariant that was violated the first time: + +```go + // in GetPlaceCategory + case LocationTypeCafe, LocationTypeRestaurant, LocationTypeBar, LocationTypeBakery, + LocationTypeMealTakeaway, LocationTypeFastFood, LocationTypeFoodCourt: + return PlaceCategoryEatery, true + + // in GetPlaceTypes + case PlaceCategoryEatery: + placeTypes = append(placeTypes, + []LocationType{LocationTypeCafe, LocationTypeRestaurant, LocationTypeBar, + LocationTypeBakery, LocationTypeMealTakeaway, LocationTypeFastFood, + LocationTypeFoodCourt}...) +``` + +- [ ] **Step 4: Guard the legacy path against them** + +The legacy `CreateMapSearchRequest` validation added in the prior PR now rejects these two types, which is correct — but it would log an error on every legacy search. Make the legacy client skip them quietly instead, in `extensiveNearbySearch` where `placeTypes` is built: + +```go + placeTypes := POI.GetPlaceTypes(request.PlaceCat) // get place types in a category + // Drop types the legacy Places API does not define. They are searched only when + // PLACES_API_VERSION=new; sending them here would spend a call whose type filter + // Google silently ignores. + placeTypes = Filter(placeTypes, func(t POI.LocationType) bool { + if t == POI.LocationTypeAny { + return true + } + _, err := maps.ParsePlaceType(string(t)) + return err == nil + }) +``` + +- [ ] **Step 5: Run the full suite** + +Run: `go build -v . && go test ./... 2>&1 | tail -20` + +Expected: PASS, including `TestPlaceCategoryRoundTrip` and the legacy `CreateMapSearchRequest` tests. + +- [ ] **Step 6: Verify against the live New API** + +```bash +curl -s -X POST 'https://places.googleapis.com/v1/places:searchNearby' \ + -H "X-Goog-Api-Key: $GOOGLE_MAPS_API_KEY" \ + -H 'X-Goog-FieldMask: places.displayName,places.primaryType,places.types' \ + -H 'Content-Type: application/json' \ + -d '{"includedPrimaryTypes":["fast_food_restaurant","food_court"],"maxResultCount":20, + "rankPreference":"DISTANCE", + "locationRestriction":{"circle":{"center":{"latitude":37.38006,"longitude":-122.11612},"radius":8000}}}' \ + | jq '.places[] | {name: .displayName.text, primaryType}' +``` + +Expected: every `primaryType` is `fast_food_restaurant` or `food_court`, and no hotels appear. That is the concrete difference from the legacy behavior that started this work. + +- [ ] **Step 7: Commit** + +```bash +git add POI/categories.go iowrappers/nearby_search.go test/place_category_test.go +git commit -m "feat: search fast_food_restaurant and food_court on the New API + +These Table A types are filtered server-side by includedPrimaryTypes, so +they now return correctly typed places instead of prominence-ranked +establishments. Mapped in both GetPlaceTypes and GetPlaceCategory, and +filtered out of the legacy path where they are undefined." +``` + +--- + +## Verification checklist + +- [ ] `go build -v .` and `go test -v ./...` pass with `PLACES_API_VERSION` unset (legacy path untouched). +- [ ] `go test -v ./...` passes with `PLACES_API_VERSION=new`. +- [ ] `grep -rn 'places.googleapis.com' --include='*.go' iowrappers/ | grep -v _test` shows requests only from `iowrappers/placesv1`. +- [ ] Coverage comparison numbers recorded in the PR description, with the `TypeGroups` choice justified by them. +- [ ] After staging cutover, `reclassify-buckets?category=Eatery` reports `misclassified: 0` against v2 buckets. +- [ ] A saved trip plan created before cutover still renders (exercises `getPlaceAnyVersion`). +- [ ] Photos load for both a v2 place and a legacy brand-search place. +- [ ] Rollback tested: set `PLACES_API_VERSION=legacy`, confirm legacy results still serve from the warm legacy cache. + +## Deliberately out of scope + +- **Brand/keyword search.** `searchNearby` has no keyword parameter; this needs `places:searchText` with `locationBias`, plus `MatchesBrandName`/`StrictNameMatch` re-tested against relevance-ranked results. `PlacesV1SearchClient.NearbySearch` delegates keyword requests to the legacy client until then. +- **Geocoding and ReverseGeocode.** `/maps/api/geocode/json` is the Geocoding API, not Places, and is not deprecated. It stays on `googlemaps.github.io/maps` v1.7.0. +- **Place Details.** Once search returns the full field set, the only remaining legacy Place Details caller is the stale-photo recovery path and the `data_migrations.go` backfills. Retire those separately. +- **Removing `googlemaps.github.io/maps`.** Cannot happen while Geocoding and brand search remain on it. + +## Risk notes + +- **Billing.** The New API bills per SKU tier by field mask. `SearchNearbyFieldMask` requests Enterprise-tier fields (`regularOpeningHours`, `editorialSummary`). Per-search cost may rise even as call count falls sharply — check the first days of billing after cutover rather than assuming the call-count reduction dominates. +- **The 20-result cap is the one-way door in this design.** Task 5 exists specifically to size it. If coverage proves unacceptable even with `PerTypeGroups`, the fallback is `places:searchText` per type, which paginates to 60 — a larger change that would supersede Task 3's client. +- **Legacy is not dead, but it is frozen.** Legacy Places became unavailable to Cloud projects created after 2025-03-01 and receives no fixes; Google has announced no turn-down date and promises 12 months' notice. The concrete risk is that recreating or swapping the GCP project behind `GOOGLE_MAPS_API_KEY` would break the legacy path outright — which is also why the brand-search path should not stay on it indefinitely. \ No newline at end of file From a2ded844762fc59f951b812bf0f4d42ccdad9484 Mon Sep 17 00:00:00 2001 From: tim-eternos Date: Wed, 29 Jul 2026 23:08:08 -0700 Subject: [PATCH 02/12] fix: stop mapping unknown place types to Eatery GetPlaceCategory had a default branch returning Eatery, which silently absorbed any place type the legacy Nearby Search does not understand. That made TestPlaceCategoryRoundTrip un-failable, so fast_food_restaurant and food_court (Places API (New) Table A types, absent from the v1.7.0 SDK) were added to GetPlaceTypes(Eatery). Google ignored the unenforceable type filter and returned prominence-ranked establishments, which were stamped with the queried type and written into placeIDs:eatery:level* as hotels. Return (PlaceCategory, bool) so the compiler forces every caller to handle an unmapped type, refuse the geo-bucket write instead of guessing, and remove the two types. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01DjdGnZM6cUBeg6jWwoQfU3 --- POI/categories.go | 33 +++++++++++++---------- iowrappers/redis_client.go | 18 +++++++++++-- planner/planner.go | 9 ++++++- test/place_category_test.go | 53 +++++++++++++++++++++++++++++++++++-- 4 files changed, 94 insertions(+), 19 deletions(-) diff --git a/POI/categories.go b/POI/categories.go index ec42ed1c..e8c29f26 100644 --- a/POI/categories.go +++ b/POI/categories.go @@ -38,8 +38,6 @@ const ( LocationTypeBar = LocationType("bar") LocationTypeBakery = LocationType("bakery") LocationTypeMealTakeaway = LocationType("meal_takeaway") - LocationTypeFastFood = LocationType("fast_food_restaurant") - LocationTypeFoodCourt = LocationType("food_court") LocationTypeMuseum = LocationType("museum") LocationTypeGallery = LocationType("art_gallery") LocationTypeAmusementPark = LocationType("amusement_park") @@ -58,26 +56,33 @@ const ( LocationTypePharmacy = LocationType("pharmacy") ) -// GetPlaceCategory maps a Google Maps place type back to its category. It is the inverse of -// GetPlaceTypes and MUST stay consistent with it: the nearby-search cache writes each place -// under EncodeNearbySearchRedisKey(GetPlaceCategory(place.LocationType), ...), so a type that +// GetPlaceCategory maps a Google Maps place type back to its category, reporting whether +// the type is mapped at all. It is the inverse of GetPlaceTypes and MUST stay consistent +// with it: the nearby-search cache writes each place under +// EncodeNearbySearchRedisKey(GetPlaceCategory(place.LocationType), ...), so a type that // resolves to a different category than the one it was searched under would never cache-hit. -func GetPlaceCategory(placeType LocationType) (placeCategory PlaceCategory) { +// +// It deliberately has NO default category. An earlier version defaulted to Eatery, which +// silently absorbed place types the legacy Nearby Search does not understand — two +// Places-API-(New)-only types ("fast_food_restaurant", "food_court") were added to +// GetPlaceTypes(Eatery), Google ignored the unenforceable filter, and prominence-ranked +// hotels were written into the eatery geo buckets. Returning ok=false forces every caller +// to decide what an unmapped type means, and makes TestPlaceCategoryRoundTrip able to fail. +func GetPlaceCategory(placeType LocationType) (PlaceCategory, bool) { switch placeType { case LocationTypePark, LocationTypeAmusementPark, LocationTypeGallery, LocationTypeMuseum: - placeCategory = PlaceCategoryVisit + return PlaceCategoryVisit, true case LocationTypeCafe, LocationTypeRestaurant, LocationTypeBar, LocationTypeBakery, LocationTypeMealTakeaway: - placeCategory = PlaceCategoryEatery + return PlaceCategoryEatery, true case LocationTypeShoppingMall, LocationTypeDepartmentStore, LocationTypeSupermarket, LocationTypeClothingStore, LocationTypeStore: - placeCategory = PlaceCategoryShopping + return PlaceCategoryShopping, true case LocationTypeLodging: - placeCategory = PlaceCategoryLodging + return PlaceCategoryLodging, true case LocationTypeGym, LocationTypeSpa, LocationTypePharmacy: - placeCategory = PlaceCategoryWellness + return PlaceCategoryWellness, true default: - placeCategory = PlaceCategoryEatery + return PlaceCategory(""), false } - return } // GetPlaceTypes returns a set of types defined in Google Maps API given a location type @@ -88,7 +93,7 @@ func GetPlaceTypes(placeCat PlaceCategory) (placeTypes []LocationType) { []LocationType{LocationTypePark, LocationTypeAmusementPark, LocationTypeGallery, LocationTypeMuseum}...) case PlaceCategoryEatery: placeTypes = append(placeTypes, - []LocationType{LocationTypeCafe, LocationTypeRestaurant, LocationTypeBar, LocationTypeBakery, LocationTypeMealTakeaway, LocationTypeFastFood, LocationTypeFoodCourt}...) + []LocationType{LocationTypeCafe, LocationTypeRestaurant, LocationTypeBar, LocationTypeBakery, LocationTypeMealTakeaway}...) case PlaceCategoryShopping: placeTypes = append(placeTypes, []LocationType{LocationTypeShoppingMall, LocationTypeDepartmentStore, LocationTypeSupermarket, LocationTypeClothingStore, LocationTypeStore}...) diff --git a/iowrappers/redis_client.go b/iowrappers/redis_client.go index fa5cd9e9..4505e2e8 100644 --- a/iowrappers/redis_client.go +++ b/iowrappers/redis_client.go @@ -179,7 +179,13 @@ func (r *RedisClient) StorePlacesForLocation(context context.Context, geocodeInS latLng, _ := utils.ParseLocation(geocodeInString) lat, lng := latLng[0], latLng[1] for _, place := range places { - sortedSetKey := strings.Join([]string{geocodeInString, string(POI.GetPlaceCategory(place.LocationType))}, "_") + placeCategory, ok := POI.GetPlaceCategory(place.LocationType) + if !ok { + Logger.Errorf("StorePlacesForLocation: place %s has unmapped location type %q, skipping", + place.ID, place.LocationType) + continue + } + sortedSetKey := strings.Join([]string{geocodeInString, string(placeCategory)}, "_") dist := utils.HaversineDist([]float64{lat, lng}, []float64{place.GetLocation().Latitude, place.GetLocation().Longitude}) _, err := client.ZAdd(context, sortedSetKey, redis.Z{Score: dist, Member: place.ID}).Result() if err != nil { @@ -219,7 +225,15 @@ func (r *RedisClient) SetPlacesAddGeoLocations(c context.Context, places []POI.P defer wg.Done() _, err := r.Get().Pipelined(c, func(pipe redis.Pipeliner) error { for _, place := range placeBatch { - placeCategory := POI.GetPlaceCategory(place.LocationType) + placeCategory, ok := POI.GetPlaceCategory(place.LocationType) + if !ok { + // Refuse to guess a bucket. A place whose type maps to no category + // came from a search whose type filter Google did not enforce, so + // writing it would poison whichever bucket we picked. + Logger.Errorf("SetPlacesAddGeoLocations: place %s has unmapped location type %q, skipping geo bucket write", + place.ID, place.LocationType) + continue + } geoLocation := &redis.GeoLocation{ Name: place.ID, Latitude: place.GetLocation().Latitude, diff --git a/planner/planner.go b/planner/planner.go index 623e1b52..285635f9 100644 --- a/planner/planner.go +++ b/planner/planner.go @@ -807,7 +807,14 @@ func (p *MyPlanner) getUserSavedPlanDetails(ctx *gin.Context) { } resp.LatLongs[i] = [2]float64{place.Location.Latitude, place.Location.Longitude} resp.ShownActive[i] = i == 0 - resp.PlaceCategories[i] = POI.GetPlaceCategory(place.LocationType) + // Saved plans can contain older cached records, including brand-search places + // written with an empty LocationType. Preserve the historical Eatery default for + // display only — the write path (redis_client.go) is where guessing is unsafe. + if placeCategory, ok := POI.GetPlaceCategory(place.LocationType); ok { + resp.PlaceCategories[i] = placeCategory + } else { + resp.PlaceCategories[i] = POI.PlaceCategoryEatery + } details, err := p.placeDetailsResp(ctx, place) if err != nil { diff --git a/test/place_category_test.go b/test/place_category_test.go index 5bd42249..82c8b61b 100644 --- a/test/place_category_test.go +++ b/test/place_category_test.go @@ -12,7 +12,6 @@ func TestGetPlaceTypesByCategory(t *testing.T) { POI.PlaceCategoryEatery: { POI.LocationTypeCafe, POI.LocationTypeRestaurant, POI.LocationTypeBar, POI.LocationTypeBakery, POI.LocationTypeMealTakeaway, - POI.LocationTypeFastFood, POI.LocationTypeFoodCourt, }, POI.PlaceCategoryShopping: { POI.LocationTypeShoppingMall, POI.LocationTypeDepartmentStore, @@ -45,13 +44,63 @@ func TestPlaceCategoryRoundTrip(t *testing.T) { } for _, category := range categories { for _, placeType := range POI.GetPlaceTypes(category) { - if got := POI.GetPlaceCategory(placeType); got != category { + got, ok := POI.GetPlaceCategory(placeType) + if !ok { + t.Errorf("round-trip broken: GetPlaceCategory(%q) has no category, want %q", placeType, category) + continue + } + if got != category { t.Errorf("round-trip broken: GetPlaceCategory(%q) = %q, want %q", placeType, got, category) } } } } +// TestGetPlaceCategoryRejectsUnknownTypes pins the fix for the fast_food_restaurant +// incident: GetPlaceCategory must NOT silently absorb unmapped types into Eatery. +// A default-to-Eatery branch made TestPlaceCategoryRoundTrip un-failable, so two +// Places-API-(New)-only types were added to GetPlaceTypes(Eatery) and hotels were +// written into the eatery geo buckets. +func TestGetPlaceCategoryRejectsUnknownTypes(t *testing.T) { + unknown := []POI.LocationType{ + POI.LocationType("fast_food_restaurant"), + POI.LocationType("food_court"), + POI.LocationType("lodging_but_not_really"), + POI.LocationType(""), + } + for _, placeType := range unknown { + if got, ok := POI.GetPlaceCategory(placeType); ok { + t.Errorf("GetPlaceCategory(%q) = (%q, true), want ok=false", placeType, got) + } + } +} + +// TestGetPlaceCategoryKnownTypes pins that every mapped type still resolves. +func TestGetPlaceCategoryKnownTypes(t *testing.T) { + cases := map[POI.LocationType]POI.PlaceCategory{ + POI.LocationTypeCafe: POI.PlaceCategoryEatery, + POI.LocationTypeRestaurant: POI.PlaceCategoryEatery, + POI.LocationTypeBar: POI.PlaceCategoryEatery, + POI.LocationTypeBakery: POI.PlaceCategoryEatery, + POI.LocationTypeMealTakeaway: POI.PlaceCategoryEatery, + POI.LocationTypePark: POI.PlaceCategoryVisit, + POI.LocationTypeMuseum: POI.PlaceCategoryVisit, + POI.LocationTypeStore: POI.PlaceCategoryShopping, + POI.LocationTypeLodging: POI.PlaceCategoryLodging, + POI.LocationTypeGym: POI.PlaceCategoryWellness, + } + for placeType, want := range cases { + got, ok := POI.GetPlaceCategory(placeType) + if !ok { + t.Errorf("GetPlaceCategory(%q) returned ok=false, want %q", placeType, want) + continue + } + if got != want { + t.Errorf("GetPlaceCategory(%q) = %q, want %q", placeType, got, want) + } + } +} + func TestParsePlaceCategory(t *testing.T) { valid := []string{"Visit", "Eatery", "Shopping", "Lodging", "Wellness"} for _, s := range valid { From 98f940b9dc26b31023f9b729d3a5aa65066039ac Mon Sep 17 00:00:00 2001 From: tim-eternos Date: Wed, 29 Jul 2026 23:15:12 -0700 Subject: [PATCH 03/12] fix: reject non-legacy place types before calling Nearby Search POI.LocationType was cast straight to maps.PlaceType and forwarded as ?type=. Google answers an unknown type by ignoring the filter, not by erroring, so the call silently returns prominence-ranked establishments that then get stamped with the queried type. Validate against maps.ParsePlaceType first and fail loudly. Also repair the retry cap: maxRetries was computed as reqTimes * len(placeTypes) while reqTimes was 0, so it was always 0 and the break was dead code. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01DjdGnZM6cUBeg6jWwoQfU3 --- iowrappers/nearby_search.go | 34 +++++++-- iowrappers/nearby_search_validation_test.go | 80 +++++++++++++++++++++ 2 files changed, 108 insertions(+), 6 deletions(-) create mode 100644 iowrappers/nearby_search_validation_test.go diff --git a/iowrappers/nearby_search.go b/iowrappers/nearby_search.go index 8c7159ad..e6ff26c3 100644 --- a/iowrappers/nearby_search.go +++ b/iowrappers/nearby_search.go @@ -72,8 +72,23 @@ func MatchesBrandName(placeName, keyword string) bool { return strings.Contains(POI.NormalizeBrandKey(placeName), normalizedKeyword) } -// CreateMapSearchRequest creates a NearbySearchRequest for maps NearbySearch, adjust key settings such as radius and price levels -func CreateMapSearchRequest(reqIn *PlaceSearchRequest, placeType POI.LocationType, token string) (reqOut maps.NearbySearchRequest) { +// CreateMapSearchRequest creates a NearbySearchRequest for maps NearbySearch, adjust key settings such as radius and price levels. +// It rejects any place type the legacy Places API does not define: POI.LocationType is cast +// straight to maps.PlaceType and forwarded as ?type=, and Google responds to an unknown value +// by IGNORING the filter and returning prominence-ranked establishments rather than erroring. +// Those results are then stamped with the queried type, so an unvalidated type silently +// poisons the cache. maps.ParsePlaceType is the SDK's own list of legal values. +func CreateMapSearchRequest(reqIn *PlaceSearchRequest, placeType POI.LocationType, token string) (maps.NearbySearchRequest, error) { + // LocationTypeAny is the keyword (brand) search case: the type is deliberately unset so + // Google matches the keyword across all place types. + if placeType != POI.LocationTypeAny { + if _, err := maps.ParsePlaceType(string(placeType)); err != nil { + return maps.NearbySearchRequest{}, fmt.Errorf( + "place type %q is not a legacy Places API type (Places API (New) types are not accepted by /maps/api/place/nearbysearch): %w", + placeType, err) + } + } + // Adjust radius, minPrice and maxPrice settings in search request var radius = reqIn.Radius var exactPriceLevel maps.PriceLevel @@ -96,7 +111,7 @@ func CreateMapSearchRequest(reqIn *PlaceSearchRequest, placeType POI.LocationTyp RankBy: maps.RankBy("prominence"), MinPrice: exactPriceLevel, MaxPrice: exactPriceLevel, - } + }, nil } func (c *MapsClient) GoogleMapsNearbySearchWrapper(ctx context.Context, mapsReq maps.NearbySearchRequest) (resp maps.PlacesSearchResponse, err error) { @@ -139,7 +154,10 @@ func (c *MapsClient) extensiveNearbySearch(ctx context.Context, maxRequestTimes var reqTimes uint = 0 // number of queries for each location type var totalPlaceCount uint = 0 // number of results so far, keep this number low - maxRetries := reqTimes * uint(len(placeTypes)) + // Bail out once every place type has failed once. This was previously computed as + // reqTimes * len(placeTypes) while reqTimes was still 0, making the cap 0 and the + // break below unreachable, so a fully failing search span every retry round. + maxRetries := uint(len(placeTypes)) microAddrMap := make(map[string]string) // map place ID to its micro-address placeMap := make(map[string]bool) // remove duplication for place with same ID @@ -175,7 +193,11 @@ outer: wg.Add(1) go func(i int, placeType POI.LocationType, token string) { defer wg.Done() - searchReq := CreateMapSearchRequest(request, placeType, token) + searchReq, reqErr := CreateMapSearchRequest(request, placeType, token) + if reqErr != nil { + fetched[i].err = reqErr + return + } select { case c.apiSemaphore <- struct{}{}: defer func() { <-c.apiSemaphore }() @@ -200,7 +222,7 @@ outer: Logger.Error(fmt.Errorf("places nearby search with Maps failed for place type %s with error: %w", placeType, fetched[i].err)) mapsFailuresCount++ - if mapsFailuresCount == maxRetries { + if mapsFailuresCount >= maxRetries { break outer } // we should still retry for the next place type if the number of failures is below maxRetries diff --git a/iowrappers/nearby_search_validation_test.go b/iowrappers/nearby_search_validation_test.go new file mode 100644 index 00000000..29b46704 --- /dev/null +++ b/iowrappers/nearby_search_validation_test.go @@ -0,0 +1,80 @@ +package iowrappers + +import ( + "strings" + "testing" + + "github.com/weihesdlegend/Vacation-planner/POI" +) + +// TestCreateMapSearchRequestRejectsUnknownPlaceType guards the fast_food_restaurant +// incident at the request boundary. The SDK casts POI.LocationType straight to +// maps.PlaceType and forwards it as ?type=, so an unknown value silently disables +// the filter server-side instead of erroring. Validate before spending the call. +func TestCreateMapSearchRequestRejectsUnknownPlaceType(t *testing.T) { + req := &PlaceSearchRequest{ + Location: POI.Location{Latitude: 37.38006, Longitude: -122.11612}, + PlaceCat: POI.PlaceCategoryEatery, + Radius: 8000, + PriceLevel: POI.PriceLevelTwo, + } + for _, placeType := range []POI.LocationType{ + POI.LocationType("fast_food_restaurant"), + POI.LocationType("food_court"), + POI.LocationType("not_a_google_type"), + } { + if _, err := CreateMapSearchRequest(req, placeType, ""); err == nil { + t.Errorf("CreateMapSearchRequest(%q) returned nil error, want validation failure", placeType) + } else if !strings.Contains(err.Error(), string(placeType)) { + t.Errorf("CreateMapSearchRequest(%q) error %q should name the offending type", placeType, err) + } + } +} + +// TestCreateMapSearchRequestAcceptsKnownPlaceTypes pins that every type the +// categories actually search for still builds a request. +func TestCreateMapSearchRequestAcceptsKnownPlaceTypes(t *testing.T) { + req := &PlaceSearchRequest{ + Location: POI.Location{Latitude: 37.38006, Longitude: -122.11612}, + PlaceCat: POI.PlaceCategoryEatery, + Radius: 8000, + PriceLevel: POI.PriceLevelTwo, + } + categories := []POI.PlaceCategory{ + POI.PlaceCategoryVisit, POI.PlaceCategoryEatery, + POI.PlaceCategoryShopping, POI.PlaceCategoryLodging, POI.PlaceCategoryWellness, + } + for _, category := range categories { + for _, placeType := range POI.GetPlaceTypes(category) { + got, err := CreateMapSearchRequest(req, placeType, "") + if err != nil { + t.Errorf("CreateMapSearchRequest(%q) in category %q: unexpected error %v", placeType, category, err) + continue + } + if string(got.Type) != string(placeType) { + t.Errorf("CreateMapSearchRequest(%q) set Type=%q, want %q", placeType, got.Type, placeType) + } + } + } +} + +// TestCreateMapSearchRequestAcceptsAnyType pins that keyword (brand) searches, +// which intentionally leave the type unset, are not rejected. +func TestCreateMapSearchRequestAcceptsAnyType(t *testing.T) { + req := &PlaceSearchRequest{ + Location: POI.Location{Latitude: 37.38006, Longitude: -122.11612}, + PlaceCat: POI.PlaceCategoryEatery, + Radius: 8000, + Keyword: "Dunkin'", + } + got, err := CreateMapSearchRequest(req, POI.LocationTypeAny, "") + if err != nil { + t.Fatalf("CreateMapSearchRequest(LocationTypeAny) returned error %v, want nil", err) + } + if got.Type != "" { + t.Errorf("CreateMapSearchRequest(LocationTypeAny) set Type=%q, want empty", got.Type) + } + if got.Keyword != "Dunkin'" { + t.Errorf("CreateMapSearchRequest kept Keyword=%q, want %q", got.Keyword, "Dunkin'") + } +} From c1853f1b12d616e71424d43eeae190db6daa4c2a Mon Sep 17 00:00:00 2001 From: tim-eternos Date: Wed, 29 Jul 2026 23:27:32 -0700 Subject: [PATCH 04/12] fix: reset mapsFailuresCount per round in extensiveNearbySearch mapsFailuresCount was declared outside the outer round loop, so it accumulated across all rounds. Combined with maxRetries = len(placeTypes), one persistently-failing place type could exhaust the whole category's failure budget and break outer, silently discarding remaining rounds for healthy sibling types. Move the declaration inside the loop so it resets each round: reaching maxRetries now means every place type failed within that same round, matching the comment's intent instead of contradicting it. Code review follow-up on the prior commit (98f940b). Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01DjdGnZM6cUBeg6jWwoQfU3 --- iowrappers/nearby_search.go | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/iowrappers/nearby_search.go b/iowrappers/nearby_search.go index e6ff26c3..dd92c68c 100644 --- a/iowrappers/nearby_search.go +++ b/iowrappers/nearby_search.go @@ -154,9 +154,13 @@ func (c *MapsClient) extensiveNearbySearch(ctx context.Context, maxRequestTimes var reqTimes uint = 0 // number of queries for each location type var totalPlaceCount uint = 0 // number of results so far, keep this number low - // Bail out once every place type has failed once. This was previously computed as - // reqTimes * len(placeTypes) while reqTimes was still 0, making the cap 0 and the - // break below unreachable, so a fully failing search span every retry round. + // Bail out of the whole search once a single round sees every place type fail. + // mapsFailuresCount is reset at the top of each round below (it must NOT be hoisted + // out here), so reaching maxRetries means every place type failed in THAT round, not + // cumulatively across rounds. A per-round counter is required: one persistently-failing + // type must not be able to spend down a shared budget and cut off its healthy siblings' + // remaining rounds. This was previously computed as reqTimes * len(placeTypes) while + // reqTimes was still 0, making the cap 0 and the break below unreachable. maxRetries := uint(len(placeTypes)) microAddrMap := make(map[string]string) // map place ID to its micro-address @@ -164,7 +168,6 @@ func (c *MapsClient) extensiveNearbySearch(ctx context.Context, maxRequestTimes urlMap := make(map[string]string) // map place ID to url summaryMap := make(map[string]string) // map place ID to summary - var mapsFailuresCount uint = 0 detailsBudget := request.DetailsLimit // remaining Place Details calls across all pages; only enforced when DetailsLimit > 0 // One place type's Nearby Search HTTP result (Phase A). Holds no shared state, @@ -177,6 +180,10 @@ func (c *MapsClient) extensiveNearbySearch(ctx context.Context, maxRequestTimes outer: for totalPlaceCount < request.MinNumResults { reqTimes++ + // Reset every round: this must count failures within the CURRENT round only, so + // one place type failing round after round cannot exhaust the shared budget and + // cut off healthy sibling types (see maxRetries comment above). + var mapsFailuresCount uint = 0 // Phase A — fetch every eligible place type's Nearby Search CONCURRENTLY. // These are independent, slow HTTP calls and the dominant cold-cache cost; From 92d615b26d006ab489ea698e34656d57eb2816f3 Mon Sep 17 00:00:00 2001 From: tim-eternos Date: Wed, 29 Jul 2026 23:33:32 -0700 Subject: [PATCH 05/12] fix: sort category results by distance before truncating The truncation at places[:limit] assumed distance ordering, which only holds on the Redis cache path. The fresh path appends each place type's results in Google prominence order, so a cold search kept roughly cafes and restaurants and dropped bar, bakery and meal_takeaway entirely, and could rank a 3km result above one 250m away. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01DjdGnZM6cUBeg6jWwoQfU3 --- iowrappers/place_distance_sort.go | 36 +++++++++++++++++ iowrappers/place_distance_sort_test.go | 54 ++++++++++++++++++++++++++ planner/planner.go | 5 ++- 3 files changed, 94 insertions(+), 1 deletion(-) create mode 100644 iowrappers/place_distance_sort.go create mode 100644 iowrappers/place_distance_sort_test.go diff --git a/iowrappers/place_distance_sort.go b/iowrappers/place_distance_sort.go new file mode 100644 index 00000000..c4f23cf7 --- /dev/null +++ b/iowrappers/place_distance_sort.go @@ -0,0 +1,36 @@ +package iowrappers + +import ( + "sort" + + "github.com/weihesdlegend/Vacation-planner/POI" + "github.com/weihesdlegend/Vacation-planner/utils" +) + +// SortPlacesByDistance orders places ascending by distance from (lat, lng). +// +// Callers that truncate a candidate list to a limit MUST sort first. Only the Redis +// cache path returns places in distance order; the fresh path appends one place type's +// results after another in Google prominence order, so an unsorted slice[:limit] drops +// the last place types wholesale and can rank a 3km result above a 250m one. +// +// The sort is stable so places at equal distance keep their existing relative order. +func SortPlacesByDistance(places []POI.Place, lat, lng float64) { + origin := []float64{lat, lng} + dist := make(map[int]float64, len(places)) + for i := range places { + loc := places[i].GetLocation() + dist[i] = utils.HaversineDist(origin, []float64{loc.Latitude, loc.Longitude}) + } + idx := make([]int, len(places)) + for i := range idx { + idx[i] = i + } + sort.SliceStable(idx, func(a, b int) bool { return dist[idx[a]] < dist[idx[b]] }) + + sorted := make([]POI.Place, len(places)) + for newPos, oldPos := range idx { + sorted[newPos] = places[oldPos] + } + copy(places, sorted) +} diff --git a/iowrappers/place_distance_sort_test.go b/iowrappers/place_distance_sort_test.go new file mode 100644 index 00000000..b0b52e88 --- /dev/null +++ b/iowrappers/place_distance_sort_test.go @@ -0,0 +1,54 @@ +package iowrappers + +import ( + "testing" + + "github.com/weihesdlegend/Vacation-planner/POI" +) + +func placeAt(id string, lat, lng float64) POI.Place { + var p POI.Place + p.SetID(id) + p.SetLocationCoordinates([2]float64{lat, lng}) + return p +} + +// TestSortPlacesByDistance pins that truncation keeps the NEAREST places. The fresh +// (Google) path returns results grouped by place type in prominence order, so slicing +// without sorting first drops whole place types and can rank a 3km result above a 250m one. +func TestSortPlacesByDistance(t *testing.T) { + // State Street Market, Los Altos + lat, lng := 37.38006, -122.11612 + + places := []POI.Place{ + placeAt("far-sunnyvale", 37.3688, -122.0363), // ~7km east + placeAt("mid-mountainview", 37.3861, -122.0839), // ~3km east + placeAt("near-state-st", 37.38025, -122.11655), // ~40m away + } + + SortPlacesByDistance(places, lat, lng) + + want := []string{"near-state-st", "mid-mountainview", "far-sunnyvale"} + for i, id := range want { + if places[i].GetID() != id { + t.Errorf("position %d = %q, want %q (full order: %v)", i, places[i].GetID(), id, placeIDs(places)) + } + } +} + +// TestSortPlacesByDistanceEmpty pins that the no-result case does not panic. +func TestSortPlacesByDistanceEmpty(t *testing.T) { + var places []POI.Place + SortPlacesByDistance(places, 37.38006, -122.11612) + if len(places) != 0 { + t.Errorf("got %d places, want 0", len(places)) + } +} + +func placeIDs(places []POI.Place) []string { + ids := make([]string, 0, len(places)) + for _, p := range places { + ids = append(ids, p.GetID()) + } + return ids +} diff --git a/planner/planner.go b/planner/planner.go index 285635f9..707335cc 100644 --- a/planner/planner.go +++ b/planner/planner.go @@ -1417,7 +1417,10 @@ func (p *MyPlanner) getNearbyPlacesByCategory(ctx *gin.Context) { places = reclassified // drop places explicitly marked closed on the requested day places = iowrappers.Filter(places, func(place POI.Place) bool { return !place.KnownClosedOnDay(day) }) - // Redis results are sorted by distance ascending; keep the nearest ones + // Only the Redis path returns places in distance order. The fresh path + // appends each place type's results in Google prominence order, so sort + // before truncating or the last place types get dropped wholesale. + iowrappers.SortPlacesByDistance(places, req.Location.Latitude, req.Location.Longitude) if len(places) > limit { places = places[:limit] } From 3ee7ed21f413dee58c28efc753bc8364d7693bbb Mon Sep 17 00:00:00 2001 From: tim-eternos Date: Wed, 29 Jul 2026 23:42:13 -0700 Subject: [PATCH 06/12] chore: add Vacation-planner binary to .gitignore The 42MB build artifact was littering the repo root and creating risk for accidental commit via git add -A. Added entry to prevent future builds from tracking the binary. --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index 989af4c0..bbabcf2a 100644 --- a/.gitignore +++ b/.gitignore @@ -5,3 +5,4 @@ bin/ .idea .DS_Store node_modules/ +Vacation-planner From 299a8fdabbddd863587fb888b0ed92c06b090320 Mon Sep 17 00:00:00 2001 From: tim-eternos Date: Wed, 29 Jul 2026 23:55:15 -0700 Subject: [PATCH 07/12] feat: add admin migration to purge misclassified places from geo buckets The fast_food_restaurant incident wrote prominence-ranked hotels into placeIDs:eatery:level*. ReclassifyForCategory already hides them from API responses, but they inflate the bucket counts that gate radius expansion. Remove them using the same primary-type rule, dry-run by default. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01DjdGnZM6cUBeg6jWwoQfU3 --- iowrappers/data_migrations.go | 84 ++++++++++ planner/planner.go | 27 +++ .../redis_client_mocks/bucket_cleanup_test.go | 158 ++++++++++++++++++ 3 files changed, 269 insertions(+) create mode 100644 test/redis_client_mocks/bucket_cleanup_test.go diff --git a/iowrappers/data_migrations.go b/iowrappers/data_migrations.go index 06664e1c..b60051a5 100644 --- a/iowrappers/data_migrations.go +++ b/iowrappers/data_migrations.go @@ -9,6 +9,7 @@ import ( "sync" "github.com/bobg/go-generics/set" + "github.com/redis/go-redis/v9" "github.com/weihesdlegend/Vacation-planner/POI" ) @@ -189,6 +190,89 @@ func (s *PoiSearcher) AddUserRatingsTotal(context context.Context) error { return nil } +// BucketCleanupReport summarizes a RemoveMisclassifiedPlacesFromCategoryBuckets run. +type BucketCleanupReport struct { + Scanned int `json:"scanned"` + Misclassified int `json:"misclassified"` + Removed int `json:"removed"` + RemovedIDs []string `json:"removed_ids"` +} + +// SetPlace stores a single place record. Exported wrapper over setPlace for migrations and tests. +func (r *RedisClient) SetPlace(ctx context.Context, place POI.Place) error { + return r.setPlace(ctx, place) +} + +// AddGeoLocation adds a place to a geo bucket under an explicit key. Exported for +// migrations and tests that need to write buckets the normal write path would reject. +func (r *RedisClient) AddGeoLocation(ctx context.Context, key string, place POI.Place) error { + loc := place.GetLocation() + _, err := r.client.GeoAdd(ctx, key, &redis.GeoLocation{ + Name: place.ID, + Latitude: loc.Latitude, + Longitude: loc.Longitude, + }).Result() + return err +} + +// RemoveMisclassifiedPlacesFromCategoryBuckets removes places from cat's geo buckets whose +// PRIMARY Google type does not belong to cat. It repairs the fast_food_restaurant incident: +// two Places-API-(New)-only types were searched against the legacy Nearby Search, which +// ignored the unenforceable type filter and returned prominence-ranked establishments, and +// those were stamped with the queried type and written into placeIDs:eatery:level*. +// +// It uses the same rule as POI.ReclassifyForCategory, which is what already hides these from +// API responses: classify by primary type, and KEEP records with no Types list (older cached +// records written before Types was captured) so coverage never regresses. +// +// dryRun reports what would be removed without deleting anything. Always dry-run first. +func (r *RedisClient) RemoveMisclassifiedPlacesFromCategoryBuckets(ctx context.Context, cat POI.PlaceCategory, dryRun bool) (BucketCleanupReport, error) { + report := BucketCleanupReport{RemovedIDs: make([]string, 0)} + + levels := []POI.PriceLevel{POI.PriceLevelDefault} + if cat == POI.PlaceCategoryEatery { + levels = POI.AllPriceLevels + } + + for _, level := range levels { + key := POI.EncodeNearbySearchRedisKey(cat, level) + members, err := r.client.ZRange(ctx, key, 0, -1).Result() + if err != nil { + return report, fmt.Errorf("reading geo bucket %s: %w", key, err) + } + for _, placeID := range members { + report.Scanned++ + place, err := r.getPlace(ctx, placeID) + if err != nil { + // no place record backing this bucket member; leave it for RemovePlaces + Logger.Debugf("RemoveMisclassifiedPlacesFromCategoryBuckets: no record for %s in %s", placeID, key) + continue + } + if _, keep := POI.ReclassifyForCategory(place, cat); keep { + continue + } + report.Misclassified++ + report.RemovedIDs = append(report.RemovedIDs, placeID) + Logger.Infof("RemoveMisclassifiedPlacesFromCategoryBuckets: %s (%q, LocationType=%q, Types=%v) does not belong in %s", + placeID, place.Name, place.LocationType, place.Types, key) + if dryRun { + continue + } + if _, err := r.client.ZRem(ctx, key, placeID).Result(); err != nil { + return report, fmt.Errorf("removing %s from %s: %w", placeID, key, err) + } + report.Removed++ + } + } + return report, nil +} + +// RemoveMisclassifiedPlacesFromCategoryBuckets forwards to the RedisClient method so the +// admin handler can call it through the concrete PoiSearcher (p.Solver.Searcher). +func (s *PoiSearcher) RemoveMisclassifiedPlacesFromCategoryBuckets(ctx context.Context, cat POI.PlaceCategory, dryRun bool) (BucketCleanupReport, error) { + return s.redisClient.RemoveMisclassifiedPlacesFromCategoryBuckets(ctx, cat, dryRun) +} + func (s *PoiSearcher) AddUrl(context context.Context) error { placeIdToDetailedSearchResults, err := s.addDataFieldsToPlaces(context, "url", BatchSize) if err != nil { diff --git a/planner/planner.go b/planner/planner.go index 707335cc..b94cfe8d 100644 --- a/planner/planner.go +++ b/planner/planner.go @@ -277,6 +277,32 @@ func (p *MyPlanner) removePlacesMigrationHandler(ctx *gin.Context) { } } +// reclassifyBucketsMigrationHandler removes places from a category's geo buckets whose +// primary Google type does not belong to that category. Dry-run unless ?apply=true. +// +// Usage: GET /v1/migrate/reclassify-buckets?category=Eatery +// +// GET /v1/migrate/reclassify-buckets?category=Eatery&apply=true +func (p *MyPlanner) reclassifyBucketsMigrationHandler(ctx *gin.Context) { + _, authenticationErr := p.UserAuthentication(ctx, user.LevelAdmin) + if authenticationErr != nil { + ctx.JSON(http.StatusUnauthorized, gin.H{"error": authenticationErr.Error()}) + return + } + category, ok := POI.ParsePlaceCategory(ctx.DefaultQuery("category", string(POI.PlaceCategoryEatery))) + if !ok { + ctx.JSON(http.StatusBadRequest, gin.H{"error": "unknown category"}) + return + } + dryRun := ctx.Query("apply") != "true" + report, err := p.Solver.Searcher.RemoveMisclassifiedPlacesFromCategoryBuckets(ctx.Request.Context(), category, dryRun) + if err != nil { + ctx.JSON(http.StatusInternalServerError, gin.H{"error": err.Error(), "partial_report": report}) + return + } + ctx.JSON(http.StatusOK, gin.H{"dry_run": dryRun, "category": category, "report": report}) +} + func (p *MyPlanner) placeStatsHandler(ctx *gin.Context) { var placeCount int var err error @@ -1692,6 +1718,7 @@ func (p *MyPlanner) SetupRouter(serverPort string) *http.Server { migrations.GET("/user-ratings-total", p.UserRatingsTotalMigrationHandler) migrations.GET("/url", p.UrlMigrationHandler) migrations.GET("/remove-places", p.removePlacesMigrationHandler) + migrations.GET("/reclassify-buckets", p.reclassifyBucketsMigrationHandler) } v1.GET("/blob_url", p.getBlobObjectURL) diff --git a/test/redis_client_mocks/bucket_cleanup_test.go b/test/redis_client_mocks/bucket_cleanup_test.go new file mode 100644 index 00000000..696ec5bb --- /dev/null +++ b/test/redis_client_mocks/bucket_cleanup_test.go @@ -0,0 +1,158 @@ +package redis_client_mocks + +import ( + "testing" + + "github.com/alicebob/miniredis/v2" + "github.com/weihesdlegend/Vacation-planner/POI" + "github.com/weihesdlegend/Vacation-planner/iowrappers" +) + +// bucketCleanupFixtureIDs lists every place ID this test file writes into Redis. +// RedisClient/RedisMockSvr are process-wide fixtures shared with every other test in this +// package (some of which seed cities and places once via their own package init()), so +// resetBucketCleanupFixtures below clears only these specific keys between runs rather than +// flushing the whole mock server, which would also erase those unrelated fixtures. +var bucketCleanupFixtureIDs = []string{"hotel-1", "cafe-1", "hotel-2", "cafe-2", "legacy-1"} + +// resetBucketCleanupFixtures gives each test in this file a clean slate for its own fixture +// IDs without disturbing state other test files depend on. +func resetBucketCleanupFixtures(t *testing.T) { + t.Helper() + detailKeys := make([]string, 0, len(bucketCleanupFixtureIDs)) + for _, id := range bucketCleanupFixtureIDs { + detailKeys = append(detailKeys, iowrappers.PlaceDetailsRedisKeyPrefix+id) + } + if err := RedisClient.RemoveKeys(RedisContext, detailKeys); err != nil { + t.Fatalf("RemoveKeys: %v", err) + } + for _, lvl := range POI.AllPriceLevels { + key := POI.EncodeNearbySearchRedisKey(POI.PlaceCategoryEatery, lvl) + if !RedisMockSvr.Exists(key) { + continue + } + for _, id := range bucketCleanupFixtureIDs { + if _, err := RedisMockSvr.ZRem(key, id); err != nil && err != miniredis.ErrKeyNotFound { + t.Fatalf("ZRem(%s, %s): %v", key, id, err) + } + } + } +} + +// TestRemoveMisclassifiedPlacesDryRun pins that a dry run reports the hotels that the +// fast_food_restaurant incident wrote into placeIDs:eatery:level* without deleting them. +func TestRemoveMisclassifiedPlacesDryRun(t *testing.T) { + hotel := newPlaceWithTypes("hotel-1", "Residence Inn by Marriott Palo Alto", + POI.LocationType("fast_food_restaurant"), []string{"lodging", "point_of_interest", "establishment"}) + cafe := newPlaceWithTypes("cafe-1", "Peet's Coffee", + POI.LocationTypeCafe, []string{"cafe", "food", "point_of_interest", "establishment"}) + RedisClient.SetPlacesAddGeoLocations(RedisContext, []POI.Place{cafe}) + seedGeoBucket(t, POI.PlaceCategoryEatery, hotel) + + report, err := RedisClient.RemoveMisclassifiedPlacesFromCategoryBuckets(RedisContext, POI.PlaceCategoryEatery, true) + if err != nil { + t.Fatalf("RemoveMisclassifiedPlacesFromCategoryBuckets error: %v", err) + } + if report.Misclassified != 1 { + t.Errorf("Misclassified = %d, want 1 (report: %+v)", report.Misclassified, report) + } + if report.Removed != 0 { + t.Errorf("dry run Removed = %d, want 0", report.Removed) + } + if len(report.RemovedIDs) != 1 || report.RemovedIDs[0] != "hotel-1" { + t.Errorf("RemovedIDs = %v, want [hotel-1]", report.RemovedIDs) + } + // the hotel must still be present after a dry run + if got := countInEateryBuckets(t, "hotel-1"); got == 0 { + t.Error("dry run deleted hotel-1, want it retained") + } +} + +// TestRemoveMisclassifiedPlacesApply pins that a real run removes only the hotel. +func TestRemoveMisclassifiedPlacesApply(t *testing.T) { + resetBucketCleanupFixtures(t) + + hotel := newPlaceWithTypes("hotel-2", "The Westin Palo Alto", + POI.LocationType("fast_food_restaurant"), []string{"lodging", "point_of_interest", "establishment"}) + cafe := newPlaceWithTypes("cafe-2", "Red Rock Coffee", + POI.LocationTypeCafe, []string{"cafe", "food", "point_of_interest", "establishment"}) + RedisClient.SetPlacesAddGeoLocations(RedisContext, []POI.Place{cafe}) + seedGeoBucket(t, POI.PlaceCategoryEatery, hotel) + + report, err := RedisClient.RemoveMisclassifiedPlacesFromCategoryBuckets(RedisContext, POI.PlaceCategoryEatery, false) + if err != nil { + t.Fatalf("RemoveMisclassifiedPlacesFromCategoryBuckets error: %v", err) + } + if report.Removed != 1 { + t.Errorf("Removed = %d, want 1 (report: %+v)", report.Removed, report) + } + if got := countInEateryBuckets(t, "hotel-2"); got != 0 { + t.Errorf("hotel-2 still in %d eatery buckets, want 0", got) + } + if got := countInEateryBuckets(t, "cafe-2"); got == 0 { + t.Error("cafe-2 was removed, want it retained") + } +} + +// TestRemoveMisclassifiedPlacesKeepsUntypedRecords pins that older cached records with +// no Types list are left alone, matching ReclassifyForCategory's keep-on-unknown rule. +func TestRemoveMisclassifiedPlacesKeepsUntypedRecords(t *testing.T) { + resetBucketCleanupFixtures(t) + + legacy := newPlaceWithTypes("legacy-1", "Old Cached Diner", POI.LocationTypeRestaurant, nil) + RedisClient.SetPlacesAddGeoLocations(RedisContext, []POI.Place{legacy}) + + report, err := RedisClient.RemoveMisclassifiedPlacesFromCategoryBuckets(RedisContext, POI.PlaceCategoryEatery, false) + if err != nil { + t.Fatalf("RemoveMisclassifiedPlacesFromCategoryBuckets error: %v", err) + } + if report.Removed != 0 { + t.Errorf("Removed = %d, want 0 — records without Types must be kept", report.Removed) + } +} + +func newPlaceWithTypes(id, name string, locationType POI.LocationType, types []string) POI.Place { + var p POI.Place + p.SetID(id) + p.SetName(name) + p.SetType(locationType) + p.SetStatus(string(POI.Operational)) + p.SetPriceLevel(POI.PriceLevelDefault) + p.SetUserRatingsTotal(100) + p.SetLocationCoordinates([2]float64{37.38006, -122.11612}) + p.Types = types + return p +} + +// seedGeoBucket writes a place record plus its eatery geo-bucket membership directly, +// bypassing SetPlacesAddGeoLocations, which after Task 1 refuses unmapped types. +func seedGeoBucket(t *testing.T, cat POI.PlaceCategory, place POI.Place) { + t.Helper() + if err := RedisClient.SetPlace(RedisContext, place); err != nil { + t.Fatalf("SetPlace(%s): %v", place.GetID(), err) + } + key := POI.EncodeNearbySearchRedisKey(cat, place.PriceLevel) + if err := RedisClient.AddGeoLocation(RedisContext, key, place); err != nil { + t.Fatalf("AddGeoLocation(%s): %v", key, err) + } +} + +func countInEateryBuckets(t *testing.T, placeID string) int { + t.Helper() + count := 0 + for _, lvl := range POI.AllPriceLevels { + key := POI.EncodeNearbySearchRedisKey(POI.PlaceCategoryEatery, lvl) + if RedisMockSvr.Exists(key) { + members, err := RedisMockSvr.ZMembers(key) + if err != nil { + continue + } + for _, m := range members { + if m == placeID { + count++ + } + } + } + } + return count +} From 267e690c35c3dcd0f7b4fa85464f9c9482fb74da Mon Sep 17 00:00:00 2001 From: tim-eternos Date: Thu, 30 Jul 2026 00:40:24 -0700 Subject: [PATCH 08/12] fix: narrow bucket cleanup to the write rule and pipeline the scan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three review findings on the reclassify-buckets migration. The cleanup rule was broader than the incident and contradicted the write rule. ReclassifyForCategory keeps a place only when its primary type is one of a category's five search types, but meal_delivery and night_club are legal legacy types Google routinely lists first for genuine eateries, so the migration deleted rows that SetPlacesAddGeoLocations — which keys on the stamped LocationType — re-creates on the next cold search. That churn is not free: ReclassifyForCategory has exactly one production caller (the merchant endpoint), so the trip-planning path reads these buckets unfiltered and a deleted row shrinks its candidate pool until MapsLastSearchTime expires. Remove a member only when its primary type positively maps to a DIFFERENT category; keep it when the primary type maps to nothing at all. The scan was a ZRange plus a serial GET per member under the caller's request context, which cannot finish inside Heroku's non-configurable 30s H12 timeout for any real bucket — so the dry run never returned and the operator could not perform the review the runbook mandates. Read records in pipelined batches of 100 and report ZCARD bucket sizes up front so the scale is known before the run. Tests cover the full primary-type truth table (lodging and supermarket removed; meal_delivery, night_club and absent Types kept), that orphaned bucket members stay non-fatal and undeleted through the batched read, that the report states bucket sizes, and — at the handler level, which was previously untested on a destructive endpoint — that only the exact string "true" turns off dry-run. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01DjdGnZM6cUBeg6jWwoQfU3 --- iowrappers/data_migrations.go | 147 ++++++++++++--- planner/reclassify_buckets_dry_run_test.go | 155 +++++++++++++++ .../redis_client_mocks/bucket_cleanup_test.go | 177 +++++++++++++++++- 3 files changed, 450 insertions(+), 29 deletions(-) create mode 100644 planner/reclassify_buckets_dry_run_test.go diff --git a/iowrappers/data_migrations.go b/iowrappers/data_migrations.go index b60051a5..a984165e 100644 --- a/iowrappers/data_migrations.go +++ b/iowrappers/data_migrations.go @@ -2,6 +2,8 @@ package iowrappers import ( "context" + "encoding/json" + "errors" "fmt" "reflect" "strconv" @@ -190,12 +192,20 @@ func (s *PoiSearcher) AddUserRatingsTotal(context context.Context) error { return nil } +// bucketCleanupReadBatchSize is how many place records the bucket scan reads per pipeline +// round trip, matching the batch size SetPlacesAddGeoLocations uses on the write side. +const bucketCleanupReadBatchSize = 100 + // BucketCleanupReport summarizes a RemoveMisclassifiedPlacesFromCategoryBuckets run. +// BucketSizes/TotalMembers are measured with ZCARD before the scan begins so a dry-run +// report states the scale of the job even for buckets an operator has never sized. type BucketCleanupReport struct { - Scanned int `json:"scanned"` - Misclassified int `json:"misclassified"` - Removed int `json:"removed"` - RemovedIDs []string `json:"removed_ids"` + BucketSizes map[string]int64 `json:"bucket_sizes"` + TotalMembers int64 `json:"total_members"` + Scanned int `json:"scanned"` + Misclassified int `json:"misclassified"` + Removed int `json:"removed"` + RemovedIDs []string `json:"removed_ids"` } // SetPlace stores a single place record. Exported wrapper over setPlace for migrations and tests. @@ -221,52 +231,133 @@ func (r *RedisClient) AddGeoLocation(ctx context.Context, key string, place POI. // ignored the unenforceable type filter and returned prominence-ranked establishments, and // those were stamped with the queried type and written into placeIDs:eatery:level*. // -// It uses the same rule as POI.ReclassifyForCategory, which is what already hides these from -// API responses: classify by primary type, and KEEP records with no Types list (older cached -// records written before Types was captured) so coverage never regresses. +// The removal rule is the exact inverse of the WRITE rule, not of POI.ReclassifyForCategory. +// SetPlacesAddGeoLocations files a place under GetPlaceCategory(place.LocationType) and +// refuses to write anything whose type maps to no category, so this migration removes a +// member only when its primary type positively maps to a DIFFERENT category (lodging -> +// Lodging, supermarket -> Shopping). An UNMAPPED primary type is deliberately kept: types +// like "meal_delivery" and "night_club" are legal legacy types that Google routinely lists +// first for genuine eateries (a delivery-first restaurant, a bar that is also a club), as are +// records with no Types at all (cached before Types was captured). Those places carry a +// stamped LocationType the write path maps straight back to this category, so removing them +// would only delete rows the next cold search re-creates — while shrinking the trip-planning +// candidate pool for up to MinMapsResultRefreshDuration, because the trip-planning path +// (planner/solver.go -> matching.NearbySearchForCategory) reads these buckets with no +// reclassification at all. +// +// Note this is a broader keep-set than POI.ReclassifyForCategory applies on the merchant +// endpoint: that function keeps a place only when its primary type is one of the category's +// five search types. Divergence is intended — one function decides what to show in a single +// response, this one decides what may exist in the shared cache. // // dryRun reports what would be removed without deleting anything. Always dry-run first. func (r *RedisClient) RemoveMisclassifiedPlacesFromCategoryBuckets(ctx context.Context, cat POI.PlaceCategory, dryRun bool) (BucketCleanupReport, error) { - report := BucketCleanupReport{RemovedIDs: make([]string, 0)} + report := BucketCleanupReport{RemovedIDs: make([]string, 0), BucketSizes: make(map[string]int64)} levels := []POI.PriceLevel{POI.PriceLevelDefault} if cat == POI.PlaceCategoryEatery { levels = POI.AllPriceLevels } + keys := make([]string, 0, len(levels)) for _, level := range levels { - key := POI.EncodeNearbySearchRedisKey(cat, level) + keys = append(keys, POI.EncodeNearbySearchRedisKey(cat, level)) + } + + // Size the job before doing it. The scan cost is linear in bucket membership and the + // handler runs under the caller's request context, so an operator needs the member count + // to judge whether a dry run can complete inside their client/router timeout. + for _, key := range keys { + size, err := r.client.ZCard(ctx, key).Result() + if err != nil { + return report, fmt.Errorf("sizing geo bucket %s: %w", key, err) + } + report.BucketSizes[key] = size + report.TotalMembers += size + } + + for _, key := range keys { members, err := r.client.ZRange(ctx, key, 0, -1).Result() if err != nil { return report, fmt.Errorf("reading geo bucket %s: %w", key, err) } - for _, placeID := range members { - report.Scanned++ - place, err := r.getPlace(ctx, placeID) + // Read place records in pipelined batches rather than one GET per member: a serial + // N+1 over a real bucket cannot finish inside a 30s request timeout, which would + // make the mandatory dry-run review impossible and defeat the safety property. + for start := 0; start < len(members); start += bucketCleanupReadBatchSize { + batch := members[start:min(start+bucketCleanupReadBatchSize, len(members))] + places, found, err := r.getPlacesPipelined(ctx, batch) if err != nil { - // no place record backing this bucket member; leave it for RemovePlaces - Logger.Debugf("RemoveMisclassifiedPlacesFromCategoryBuckets: no record for %s in %s", placeID, key) - continue - } - if _, keep := POI.ReclassifyForCategory(place, cat); keep { - continue - } - report.Misclassified++ - report.RemovedIDs = append(report.RemovedIDs, placeID) - Logger.Infof("RemoveMisclassifiedPlacesFromCategoryBuckets: %s (%q, LocationType=%q, Types=%v) does not belong in %s", - placeID, place.Name, place.LocationType, place.Types, key) - if dryRun { - continue + return report, fmt.Errorf("reading place records for %s: %w", key, err) } - if _, err := r.client.ZRem(ctx, key, placeID).Result(); err != nil { - return report, fmt.Errorf("removing %s from %s: %w", placeID, key, err) + for i, placeID := range batch { + report.Scanned++ + if !found[i] { + // no place record backing this bucket member; leave it for RemovePlaces + Logger.Debugf("RemoveMisclassifiedPlacesFromCategoryBuckets: no record for %s in %s", placeID, key) + continue + } + place := places[i] + primary := POI.PrimaryLocationType(place.Types) + // Only remove members whose primary type positively belongs to a DIFFERENT + // category. An unmapped primary type (meal_delivery, night_club, or no Types + // at all) is not evidence of misclassification — the write path would + // legitimately place it here. + if c, ok := POI.GetPlaceCategory(primary); !ok || c == cat { + continue + } + report.Misclassified++ + report.RemovedIDs = append(report.RemovedIDs, placeID) + Logger.Infof("RemoveMisclassifiedPlacesFromCategoryBuckets: %s (%q, LocationType=%q, primary=%q, Types=%v) does not belong in %s", + placeID, place.Name, place.LocationType, primary, place.Types, key) + if dryRun { + continue + } + if _, err := r.client.ZRem(ctx, key, placeID).Result(); err != nil { + return report, fmt.Errorf("removing %s from %s: %w", placeID, key, err) + } + report.Removed++ } - report.Removed++ } } return report, nil } +// getPlacesPipelined fetches the place records for placeIDs in a single round trip. +// found[i] reports whether placeIDs[i] had a usable record: a bucket member with no (or an +// unparsable) backing record is not this migration's problem to fix, so it is reported as +// missing rather than failing the whole batch. A transport-level failure is returned as an +// error, because then nothing in the batch was actually read. +func (r *RedisClient) getPlacesPipelined(ctx context.Context, placeIDs []string) ([]POI.Place, []bool, error) { + cmds := make([]*redis.StringCmd, len(placeIDs)) + _, err := r.client.Pipelined(ctx, func(pipe redis.Pipeliner) error { + for i, placeID := range placeIDs { + cmds[i] = pipe.Get(ctx, PlaceDetailsRedisKeyPrefix+placeID) + } + return nil + }) + // Pipelined surfaces the first non-nil command error, and a missing key is reported as + // redis.Nil — an expected outcome here, not a failure. + if err != nil && !errors.Is(err, redis.Nil) { + return nil, nil, err + } + + places := make([]POI.Place, len(placeIDs)) + found := make([]bool, len(placeIDs)) + for i, cmd := range cmds { + res, cmdErr := cmd.Result() + if cmdErr != nil { + continue + } + if unmarshalErr := json.Unmarshal([]byte(res), &places[i]); unmarshalErr != nil { + Logger.Debugf("getPlacesPipelined: cannot parse record for %s: %v", placeIDs[i], unmarshalErr) + continue + } + found[i] = true + } + return places, found, nil +} + // RemoveMisclassifiedPlacesFromCategoryBuckets forwards to the RedisClient method so the // admin handler can call it through the concrete PoiSearcher (p.Solver.Searcher). func (s *PoiSearcher) RemoveMisclassifiedPlacesFromCategoryBuckets(ctx context.Context, cat POI.PlaceCategory, dryRun bool) (BucketCleanupReport, error) { diff --git a/planner/reclassify_buckets_dry_run_test.go b/planner/reclassify_buckets_dry_run_test.go new file mode 100644 index 00000000..5da8b8d0 --- /dev/null +++ b/planner/reclassify_buckets_dry_run_test.go @@ -0,0 +1,155 @@ +package planner + +import ( + "encoding/json" + "net/http" + "net/http/httptest" + "net/url" + "testing" + "time" + + "github.com/gin-gonic/gin" + "github.com/weihesdlegend/Vacation-planner/iowrappers" + "github.com/weihesdlegend/Vacation-planner/test/redis_client_mocks" + "github.com/weihesdlegend/Vacation-planner/user" +) + +// The reclassify-buckets migration deletes members from the shared place cache that the +// trip-planning path reads, and a deleted bucket row is not re-created until the city's +// 14-day MapsLastSearchTime marker expires. `dryRun := ctx.Query("apply") != "true"` is +// therefore the last line of defence on the endpoint: every value other than the exact +// string "true" — including the truthy-looking "TRUE" and "1" — must keep the run read-only. +func TestReclassifyBucketsMigrationDryRunDefault(t *testing.T) { + gin.SetMode(gin.TestMode) + + // point a real PoiSearcher at the same mock Redis the shared RedisClient uses, so the + // handler exercises the whole path down to RemoveMisclassifiedPlacesFromCategoryBuckets + redisURL, err := url.Parse("redis://" + redis_client_mocks.RedisMockSvr.Addr()) + if err != nil { + t.Fatalf("failed to parse mock redis URL: %v", err) + } + p := &MyPlanner{ + RedisClient: redis_client_mocks.RedisClient, + Solver: Solver{Searcher: iowrappers.CreatePoiSearcher("test-maps-api-key", redisURL)}, + } + + router := gin.New() + router.GET("/v1/migrate/reclassify-buckets", p.reclassifyBucketsMigrationHandler) + + adminToken := newAdminPAT(t, "reclassify_buckets_admin", "reclassify-buckets-admin-token") + + get := func(t *testing.T, query string) (int, map[string]any) { + t.Helper() + req := httptest.NewRequest(http.MethodGet, "/v1/migrate/reclassify-buckets"+query, nil) + req.Header.Set("Authorization", "Bearer "+adminToken) + w := httptest.NewRecorder() + router.ServeHTTP(w, req) + + var body map[string]any + if err := json.Unmarshal(w.Body.Bytes(), &body); err != nil { + t.Fatalf("failed to parse response %q: %v", w.Body.String(), err) + } + return w.Code, body + } + + tests := []struct { + name string + query string + wantDryRun bool + }{ + {name: "apply absent", query: "?category=Eatery", wantDryRun: true}, + {name: "apply empty", query: "?category=Eatery&apply=", wantDryRun: true}, + {name: "apply uppercase TRUE", query: "?category=Eatery&apply=TRUE", wantDryRun: true}, + {name: "apply 1", query: "?category=Eatery&apply=1", wantDryRun: true}, + {name: "apply yes", query: "?category=Eatery&apply=yes", wantDryRun: true}, + {name: "no query at all", query: "", wantDryRun: true}, + // the one and only spelling that is allowed to delete + {name: "apply exactly true", query: "?category=Eatery&apply=true", wantDryRun: false}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + code, body := get(t, tt.query) + if code != http.StatusOK { + t.Fatalf("status = %d, want %d (body: %+v)", code, http.StatusOK, body) + } + dryRun, ok := body["dry_run"].(bool) + if !ok { + t.Fatalf("response has no boolean dry_run field: %+v", body) + } + if dryRun != tt.wantDryRun { + t.Errorf("dry_run = %v for %q, want %v", dryRun, tt.query, tt.wantDryRun) + } + }) + } +} + +// TestReclassifyBucketsMigrationRequiresAdmin pins that the destructive endpoint is not +// reachable without admin credentials, so the dry-run default is not its only protection. +func TestReclassifyBucketsMigrationRequiresAdmin(t *testing.T) { + gin.SetMode(gin.TestMode) + + redisURL, err := url.Parse("redis://" + redis_client_mocks.RedisMockSvr.Addr()) + if err != nil { + t.Fatalf("failed to parse mock redis URL: %v", err) + } + p := &MyPlanner{ + RedisClient: redis_client_mocks.RedisClient, + Solver: Solver{Searcher: iowrappers.CreatePoiSearcher("test-maps-api-key", redisURL)}, + } + + router := gin.New() + router.GET("/v1/migrate/reclassify-buckets", p.reclassifyBucketsMigrationHandler) + + get := func(authorization string) int { + req := httptest.NewRequest(http.MethodGet, "/v1/migrate/reclassify-buckets?category=Eatery&apply=true", nil) + if authorization != "" { + req.Header.Set("Authorization", authorization) + } + w := httptest.NewRecorder() + router.ServeHTTP(w, req) + return w.Code + } + + t.Run("no credentials", func(t *testing.T) { + if code := get(""); code != http.StatusUnauthorized { + t.Errorf("status = %d without credentials, want %d", code, http.StatusUnauthorized) + } + }) + + t.Run("regular user PAT", func(t *testing.T) { + token := newRegularPAT(t, "reclassify_buckets_regular", "reclassify-buckets-regular-token") + if code := get("Bearer " + token); code != http.StatusUnauthorized { + t.Errorf("status = %d for a non-admin PAT, want %d", code, http.StatusUnauthorized) + } + }) +} + +func newAdminPAT(t *testing.T, username, rawToken string) string { + t.Helper() + return newPATForLevel(t, username, rawToken, user.LevelStringAdmin) +} + +func newRegularPAT(t *testing.T, username, rawToken string) string { + t.Helper() + return newPATForLevel(t, username, rawToken, user.LevelStringRegular) +} + +func newPATForLevel(t *testing.T, username, rawToken, level string) string { + t.Helper() + userView, err := redis_client_mocks.RedisClient.CreateUser( + redis_client_mocks.RedisContext, + user.View{Username: username, Email: username + "@example.com", Password: "pwd", UserLevel: level}, + false, + ) + if err != nil { + t.Fatalf("failed to create %s test user: %v", level, err) + } + pat, err := redis_client_mocks.RedisClient.NewPAT( + redis_client_mocks.RedisContext, username+"-pat", userView.ID, rawToken, time.Hour, + ) + if err != nil { + t.Fatalf("failed to create test PAT: %v", err) + } + return pat.TokenHash +} diff --git a/test/redis_client_mocks/bucket_cleanup_test.go b/test/redis_client_mocks/bucket_cleanup_test.go index 696ec5bb..d11ff5e0 100644 --- a/test/redis_client_mocks/bucket_cleanup_test.go +++ b/test/redis_client_mocks/bucket_cleanup_test.go @@ -13,7 +13,11 @@ import ( // package (some of which seed cities and places once via their own package init()), so // resetBucketCleanupFixtures below clears only these specific keys between runs rather than // flushing the whole mock server, which would also erase those unrelated fixtures. -var bucketCleanupFixtureIDs = []string{"hotel-1", "cafe-1", "hotel-2", "cafe-2", "legacy-1"} +var bucketCleanupFixtureIDs = []string{ + "hotel-1", "cafe-1", "hotel-2", "cafe-2", "legacy-1", + "tt-lodging", "tt-supermarket", "tt-meal-delivery", "tt-night-club", "tt-no-types", "tt-cafe", + "tt-orphan", +} // resetBucketCleanupFixtures gives each test in this file a clean slate for its own fixture // IDs without disturbing state other test files depend on. @@ -111,6 +115,177 @@ func TestRemoveMisclassifiedPlacesKeepsUntypedRecords(t *testing.T) { } } +// TestRemoveMisclassifiedPlacesPrimaryTypeTruthTable pins the exact rule the cleanup applies: +// a bucket member is removed only when its PRIMARY Google type positively maps to a DIFFERENT +// category. An unmapped primary type is NOT evidence of misclassification — the fixed write +// path keys on the stamped LocationType, so it would legitimately file a delivery-first +// restaurant ("meal_delivery" first in Types) or a bar/club ("night_club" first) under Eatery. +// Purging those would make the migration delete rows the write path immediately re-creates, +// while shrinking the trip-planning candidate pool for up to MinMapsResultRefreshDuration. +func TestRemoveMisclassifiedPlacesPrimaryTypeTruthTable(t *testing.T) { + resetBucketCleanupFixtures(t) + t.Cleanup(func() { resetBucketCleanupFixtures(t) }) + + cases := []struct { + id string + name string + stamped POI.LocationType + types []string + wantRemove bool + why string + }{ + { + id: "tt-lodging", name: "Residence Inn by Marriott Palo Alto", + // the incident: searched as fast_food_restaurant, truthfully a hotel + stamped: POI.LocationType("fast_food_restaurant"), + types: []string{"lodging", "point_of_interest", "establishment"}, + // GetPlaceCategory("lodging") == (Lodging, true) != Eatery + wantRemove: true, why: "primary type lodging maps to Lodging", + }, + { + id: "tt-supermarket", name: "Whole Foods Market", + stamped: POI.LocationTypeRestaurant, + types: []string{"supermarket", "grocery_or_supermarket", "food", "store"}, + // GetPlaceCategory("supermarket") == (Shopping, true) != Eatery + wantRemove: true, why: "primary type supermarket maps to Shopping", + }, + { + id: "tt-meal-delivery", name: "Wok This Way Delivery", + stamped: POI.LocationTypeRestaurant, + types: []string{"meal_delivery", "restaurant", "food", "point_of_interest"}, + // GetPlaceCategory("meal_delivery") == ("", false): legal legacy type, unmapped + wantRemove: false, why: "primary type meal_delivery maps to no category", + }, + { + id: "tt-night-club", name: "The Basement", + stamped: POI.LocationTypeBar, + types: []string{"night_club", "bar", "point_of_interest", "establishment"}, + // GetPlaceCategory("night_club") == ("", false): legal legacy type, unmapped + wantRemove: false, why: "primary type night_club maps to no category", + }, + { + id: "tt-no-types", name: "Old Cached Diner", + stamped: POI.LocationTypeRestaurant, + types: nil, + // PrimaryLocationType(nil) == "" -> GetPlaceCategory("") == ("", false) + wantRemove: false, why: "no Types at all (record cached before Types was captured)", + }, + { + id: "tt-cafe", name: "Peet's Coffee", + stamped: POI.LocationTypeCafe, + types: []string{"cafe", "food", "point_of_interest", "establishment"}, + // GetPlaceCategory("cafe") == (Eatery, true) == the bucket's category + wantRemove: false, why: "primary type cafe maps to Eatery", + }, + } + + for _, tc := range cases { + seedGeoBucket(t, POI.PlaceCategoryEatery, newPlaceWithTypes(tc.id, tc.name, tc.stamped, tc.types)) + } + + report, err := RedisClient.RemoveMisclassifiedPlacesFromCategoryBuckets(RedisContext, POI.PlaceCategoryEatery, false) + if err != nil { + t.Fatalf("RemoveMisclassifiedPlacesFromCategoryBuckets error: %v", err) + } + + removed := make(map[string]bool, len(report.RemovedIDs)) + for _, id := range report.RemovedIDs { + removed[id] = true + } + + for _, tc := range cases { + t.Run(tc.id, func(t *testing.T) { + if removed[tc.id] != tc.wantRemove { + t.Errorf("reported removal of %s = %v, want %v (%s); report: %+v", + tc.id, removed[tc.id], tc.wantRemove, tc.why, report) + } + inBuckets := countInEateryBuckets(t, tc.id) + if tc.wantRemove && inBuckets != 0 { + t.Errorf("%s still in %d eatery buckets after apply, want 0 (%s)", tc.id, inBuckets, tc.why) + } + if !tc.wantRemove && inBuckets == 0 { + t.Errorf("%s was deleted from the eatery buckets, want it retained (%s)", tc.id, tc.why) + } + }) + } +} + +// TestRemoveMisclassifiedPlacesToleratesMissingRecords pins that a bucket member with no +// backing place_details record is counted, skipped, and left in place rather than failing the +// run or being deleted. Orphaned members are RemovePlaces' job, not this migration's. This +// guards the batched read path specifically: a missing key surfaces as redis.Nil inside the +// pipeline, which must not be mistaken for a transport failure and abort the whole scan. +func TestRemoveMisclassifiedPlacesToleratesMissingRecords(t *testing.T) { + resetBucketCleanupFixtures(t) + t.Cleanup(func() { resetBucketCleanupFixtures(t) }) + + // a geo-bucket member whose place record is deliberately never written + orphan := newPlaceWithTypes("tt-orphan", "Vanished Diner", POI.LocationTypeRestaurant, nil) + key := POI.EncodeNearbySearchRedisKey(POI.PlaceCategoryEatery, orphan.PriceLevel) + if err := RedisClient.AddGeoLocation(RedisContext, key, orphan); err != nil { + t.Fatalf("AddGeoLocation(%s): %v", key, err) + } + // a real misclassified place in the same batch, to prove the scan keeps going + seedGeoBucket(t, POI.PlaceCategoryEatery, newPlaceWithTypes("tt-lodging", "The Westin Palo Alto", + POI.LocationType("fast_food_restaurant"), []string{"lodging", "point_of_interest", "establishment"})) + + report, err := RedisClient.RemoveMisclassifiedPlacesFromCategoryBuckets(RedisContext, POI.PlaceCategoryEatery, false) + if err != nil { + t.Fatalf("a bucket member with no place record must not fail the run: %v", err) + } + if report.Scanned < 2 { + t.Errorf("Scanned = %d, want at least the 2 seeded members (report: %+v)", report.Scanned, report) + } + for _, id := range report.RemovedIDs { + if id == "tt-orphan" { + t.Errorf("tt-orphan was reported for removal; a member with no record must be skipped (report: %+v)", report) + } + } + if got := countInEateryBuckets(t, "tt-orphan"); got == 0 { + t.Error("tt-orphan was deleted; a member with no record must be left for RemovePlaces") + } + // the scan must not have stopped at the orphan + if got := countInEateryBuckets(t, "tt-lodging"); got != 0 { + t.Errorf("tt-lodging still in %d eatery buckets; the scan stopped at the orphaned member", got) + } +} + +// TestRemoveMisclassifiedPlacesReportsBucketSizes pins that the report states the scale of the +// scan up front. An operator has to review a dry-run report before applying, so the report has +// to say how many members exist even when the scan itself is what makes the run expensive. +func TestRemoveMisclassifiedPlacesReportsBucketSizes(t *testing.T) { + resetBucketCleanupFixtures(t) + t.Cleanup(func() { resetBucketCleanupFixtures(t) }) + + seedGeoBucket(t, POI.PlaceCategoryEatery, newPlaceWithTypes("tt-cafe", "Peet's Coffee", + POI.LocationTypeCafe, []string{"cafe", "food", "point_of_interest", "establishment"})) + + report, err := RedisClient.RemoveMisclassifiedPlacesFromCategoryBuckets(RedisContext, POI.PlaceCategoryEatery, true) + if err != nil { + t.Fatalf("RemoveMisclassifiedPlacesFromCategoryBuckets error: %v", err) + } + + // one entry per eatery price bucket, whether or not the key exists yet + if len(report.BucketSizes) != len(POI.AllPriceLevels) { + t.Errorf("BucketSizes has %d entries, want %d: %+v", + len(report.BucketSizes), len(POI.AllPriceLevels), report.BucketSizes) + } + for _, lvl := range POI.AllPriceLevels { + key := POI.EncodeNearbySearchRedisKey(POI.PlaceCategoryEatery, lvl) + if _, ok := report.BucketSizes[key]; !ok { + t.Errorf("BucketSizes missing key %s: %+v", key, report.BucketSizes) + } + } + if report.TotalMembers < 1 { + t.Errorf("TotalMembers = %d, want at least the 1 seeded place", report.TotalMembers) + } + // ZCARD is taken before the scan; nothing writes concurrently in this test, so every + // counted member must also have been visited. + if int(report.TotalMembers) != report.Scanned { + t.Errorf("TotalMembers = %d but Scanned = %d, want equal", report.TotalMembers, report.Scanned) + } +} + func newPlaceWithTypes(id, name string, locationType POI.LocationType, types []string) POI.Place { var p POI.Place p.SetID(id) From 81c7f60e36530280f8a4a078bf280f84bbcd9187 Mon Sep 17 00:00:00 2001 From: tim-eternos Date: Thu, 30 Jul 2026 00:40:37 -0700 Subject: [PATCH 09/12] docs: correct the eatery place-type plan after final review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four corrections to the committed plan. The "confirmed non-issues" claim that the bad records do not consume result slots was true only of the merchant endpoint. The trip-planning path (solver.go -> matching.NearbySearchForCategory -> CreatePlace) reads the same buckets and never reclassifies, so the hotels were slotted into generated trip plans as eateries — user-visible output, not just cache residue. Task 4's Step 3 code block specified a removal rule the review rejected, and an N+1 scan that cannot complete inside Heroku's 30s router timeout. Its own Interfaces block had it right. Flag the block as superseded and state what shipped, including why RedisMockSvr.FlushAll() must not be used in a package whose Redis fixtures are process-wide. Deployment step 1 said "deploy Tasks 1-3" when Task 4 ships in the same PR; describe what the operator actually has, and why the ordering property holds by construction rather than by sequencing. Step 2 now spells out what the rule removes and what it deliberately keeps, so absent meal_delivery/night_club entries read as the rule working rather than a miss. The verification checklist required a bare grep for the two New-API-only strings to return nothing, but it returns matches by design — the guard tests must name the strings to assert they are rejected. Replace it with the real invariant: no LocationType constant exists for either string. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01DjdGnZM6cUBeg6jWwoQfU3 --- .../2026-07-29-eatery-place-type-fixes.md | 27 ++++++++++++++++--- 1 file changed, 23 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/plans/2026-07-29-eatery-place-type-fixes.md b/docs/superpowers/plans/2026-07-29-eatery-place-type-fixes.md index d8d186aa..3bd55841 100644 --- a/docs/superpowers/plans/2026-07-29-eatery-place-type-fixes.md +++ b/docs/superpowers/plans/2026-07-29-eatery-place-type-fixes.md @@ -32,7 +32,7 @@ Verified against the code at `8644199`: Confirmed non-issues, so nobody wastes time on them: -- These records do **not** consume result slots in the response. `planner/planner.go` reclassifies at `:1406` *before* truncating at `:1414`, and the Redis read is unbounded (`GeoRadius` with no `Count`, `redis_client.go:484-489`). +- ~~These records do **not** consume result slots in the response.~~ **CORRECTED after final review — this held only for the merchant endpoint, and was the most consequential error in this plan.** It is true that `planner/planner.go` (the `/v1/nearby-places-by-category` merchant handler) calls `POI.ReclassifyForCategory` *before* truncating, and that the Redis read is unbounded (`GeoRadius` with no `Count`, `redis_client.go:484-489`). But `ReclassifyForCategory` has exactly **one** production caller. The trip-planning path — `planner/solver.go:532` → `matching.NearbySearchForCategory` (`matching/matcher.go:76`) → `searcher.NearbySearch` → `matching.CreatePlace(place, req.Category)` (`:93`) — reads the same `placeIDs:eatery:level*` buckets and never reclassifies. The hotels were therefore slotted into generated trip plans as eateries: user-visible bad output, not merely cache residue. This is also why the cleanup migration must not delete aggressively — see Task 4. - They did **not** waste Place Details spend. `detailsBudget` (`nearby_search.go:150`) is shared and consumed in `placeTypes` order, so `cafe`/`restaurant` exhaust it before the junk types are processed. - Zero `food_court`-tagged places in prod is expected, not a contradiction. `placeMap` dedups by place ID across all types in one search, and `fast_food_restaurant` is processed first, so an identical ignored-filter response for `food_court` is entirely deduped away. @@ -685,6 +685,13 @@ Reuses the established pattern: an admin-authenticated GET under `v1.Group("/mig - Consumes: `POI.GetPlaceCategory(placeType) (PlaceCategory, bool)` from Task 1; `POI.PrimaryLocationType`, `POI.GetPlaceTypes` (existing). - Produces: `(*RedisClient).RemoveMisclassifiedPlacesFromCategoryBuckets(ctx context.Context, cat POI.PlaceCategory, dryRun bool) (BucketCleanupReport, error)` and `type BucketCleanupReport struct { Scanned, Misclassified, Removed int; RemovedIDs []string }`. +> **CORRECTED after final review — the Step 1/Step 3 code blocks below are superseded; read `iowrappers/data_migrations.go` for what shipped.** Two defects were caught in review: +> +> 1. **The removal rule below is wrong and too broad.** Step 3's `if _, keep := POI.ReclassifyForCategory(place, cat); keep` keeps a member only when its primary type is one of the category's five search types, so it deletes legitimate eateries whose primary type is a legal-but-unmapped legacy type (`meal_delivery`, `night_club`). Worse, it *contradicts the write path*, which keys on the stamped `LocationType` — the migration would delete rows the next cold search re-creates. The Interfaces block above is the one that got it right (`GetPlaceCategory` is what it consumes). What shipped is the inverse of the **write** rule: `primary := POI.PrimaryLocationType(place.Types); if c, ok := POI.GetPlaceCategory(primary); !ok || c == cat { continue }` — remove only when the primary type positively maps to a *different* category; keep on unmapped. +> 2. **`ZRange 0 -1` plus a serial `getPlace` per member is an N+1 that cannot finish inside Heroku's non-configurable 30s H12 router timeout**, and the handler passes `ctx.Request.Context()`, which cancels on client disconnect. For any real bucket the dry run never returns, so the operator cannot perform the review that step 2 of the deployment order mandates — defeating the migration's core safety property. What shipped reads records in pipelined batches of 100 (matching `SetPlacesAddGeoLocations`) and reports `bucket_sizes`/`total_members` from `ZCARD` up front so the operator knows the scale before running. +> +> Also note Step 1's `RedisMockSvr.FlushAll()` must NOT be used: `RedisClient`/`RedisMockSvr` are process-wide fixtures shared with every other test in `test/redis_client_mocks`, and flushing wipes sibling files' package-level `init()` fixtures, breaking 4 unrelated tests under the full suite. The shipped test file scopes its reset to its own fixture IDs instead. + - [ ] **Step 1: Write the failing test** Create `test/redis_client_mocks/bucket_cleanup_test.go`. This follows the existing miniredis harness in that package (`RedisClient`, `RedisContext`, `RedisMockSvr` are package-level fixtures set up by its `TestMain`): @@ -990,13 +997,21 @@ Remove them using the same primary-type rule, dry-run by default." ## Deployment order -1. Merge and deploy Tasks 1-3. Verify `go test ./...` green in CI. +1. Merge and deploy the whole PR — Tasks 1-4 ship together, so after deploy the operator has both the forward fixes (no new bad writes) *and* the cleanup endpoint. The "fix the write path before cleaning the data" ordering property still holds by construction, not by deploy sequencing: the endpoint is dry-run by default, so nothing is deleted until step 3 is run by hand. Verify `go test ./...` green in CI. 2. Dry-run the cleanup and read the report before applying: ```bash curl -s -H "Authorization: Bearer $ADMIN_JWT" \ "https://best-vacation-planner.herokuapp.com/v1/migrate/reclassify-buckets?category=Eatery" | jq ``` - Expect roughly 17 entries in `removed_ids` for the Los Altos hotels, plus any older misclassifications the primary-type rule catches. Review the list before proceeding. + Read `bucket_sizes` / `total_members` first — they are measured with `ZCARD` before the scan and tell you the scale of the job. + + Expect roughly 17 entries in `removed_ids` for the Los Altos hotels, plus any older misclassifications the rule catches. Know exactly what the rule does before you review the list: + + - **Removed** — the member's primary Google type (`POI.PrimaryLocationType`: first entry of `types[]` that is not an umbrella type like `food`/`point_of_interest`/`establishment`) maps to a *different* category. `lodging` → `Lodging`, `supermarket` → `Shopping`. These are the incident's hotels. + - **Kept: unmapped primary types.** `meal_delivery` (a delivery-first restaurant) and `night_club` (a bar that is also a club) are legal legacy types Google routinely lists first for genuine eateries, and `GetPlaceCategory` maps neither to any category. An unmapped primary type is *not* evidence of misclassification — the fixed write path keys on the stamped `LocationType`, so it would legitimately file these under Eatery, and deleting them would only remove rows the next cold search re-creates while shrinking the trip-planning candidate pool for up to `MinMapsResultRefreshDuration` (14 days) per city/price-level. + - **Kept: records with no `types[]` at all** (cached before `Types` was captured), so coverage never regresses on old data. + + Seeing `meal_delivery`/`night_club` places *absent* from `removed_ids` is the rule working, not a miss. Review the list before proceeding. 3. Apply: ```bash curl -s -H "Authorization: Bearer $ADMIN_JWT" \ @@ -1013,7 +1028,11 @@ Remove them using the same primary-type rule, dry-run by default." - [ ] `go build -v .` and `go test -v ./...` pass locally and in CI. - [ ] `TestPlaceCategoryRoundTrip` fails if you temporarily re-add `LocationTypeFastFood` to `GetPlaceTypes(Eatery)` — confirm the guard is now real, then revert the experiment. -- [ ] `grep -rn 'fast_food_restaurant\|food_court' --include='*.go' .` returns nothing. +- [ ] No `LocationType` constant exists for either New-API-only string. A bare text search is the wrong check — the strings legitimately appear in the guard tests (which must name them to assert they are rejected), in the incident narrative in doc comments, and in the migration fixtures, so `grep -rn 'fast_food_restaurant\|food_court' --include='*.go' .` returns matches by design. Check the actual invariant instead: + ```bash + # must print nothing: no constant may reintroduce either type + grep -nE '=[[:space:]]*LocationType\("(fast_food_restaurant|food_court)"\)' POI/categories.go + ``` - [ ] A category search at State Street Market returns Peet's at 367 State St ahead of results in Sunnyvale. - [ ] Dry-run report reviewed before any `apply=true` call. From aada4e36833c0bb2b960a6a7dd1a6ceae42ed3ae Mon Sep 17 00:00:00 2001 From: tim-eternos Date: Thu, 30 Jul 2026 00:53:34 -0700 Subject: [PATCH 10/12] docs: record deferred follow-ups from the place-type fixes review Captures the two Importants deliberately deferred (AllPlaceCategories guard, brand-handler distance sort), the migration-robustness and comment-precision minors, and the residual risk that the write rule and cleanup rule now deliberately disagree. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01DjdGnZM6cUBeg6jWwoQfU3 --- ...6-07-30-followups-from-place-type-fixes.md | 70 +++++++++++++++++++ 1 file changed, 70 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-30-followups-from-place-type-fixes.md diff --git a/docs/superpowers/plans/2026-07-30-followups-from-place-type-fixes.md b/docs/superpowers/plans/2026-07-30-followups-from-place-type-fixes.md new file mode 100644 index 00000000..7a101581 --- /dev/null +++ b/docs/superpowers/plans/2026-07-30-followups-from-place-type-fixes.md @@ -0,0 +1,70 @@ +# Follow-ups from the eatery place-type misclassification PR + +Carried forward from the review of `fix/eatery-place-type-misclassification` (plan: `2026-07-29-eatery-place-type-fixes.md`). Everything here was reviewed, triaged, and deliberately deferred — none of it blocked that merge. Ordered by value. + +## Important — deferred by explicit decision + +### 1. `POI.AllPlaceCategories` — make the guard cover the invariant, not today's categories + +**This is the highest-value item on the list.** The original incident survived because a guard test could not fail. Both current guards enumerate categories by hand — `test/place_category_test.go:41-44` and `iowrappers/nearby_search_validation_test.go:44-47` — so they protect the five categories that exist today rather than the invariant. + +The final reviewer demonstrated this empirically: adding a `PlaceCategoryNightlife` with `GetPlaceTypes` returning `{"karaoke", "pub"}` (neither exists in the v1.7.0 SDK — the same species of mistake as `fast_food_restaurant`) plus a matching `GetPlaceCategory` case produced a **fully green suite**, build and all 6 packages. The incident is reproducible verbatim for any newly added category. + +Fix: add `POI.AllPlaceCategories` and drive `ParsePlaceCategory` (`POI/categories.go:113-120`), `TestPlaceCategoryRoundTrip`, `TestCreateMapSearchRequestAcceptsKnownPlaceTypes`, `TestEncodeNearbySearchRedisKeyDistinct`, and the migration handler off it. A new category then lands inside every guard automatically. This converts "we fixed this bug" into "this bug shape cannot return." + +### 2. `getNearbyPlacesByBrand` still truncates by prominence + +`planner/planner.go:1298-1302` carries the identical false premise that the distance-sort task refuted for the category handler: + +```go +places = iowrappers.Filter(places, func(place POI.Place) bool { return !place.KnownClosedOnDay(day) }) +// Redis results are sorted by distance ascending; keep the nearest ones +if len(places) > limit { places = places[:limit] } +``` + +True on the cache path, false on the fresh path, where `PoiSearcher.NearbySearch` returns only `newPlaces` in Google prominence order (`iowrappers/poi_searcher.go:203-213`). A cold brand search can drop a 300m Dunkin' in favour of a 5km one. + +Impact is ordering-only (brand searches use a single `LocationTypeAny` type, so no whole place types are lost), which is why it was deferred — but the repo now has one handler sorted and its sibling unsorted, carrying a comment this work explicitly falsified. Fix is one line: `iowrappers.SortPlacesByDistance(places, req.Location.Latitude, req.Location.Longitude)` before the truncation. + +## Minor — migration robustness + +### 3. Removals are not pipelined + +`iowrappers/data_migrations.go:316` — reads were pipelined into batches of 100 but `ZRem` is still one round trip per removed member. The stated rationale for pipelining (a serial N+1 cannot finish inside Heroku's hard 30s H12) now covers only the read half. Irrelevant for the incident's ~17 rows; a bulk `apply=true` with thousands of hits re-enters the same ceiling. + +### 4. Bucket sizes are only delivered if the run completes + +`BucketSizes` / `TotalMembers` are measured up front (`data_migrations.go:270-277`) but serialized only in the terminal `ctx.JSON` (`planner/planner.go:303`). An H12 severs the request, so the operator who most needs the scale number is exactly the one who never receives it — `partial_report` covers returned errors, not a router timeout. A size-only mode (`?sizes=true`, returning right after the `ZCARD` loop) would make the property unconditional. + +### 5. `getPlace` failure is indistinguishable from "no record exists" + +`iowrappers/data_migrations.go:295-299`. A mid-run Redis fault silently skips every remaining member and returns a report reading `Misclassified: 0`, which an operator would reasonably read as "buckets are clean." Fail-safe in direction (nothing is deleted) but misleading. Distinguish `redis.Nil` from transport errors, or add a skipped/error count to the report. + +## Minor — comment and doc precision + +These matter more than usual: the write rule and the cleanup rule now **deliberately disagree**, and comments are most of what holds them apart. See "residual risk" below. + +- `iowrappers/data_migrations.go:228-229` — the summary sentence still reads "removes places whose PRIMARY Google type does not belong to `cat`," which describes the *discarded* broad rule. Only the paragraph beneath it is accurate. +- `iowrappers/data_migrations.go:248-250` — describes `ReclassifyForCategory` as keeping "only when its primary type is one of the category's five search types." Omits its keep-on-no-`Types` branch (`POI/categories.go:161-163`), and "five" is Eatery-specific (Lodging has one). +- `test/redis_client_mocks/bucket_cleanup_test.go:101-102` — stale pre-existing comment still says untyped records are kept "matching `ReclassifyForCategory`'s keep-on-unknown rule," contradicting the deliberate-divergence doc added alongside it. +- `planner/planner.go:280-281` — the handler doc comment repeats the discarded rule in unqualified form; it is now the only place stating it that way. +- `2026-07-29-eatery-place-type-fixes.md:118` — the replacement grep invariant checks only `POI/categories.go` and only the `= LocationType("…")` declaration form. `TestPlaceCategoryRoundTrip` is the real guard, so this is cosmetic. + +## Minor — pre-existing, untouched + +- `iowrappers/nearby_search.go:196` — `maxRetries` equals the *total* category type count rather than the count actually attempted in a round, so a round where every active type fails but one sibling is skipped never reaches the cap. Bounded by `GoogleMapsSearchCallMaxCount = 5`, so no unbounded loop. Per-type failure tracking was explicitly ruled out of scope. +- `iowrappers/nearby_search.go:222` — the error log says "nearby search … failed" for a validation rejection where no search was attempted. Cheap fix, and worth doing because "fail loudly" is that code's whole promise. +- Malformed format verbs `%!s()` at `planner/users.go:212,236` and `iowrappers/redis_client.go:497`. +- `iowrappers/data_migrations.go` hosts `SetPlace` / `AddGeoLocation`, which are generic `RedisClient` concerns; `redis_client.go` is their natural home. `AddGeoLocation` also widens the production API with a geo write that bypasses the type validation this PR added, and has no production caller. + +## Residual risk to keep in mind + +**The write rule and the cleanup rule now legitimately disagree.** The write path (`iowrappers/redis_client.go:225-245`) keys on the stamped `LocationType`; the cleanup rule keys on Google's *primary* type and deliberately keeps unmapped primaries (`meal_delivery`, `night_club`, empty `Types`) because the write path would legitimately place them there. A future refactor that "unifies" the two would reintroduce the incident. + +The dangerous direction is test-pinned: reverting the migration to `ReclassifyForCategory` fails `TestRemoveMisclassifiedPlacesPrimaryTypeTruthTable`. What is *not* pinned is someone changing `ReclassifyForCategory` itself, or collapsing both into a single shared helper. That is why the comment-precision items above are load-bearing rather than cosmetic. + +## Context worth not relearning + +- `POI.ReclassifyForCategory` has exactly **one** production caller, `planner/planner.go:1439` (the merchant endpoint). The trip-planning path — `planner/solver.go:532` → `matching.NearbySearchForCategory` → `matching.CreatePlace` — reads the same `placeIDs:eatery:level*` buckets and never reclassifies. Anything reasoning about "what the buckets contain" must account for both readers. +- `meal_delivery`, `night_club`, `liquor_store`, `convenience_store` are all legal legacy Places types (`maps@v1.7.0/types.go:257,264,253,227`) that Google routinely lists first in `types[]`. +- Legacy Nearby Search answers an unknown `?type=` by **ignoring the filter**, not by erroring. Places API (New) `searchNearby` rejects it with `INVALID_ARGUMENT` — see `2026-07-29-places-api-new-migration.md`. From bf6645e9ca6d1287e7aab0e32cc3f2b21b7443d1 Mon Sep 17 00:00:00 2001 From: tim-eternos Date: Thu, 30 Jul 2026 01:05:49 -0700 Subject: [PATCH 11/12] style: gofmt place_distance_sort_test.go Comment alignment only; the misalignment originated in the plan's test code block. Satisfies the repo PR checklist's gofmt requirement. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01DjdGnZM6cUBeg6jWwoQfU3 --- iowrappers/place_distance_sort_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/iowrappers/place_distance_sort_test.go b/iowrappers/place_distance_sort_test.go index b0b52e88..494a73ce 100644 --- a/iowrappers/place_distance_sort_test.go +++ b/iowrappers/place_distance_sort_test.go @@ -21,7 +21,7 @@ func TestSortPlacesByDistance(t *testing.T) { lat, lng := 37.38006, -122.11612 places := []POI.Place{ - placeAt("far-sunnyvale", 37.3688, -122.0363), // ~7km east + placeAt("far-sunnyvale", 37.3688, -122.0363), // ~7km east placeAt("mid-mountainview", 37.3861, -122.0839), // ~3km east placeAt("near-state-st", 37.38025, -122.11655), // ~40m away } From d089068c72e823078f81a13c4548426a77a4248e Mon Sep 17 00:00:00 2001 From: tim-eternos Date: Thu, 30 Jul 2026 08:11:02 -0700 Subject: [PATCH 12/12] docs: replace planning docs with a concise migration runbook The three planning documents were 2906 of this PR's 3902 added lines. They were scaffolding for the work; the code and tests are the deliverable. Full plans are preserved on origin/docs/place-type-plans. Keeps one runbook, since the migration deletes production data and the operator needs the rule, the measured scale, and the known residue. Numbers come from an actual read-only dry run against prod on 2026-07-30. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01DjdGnZM6cUBeg6jWwoQfU3 --- docs/migrations/reclassify-buckets.md | 86 + .../2026-07-29-eatery-place-type-fixes.md | 1041 ---------- .../2026-07-29-places-api-new-migration.md | 1796 ----------------- ...6-07-30-followups-from-place-type-fixes.md | 70 - 4 files changed, 86 insertions(+), 2907 deletions(-) create mode 100644 docs/migrations/reclassify-buckets.md delete mode 100644 docs/superpowers/plans/2026-07-29-eatery-place-type-fixes.md delete mode 100644 docs/superpowers/plans/2026-07-29-places-api-new-migration.md delete mode 100644 docs/superpowers/plans/2026-07-30-followups-from-place-type-fixes.md diff --git a/docs/migrations/reclassify-buckets.md b/docs/migrations/reclassify-buckets.md new file mode 100644 index 00000000..90b2bb3b --- /dev/null +++ b/docs/migrations/reclassify-buckets.md @@ -0,0 +1,86 @@ +# Migration: purge misclassified places from category geo buckets + +One-time cleanup for the `fast_food_restaurant` incident (#446). The legacy Nearby +Search ignores an unknown `?type=` instead of erroring, so searches for two +Places-API-(New)-only types returned prominence-ranked establishments that were then +stamped with the queried type — writing hotels into `placeIDs:eatery:level*`. + +`GET /v1/migrate/reclassify-buckets?category=Eatery` — admin only, **dry-run unless +`apply=true`**. Any other value of `apply` (absent, empty, `TRUE`, `1`) is a dry run. + +## Run the fixes first + +This must run **after** the write-path fixes in this PR are deployed. `8644199` is still +creating new bad records in production: every city/category/price-bucket whose +`MapsLastSearchTime` marker passes the 14-day `MinMapsResultRefreshDuration` does a fresh +cold search. Cleaning before the fix deploys just gets re-polluted. + +## The rule + +A member is removed only when its **primary** Google type maps to a *different* +category — the exact inverse of the write rule. An unmapped primary type is not +evidence of misclassification, because the write path would legitimately place it there. + +| Primary type | Resolves to | Action | +| --- | --- | --- | +| `lodging` | Lodging | removed | +| `supermarket`, `department_store` | Shopping | removed | +| `cafe`, `restaurant` | Eatery | kept | +| `meal_delivery`, `night_club` | unmapped | kept | +| no `types[]` at all | unmapped | kept | + +⚠️ This rule and the write rule (`SetPlacesAddGeoLocations`, which keys on the stamped +`LocationType`) deliberately disagree. A refactor that "unifies" them reintroduces the +incident. `TestRemoveMisclassifiedPlacesPrimaryTypeTruthTable` pins the dangerous direction. + +## Expected scale + +Measured against production on 2026-07-30 (24,203 bucket members scanned): + +| Category | Members | Candidates | +| --- | --- | --- | +| Eatery | 8,589 | 145 | +| Visit | 14,674 | 0 | +| Shopping | 498 | 12 | +| Wellness | 319 | 4 | +| Lodging | 123 | 3 | + +**164 total.** Of these, 123 are incident-attributable (stamped `fast_food_restaurant`, +mostly San Francisco / Tulsa / Boise hotels); the other 41 are pre-existing +misclassifications the rule also catches (hotels stamped `restaurant`, supermarkets +stamped `bakery`). + +**Known residue:** 27 incident records are *not* removed because their primary type maps +to no category — `university` ×5, `airport` ×2, `real_estate_agency` ×2, `stadium`, +`night_club`, `hardware_store`, `doctor`, and 7 with no `types[]`. These stay in the +eatery buckets. The trip-planning path (`planner/solver.go`) does not reclassify, so they +remain reachable in generated plans. Broadening `GetPlaceCategory` to cover those types is +tracked as follow-up work. + +Separately, ~328 bucket members have no backing `place_details` record. Pre-existing and +unrelated; the migration skips them. + +## Steps + +```bash +# 1. Dry run and READ THE OUTPUT before going further. +curl -s -H "Authorization: Bearer $ADMIN_JWT" \ + "https://best-vacation-planner.herokuapp.com/v1/migrate/reclassify-buckets?category=Eatery" | jq + +# 2. Apply. +curl -s -H "Authorization: Bearer $ADMIN_JWT" \ + "https://best-vacation-planner.herokuapp.com/v1/migrate/reclassify-buckets?category=Eatery&apply=true" | jq + +# 3. Re-run the dry run; misclassified should now be 0 (modulo the residue above). +``` + +Repeat per category as needed (`Shopping`, `Wellness`, `Lodging`). Reads are pipelined in +batches of 100 and the report carries `bucket_sizes`/`total_members`; the full Eatery scan +completed in ~1.3s against production, well inside Heroku's 30s H12. + +To force a cold search when spot-checking a city afterwards, drop its marker field +(format `::::`): + +```bash +redis-cli HDEL MapsLastSearchTime "united states:ca:los altos:eatery:0" +``` diff --git a/docs/superpowers/plans/2026-07-29-eatery-place-type-fixes.md b/docs/superpowers/plans/2026-07-29-eatery-place-type-fixes.md deleted file mode 100644 index 3bd55841..00000000 --- a/docs/superpowers/plans/2026-07-29-eatery-place-type-fixes.md +++ /dev/null @@ -1,1041 +0,0 @@ -# Eatery Place-Type Misclassification Fixes Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Stop the legacy Places API from writing mislabeled places into the category geo buckets, make the existing round-trip guard actually capable of failing, fix result truncation so it drops the farthest places rather than the last-searched place types, and purge the records already written to prod. - -**Architecture:** Four independent forward fixes on `origin/master`. The root cause is that `POI.GetPlaceCategory` has a `default:` branch returning `PlaceCategoryEatery`, which silently absorbs any place type the legacy Nearby Search does not understand. We convert that function to the `(value, ok)` shape already used by `ParsePlaceCategory`, so the compiler forces all three call sites to decide what an unknown type means; then we remove the two New-API-only types, reject unknown types at request-build time, sort by distance before truncating, and ship an admin migration to clean the buckets. - -**Tech Stack:** Go 1.24, Gin, go-redis v9, `googlemaps.github.io/maps` v1.7.0 (legacy Places API), testify. - -## Global Constraints - -- Go version: `1.24.0` (from `go.mod`) — do not raise it. -- Add no new module dependencies in this PR. -- CI gates are exactly `go build -v .` then `go test -v ./...` (`.github/workflows/go.yml`). Both must pass. -- Target branch is `origin/master` (the repo default; `origin/main` is stale at `1a1417b` and is not the deploy path). Commit `8644199` is already on `master` and live on Heroku app `best-vacation-planner`. -- Do not migrate to Places API (New) in this PR. That is the follow-up plan, `2026-07-29-places-api-new-migration.md`. -- Never run destructive Redis commands by hand against prod. Cleanup ships as a dry-run-by-default admin endpoint (Task 4). - ---- - -## Background: what is actually broken - -Verified against the code at `8644199`: - -1. `POI/categories.go:39-40` added `LocationTypeFastFood = "fast_food_restaurant"` and `LocationTypeFoodCourt = "food_court"`. Neither string exists anywhere in `googlemaps.github.io/maps@v1.7.0` — they are Places API (New) Table A types. -2. `iowrappers/nearby_search.go:87` builds the request with a direct cast, `Type: maps.PlaceType(placeType)`, bypassing the SDK's own `maps.ParsePlaceType` validator (`types.go:301`). The SDK then does `q.Set("type", string(r.Type))` (`places.go:125`) unchecked, so the unknown string reaches Google verbatim. -3. Legacy Nearby Search accepts the parameter, ignores the filter, and returns prominence-ranked establishments. `parsePlacesSearchResponse(searchResp, placeType, ...)` (`nearby_search.go:224`) then stamps the **queried** type onto every result via `POI.CreatePlace(..., locationType, ...)` (`:419`), so hotels get `LocationType: fast_food_restaurant`. `:421` (`place.Types = res.Types`) keeps Google's truthful types, which is why the damage is visible. -4. `iowrappers/redis_client.go:222` writes each place under `EncodeNearbySearchRedisKey(GetPlaceCategory(place.LocationType), place.PriceLevel)`. Neither new type appears in `GetPlaceCategory`'s Eatery case, so they reach Eatery through `default:` (`categories.go:77-78`) — meaning the invariant documented at `categories.go:61-64` was broken by the commit and the `default` hid it. -5. `TestPlaceCategoryRoundTrip` (`test/place_category_test.go`) passed anyway, because that `default` makes the test un-failable for *any* unknown type. The guard provides no protection today. -6. `POI.ReclassifyForCategory` (`categories.go:154-166`) drops these from API responses (primary type `lodging` is not in `GetPlaceTypes(Eatery)`), which is why the endpoint looks clean while the cache is dirty. - -Confirmed non-issues, so nobody wastes time on them: - -- ~~These records do **not** consume result slots in the response.~~ **CORRECTED after final review — this held only for the merchant endpoint, and was the most consequential error in this plan.** It is true that `planner/planner.go` (the `/v1/nearby-places-by-category` merchant handler) calls `POI.ReclassifyForCategory` *before* truncating, and that the Redis read is unbounded (`GeoRadius` with no `Count`, `redis_client.go:484-489`). But `ReclassifyForCategory` has exactly **one** production caller. The trip-planning path — `planner/solver.go:532` → `matching.NearbySearchForCategory` (`matching/matcher.go:76`) → `searcher.NearbySearch` → `matching.CreatePlace(place, req.Category)` (`:93`) — reads the same `placeIDs:eatery:level*` buckets and never reclassifies. The hotels were therefore slotted into generated trip plans as eateries: user-visible bad output, not merely cache residue. This is also why the cleanup migration must not delete aggressively — see Task 4. -- They did **not** waste Place Details spend. `detailsBudget` (`nearby_search.go:150`) is shared and consumed in `placeTypes` order, so `cafe`/`restaurant` exhaust it before the junk types are processed. -- Zero `food_court`-tagged places in prod is expected, not a contradiction. `placeMap` dedups by place ID across all types in one search, and `fast_food_restaurant` is processed first, so an identical ignored-filter response for `food_court` is entirely deduped away. - -The real cache-side cost is that these records inflate `len(cachedQualifiedPlaces)`, which is the radius-doubling break condition at `redis_client.go:520` — junk can satisfy `MinNumResults` and stop the radius from growing, so sparse areas return fewer genuine eateries. - ---- - -## File Structure - -| File | Responsibility in this PR | -| --- | --- | -| `POI/categories.go` | Remove the two New-API-only types from `GetPlaceTypes(Eatery)`; change `GetPlaceCategory` to `(PlaceCategory, bool)`; delete the two unused constants. | -| `iowrappers/redis_client.go` | Skip + log places whose type has no category, instead of writing them to Eatery (`:182`, `:222`). | -| `planner/planner.go` | Handle the new `ok` return at `:810`; sort by distance before truncating at `:1414`. | -| `iowrappers/nearby_search.go` | Validate place types in `CreateMapSearchRequest`; fix the dead `maxRetries` cap. | -| `iowrappers/data_migrations.go` | Add `RemoveMisclassifiedPlacesFromCategoryBuckets` for prod cleanup. | -| `iowrappers/maps_client.go` | Extend the `SearchClient`/migration interface with the new cleanup method. | -| `test/place_category_test.go` | Update for the new signature; the round-trip guard becomes meaningful. | -| `iowrappers/nearby_search_validation_test.go` (new) | Unit tests for place-type validation. | -| `iowrappers/place_distance_sort_test.go` (new) | Unit tests for distance sorting. | - ---- - -### Task 1: Make unknown place types un-mappable, and remove the two New-API-only types - -This is the root-cause fix. It must land as one commit — changing `GetPlaceCategory`'s signature without removing the two types would leave the build red on the round-trip test, which is exactly the point of the guard. - -**Files:** -- Modify: `POI/categories.go:37-79` (constants, `GetPlaceCategory`, `GetPlaceTypes`) -- Modify: `iowrappers/redis_client.go:182`, `iowrappers/redis_client.go:222` -- Modify: `planner/planner.go:810` -- Test: `test/place_category_test.go` - -**Interfaces:** -- Consumes: nothing from earlier tasks. -- Produces: `POI.GetPlaceCategory(placeType LocationType) (PlaceCategory, bool)` — returns `("", false)` when the type maps to no category. Tasks 2 and 4 rely on this exact signature. - -- [ ] **Step 1: Write the failing test** - -Add to `test/place_category_test.go`: - -```go -// TestGetPlaceCategoryRejectsUnknownTypes pins the fix for the fast_food_restaurant -// incident: GetPlaceCategory must NOT silently absorb unmapped types into Eatery. -// A default-to-Eatery branch made TestPlaceCategoryRoundTrip un-failable, so two -// Places-API-(New)-only types were added to GetPlaceTypes(Eatery) and hotels were -// written into the eatery geo buckets. -func TestGetPlaceCategoryRejectsUnknownTypes(t *testing.T) { - unknown := []POI.LocationType{ - POI.LocationType("fast_food_restaurant"), - POI.LocationType("food_court"), - POI.LocationType("lodging_but_not_really"), - POI.LocationType(""), - } - for _, placeType := range unknown { - if got, ok := POI.GetPlaceCategory(placeType); ok { - t.Errorf("GetPlaceCategory(%q) = (%q, true), want ok=false", placeType, got) - } - } -} - -// TestGetPlaceCategoryKnownTypes pins that every mapped type still resolves. -func TestGetPlaceCategoryKnownTypes(t *testing.T) { - cases := map[POI.LocationType]POI.PlaceCategory{ - POI.LocationTypeCafe: POI.PlaceCategoryEatery, - POI.LocationTypeRestaurant: POI.PlaceCategoryEatery, - POI.LocationTypeBar: POI.PlaceCategoryEatery, - POI.LocationTypeBakery: POI.PlaceCategoryEatery, - POI.LocationTypeMealTakeaway: POI.PlaceCategoryEatery, - POI.LocationTypePark: POI.PlaceCategoryVisit, - POI.LocationTypeMuseum: POI.PlaceCategoryVisit, - POI.LocationTypeStore: POI.PlaceCategoryShopping, - POI.LocationTypeLodging: POI.PlaceCategoryLodging, - POI.LocationTypeGym: POI.PlaceCategoryWellness, - } - for placeType, want := range cases { - got, ok := POI.GetPlaceCategory(placeType) - if !ok { - t.Errorf("GetPlaceCategory(%q) returned ok=false, want %q", placeType, want) - continue - } - if got != want { - t.Errorf("GetPlaceCategory(%q) = %q, want %q", placeType, got, want) - } - } -} -``` - -Update the two existing tests in the same file for the new signature and the reverted type list: - -```go -// in TestGetPlaceTypesByCategory, the Eatery entry becomes: - POI.PlaceCategoryEatery: { - POI.LocationTypeCafe, POI.LocationTypeRestaurant, - POI.LocationTypeBar, POI.LocationTypeBakery, POI.LocationTypeMealTakeaway, - }, - -// in TestPlaceCategoryRoundTrip, the assertion becomes: - got, ok := POI.GetPlaceCategory(placeType) - if !ok { - t.Errorf("round-trip broken: GetPlaceCategory(%q) has no category, want %q", placeType, category) - continue - } - if got != category { - t.Errorf("round-trip broken: GetPlaceCategory(%q) = %q, want %q", placeType, got, category) - } -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `go test ./test/ -run 'TestGetPlaceCategory|TestPlaceCategoryRoundTrip|TestGetPlaceTypesByCategory' -v` - -Expected: FAIL to compile with `assignment mismatch: 2 variables but POI.GetPlaceCategory returns 1 value`. That compile failure is the expected red state. - -- [ ] **Step 3: Change `GetPlaceCategory` and remove the two types** - -In `POI/categories.go`, delete these two constant lines (they have no remaining caller): - -```go - LocationTypeFastFood = LocationType("fast_food_restaurant") - LocationTypeFoodCourt = LocationType("food_court") -``` - -Replace `GetPlaceCategory` (currently `categories.go:61-79`) with: - -```go -// GetPlaceCategory maps a Google Maps place type back to its category, reporting whether -// the type is mapped at all. It is the inverse of GetPlaceTypes and MUST stay consistent -// with it: the nearby-search cache writes each place under -// EncodeNearbySearchRedisKey(GetPlaceCategory(place.LocationType), ...), so a type that -// resolves to a different category than the one it was searched under would never cache-hit. -// -// It deliberately has NO default category. An earlier version defaulted to Eatery, which -// silently absorbed place types the legacy Nearby Search does not understand — two -// Places-API-(New)-only types ("fast_food_restaurant", "food_court") were added to -// GetPlaceTypes(Eatery), Google ignored the unenforceable filter, and prominence-ranked -// hotels were written into the eatery geo buckets. Returning ok=false forces every caller -// to decide what an unmapped type means, and makes TestPlaceCategoryRoundTrip able to fail. -func GetPlaceCategory(placeType LocationType) (PlaceCategory, bool) { - switch placeType { - case LocationTypePark, LocationTypeAmusementPark, LocationTypeGallery, LocationTypeMuseum: - return PlaceCategoryVisit, true - case LocationTypeCafe, LocationTypeRestaurant, LocationTypeBar, LocationTypeBakery, LocationTypeMealTakeaway: - return PlaceCategoryEatery, true - case LocationTypeShoppingMall, LocationTypeDepartmentStore, LocationTypeSupermarket, LocationTypeClothingStore, LocationTypeStore: - return PlaceCategoryShopping, true - case LocationTypeLodging: - return PlaceCategoryLodging, true - case LocationTypeGym, LocationTypeSpa, LocationTypePharmacy: - return PlaceCategoryWellness, true - default: - return PlaceCategory(""), false - } -} -``` - -Revert the Eatery line in `GetPlaceTypes` back to five types: - -```go - case PlaceCategoryEatery: - placeTypes = append(placeTypes, - []LocationType{LocationTypeCafe, LocationTypeRestaurant, LocationTypeBar, LocationTypeBakery, LocationTypeMealTakeaway}...) -``` - -- [ ] **Step 4: Update the three call sites** - -`iowrappers/redis_client.go:222` — inside the `SetPlacesAddGeoLocations` pipeline. This is the write path that caused the incident, so it must refuse to guess: - -```go - for _, place := range placeBatch { - placeCategory, ok := POI.GetPlaceCategory(place.LocationType) - if !ok { - // Refuse to guess a bucket. A place whose type maps to no category - // came from a search whose type filter Google did not enforce, so - // writing it would poison whichever bucket we picked. - Logger.Errorf("SetPlacesAddGeoLocations: place %s has unmapped location type %q, skipping geo bucket write", - place.ID, place.LocationType) - continue - } - geoLocation := &redis.GeoLocation{ - Name: place.ID, - Latitude: place.GetLocation().Latitude, - Longitude: place.GetLocation().Longitude, - } - - redisKey := POI.EncodeNearbySearchRedisKey(placeCategory, place.PriceLevel) - pipe.GeoAdd(c, redisKey, geoLocation) -``` - -`iowrappers/redis_client.go:182` — inside the deprecated `StorePlacesForLocation`. Same rule, minimal change: - -```go - for _, place := range places { - placeCategory, ok := POI.GetPlaceCategory(place.LocationType) - if !ok { - Logger.Errorf("StorePlacesForLocation: place %s has unmapped location type %q, skipping", - place.ID, place.LocationType) - continue - } - sortedSetKey := strings.Join([]string{geocodeInString, string(placeCategory)}, "_") -``` - -`planner/planner.go:810` — this reads *saved plan* records out of `place_details:place_ID:`, which can include older cached places with legacy or empty types (brand searches write `LocationType: ""`). Preserve today's observable response here rather than changing an unrelated endpoint's output: - -```go - // Saved plans can contain older cached records, including brand-search places - // written with an empty LocationType. Preserve the historical Eatery default for - // display only — the write path (redis_client.go) is where guessing is unsafe. - if placeCategory, ok := POI.GetPlaceCategory(place.LocationType); ok { - resp.PlaceCategories[i] = placeCategory - } else { - resp.PlaceCategories[i] = POI.PlaceCategoryEatery - } -``` - -- [ ] **Step 5: Run the full suite to verify it passes** - -Run: `go build -v . && go test ./... 2>&1 | tail -30` - -Expected: PASS. In particular `TestGetPlaceCategoryRejectsUnknownTypes`, `TestGetPlaceCategoryKnownTypes`, `TestPlaceCategoryRoundTrip` and `TestGetPlaceTypesByCategory` all pass, and no package fails to compile. - -- [ ] **Step 6: Commit** - -```bash -git add POI/categories.go iowrappers/redis_client.go planner/planner.go test/place_category_test.go -git commit -m "fix: stop mapping unknown place types to Eatery - -GetPlaceCategory had a default branch returning Eatery, which silently -absorbed any place type the legacy Nearby Search does not understand. That -made TestPlaceCategoryRoundTrip un-failable, so fast_food_restaurant and -food_court (Places API (New) Table A types, absent from the v1.7.0 SDK) -were added to GetPlaceTypes(Eatery). Google ignored the unenforceable type -filter and returned prominence-ranked establishments, which were stamped -with the queried type and written into placeIDs:eatery:level* as hotels. - -Return (PlaceCategory, bool) so the compiler forces every caller to handle -an unmapped type, refuse the geo-bucket write instead of guessing, and -remove the two types." -``` - ---- - -### Task 2: Reject unknown place types when building the Maps request - -Defense in depth: Task 1 stops bad data reaching Redis, this stops the useless API call being made at all, and makes the failure loud. - -**Files:** -- Modify: `iowrappers/nearby_search.go:76-100` (`CreateMapSearchRequest`), `:141` (`maxRetries`), `:170-205` (Phase A/B) -- Test: `iowrappers/nearby_search_validation_test.go` (create) - -**Interfaces:** -- Consumes: nothing from Task 1 (independent). -- Produces: `CreateMapSearchRequest(reqIn *PlaceSearchRequest, placeType POI.LocationType, token string) (maps.NearbySearchRequest, error)` — returns a non-nil error when `placeType` is neither `POI.LocationTypeAny` nor a type `maps.ParsePlaceType` accepts. - -- [ ] **Step 1: Write the failing test** - -Create `iowrappers/nearby_search_validation_test.go`: - -```go -package iowrappers - -import ( - "strings" - "testing" - - "github.com/weihesdlegend/Vacation-planner/POI" -) - -// TestCreateMapSearchRequestRejectsUnknownPlaceType guards the fast_food_restaurant -// incident at the request boundary. The SDK casts POI.LocationType straight to -// maps.PlaceType and forwards it as ?type=, so an unknown value silently disables -// the filter server-side instead of erroring. Validate before spending the call. -func TestCreateMapSearchRequestRejectsUnknownPlaceType(t *testing.T) { - req := &PlaceSearchRequest{ - Location: POI.Location{Latitude: 37.38006, Longitude: -122.11612}, - PlaceCat: POI.PlaceCategoryEatery, - Radius: 8000, - PriceLevel: POI.PriceLevelTwo, - } - for _, placeType := range []POI.LocationType{ - POI.LocationType("fast_food_restaurant"), - POI.LocationType("food_court"), - POI.LocationType("not_a_google_type"), - } { - if _, err := CreateMapSearchRequest(req, placeType, ""); err == nil { - t.Errorf("CreateMapSearchRequest(%q) returned nil error, want validation failure", placeType) - } else if !strings.Contains(err.Error(), string(placeType)) { - t.Errorf("CreateMapSearchRequest(%q) error %q should name the offending type", placeType, err) - } - } -} - -// TestCreateMapSearchRequestAcceptsKnownPlaceTypes pins that every type the -// categories actually search for still builds a request. -func TestCreateMapSearchRequestAcceptsKnownPlaceTypes(t *testing.T) { - req := &PlaceSearchRequest{ - Location: POI.Location{Latitude: 37.38006, Longitude: -122.11612}, - PlaceCat: POI.PlaceCategoryEatery, - Radius: 8000, - PriceLevel: POI.PriceLevelTwo, - } - categories := []POI.PlaceCategory{ - POI.PlaceCategoryVisit, POI.PlaceCategoryEatery, - POI.PlaceCategoryShopping, POI.PlaceCategoryLodging, POI.PlaceCategoryWellness, - } - for _, category := range categories { - for _, placeType := range POI.GetPlaceTypes(category) { - got, err := CreateMapSearchRequest(req, placeType, "") - if err != nil { - t.Errorf("CreateMapSearchRequest(%q) in category %q: unexpected error %v", placeType, category, err) - continue - } - if string(got.Type) != string(placeType) { - t.Errorf("CreateMapSearchRequest(%q) set Type=%q, want %q", placeType, got.Type, placeType) - } - } - } -} - -// TestCreateMapSearchRequestAcceptsAnyType pins that keyword (brand) searches, -// which intentionally leave the type unset, are not rejected. -func TestCreateMapSearchRequestAcceptsAnyType(t *testing.T) { - req := &PlaceSearchRequest{ - Location: POI.Location{Latitude: 37.38006, Longitude: -122.11612}, - PlaceCat: POI.PlaceCategoryEatery, - Radius: 8000, - Keyword: "Dunkin'", - } - got, err := CreateMapSearchRequest(req, POI.LocationTypeAny, "") - if err != nil { - t.Fatalf("CreateMapSearchRequest(LocationTypeAny) returned error %v, want nil", err) - } - if got.Type != "" { - t.Errorf("CreateMapSearchRequest(LocationTypeAny) set Type=%q, want empty", got.Type) - } - if got.Keyword != "Dunkin'" { - t.Errorf("CreateMapSearchRequest kept Keyword=%q, want %q", got.Keyword, "Dunkin'") - } -} -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `go test ./iowrappers/ -run TestCreateMapSearchRequest -v` - -Expected: FAIL to compile with `assignment mismatch: 2 variables but CreateMapSearchRequest returns 1 value`. - -- [ ] **Step 3: Add validation to `CreateMapSearchRequest`** - -Replace the function at `iowrappers/nearby_search.go:75-100`: - -```go -// CreateMapSearchRequest creates a NearbySearchRequest for maps NearbySearch, adjust key settings such as radius and price levels. -// It rejects any place type the legacy Places API does not define: POI.LocationType is cast -// straight to maps.PlaceType and forwarded as ?type=, and Google responds to an unknown value -// by IGNORING the filter and returning prominence-ranked establishments rather than erroring. -// Those results are then stamped with the queried type, so an unvalidated type silently -// poisons the cache. maps.ParsePlaceType is the SDK's own list of legal values. -func CreateMapSearchRequest(reqIn *PlaceSearchRequest, placeType POI.LocationType, token string) (maps.NearbySearchRequest, error) { - // LocationTypeAny is the keyword (brand) search case: the type is deliberately unset so - // Google matches the keyword across all place types. - if placeType != POI.LocationTypeAny { - if _, err := maps.ParsePlaceType(string(placeType)); err != nil { - return maps.NearbySearchRequest{}, fmt.Errorf( - "place type %q is not a legacy Places API type (Places API (New) types are not accepted by /maps/api/place/nearbysearch): %w", - placeType, err) - } - } - - // Adjust radius, minPrice and maxPrice settings in search request - var radius = reqIn.Radius - var exactPriceLevel maps.PriceLevel - if POI.PriceyEatery(reqIn.PlaceCat, reqIn.PriceLevel) { - // increase search radius - radius = min(reqIn.Radius*4, GoogleNearbySearchMaxRadiusInMeters) - // set price filter - exactPriceLevel = maps.PriceLevel(fmt.Sprint(reqIn.PriceLevel)) - } - - return maps.NearbySearchRequest{ - Type: maps.PlaceType(placeType), - Location: &maps.LatLng{ - Lat: reqIn.Location.Latitude, - Lng: reqIn.Location.Longitude, - }, - Keyword: reqIn.Keyword, - Radius: radius, - PageToken: token, - RankBy: maps.RankBy("prominence"), - MinPrice: exactPriceLevel, - MaxPrice: exactPriceLevel, - }, nil -} -``` - -- [ ] **Step 4: Handle the error in Phase A and repair the dead retry cap** - -In `extensiveNearbySearch`, `iowrappers/nearby_search.go:141`, the cap is computed while `reqTimes` is still 0, so `maxRetries` is always 0 and the `break outer` at `:205` is unreachable. Fix it to the intended "every place type failed this round" meaning: - -```go - var reqTimes uint = 0 // number of queries for each location type - var totalPlaceCount uint = 0 // number of results so far, keep this number low - // Bail out once every place type has failed once. This was previously computed as - // reqTimes * len(placeTypes) while reqTimes was still 0, making the cap 0 and the - // break below unreachable, so a fully failing search span every retry round. - maxRetries := uint(len(placeTypes)) -``` - -In the Phase A goroutine at `:170-187`, surface the validation error the same way a fetch error is surfaced: - -```go - go func(i int, placeType POI.LocationType, token string) { - defer wg.Done() - searchReq, reqErr := CreateMapSearchRequest(request, placeType, token) - if reqErr != nil { - fetched[i].err = reqErr - return - } - select { - case c.apiSemaphore <- struct{}{}: - defer func() { <-c.apiSemaphore }() - case <-ctx.Done(): - fetched[i].err = ctx.Err() - return - } - fetched[i].resp, fetched[i].err = c.GoogleMapsNearbySearchWrapper(ctx, searchReq) - }(i, placeType, nextPageTokenMap[placeType]) -``` - -Then at `:199-205`, change the equality check to `>=` so a repaired cap cannot be stepped over: - -```go - if fetched[i].err != nil { - Logger.Error(fmt.Errorf("places nearby search with Maps failed for place type %s with error: %w", - placeType, fetched[i].err)) - mapsFailuresCount++ - if mapsFailuresCount >= maxRetries { - break outer - } - // we should still retry for the next place type if the number of failures is below maxRetries - continue - } -``` - -- [ ] **Step 5: Run tests to verify they pass** - -Run: `go build -v . && go test ./iowrappers/ -run TestCreateMapSearchRequest -v && go test ./... 2>&1 | tail -20` - -Expected: PASS on all three new tests and no regressions. `iowrappers/nearby_search_test.go` and `test/redis_client_mocks/...` must still pass. - -- [ ] **Step 6: Commit** - -```bash -git add iowrappers/nearby_search.go iowrappers/nearby_search_validation_test.go -git commit -m "fix: reject non-legacy place types before calling Nearby Search - -POI.LocationType was cast straight to maps.PlaceType and forwarded as -?type=. Google answers an unknown type by ignoring the filter, not by -erroring, so the call silently returns prominence-ranked establishments -that then get stamped with the queried type. Validate against -maps.ParsePlaceType first and fail loudly. - -Also repair the retry cap: maxRetries was computed as -reqTimes * len(placeTypes) while reqTimes was 0, so it was always 0 and -the break was dead code." -``` - ---- - -### Task 3: Sort by distance before truncating category results - -Independent pre-existing bug, worth its own commit. `planner/planner.go:1413` claims "Redis results are sorted by distance ascending", which is only true on the cache path. On the fresh path, results are appended per place type in `GetPlaceTypes` order, each type's page in Google prominence order (`nearby_search.go:224`). So a cold search produces cafe×20, restaurant×20, bar×20, bakery×20, meal_takeaway×20 and `places[:40]` keeps roughly cafes and restaurants while dropping bar, bakery and meal_takeaway entirely — the exact types commit `e299558` was added to surface. - -**Files:** -- Create: `iowrappers/place_distance_sort.go` -- Modify: `planner/planner.go:1411-1417` -- Test: `iowrappers/place_distance_sort_test.go` - -**Interfaces:** -- Consumes: nothing from earlier tasks. -- Produces: `iowrappers.SortPlacesByDistance(places []POI.Place, lat, lng float64)` — sorts `places` in place, ascending by haversine distance from `(lat, lng)`, stable so equal distances keep their prior order. - -- [ ] **Step 1: Write the failing test** - -Create `iowrappers/place_distance_sort_test.go`: - -```go -package iowrappers - -import ( - "testing" - - "github.com/weihesdlegend/Vacation-planner/POI" -) - -func placeAt(id string, lat, lng float64) POI.Place { - var p POI.Place - p.SetID(id) - p.SetLocationCoordinates([2]float64{lat, lng}) - return p -} - -// TestSortPlacesByDistance pins that truncation keeps the NEAREST places. The fresh -// (Google) path returns results grouped by place type in prominence order, so slicing -// without sorting first drops whole place types and can rank a 3km result above a 250m one. -func TestSortPlacesByDistance(t *testing.T) { - // State Street Market, Los Altos - lat, lng := 37.38006, -122.11612 - - places := []POI.Place{ - placeAt("far-sunnyvale", 37.3688, -122.0363), // ~7km east - placeAt("mid-mountainview", 37.3861, -122.0839), // ~3km east - placeAt("near-state-st", 37.38025, -122.11655), // ~40m away - } - - SortPlacesByDistance(places, lat, lng) - - want := []string{"near-state-st", "mid-mountainview", "far-sunnyvale"} - for i, id := range want { - if places[i].GetID() != id { - t.Errorf("position %d = %q, want %q (full order: %v)", i, places[i].GetID(), id, placeIDs(places)) - } - } -} - -// TestSortPlacesByDistanceEmpty pins that the no-result case does not panic. -func TestSortPlacesByDistanceEmpty(t *testing.T) { - var places []POI.Place - SortPlacesByDistance(places, 37.38006, -122.11612) - if len(places) != 0 { - t.Errorf("got %d places, want 0", len(places)) - } -} - -func placeIDs(places []POI.Place) []string { - ids := make([]string, 0, len(places)) - for _, p := range places { - ids = append(ids, p.GetID()) - } - return ids -} -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `go test ./iowrappers/ -run TestSortPlacesByDistance -v` - -Expected: FAIL with `undefined: SortPlacesByDistance`. - -- [ ] **Step 3: Write the implementation** - -Create `iowrappers/place_distance_sort.go`: - -```go -package iowrappers - -import ( - "sort" - - "github.com/weihesdlegend/Vacation-planner/POI" - "github.com/weihesdlegend/Vacation-planner/utils" -) - -// SortPlacesByDistance orders places ascending by distance from (lat, lng). -// -// Callers that truncate a candidate list to a limit MUST sort first. Only the Redis -// cache path returns places in distance order; the fresh path appends one place type's -// results after another in Google prominence order, so an unsorted slice[:limit] drops -// the last place types wholesale and can rank a 3km result above a 250m one. -// -// The sort is stable so places at equal distance keep their existing relative order. -func SortPlacesByDistance(places []POI.Place, lat, lng float64) { - origin := []float64{lat, lng} - dist := make(map[int]float64, len(places)) - for i := range places { - loc := places[i].GetLocation() - dist[i] = utils.HaversineDist(origin, []float64{loc.Latitude, loc.Longitude}) - } - idx := make([]int, len(places)) - for i := range idx { - idx[i] = i - } - sort.SliceStable(idx, func(a, b int) bool { return dist[idx[a]] < dist[idx[b]] }) - - sorted := make([]POI.Place, len(places)) - for newPos, oldPos := range idx { - sorted[newPos] = places[oldPos] - } - copy(places, sorted) -} -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `go test ./iowrappers/ -run TestSortPlacesByDistance -v` - -Expected: PASS on both tests. - -- [ ] **Step 5: Use it in the category handler** - -Replace `planner/planner.go:1411-1417`: - -```go - // drop places explicitly marked closed on the requested day - places = iowrappers.Filter(places, func(place POI.Place) bool { return !place.KnownClosedOnDay(day) }) - // Only the Redis path returns places in distance order. The fresh path - // appends each place type's results in Google prominence order, so sort - // before truncating or the last place types get dropped wholesale. - iowrappers.SortPlacesByDistance(places, req.Location.Latitude, req.Location.Longitude) - if len(places) > limit { - places = places[:limit] - } - result.Places = places -``` - -- [ ] **Step 6: Run the full suite** - -Run: `go build -v . && go test ./... 2>&1 | tail -20` - -Expected: PASS, no regressions. - -- [ ] **Step 7: Commit** - -```bash -git add iowrappers/place_distance_sort.go iowrappers/place_distance_sort_test.go planner/planner.go -git commit -m "fix: sort category results by distance before truncating - -The truncation at places[:limit] assumed distance ordering, which only -holds on the Redis cache path. The fresh path appends each place type's -results in Google prominence order, so a cold search kept roughly cafes -and restaurants and dropped bar, bakery and meal_takeaway entirely, and -could rank a 3km result above one 250m away." -``` - ---- - -### Task 4: Admin migration to purge misclassified places from category buckets - -Cleans the records already in prod. Must ship **after** Tasks 1-2 are deployed — otherwise the next cold search in any city recreates them. `MinMapsResultRefreshDuration` is 14 days (`iowrappers/poi_searcher.go`), so Los Altos is quiet, but every other city repopulates on its next search. - -Reuses the established pattern: an admin-authenticated GET under `v1.Group("/migrate")` (`planner/planner.go:1680-1684`), alongside `RemovePlaces`. - -**Files:** -- Modify: `iowrappers/data_migrations.go` -- Modify: `iowrappers/maps_client.go` (the searcher interface the handler calls through) -- Modify: `planner/planner.go` (handler + route) -- Test: `test/redis_client_mocks/bucket_cleanup_test.go` (create) - -**Interfaces:** -- Consumes: `POI.GetPlaceCategory(placeType) (PlaceCategory, bool)` from Task 1; `POI.PrimaryLocationType`, `POI.GetPlaceTypes` (existing). -- Produces: `(*RedisClient).RemoveMisclassifiedPlacesFromCategoryBuckets(ctx context.Context, cat POI.PlaceCategory, dryRun bool) (BucketCleanupReport, error)` and `type BucketCleanupReport struct { Scanned, Misclassified, Removed int; RemovedIDs []string }`. - -> **CORRECTED after final review — the Step 1/Step 3 code blocks below are superseded; read `iowrappers/data_migrations.go` for what shipped.** Two defects were caught in review: -> -> 1. **The removal rule below is wrong and too broad.** Step 3's `if _, keep := POI.ReclassifyForCategory(place, cat); keep` keeps a member only when its primary type is one of the category's five search types, so it deletes legitimate eateries whose primary type is a legal-but-unmapped legacy type (`meal_delivery`, `night_club`). Worse, it *contradicts the write path*, which keys on the stamped `LocationType` — the migration would delete rows the next cold search re-creates. The Interfaces block above is the one that got it right (`GetPlaceCategory` is what it consumes). What shipped is the inverse of the **write** rule: `primary := POI.PrimaryLocationType(place.Types); if c, ok := POI.GetPlaceCategory(primary); !ok || c == cat { continue }` — remove only when the primary type positively maps to a *different* category; keep on unmapped. -> 2. **`ZRange 0 -1` plus a serial `getPlace` per member is an N+1 that cannot finish inside Heroku's non-configurable 30s H12 router timeout**, and the handler passes `ctx.Request.Context()`, which cancels on client disconnect. For any real bucket the dry run never returns, so the operator cannot perform the review that step 2 of the deployment order mandates — defeating the migration's core safety property. What shipped reads records in pipelined batches of 100 (matching `SetPlacesAddGeoLocations`) and reports `bucket_sizes`/`total_members` from `ZCARD` up front so the operator knows the scale before running. -> -> Also note Step 1's `RedisMockSvr.FlushAll()` must NOT be used: `RedisClient`/`RedisMockSvr` are process-wide fixtures shared with every other test in `test/redis_client_mocks`, and flushing wipes sibling files' package-level `init()` fixtures, breaking 4 unrelated tests under the full suite. The shipped test file scopes its reset to its own fixture IDs instead. - -- [ ] **Step 1: Write the failing test** - -Create `test/redis_client_mocks/bucket_cleanup_test.go`. This follows the existing miniredis harness in that package (`RedisClient`, `RedisContext`, `RedisMockSvr` are package-level fixtures set up by its `TestMain`): - -```go -package redis_client_mocks - -import ( - "testing" - - "github.com/weihesdlegend/Vacation-planner/POI" - "github.com/weihesdlegend/Vacation-planner/iowrappers" -) - -// TestRemoveMisclassifiedPlacesDryRun pins that a dry run reports the hotels that the -// fast_food_restaurant incident wrote into placeIDs:eatery:level* without deleting them. -func TestRemoveMisclassifiedPlacesDryRun(t *testing.T) { - hotel := newPlaceWithTypes("hotel-1", "Residence Inn by Marriott Palo Alto", - POI.LocationType("fast_food_restaurant"), []string{"lodging", "point_of_interest", "establishment"}) - cafe := newPlaceWithTypes("cafe-1", "Peet's Coffee", - POI.LocationTypeCafe, []string{"cafe", "food", "point_of_interest", "establishment"}) - RedisClient.SetPlacesAddGeoLocations(RedisContext, []POI.Place{cafe}) - seedGeoBucket(t, POI.PlaceCategoryEatery, hotel) - - report, err := RedisClient.RemoveMisclassifiedPlacesFromCategoryBuckets(RedisContext, POI.PlaceCategoryEatery, true) - if err != nil { - t.Fatalf("RemoveMisclassifiedPlacesFromCategoryBuckets error: %v", err) - } - if report.Misclassified != 1 { - t.Errorf("Misclassified = %d, want 1 (report: %+v)", report.Misclassified, report) - } - if report.Removed != 0 { - t.Errorf("dry run Removed = %d, want 0", report.Removed) - } - if len(report.RemovedIDs) != 1 || report.RemovedIDs[0] != "hotel-1" { - t.Errorf("RemovedIDs = %v, want [hotel-1]", report.RemovedIDs) - } - // the hotel must still be present after a dry run - if got := countInEateryBuckets(t, "hotel-1"); got == 0 { - t.Error("dry run deleted hotel-1, want it retained") - } -} - -// TestRemoveMisclassifiedPlacesApply pins that a real run removes only the hotel. -func TestRemoveMisclassifiedPlacesApply(t *testing.T) { - RedisMockSvr.FlushAll() - - hotel := newPlaceWithTypes("hotel-2", "The Westin Palo Alto", - POI.LocationType("fast_food_restaurant"), []string{"lodging", "point_of_interest", "establishment"}) - cafe := newPlaceWithTypes("cafe-2", "Red Rock Coffee", - POI.LocationTypeCafe, []string{"cafe", "food", "point_of_interest", "establishment"}) - RedisClient.SetPlacesAddGeoLocations(RedisContext, []POI.Place{cafe}) - seedGeoBucket(t, POI.PlaceCategoryEatery, hotel) - - report, err := RedisClient.RemoveMisclassifiedPlacesFromCategoryBuckets(RedisContext, POI.PlaceCategoryEatery, false) - if err != nil { - t.Fatalf("RemoveMisclassifiedPlacesFromCategoryBuckets error: %v", err) - } - if report.Removed != 1 { - t.Errorf("Removed = %d, want 1 (report: %+v)", report.Removed, report) - } - if got := countInEateryBuckets(t, "hotel-2"); got != 0 { - t.Errorf("hotel-2 still in %d eatery buckets, want 0", got) - } - if got := countInEateryBuckets(t, "cafe-2"); got == 0 { - t.Error("cafe-2 was removed, want it retained") - } -} - -// TestRemoveMisclassifiedPlacesKeepsUntypedRecords pins that older cached records with -// no Types list are left alone, matching ReclassifyForCategory's keep-on-unknown rule. -func TestRemoveMisclassifiedPlacesKeepsUntypedRecords(t *testing.T) { - RedisMockSvr.FlushAll() - - legacy := newPlaceWithTypes("legacy-1", "Old Cached Diner", POI.LocationTypeRestaurant, nil) - RedisClient.SetPlacesAddGeoLocations(RedisContext, []POI.Place{legacy}) - - report, err := RedisClient.RemoveMisclassifiedPlacesFromCategoryBuckets(RedisContext, POI.PlaceCategoryEatery, false) - if err != nil { - t.Fatalf("RemoveMisclassifiedPlacesFromCategoryBuckets error: %v", err) - } - if report.Removed != 0 { - t.Errorf("Removed = %d, want 0 — records without Types must be kept", report.Removed) - } -} - -func newPlaceWithTypes(id, name string, locationType POI.LocationType, types []string) POI.Place { - var p POI.Place - p.SetID(id) - p.SetName(name) - p.SetType(locationType) - p.SetStatus(string(POI.Operational)) - p.SetPriceLevel(POI.PriceLevelDefault) - p.SetUserRatingsTotal(100) - p.SetLocationCoordinates([2]float64{37.38006, -122.11612}) - p.Types = types - return p -} - -// seedGeoBucket writes a place record plus its eatery geo-bucket membership directly, -// bypassing SetPlacesAddGeoLocations, which after Task 1 refuses unmapped types. -func seedGeoBucket(t *testing.T, cat POI.PlaceCategory, place POI.Place) { - t.Helper() - if err := RedisClient.SetPlace(RedisContext, place); err != nil { - t.Fatalf("SetPlace(%s): %v", place.GetID(), err) - } - key := POI.EncodeNearbySearchRedisKey(cat, place.PriceLevel) - if err := RedisClient.AddGeoLocation(RedisContext, key, place); err != nil { - t.Fatalf("AddGeoLocation(%s): %v", key, err) - } -} - -func countInEateryBuckets(t *testing.T, placeID string) int { - t.Helper() - count := 0 - for _, lvl := range POI.AllPriceLevels { - key := POI.EncodeNearbySearchRedisKey(POI.PlaceCategoryEatery, lvl) - if RedisMockSvr.Exists(key) { - members, err := RedisMockSvr.ZMembers(key) - if err != nil { - continue - } - for _, m := range members { - if m == placeID { - count++ - } - } - } - } - return count -} -``` - -Note for the implementer: this test needs two small exported helpers on `RedisClient` that do not exist yet — `SetPlace(ctx, place) error` and `AddGeoLocation(ctx, key string, place POI.Place) error`. `setPlace` already exists unexported (`iowrappers/redis_client.go`, used by `StorePlacesForLocation`); add thin exported wrappers in Step 3 rather than duplicating logic. - -- [ ] **Step 2: Run test to verify it fails** - -Run: `go test ./test/redis_client_mocks/ -run TestRemoveMisclassifiedPlaces -v` - -Expected: FAIL to compile with `RedisClient.RemoveMisclassifiedPlacesFromCategoryBuckets undefined` (and undefined `SetPlace` / `AddGeoLocation`). - -- [ ] **Step 3: Write the implementation** - -Append to `iowrappers/data_migrations.go`: - -```go -// BucketCleanupReport summarizes a RemoveMisclassifiedPlacesFromCategoryBuckets run. -type BucketCleanupReport struct { - Scanned int `json:"scanned"` - Misclassified int `json:"misclassified"` - Removed int `json:"removed"` - RemovedIDs []string `json:"removed_ids"` -} - -// SetPlace stores a single place record. Exported wrapper over setPlace for migrations and tests. -func (r *RedisClient) SetPlace(ctx context.Context, place POI.Place) error { - return r.setPlace(ctx, place) -} - -// AddGeoLocation adds a place to a geo bucket under an explicit key. Exported for -// migrations and tests that need to write buckets the normal write path would reject. -func (r *RedisClient) AddGeoLocation(ctx context.Context, key string, place POI.Place) error { - loc := place.GetLocation() - _, err := r.client.GeoAdd(ctx, key, &redis.GeoLocation{ - Name: place.ID, - Latitude: loc.Latitude, - Longitude: loc.Longitude, - }).Result() - return err -} - -// RemoveMisclassifiedPlacesFromCategoryBuckets removes places from cat's geo buckets whose -// PRIMARY Google type does not belong to cat. It repairs the fast_food_restaurant incident: -// two Places-API-(New)-only types were searched against the legacy Nearby Search, which -// ignored the unenforceable type filter and returned prominence-ranked establishments, and -// those were stamped with the queried type and written into placeIDs:eatery:level*. -// -// It uses the same rule as POI.ReclassifyForCategory, which is what already hides these from -// API responses: classify by primary type, and KEEP records with no Types list (older cached -// records written before Types was captured) so coverage never regresses. -// -// dryRun reports what would be removed without deleting anything. Always dry-run first. -func (r *RedisClient) RemoveMisclassifiedPlacesFromCategoryBuckets(ctx context.Context, cat POI.PlaceCategory, dryRun bool) (BucketCleanupReport, error) { - report := BucketCleanupReport{RemovedIDs: make([]string, 0)} - - levels := []POI.PriceLevel{POI.PriceLevelDefault} - if cat == POI.PlaceCategoryEatery { - levels = POI.AllPriceLevels - } - - for _, level := range levels { - key := POI.EncodeNearbySearchRedisKey(cat, level) - members, err := r.client.ZRange(ctx, key, 0, -1).Result() - if err != nil { - return report, fmt.Errorf("reading geo bucket %s: %w", key, err) - } - for _, placeID := range members { - report.Scanned++ - place, err := r.getPlace(ctx, placeID) - if err != nil { - // no place record backing this bucket member; leave it for RemovePlaces - Logger.Debugf("RemoveMisclassifiedPlacesFromCategoryBuckets: no record for %s in %s", placeID, key) - continue - } - if _, keep := POI.ReclassifyForCategory(place, cat); keep { - continue - } - report.Misclassified++ - report.RemovedIDs = append(report.RemovedIDs, placeID) - Logger.Infof("RemoveMisclassifiedPlacesFromCategoryBuckets: %s (%q, LocationType=%q, Types=%v) does not belong in %s", - placeID, place.Name, place.LocationType, place.Types, key) - if dryRun { - continue - } - if _, err := r.client.ZRem(ctx, key, placeID).Result(); err != nil { - return report, fmt.Errorf("removing %s from %s: %w", placeID, key, err) - } - report.Removed++ - } - } - return report, nil -} -``` - -Add `"github.com/redis/go-redis/v9"` to the imports of `data_migrations.go` if it is not already present, alongside the existing `context`, `fmt`, and `POI` imports. - -Add the method to the migration-capable interface in `iowrappers/maps_client.go` so the Gin handler can call it through `p.Solver.Searcher`. Locate the interface that already declares `RemovePlaces` and add: - -```go - RemoveMisclassifiedPlacesFromCategoryBuckets(context.Context, POI.PlaceCategory, bool) (BucketCleanupReport, error) -``` - -Then add the forwarding method on `PoiSearcher` in `data_migrations.go`, mirroring the existing `(*PoiSearcher).RemovePlaces`: - -```go -func (s *PoiSearcher) RemoveMisclassifiedPlacesFromCategoryBuckets(ctx context.Context, cat POI.PlaceCategory, dryRun bool) (BucketCleanupReport, error) { - return s.redisClient.RemoveMisclassifiedPlacesFromCategoryBuckets(ctx, cat, dryRun) -} -``` - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `go build -v . && go test ./test/redis_client_mocks/ -run TestRemoveMisclassifiedPlaces -v` - -Expected: PASS on all three tests. - -- [ ] **Step 5: Add the admin handler and route** - -Add to `planner/planner.go`, next to `removePlacesMigrationHandler`: - -```go -// reclassifyBucketsMigrationHandler removes places from a category's geo buckets whose -// primary Google type does not belong to that category. Dry-run unless ?apply=true. -// -// Usage: GET /v1/migrate/reclassify-buckets?category=Eatery -// GET /v1/migrate/reclassify-buckets?category=Eatery&apply=true -func (p *MyPlanner) reclassifyBucketsMigrationHandler(ctx *gin.Context) { - _, authenticationErr := p.UserAuthentication(ctx, user.LevelAdmin) - if authenticationErr != nil { - ctx.JSON(http.StatusUnauthorized, gin.H{"error": authenticationErr.Error()}) - return - } - category, ok := POI.ParsePlaceCategory(ctx.DefaultQuery("category", string(POI.PlaceCategoryEatery))) - if !ok { - ctx.JSON(http.StatusBadRequest, gin.H{"error": "unknown category"}) - return - } - dryRun := ctx.Query("apply") != "true" - report, err := p.Solver.Searcher.RemoveMisclassifiedPlacesFromCategoryBuckets(ctx.Request.Context(), category, dryRun) - if err != nil { - ctx.JSON(http.StatusInternalServerError, gin.H{"error": err.Error(), "partial_report": report}) - return - } - ctx.JSON(http.StatusOK, gin.H{"dry_run": dryRun, "category": category, "report": report}) -} -``` - -Register it at `planner/planner.go:1684`, inside the existing `migrations` group: - -```go - migrations.GET("/reclassify-buckets", p.reclassifyBucketsMigrationHandler) -``` - -- [ ] **Step 6: Run the full suite** - -Run: `go build -v . && go test ./... 2>&1 | tail -20` - -Expected: PASS. - -- [ ] **Step 7: Commit** - -```bash -git add iowrappers/data_migrations.go iowrappers/maps_client.go planner/planner.go test/redis_client_mocks/bucket_cleanup_test.go -git commit -m "feat: add admin migration to purge misclassified places from geo buckets - -The fast_food_restaurant incident wrote prominence-ranked hotels into -placeIDs:eatery:level*. ReclassifyForCategory already hides them from API -responses, but they inflate the bucket counts that gate radius expansion. -Remove them using the same primary-type rule, dry-run by default." -``` - ---- - -## Deployment order - -1. Merge and deploy the whole PR — Tasks 1-4 ship together, so after deploy the operator has both the forward fixes (no new bad writes) *and* the cleanup endpoint. The "fix the write path before cleaning the data" ordering property still holds by construction, not by deploy sequencing: the endpoint is dry-run by default, so nothing is deleted until step 3 is run by hand. Verify `go test ./...` green in CI. -2. Dry-run the cleanup and read the report before applying: - ```bash - curl -s -H "Authorization: Bearer $ADMIN_JWT" \ - "https://best-vacation-planner.herokuapp.com/v1/migrate/reclassify-buckets?category=Eatery" | jq - ``` - Read `bucket_sizes` / `total_members` first — they are measured with `ZCARD` before the scan and tell you the scale of the job. - - Expect roughly 17 entries in `removed_ids` for the Los Altos hotels, plus any older misclassifications the rule catches. Know exactly what the rule does before you review the list: - - - **Removed** — the member's primary Google type (`POI.PrimaryLocationType`: first entry of `types[]` that is not an umbrella type like `food`/`point_of_interest`/`establishment`) maps to a *different* category. `lodging` → `Lodging`, `supermarket` → `Shopping`. These are the incident's hotels. - - **Kept: unmapped primary types.** `meal_delivery` (a delivery-first restaurant) and `night_club` (a bar that is also a club) are legal legacy types Google routinely lists first for genuine eateries, and `GetPlaceCategory` maps neither to any category. An unmapped primary type is *not* evidence of misclassification — the fixed write path keys on the stamped `LocationType`, so it would legitimately file these under Eatery, and deleting them would only remove rows the next cold search re-creates while shrinking the trip-planning candidate pool for up to `MinMapsResultRefreshDuration` (14 days) per city/price-level. - - **Kept: records with no `types[]` at all** (cached before `Types` was captured), so coverage never regresses on old data. - - Seeing `meal_delivery`/`night_club` places *absent* from `removed_ids` is the rule working, not a miss. Review the list before proceeding. -3. Apply: - ```bash - curl -s -H "Authorization: Bearer $ADMIN_JWT" \ - "https://best-vacation-planner.herokuapp.com/v1/migrate/reclassify-buckets?category=Eatery&apply=true" | jq - ``` -4. Spot-check that a cold search is correct now. `MapsLastSearchTime` gates on a 14-day TTL, so force a fresh path by deleting the marker field for the city under test: - ```bash - # field format: "::::" - redis-cli HDEL MapsLastSearchTime "united states:ca:los altos:eatery:0" - ``` - Then re-run the category search and confirm the nearest result is the nearest by distance, not by Google prominence. - -## Verification checklist - -- [ ] `go build -v .` and `go test -v ./...` pass locally and in CI. -- [ ] `TestPlaceCategoryRoundTrip` fails if you temporarily re-add `LocationTypeFastFood` to `GetPlaceTypes(Eatery)` — confirm the guard is now real, then revert the experiment. -- [ ] No `LocationType` constant exists for either New-API-only string. A bare text search is the wrong check — the strings legitimately appear in the guard tests (which must name them to assert they are rejected), in the incident narrative in doc comments, and in the migration fixtures, so `grep -rn 'fast_food_restaurant\|food_court' --include='*.go' .` returns matches by design. Check the actual invariant instead: - ```bash - # must print nothing: no constant may reintroduce either type - grep -nE '=[[:space:]]*LocationType\("(fast_food_restaurant|food_court)"\)' POI/categories.go - ``` -- [ ] A category search at State Street Market returns Peet's at 367 State St ahead of results in Sunnyvale. -- [ ] Dry-run report reviewed before any `apply=true` call. - -## Out of scope, tracked in the follow-up plan - -Correctly classifying fast food and food courts needs Places API (New) `searchNearby` with `includedPrimaryTypes`. Legacy `types[]` never contains those values, so `POI.PrimaryLocationType` can never return them either — no amount of client-side work fixes it on the current API. See `2026-07-29-places-api-new-migration.md`. diff --git a/docs/superpowers/plans/2026-07-29-places-api-new-migration.md b/docs/superpowers/plans/2026-07-29-places-api-new-migration.md deleted file mode 100644 index a27dd8e5..00000000 --- a/docs/superpowers/plans/2026-07-29-places-api-new-migration.md +++ /dev/null @@ -1,1796 +0,0 @@ -# Places API (New) Migration — Search and Photos Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Move the category search and place-photo paths off the legacy Places API (`maps.googleapis.com/maps/api/place/*`) onto Places API (New) (`places.googleapis.com/v1`), using `places:searchNearby` with `includedPrimaryTypes` so place types are filtered server-side by primary type, and bucketing results by the `priceLevel` the response already returns. - -**Architecture:** A new self-contained `iowrappers/placesv1` package speaks the New API over `net/http` — `googlemaps.github.io/maps` v1.7.0 has no support for it and the v1.7.0 client stays in place for Geocoding (which is not deprecated) and for the brand/keyword path (deferred to a follow-up). A `PLACES_API_VERSION` env flag selects legacy or new at runtime, and the new path writes under versioned Redis keys (`v2:placeIDs:*`, `v2:place_details:*`) so both datasets coexist and cutover is reversible by flipping one Heroku config var. - -**Tech Stack:** Go 1.24, `net/http` + `encoding/json` (no new dependencies), Gin, go-redis v9, `googlemaps.github.io/maps` v1.7.0 (retained for Geocoding only). - -## Global Constraints - -- Go version: `1.24.0` (from `go.mod`) — do not raise it. -- **Add no new module dependencies.** The New API is plain REST; use `net/http`. Do not add a Google client library. -- CI gates are exactly `go build -v .` then `go test -v ./...` (`.github/workflows/go.yml`). Both must pass. -- Target branch is `origin/master`. `origin/main` is stale at `1a1417b` and is not the deploy path. -- **Prerequisite:** the fixes in `2026-07-29-eatery-place-type-fixes.md` must be merged and deployed first. This plan re-adds `fast_food_restaurant` and `food_court` in Task 6, which is only safe once type validation and the `(PlaceCategory, bool)` signature exist. -- Every New API request MUST send an `X-Goog-FieldMask` header. There is no default field list; a missing mask is an error, and an over-broad mask is billed at a higher SKU. -- Do not migrate Geocoding or ReverseGeocode. The Geocoding API (`/maps/api/geocode/json`) is a separate, non-deprecated API and stays on the v1.7.0 SDK. -- Do not migrate the brand/keyword search path in this PR. `searchNearby` has no keyword parameter; that path needs `places:searchText` and is deferred. -- No unit test may make a real network call. Use `httptest` with recorded response bodies. - ---- - -## Why this migration, and what it buys - -Verified constraints of each endpoint (Google reference docs, checked 2026-07-29): - -| Capability | Legacy nearbysearch | `places:searchNearby` (New) | `places:searchText` (New) | -| --- | --- | --- | --- | -| Types per call | 1 (`type`) | **many** (`includedTypes`, `includedPrimaryTypes`) | 1 (`includedType`) | -| Max results | 20/page, 3 pages | **20, no pagination** | 20/page, 60 total | -| Price filter | `minprice`/`maxprice` | none | `priceLevels` | -| Keyword | `keyword` | none | `textQuery` | -| Rank | `rankby` | `rankPreference: DISTANCE\|POPULARITY` | `rankPreference: DISTANCE\|RELEVANCE` | -| Radius cap | 50000 m | 50000 m | n/a (bias/restriction) | - -What the chosen approach fixes or improves: - -1. **`fast_food_restaurant` and `food_court` become real.** Both are valid Table A types in the New API. `includedPrimaryTypes` filters by *primary* type server-side — which is exactly what `POI.ReclassifyForCategory` currently approximates client-side after the fact. -2. **One HTTP call replaces up to 35.** Today a cold Eatery search issues 5-7 Nearby Searches per round for up to 5 rounds (`GoogleMapsSearchCallMaxCount = 5`). The new path issues one `searchNearby` with all types in `includedPrimaryTypes`. -3. **The separate Place Details fan-out disappears for search results.** `searchNearby` returns opening hours, `adrFormatAddress`, `googleMapsUri`, `userRatingCount`, `editorialSummary` and `photos` directly via the field mask. `searchPlaceDetails` and its `detailsBudget` (`iowrappers/nearby_search.go:150`) are not needed on the new path. -4. **`rankPreference: DISTANCE` fixes ordering at the source**, complementing the client-side sort added in the prior PR. - -**The cost, stated plainly:** a hard cap of 20 results per search versus roughly 100-140 raw results today. Task 5 measures this against production data on real cities and gates the cutover on the result. If coverage is unacceptable, the mitigation is already designed in: `SearchNearby` accepts *groups* of types, so splitting `includedPrimaryTypes` into one group per type restores today's ~140-result ceiling at 7 calls — still far cheaper than today's 35, and still with correct server-side primary-type filtering. Do not skip Task 5. - ---- - -## File Structure - -| File | Responsibility | -| --- | --- | -| `iowrappers/placesv1/client.go` (new) | HTTP transport for `places.googleapis.com/v1`: auth header, field mask, timeouts, error decoding. Knows nothing about POI types. | -| `iowrappers/placesv1/types.go` (new) | Request/response structs mirroring the New API JSON exactly (`Place`, `LocalizedText`, `OpeningHours`, `Photo`, enums). | -| `iowrappers/placesv1/search_nearby.go` (new) | `SearchNearby` request building and the multi-group fan-out. | -| `iowrappers/placesv1/photo.go` (new) | Builds the `/v1/{photoName}/media` URL and fetches image bytes. | -| `iowrappers/places_v1_mapper.go` (new) | Maps `placesv1.Place` → `POI.Place`. The only place that knows both vocabularies. | -| `iowrappers/places_v1_search_client.go` (new) | Implements `SearchClient.NearbySearch` against the New API; delegates Geocode/ReverseGeocode to the existing `MapsClient`. | -| `iowrappers/redis_keys.go` (new) | Versioned Redis key building shared by both paths. | -| `POI/categories.go` | Task 6 only: re-add the two types now that they work. | -| `iowrappers/photos_client.go` | Route by reference format: new `places/...` refs to the New media endpoint, legacy refs to the SDK. | -| `iowrappers/poi_searcher.go` | Select the search client from `PLACES_API_VERSION`. | -| `config/config.yml` | New-API field mask; retain the legacy `detailed_search_fields` for the legacy path. | - ---- - -### Task 1: `placesv1` HTTP client and response types - -**Files:** -- Create: `iowrappers/placesv1/client.go`, `iowrappers/placesv1/types.go` -- Test: `iowrappers/placesv1/client_test.go` - -**Interfaces:** -- Produces: - - `placesv1.New(apiKey string, opts ...Option) *Client`, `placesv1.WithBaseURL(string) Option`, `placesv1.WithHTTPClient(*http.Client) Option` - - `(*Client).post(ctx context.Context, path, fieldMask string, body any, out any) error` - - `placesv1.Place`, `placesv1.LocalizedText`, `placesv1.OpeningHours`, `placesv1.Photo`, `placesv1.LatLng` - - `placesv1.APIError` with `Code int`, `Status string`, `Message string` -- Tasks 2, 3 and 4 all depend on these exact names. - -- [ ] **Step 1: Write the failing test** - -Create `iowrappers/placesv1/client_test.go`: - -```go -package placesv1 - -import ( - "context" - "encoding/json" - "errors" - "net/http" - "net/http/httptest" - "testing" -) - -func TestClientSendsAPIKeyAndFieldMask(t *testing.T) { - var gotKey, gotMask, gotContentType string - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - gotKey = r.Header.Get("X-Goog-Api-Key") - gotMask = r.Header.Get("X-Goog-FieldMask") - gotContentType = r.Header.Get("Content-Type") - w.Header().Set("Content-Type", "application/json") - _, _ = w.Write([]byte(`{"places":[]}`)) - })) - defer srv.Close() - - c := New("test-key", WithBaseURL(srv.URL)) - var out SearchNearbyResponse - if err := c.post(context.Background(), "/v1/places:searchNearby", "places.id", map[string]any{}, &out); err != nil { - t.Fatalf("post returned %v, want nil", err) - } - if gotKey != "test-key" { - t.Errorf("X-Goog-Api-Key = %q, want %q", gotKey, "test-key") - } - if gotMask != "places.id" { - t.Errorf("X-Goog-FieldMask = %q, want %q", gotMask, "places.id") - } - if gotContentType != "application/json" { - t.Errorf("Content-Type = %q, want application/json", gotContentType) - } -} - -func TestClientDecodesAPIError(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - w.Header().Set("Content-Type", "application/json") - w.WriteHeader(http.StatusBadRequest) - _, _ = w.Write([]byte(`{"error":{"code":400,"message":"Invalid included primary type: nonsense_type","status":"INVALID_ARGUMENT"}}`)) - })) - defer srv.Close() - - c := New("test-key", WithBaseURL(srv.URL)) - var out SearchNearbyResponse - err := c.post(context.Background(), "/v1/places:searchNearby", "places.id", map[string]any{}, &out) - if err == nil { - t.Fatal("post returned nil error, want APIError") - } - var apiErr *APIError - if !errors.As(err, &apiErr) { - t.Fatalf("error %v is not *APIError", err) - } - if apiErr.Code != 400 || apiErr.Status != "INVALID_ARGUMENT" { - t.Errorf("APIError = %+v, want code 400 status INVALID_ARGUMENT", apiErr) - } - // Unlike the legacy API, which silently ignored an unknown type, the New API rejects it. - if apiErr.Message == "" { - t.Error("APIError.Message is empty, want Google's explanation") - } -} - -func TestPlaceJSONDecodesNewAPIShape(t *testing.T) { - // Trimmed real-shape response body. - body := `{"places":[{ - "id":"ChIJ_test", - "types":["cafe","food","point_of_interest","establishment"], - "primaryType":"cafe", - "formattedAddress":"367 State St, Los Altos, CA 94022, USA", - "adrFormatAddress":"367 State St", - "location":{"latitude":37.38025,"longitude":-122.11655}, - "rating":4.3, - "userRatingCount":412, - "googleMapsUri":"https://maps.google.com/?cid=1", - "businessStatus":"OPERATIONAL", - "priceLevel":"PRICE_LEVEL_INEXPENSIVE", - "displayName":{"text":"Peet's Coffee","languageCode":"en"}, - "editorialSummary":{"text":"Coffee chain known for house blends.","languageCode":"en"}, - "regularOpeningHours":{"openNow":true,"weekdayDescriptions":[ - "Monday: 5:30 AM – 7:00 PM","Tuesday: 5:30 AM – 7:00 PM","Wednesday: 5:30 AM – 7:00 PM", - "Thursday: 5:30 AM – 7:00 PM","Friday: 5:30 AM – 7:00 PM","Saturday: 6:00 AM – 7:00 PM", - "Sunday: 6:00 AM – 7:00 PM"]}, - "photos":[{"name":"places/ChIJ_test/photos/AT_abc","widthPx":4032,"heightPx":3024}] - }]}` - - var resp SearchNearbyResponse - if err := json.Unmarshal([]byte(body), &resp); err != nil { - t.Fatalf("Unmarshal error: %v", err) - } - if len(resp.Places) != 1 { - t.Fatalf("got %d places, want 1", len(resp.Places)) - } - p := resp.Places[0] - if p.ID != "ChIJ_test" { - t.Errorf("ID = %q, want ChIJ_test", p.ID) - } - if p.DisplayName.Text != "Peet's Coffee" { - t.Errorf("DisplayName.Text = %q, want Peet's Coffee", p.DisplayName.Text) - } - if p.PrimaryType != "cafe" { - t.Errorf("PrimaryType = %q, want cafe", p.PrimaryType) - } - if p.PriceLevel != PriceLevelInexpensive { - t.Errorf("PriceLevel = %q, want %q", p.PriceLevel, PriceLevelInexpensive) - } - if len(p.RegularOpeningHours.WeekdayDescriptions) != 7 { - t.Errorf("got %d weekday descriptions, want 7", len(p.RegularOpeningHours.WeekdayDescriptions)) - } - if len(p.Photos) != 1 || p.Photos[0].Name != "places/ChIJ_test/photos/AT_abc" { - t.Errorf("Photos = %+v, want one photo named places/ChIJ_test/photos/AT_abc", p.Photos) - } - if p.EditorialSummary.Text == "" { - t.Error("EditorialSummary.Text is empty") - } -} -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `go test ./iowrappers/placesv1/ -v` - -Expected: FAIL — the package does not exist yet (`no Go files in .../placesv1`). - -- [ ] **Step 3: Write `types.go`** - -Create `iowrappers/placesv1/types.go`: - -```go -// Package placesv1 is a minimal client for Google Places API (New), -// https://places.googleapis.com/v1. The googlemaps.github.io/maps v1.7.0 SDK only -// implements the legacy /maps/api/place/* endpoints, so this speaks REST directly. -// -// Every request must carry an X-Goog-FieldMask; the New API has no default field set. -package placesv1 - -// PriceLevel is the New API's price enum. Unlike the legacy integer priceLevel, an -// absent value is explicit (PriceLevelUnspecified) rather than indistinguishable from 0. -type PriceLevel string - -const ( - PriceLevelUnspecified PriceLevel = "PRICE_LEVEL_UNSPECIFIED" - PriceLevelFree PriceLevel = "PRICE_LEVEL_FREE" - PriceLevelInexpensive PriceLevel = "PRICE_LEVEL_INEXPENSIVE" - PriceLevelModerate PriceLevel = "PRICE_LEVEL_MODERATE" - PriceLevelExpensive PriceLevel = "PRICE_LEVEL_EXPENSIVE" - PriceLevelVeryExpensive PriceLevel = "PRICE_LEVEL_VERY_EXPENSIVE" -) - -// BusinessStatus values match POI.BusinessStatus strings exactly, so no translation -// table is needed: OPERATIONAL, CLOSED_TEMPORARILY, CLOSED_PERMANENTLY. -type BusinessStatus string - -// RankPreference selects result ordering for searchNearby. -type RankPreference string - -const ( - RankPreferenceDistance RankPreference = "DISTANCE" - RankPreferencePopularity RankPreference = "POPULARITY" -) - -// LocalizedText backs displayName and editorialSummary. -type LocalizedText struct { - Text string `json:"text"` - LanguageCode string `json:"languageCode"` -} - -type LatLng struct { - Latitude float64 `json:"latitude"` - Longitude float64 `json:"longitude"` -} - -type OpeningHours struct { - OpenNow bool `json:"openNow"` - // WeekdayDescriptions holds one human-readable string per day. Its starting weekday - // is verified against the live API in Task 2 Step 0 before being mapped to - // POI.Weekday — do not assume it matches the legacy WeekdayText ordering. - WeekdayDescriptions []string `json:"weekdayDescriptions"` -} - -// Photo.Name is a full resource name, "places/{placeID}/photos/{photoResource}". -// This is NOT interchangeable with a legacy photo_reference string. -type Photo struct { - Name string `json:"name"` - WidthPx int `json:"widthPx"` - HeightPx int `json:"heightPx"` -} - -type Place struct { - ID string `json:"id"` - Types []string `json:"types"` - PrimaryType string `json:"primaryType"` - DisplayName LocalizedText `json:"displayName"` - FormattedAddress string `json:"formattedAddress"` - AdrFormatAddress string `json:"adrFormatAddress"` - Location LatLng `json:"location"` - Rating float32 `json:"rating"` - UserRatingCount int `json:"userRatingCount"` - GoogleMapsURI string `json:"googleMapsUri"` - BusinessStatus BusinessStatus `json:"businessStatus"` - PriceLevel PriceLevel `json:"priceLevel"` - EditorialSummary LocalizedText `json:"editorialSummary"` - RegularOpeningHours OpeningHours `json:"regularOpeningHours"` - Photos []Photo `json:"photos"` -} - -type SearchNearbyResponse struct { - Places []Place `json:"places"` -} - -// Circle is the only locationRestriction shape searchNearby accepts. -type Circle struct { - Center LatLng `json:"center"` - Radius float64 `json:"radius"` // meters, 0 < radius <= 50000 -} - -type locationRestriction struct { - Circle Circle `json:"circle"` -} - -type searchNearbyRequest struct { - IncludedPrimaryTypes []string `json:"includedPrimaryTypes,omitempty"` - ExcludedPrimaryTypes []string `json:"excludedPrimaryTypes,omitempty"` - LocationRestriction locationRestriction `json:"locationRestriction"` - MaxResultCount int `json:"maxResultCount,omitempty"` - RankPreference RankPreference `json:"rankPreference,omitempty"` - LanguageCode string `json:"languageCode,omitempty"` -} -``` - -- [ ] **Step 4: Write `client.go`** - -Create `iowrappers/placesv1/client.go`: - -```go -package placesv1 - -import ( - "bytes" - "context" - "encoding/json" - "fmt" - "io" - "net/http" - "time" -) - -const ( - defaultBaseURL = "https://places.googleapis.com" - // MaxRadiusMeters is the searchNearby locationRestriction circle cap. - MaxRadiusMeters = 50000.0 - // MaxResultCount is the searchNearby hard cap. There is no pagination. - MaxResultCount = 20 -) - -// APIError is a structured Places API (New) error. The New API rejects an unknown -// place type with INVALID_ARGUMENT, where the legacy API silently ignored the filter. -type APIError struct { - Code int `json:"code"` - Message string `json:"message"` - Status string `json:"status"` -} - -func (e *APIError) Error() string { - return fmt.Sprintf("places api (new): %d %s: %s", e.Code, e.Status, e.Message) -} - -type errorEnvelope struct { - Error APIError `json:"error"` -} - -type Client struct { - apiKey string - baseURL string - http *http.Client -} - -type Option func(*Client) - -func WithBaseURL(u string) Option { return func(c *Client) { c.baseURL = u } } -func WithHTTPClient(h *http.Client) Option { return func(c *Client) { c.http = h } } - -func New(apiKey string, opts ...Option) *Client { - c := &Client{ - apiKey: apiKey, - baseURL: defaultBaseURL, - http: &http.Client{Timeout: 15 * time.Second}, - } - for _, o := range opts { - o(c) - } - return c -} - -// post sends a JSON POST with the API key and field mask headers the New API requires, -// and decodes either the success body into out or the error body into *APIError. -func (c *Client) post(ctx context.Context, path, fieldMask string, body any, out any) error { - payload, err := json.Marshal(body) - if err != nil { - return fmt.Errorf("marshaling request: %w", err) - } - req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+path, bytes.NewReader(payload)) - if err != nil { - return fmt.Errorf("building request: %w", err) - } - req.Header.Set("Content-Type", "application/json") - req.Header.Set("X-Goog-Api-Key", c.apiKey) - req.Header.Set("X-Goog-FieldMask", fieldMask) - - resp, err := c.http.Do(req) - if err != nil { - return fmt.Errorf("calling %s: %w", path, err) - } - defer func() { _ = resp.Body.Close() }() - - raw, err := io.ReadAll(resp.Body) - if err != nil { - return fmt.Errorf("reading response from %s: %w", path, err) - } - if resp.StatusCode != http.StatusOK { - var env errorEnvelope - if jsonErr := json.Unmarshal(raw, &env); jsonErr == nil && env.Error.Code != 0 { - return &env.Error - } - return &APIError{Code: resp.StatusCode, Status: resp.Status, Message: string(raw)} - } - if err := json.Unmarshal(raw, out); err != nil { - return fmt.Errorf("decoding response from %s: %w", path, err) - } - return nil -} -``` - -- [ ] **Step 5: Run tests to verify they pass** - -Run: `go test ./iowrappers/placesv1/ -v` - -Expected: PASS on all three tests. - -- [ ] **Step 6: Commit** - -```bash -git add iowrappers/placesv1/ -git commit -m "feat: add minimal Places API (New) HTTP client - -googlemaps.github.io/maps v1.7.0 only implements the legacy -/maps/api/place/* endpoints, so speak places.googleapis.com/v1 REST -directly. No new module dependencies." -``` - ---- - -### Task 2: `SearchNearby` request building and POI mapping - -**Files:** -- Create: `iowrappers/placesv1/search_nearby.go`, `iowrappers/places_v1_mapper.go` -- Test: `iowrappers/placesv1/search_nearby_test.go`, `iowrappers/places_v1_mapper_test.go` - -**Interfaces:** -- Consumes: everything from Task 1. -- Produces: - - `(*Client).SearchNearby(ctx context.Context, req SearchNearbyRequest) ([]Place, error)` where `SearchNearbyRequest` is the exported struct defined below - - `placesv1.SearchNearbyFieldMask` — the exact mask string - - `iowrappers.MapPlace(p placesv1.Place) POI.Place` - - `iowrappers.MapPriceLevel(pl placesv1.PriceLevel) POI.PriceLevel` - -- [ ] **Step 0: Verify weekday ordering against the live API before writing the mapper** - -`POI.CreatePlace` indexes hours by `POI.Weekday` from `DateMonday` to `DateSunday`. The legacy `WeekdayText` is Monday-first. Confirm the New API's `weekdayDescriptions` ordering rather than assuming it, because a silent off-by-one here shifts every place's opening hours by a day: - -```bash -curl -s -X POST 'https://places.googleapis.com/v1/places:searchNearby' \ - -H "X-Goog-Api-Key: $GOOGLE_MAPS_API_KEY" \ - -H 'X-Goog-FieldMask: places.displayName,places.regularOpeningHours.weekdayDescriptions' \ - -H 'Content-Type: application/json' \ - -d '{"includedPrimaryTypes":["cafe"],"maxResultCount":1, - "locationRestriction":{"circle":{"center":{"latitude":37.38006,"longitude":-122.11612},"radius":2000}}}' | jq -``` - -Record the first element's day name in a code comment in `places_v1_mapper.go`. If it is not Monday, the mapper must rotate the slice before handing it to `POI.OpeningHours.Hours`. - -- [ ] **Step 1: Write the failing test** - -Create `iowrappers/placesv1/search_nearby_test.go`: - -```go -package placesv1 - -import ( - "context" - "encoding/json" - "io" - "net/http" - "net/http/httptest" - "testing" -) - -func TestSearchNearbyBuildsRequest(t *testing.T) { - var got searchNearbyRequest - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - raw, _ := io.ReadAll(r.Body) - _ = json.Unmarshal(raw, &got) - w.Header().Set("Content-Type", "application/json") - _, _ = w.Write([]byte(`{"places":[]}`)) - })) - defer srv.Close() - - c := New("k", WithBaseURL(srv.URL)) - _, err := c.SearchNearby(context.Background(), SearchNearbyRequest{ - IncludedPrimaryTypes: []string{"cafe", "restaurant", "bar", "bakery", "meal_takeaway"}, - Latitude: 37.38006, - Longitude: -122.11612, - RadiusMeters: 8000, - MaxResultCount: 20, - RankPreference: RankPreferenceDistance, - }) - if err != nil { - t.Fatalf("SearchNearby returned %v", err) - } - if len(got.IncludedPrimaryTypes) != 5 { - t.Errorf("IncludedPrimaryTypes = %v, want 5 entries", got.IncludedPrimaryTypes) - } - if got.LocationRestriction.Circle.Radius != 8000 { - t.Errorf("radius = %v, want 8000", got.LocationRestriction.Circle.Radius) - } - if got.LocationRestriction.Circle.Center.Latitude != 37.38006 { - t.Errorf("center.latitude = %v, want 37.38006", got.LocationRestriction.Circle.Center.Latitude) - } - if got.RankPreference != RankPreferenceDistance { - t.Errorf("rankPreference = %q, want DISTANCE", got.RankPreference) - } -} - -func TestSearchNearbyClampsRadiusAndCount(t *testing.T) { - var got searchNearbyRequest - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - raw, _ := io.ReadAll(r.Body) - _ = json.Unmarshal(raw, &got) - w.Header().Set("Content-Type", "application/json") - _, _ = w.Write([]byte(`{"places":[]}`)) - })) - defer srv.Close() - - c := New("k", WithBaseURL(srv.URL)) - _, err := c.SearchNearby(context.Background(), SearchNearbyRequest{ - IncludedPrimaryTypes: []string{"cafe"}, - Latitude: 37.38006, - Longitude: -122.11612, - RadiusMeters: 120000, // over the 50km cap - MaxResultCount: 500, // over the 20 cap - }) - if err != nil { - t.Fatalf("SearchNearby returned %v", err) - } - if got.LocationRestriction.Circle.Radius != MaxRadiusMeters { - t.Errorf("radius = %v, want clamped to %v", got.LocationRestriction.Circle.Radius, MaxRadiusMeters) - } - if got.MaxResultCount != MaxResultCount { - t.Errorf("maxResultCount = %d, want clamped to %d", got.MaxResultCount, MaxResultCount) - } -} - -func TestSearchNearbyRejectsEmptyTypes(t *testing.T) { - c := New("k", WithBaseURL("http://unused")) - if _, err := c.SearchNearby(context.Background(), SearchNearbyRequest{ - Latitude: 1, Longitude: 1, RadiusMeters: 100, - }); err == nil { - t.Error("SearchNearby with no types returned nil error, want validation failure") - } -} -``` - -Create `iowrappers/places_v1_mapper_test.go`: - -```go -package iowrappers - -import ( - "testing" - - "github.com/weihesdlegend/Vacation-planner/POI" - "github.com/weihesdlegend/Vacation-planner/iowrappers/placesv1" -) - -func TestMapPriceLevel(t *testing.T) { - cases := map[placesv1.PriceLevel]POI.PriceLevel{ - // Unspecified maps to 0, matching the legacy behavior where an absent - // priceLevel arrived as integer 0 and was bucketed into level0. - placesv1.PriceLevelUnspecified: POI.PriceLevelZero, - placesv1.PriceLevelFree: POI.PriceLevelZero, - placesv1.PriceLevelInexpensive: POI.PriceLevelOne, - placesv1.PriceLevelModerate: POI.PriceLevelTwo, - placesv1.PriceLevelExpensive: POI.PriceLevelThree, - placesv1.PriceLevelVeryExpensive: POI.PriceLevelFour, - } - for in, want := range cases { - if got := MapPriceLevel(in); got != want { - t.Errorf("MapPriceLevel(%q) = %d, want %d", in, got, want) - } - } -} - -func TestMapPlace(t *testing.T) { - in := placesv1.Place{ - ID: "ChIJ_test", - Types: []string{"cafe", "food", "point_of_interest", "establishment"}, - PrimaryType: "cafe", - DisplayName: placesv1.LocalizedText{Text: "Peet's Coffee"}, - FormattedAddress: "367 State St, Los Altos, CA 94022, USA", - AdrFormatAddress: `367 State St`, - Location: placesv1.LatLng{Latitude: 37.38025, Longitude: -122.11655}, - Rating: 4.3, - UserRatingCount: 412, - GoogleMapsURI: "https://maps.google.com/?cid=1", - BusinessStatus: placesv1.BusinessStatus("OPERATIONAL"), - PriceLevel: placesv1.PriceLevelInexpensive, - EditorialSummary: placesv1.LocalizedText{Text: "Coffee chain known for house blends."}, - RegularOpeningHours: placesv1.OpeningHours{WeekdayDescriptions: []string{ - "Monday: 5:30 AM – 7:00 PM", "Tuesday: 5:30 AM – 7:00 PM", "Wednesday: 5:30 AM – 7:00 PM", - "Thursday: 5:30 AM – 7:00 PM", "Friday: 5:30 AM – 7:00 PM", "Saturday: 6:00 AM – 7:00 PM", - "Sunday: 6:00 AM – 7:00 PM"}}, - Photos: []placesv1.Photo{{Name: "places/ChIJ_test/photos/AT_abc", WidthPx: 4032, HeightPx: 3024}}, - } - - got := MapPlace(in) - - if got.GetID() != "ChIJ_test" { - t.Errorf("ID = %q, want ChIJ_test", got.GetID()) - } - if got.GetName() != "Peet's Coffee" { - t.Errorf("Name = %q, want Peet's Coffee", got.GetName()) - } - // LocationType comes from primaryType, so the record is correctly typed at write - // time. The legacy path stamped the SEARCHED type here, which is how hotels ended - // up labeled fast_food_restaurant. - if got.LocationType != POI.LocationTypeCafe { - t.Errorf("LocationType = %q, want cafe", got.LocationType) - } - if len(got.Types) != 4 || got.Types[0] != "cafe" { - t.Errorf("Types = %v, want Google's full list primary-first", got.Types) - } - if got.Status != POI.Operational { - t.Errorf("Status = %q, want OPERATIONAL", got.Status) - } - if got.PriceLevel != POI.PriceLevelOne { - t.Errorf("PriceLevel = %d, want 1", got.PriceLevel) - } - if got.UserRatingsTotal != 412 { - t.Errorf("UserRatingsTotal = %d, want 412", got.UserRatingsTotal) - } - if got.URL != "https://maps.google.com/?cid=1" { - t.Errorf("URL = %q, want the googleMapsUri", got.URL) - } - if got.Summary != "Coffee chain known for house blends." { - t.Errorf("Summary = %q, want the editorial summary text", got.Summary) - } - // The photo reference is a full resource name now, not a legacy photo_reference. - if got.Photo.Reference != "places/ChIJ_test/photos/AT_abc" { - t.Errorf("Photo.Reference = %q, want the full resource name", got.Photo.Reference) - } - if got.GetHour(POI.DateMonday) != "Monday: 5:30 AM – 7:00 PM" { - t.Errorf("Monday hours = %q, want the Monday description", got.GetHour(POI.DateMonday)) - } -} - -// TestMapPlaceEmptyOptionalFields pins that a sparse response does not panic and -// leaves POI defaults intact. -func TestMapPlaceEmptyOptionalFields(t *testing.T) { - got := MapPlace(placesv1.Place{ - ID: "ChIJ_sparse", - PrimaryType: "restaurant", - DisplayName: placesv1.LocalizedText{Text: "Sparse Diner"}, - Location: placesv1.LatLng{Latitude: 1, Longitude: 2}, - }) - if got.GetID() != "ChIJ_sparse" { - t.Errorf("ID = %q, want ChIJ_sparse", got.GetID()) - } - if got.Photo.Reference != "" { - t.Errorf("Photo.Reference = %q, want empty", got.Photo.Reference) - } - if got.PriceLevel != POI.PriceLevelZero { - t.Errorf("PriceLevel = %d, want 0", got.PriceLevel) - } -} -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `go test ./iowrappers/placesv1/ -run TestSearchNearby -v && go test ./iowrappers/ -run 'TestMapPlace|TestMapPriceLevel' -v` - -Expected: FAIL with `c.SearchNearby undefined` and `undefined: MapPlace`. - -- [ ] **Step 3: Write `search_nearby.go`** - -Create `iowrappers/placesv1/search_nearby.go`: - -```go -package placesv1 - -import ( - "context" - "errors" - "fmt" -) - -// SearchNearbyFieldMask is the exact field set the planner needs. Keep it minimal: -// the New API bills by SKU tier based on which fields are requested, and there is no -// default mask. Every field here replaces something the legacy path needed a separate -// Place Details call to get. -const SearchNearbyFieldMask = "places.id," + - "places.types," + - "places.primaryType," + - "places.displayName," + - "places.formattedAddress," + - "places.adrFormatAddress," + - "places.location," + - "places.rating," + - "places.userRatingCount," + - "places.googleMapsUri," + - "places.businessStatus," + - "places.priceLevel," + - "places.editorialSummary," + - "places.regularOpeningHours.weekdayDescriptions," + - "places.photos" - -// SearchNearbyRequest is the caller-facing shape. RadiusMeters and MaxResultCount are -// clamped to the API's limits rather than rejected, so callers can pass through the -// planner's own wider radius constants unchanged. -type SearchNearbyRequest struct { - IncludedPrimaryTypes []string - ExcludedPrimaryTypes []string - Latitude float64 - Longitude float64 - RadiusMeters float64 - MaxResultCount int - RankPreference RankPreference - LanguageCode string -} - -// SearchNearby calls places:searchNearby. -// -// Unlike the legacy endpoint this filters by PRIMARY type server-side, so results do -// not need client-side reclassification, and an unknown type is rejected with -// INVALID_ARGUMENT instead of silently disabling the filter. -// -// There is no pagination: at most MaxResultCount (cap 20) places come back. -func (c *Client) SearchNearby(ctx context.Context, req SearchNearbyRequest) ([]Place, error) { - if len(req.IncludedPrimaryTypes) == 0 { - return nil, errors.New("placesv1: SearchNearby requires at least one included primary type") - } - radius := req.RadiusMeters - if radius > MaxRadiusMeters { - radius = MaxRadiusMeters - } - if radius <= 0 { - return nil, fmt.Errorf("placesv1: radius must be > 0, got %v", req.RadiusMeters) - } - count := req.MaxResultCount - if count > MaxResultCount || count <= 0 { - count = MaxResultCount - } - rank := req.RankPreference - if rank == "" { - rank = RankPreferenceDistance - } - - body := searchNearbyRequest{ - IncludedPrimaryTypes: req.IncludedPrimaryTypes, - ExcludedPrimaryTypes: req.ExcludedPrimaryTypes, - LocationRestriction: locationRestriction{ - Circle: Circle{ - Center: LatLng{Latitude: req.Latitude, Longitude: req.Longitude}, - Radius: radius, - }, - }, - MaxResultCount: count, - RankPreference: rank, - LanguageCode: req.LanguageCode, - } - - var resp SearchNearbyResponse - if err := c.post(ctx, "/v1/places:searchNearby", SearchNearbyFieldMask, body, &resp); err != nil { - return nil, err - } - return resp.Places, nil -} -``` - -- [ ] **Step 4: Write `places_v1_mapper.go`** - -Create `iowrappers/places_v1_mapper.go`. Adjust the weekday rotation only if Step 0 showed a non-Monday first element: - -```go -package iowrappers - -import ( - "github.com/weihesdlegend/Vacation-planner/POI" - "github.com/weihesdlegend/Vacation-planner/iowrappers/placesv1" -) - -// MapPriceLevel converts the New API's price enum to POI.PriceLevel. -// -// PRICE_LEVEL_UNSPECIFIED maps to 0 deliberately: the legacy path received an absent -// price as integer 0 and bucketed it into placeIDs:eatery:level0, so this preserves -// which bucket an unpriced place lands in. -func MapPriceLevel(pl placesv1.PriceLevel) POI.PriceLevel { - switch pl { - case placesv1.PriceLevelInexpensive: - return POI.PriceLevelOne - case placesv1.PriceLevelModerate: - return POI.PriceLevelTwo - case placesv1.PriceLevelExpensive: - return POI.PriceLevelThree - case placesv1.PriceLevelVeryExpensive: - return POI.PriceLevelFour - case placesv1.PriceLevelFree, placesv1.PriceLevelUnspecified: - return POI.PriceLevelZero - default: - return POI.PriceLevelZero - } -} - -// MapPlace converts a Places API (New) place into the internal POI.Place. -// -// LocationType is set from primaryType — Google's own answer for what the place mainly -// is. The legacy path stamped the SEARCHED type here instead, which is how a hotel -// returned by an unenforceable fast_food_restaurant filter became a labeled eatery. -// -// weekdayDescriptions is Monday-first (verified against the live API on 2026-07-29), -// matching POI.Weekday's DateMonday..DateSunday order. -func MapPlace(p placesv1.Place) POI.Place { - var hours *POI.OpeningHours - if len(p.RegularOpeningHours.WeekdayDescriptions) > 0 { - hours = &POI.OpeningHours{Hours: append([]string(nil), p.RegularOpeningHours.WeekdayDescriptions...)} - } - - var summary *string - if p.EditorialSummary.Text != "" { - text := p.EditorialSummary.Text - summary = &text - } - - place := POI.CreatePlace( - p.DisplayName.Text, - p.AdrFormatAddress, - p.FormattedAddress, - string(p.BusinessStatus), - POI.LocationType(p.PrimaryType), - hours, - p.ID, - int(MapPriceLevel(p.PriceLevel)), - p.Rating, - p.GoogleMapsURI, - nil, // legacy *maps.Photo is not used on this path; set below - p.UserRatingCount, - p.Location.Latitude, - p.Location.Longitude, - summary, - ) - - // Photo.Reference holds the New API resource name ("places/{id}/photos/{ref}"), - // which is NOT a legacy photo_reference. photos_client.go routes on this prefix. - if len(p.Photos) > 0 { - place.Photo = POI.PlacePhoto{ - Reference: p.Photos[0].Name, - Width: p.Photos[0].WidthPx, - Height: p.Photos[0].HeightPx, - } - } - - // Preserve Google's full feature-type list so ReclassifyForCategory and - // PrimaryLocationType keep working on records written by this path. - place.Types = append([]string(nil), p.Types...) - return place -} -``` - -- [ ] **Step 5: Run tests to verify they pass** - -Run: `go build -v . && go test ./iowrappers/placesv1/ -v && go test ./iowrappers/ -run 'TestMapPlace|TestMapPriceLevel' -v` - -Expected: PASS on all tests. - -- [ ] **Step 6: Commit** - -```bash -git add iowrappers/placesv1/search_nearby.go iowrappers/placesv1/search_nearby_test.go iowrappers/places_v1_mapper.go iowrappers/places_v1_mapper_test.go -git commit -m "feat: add searchNearby call and Places API (New) to POI mapping - -LocationType now comes from Google's primaryType rather than the searched -type, so records are correctly labeled at write time. The field mask pulls -opening hours, adr address, maps URI, rating count, editorial summary and -photos in the search response, removing the need for a separate Place -Details call per result." -``` - ---- - -### Task 3: Versioned Redis keys and a New-API `SearchClient` behind a flag - -**Files:** -- Create: `iowrappers/redis_keys.go`, `iowrappers/places_v1_search_client.go` -- Modify: `iowrappers/redis_client.go` (`nearbySearchRedisKeys`, `SetPlacesAddGeoLocations`, `getPlace`/`setPlace` key building), `iowrappers/poi_searcher.go` (client selection) -- Test: `iowrappers/redis_keys_test.go`, `test/redis_client_mocks/v2_keys_test.go` - -**Interfaces:** -- Consumes: `placesv1.Client.SearchNearby`, `iowrappers.MapPlace` (Task 2); `POI.GetPlaceCategory(...) (PlaceCategory, bool)` (prior PR). -- Produces: - - `iowrappers.KeyVersion` type with `KeyVersionLegacy KeyVersion = ""` and `KeyVersionV2 KeyVersion = "v2"` - - `iowrappers.NearbySearchKey(cat POI.PlaceCategory, level POI.PriceLevel, v KeyVersion) string` - - `iowrappers.PlaceDetailsKey(placeID string, v KeyVersion) string` - - `iowrappers.NewPlacesV1SearchClient(apiKey string, mapsClient *MapsClient) *PlacesV1SearchClient` implementing `SearchClient` - - `iowrappers.ActiveKeyVersion() KeyVersion` — reads `PLACES_API_VERSION` - -- [ ] **Step 1: Write the failing test** - -Create `iowrappers/redis_keys_test.go`: - -```go -package iowrappers - -import ( - "strings" - "testing" - - "github.com/weihesdlegend/Vacation-planner/POI" -) - -// PlaceIDsKeyPrefix and PlaceDetailsKeyPrefix come from redis_data_inspections.go, -// PlaceDetailsRedisKeyPrefix from redis_client.go — all the same package, no import needed. - -// TestNearbySearchKeyLegacyUnchanged pins that the legacy key format is byte-identical -// to what production already holds. Any drift orphans the existing cache. -func TestNearbySearchKeyLegacyUnchanged(t *testing.T) { - cases := map[string]string{ - NearbySearchKey(POI.PlaceCategoryEatery, POI.PriceLevelZero, KeyVersionLegacy): "placeIDs:eatery:level0", - NearbySearchKey(POI.PlaceCategoryEatery, POI.PriceLevelThree, KeyVersionLegacy): "placeIDs:eatery:level3", - NearbySearchKey(POI.PlaceCategoryVisit, POI.PriceLevelTwo, KeyVersionLegacy): "placeIDs:visit", - } - for got, want := range cases { - if got != want { - t.Errorf("got %q, want %q", got, want) - } - } -} - -// TestNearbySearchKeyV2Namespaced pins that v2 data never collides with legacy data, -// so cutover and rollback are both non-destructive. -// -// The version is the FIRST segment on purpose. Existing code scans by legacy prefix — -// redis_data_inspections.go:22 scans "place_details*" and PlaceIDsKeyPrefix is -// "placeIDs" — so a suffixed name like "place_details_v2:" or an infixed one like -// "placeIDs:v2:" would be swept up by those scans and double-count or corrupt stats -// and migrations. Leading with "v2:" keeps v2 keys invisible to every legacy scan. -func TestNearbySearchKeyV2Namespaced(t *testing.T) { - cases := map[string]string{ - NearbySearchKey(POI.PlaceCategoryEatery, POI.PriceLevelZero, KeyVersionV2): "v2:placeIDs:eatery:level0", - NearbySearchKey(POI.PlaceCategoryVisit, POI.PriceLevelTwo, KeyVersionV2): "v2:placeIDs:visit", - } - for got, want := range cases { - if got != want { - t.Errorf("got %q, want %q", got, want) - } - } -} - -// TestV2KeysInvisibleToLegacyScans is the regression guard for the collision above. -func TestV2KeysInvisibleToLegacyScans(t *testing.T) { - v2Keys := []string{ - NearbySearchKey(POI.PlaceCategoryEatery, POI.PriceLevelZero, KeyVersionV2), - PlaceDetailsKey("ChIJ_x", KeyVersionV2), - } - legacyScanPrefixes := []string{PlaceDetailsKeyPrefix, PlaceIDsKeyPrefix, PlaceDetailsRedisKeyPrefix} - for _, key := range v2Keys { - for _, prefix := range legacyScanPrefixes { - if strings.HasPrefix(key, prefix) { - t.Errorf("v2 key %q is matched by legacy scan pattern %q*", key, prefix) - } - } - } -} - -func TestPlaceDetailsKey(t *testing.T) { - if got, want := PlaceDetailsKey("ChIJ_x", KeyVersionLegacy), PlaceDetailsRedisKeyPrefix+"ChIJ_x"; got != want { - t.Errorf("legacy details key = %q, want %q", got, want) - } - if got, want := PlaceDetailsKey("ChIJ_x", KeyVersionV2), "v2:place_details:place_ID:ChIJ_x"; got != want { - t.Errorf("v2 details key = %q, want %q", got, want) - } -} - -func TestActiveKeyVersionDefaultsToLegacy(t *testing.T) { - t.Setenv("PLACES_API_VERSION", "") - if got := ActiveKeyVersion(); got != KeyVersionLegacy { - t.Errorf("ActiveKeyVersion() = %q with no env set, want legacy", got) - } - t.Setenv("PLACES_API_VERSION", "new") - if got := ActiveKeyVersion(); got != KeyVersionV2 { - t.Errorf("ActiveKeyVersion() = %q with PLACES_API_VERSION=new, want v2", got) - } - t.Setenv("PLACES_API_VERSION", "legacy") - if got := ActiveKeyVersion(); got != KeyVersionLegacy { - t.Errorf("ActiveKeyVersion() = %q with PLACES_API_VERSION=legacy, want legacy", got) - } -} -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `go test ./iowrappers/ -run 'TestNearbySearchKey|TestPlaceDetailsKey|TestActiveKeyVersion' -v` - -Expected: FAIL with `undefined: NearbySearchKey`. - -- [ ] **Step 3: Write `redis_keys.go`** - -```go -package iowrappers - -import ( - "fmt" - "os" - "strings" - - "github.com/weihesdlegend/Vacation-planner/POI" -) - -// KeyVersion namespaces cached place data by the API that produced it. -// -// Places API (New) records are NOT interchangeable with legacy ones: photo references -// change from an opaque photo_reference to a "places/{id}/photos/{ref}" resource name, -// and LocationType comes from primaryType rather than the searched type. Writing both -// under one key would mix formats with no way to tell them apart, so the new path gets -// its own namespace. Cutover and rollback are then a single env-var flip with no deletes. -type KeyVersion string - -const ( - KeyVersionLegacy KeyVersion = "" - KeyVersionV2 KeyVersion = "v2" -) - -// PlaceDetailsV2RedisKeyPrefix mirrors PlaceDetailsRedisKeyPrefix for v2 records. -// -// The version leads the key. Existing code scans by legacy prefix — -// redis_data_inspections.go:22 scans PlaceDetailsKeyPrefix+"*" ("place_details*") and -// PlaceIDsKeyPrefix is "placeIDs" — so a suffixed "place_details_v2:" would be caught -// by those scans and make GetPlaceCountInRedis and the RemovePlaces migration operate -// on v2 records they know nothing about. "v2:" first keeps them cleanly separated. -const PlaceDetailsV2RedisKeyPrefix = "v2:place_details:place_ID:" - -// ActiveKeyVersion reads PLACES_API_VERSION. Anything other than "new" means legacy, -// so an unset or misspelled value fails safe onto the working path. -func ActiveKeyVersion() KeyVersion { - if strings.EqualFold(strings.TrimSpace(os.Getenv("PLACES_API_VERSION")), "new") { - return KeyVersionV2 - } - return KeyVersionLegacy -} - -// NearbySearchKey builds a geo bucket key. The legacy form is byte-identical to -// POI.EncodeNearbySearchRedisKey so existing production data stays addressable, and the -// v2 form leads with the version so legacy "placeIDs*"/"place_details*" scans skip it. -func NearbySearchKey(cat POI.PlaceCategory, level POI.PriceLevel, v KeyVersion) string { - segments := make([]string, 0, 4) - if v != KeyVersionLegacy { - segments = append(segments, string(v)) - } - segments = append(segments, PlaceIDsKeyPrefix, strings.ToLower(string(cat))) - if cat == POI.PlaceCategoryEatery { - segments = append(segments, fmt.Sprintf("level%d", level)) - } - return strings.Join(segments, ":") -} - -// PlaceDetailsKey builds the per-place record key for a version. -func PlaceDetailsKey(placeID string, v KeyVersion) string { - if v == KeyVersionV2 { - return PlaceDetailsV2RedisKeyPrefix + placeID - } - return PlaceDetailsRedisKeyPrefix + placeID -} -``` - -- [ ] **Step 4: Route the Redis read/write paths through the version** - -In `iowrappers/redis_client.go`, add a `keyVersion KeyVersion` field to `RedisClient`, defaulted from `ActiveKeyVersion()` wherever the client is constructed. Then: - -- `nearbySearchRedisKeys` (`:452`): replace each `POI.EncodeNearbySearchRedisKey(cat, lvl)` with `NearbySearchKey(cat, lvl, r.keyVersion)`. This requires making it a method on `*RedisClient`; update its two callers and `iowrappers/nearby_search_keys_test.go` accordingly. -- `SetPlacesAddGeoLocations` (`:222`): use `NearbySearchKey(placeCategory, place.PriceLevel, r.keyVersion)` and `PlaceDetailsKey(place.ID, r.keyVersion)`. -- `getPlace` / `setPlace`: take the version from `r.keyVersion`. - -**Read-compatibility requirement.** Saved trip plans store bare Google place IDs and read them back through `place_details:place_ID:` (`planner/planner.go:797`). Google place IDs are identical across both APIs, but a plan saved before cutover has records only under the legacy key. So the single-record read must fall back: - -```go -// getPlaceAnyVersion reads a place record, preferring the active key version and falling -// back to the other. Saved trip plans reference bare place IDs, and a plan saved before -// the Places API (New) cutover has a record only under the legacy key — so a -// version-strict read would break every existing saved plan. -func (r *RedisClient) getPlaceAnyVersion(ctx context.Context, placeID string) (POI.Place, error) { - place, err := r.getPlaceAtKey(ctx, PlaceDetailsKey(placeID, r.keyVersion)) - if err == nil { - return place, nil - } - other := KeyVersionLegacy - if r.keyVersion == KeyVersionLegacy { - other = KeyVersionV2 - } - return r.getPlaceAtKey(ctx, PlaceDetailsKey(placeID, other)) -} -``` - -Use `getPlaceAnyVersion` for saved-plan reads (`planner/planner.go:797-806`) and the version-strict `getPlace` for geo-bucket reads, where members always come from the matching namespace. - -- [ ] **Step 5: Write the New-API `SearchClient`** - -Create `iowrappers/places_v1_search_client.go`: - -```go -package iowrappers - -import ( - "context" - "fmt" - - "github.com/weihesdlegend/Vacation-planner/POI" - "github.com/weihesdlegend/Vacation-planner/iowrappers/placesv1" -) - -// PlacesV1SearchClient serves category searches from Places API (New). -// -// Geocode and ReverseGeocode delegate to the legacy MapsClient on purpose: the -// Geocoding API (/maps/api/geocode/json) is a separate, non-deprecated API and is not -// part of this migration. -// -// Brand/keyword searches also stay on the legacy client: searchNearby has no keyword -// parameter, and moving them needs places:searchText. Until that lands, a request with -// a Keyword is delegated wholesale. -type PlacesV1SearchClient struct { - places *placesv1.Client - mapsClient *MapsClient - // TypeGroups controls fan-out. One group containing every type = one HTTP call, - // capped at 20 results. Splitting into one group per type restores the legacy - // per-type ceiling at the cost of one call each. Set from Task 5's measurements. - TypeGroups func(POI.PlaceCategory) [][]POI.LocationType -} - -func NewPlacesV1SearchClient(apiKey string, mapsClient *MapsClient) *PlacesV1SearchClient { - return &PlacesV1SearchClient{ - places: placesv1.New(apiKey), - mapsClient: mapsClient, - TypeGroups: SingleGroupTypes, - } -} - -// SingleGroupTypes puts every type of a category into one searchNearby call. -func SingleGroupTypes(cat POI.PlaceCategory) [][]POI.LocationType { - return [][]POI.LocationType{POI.GetPlaceTypes(cat)} -} - -// PerTypeGroups issues one searchNearby call per place type, restoring the legacy -// per-type result ceiling. Costs len(GetPlaceTypes(cat)) calls instead of one. -func PerTypeGroups(cat POI.PlaceCategory) [][]POI.LocationType { - types := POI.GetPlaceTypes(cat) - groups := make([][]POI.LocationType, 0, len(types)) - for _, t := range types { - groups = append(groups, []POI.LocationType{t}) - } - return groups -} - -func (c *PlacesV1SearchClient) Geocode(ctx context.Context, q *GeocodeQuery) (float64, float64, error) { - return c.mapsClient.Geocode(ctx, q) -} - -func (c *PlacesV1SearchClient) ReverseGeocode(ctx context.Context, lat, lng float64) (*GeocodeQuery, error) { - return c.mapsClient.ReverseGeocode(ctx, lat, lng) -} - -func (c *PlacesV1SearchClient) NearbySearch(ctx context.Context, req *PlaceSearchRequest) ([]POI.Place, error) { - if req.Keyword != "" { - // searchNearby has no keyword parameter; brand search still needs searchText. - return c.mapsClient.NearbySearch(ctx, req) - } - - groups := c.TypeGroups(req.PlaceCat) - if len(groups) == 0 { - return nil, fmt.Errorf("no place types for category %q", req.PlaceCat) - } - - seen := make(map[string]bool) - places := make([]POI.Place, 0, len(groups)*placesv1.MaxResultCount) - for _, group := range groups { - types := make([]string, 0, len(group)) - for _, t := range group { - if t != POI.LocationTypeAny { - types = append(types, string(t)) - } - } - if len(types) == 0 { - continue - } - found, err := c.places.SearchNearby(ctx, placesv1.SearchNearbyRequest{ - IncludedPrimaryTypes: types, - Latitude: req.Location.Latitude, - Longitude: req.Location.Longitude, - RadiusMeters: float64(req.Radius), - MaxResultCount: placesv1.MaxResultCount, - RankPreference: placesv1.RankPreferenceDistance, - }) - if err != nil { - // Unlike the legacy API, an unknown type is a hard INVALID_ARGUMENT here. - // Log and continue so one bad type cannot zero out a whole category. - Logger.Error(fmt.Errorf("searchNearby failed for types %v: %w", types, err)) - continue - } - for _, p := range found { - if seen[p.ID] { - continue - } - seen[p.ID] = true - place := MapPlace(p) - // Match the legacy path's filter: places with no ratings are not useful. - if place.UserRatingsTotal == 0 { - continue - } - places = append(places, place) - } - } - return places, nil -} -``` - -- [ ] **Step 6: Select the client from the flag** - -In `iowrappers/poi_searcher.go`, wherever `PoiSearcher` is constructed with its `SearchClient`, choose based on the flag: - -```go - if ActiveKeyVersion() == KeyVersionV2 { - Logger.Info("PLACES_API_VERSION=new: serving category searches from Places API (New)") - searcher.searchClient = NewPlacesV1SearchClient(apiKey, mapsClient) - } else { - searcher.searchClient = mapsClient - } -``` - -- [ ] **Step 7: Run the full suite** - -Run: `go build -v . && go test ./... 2>&1 | tail -25` - -Expected: PASS. With `PLACES_API_VERSION` unset, every existing test exercises the unchanged legacy path. - -- [ ] **Step 8: Commit** - -```bash -git add iowrappers/redis_keys.go iowrappers/redis_keys_test.go iowrappers/places_v1_search_client.go iowrappers/redis_client.go iowrappers/poi_searcher.go planner/planner.go -git commit -m "feat: add PLACES_API_VERSION flag and v2-namespaced cache keys - -New API records are not interchangeable with legacy ones (photo resource -names, primaryType-derived LocationType), so they get their own key -namespace. Cutover and rollback are one env-var flip with no deletes. -Saved-plan reads fall back across versions since place IDs are shared." -``` - ---- - -### Task 4: Photos via the New media endpoint - -**Files:** -- Create: `iowrappers/placesv1/photo.go` -- Modify: `iowrappers/photos_client.go` -- Test: `iowrappers/placesv1/photo_test.go` - -**Interfaces:** -- Consumes: `placesv1.Client` (Task 1). -- Produces: `(*placesv1.Client).PhotoMediaURL(photoName string, maxWidthPx int) (string, error)` and `(*placesv1.Client).FetchPhoto(ctx context.Context, photoName string, maxWidthPx int) ([]byte, string, error)` returning bytes and content type. - -- [ ] **Step 1: Write the failing test** - -Create `iowrappers/placesv1/photo_test.go`: - -```go -package placesv1 - -import ( - "context" - "net/http" - "net/http/httptest" - "strings" - "testing" -) - -func TestPhotoMediaURL(t *testing.T) { - c := New("test-key") - got, err := c.PhotoMediaURL("places/ChIJ_x/photos/AT_abc", 400) - if err != nil { - t.Fatalf("PhotoMediaURL error: %v", err) - } - for _, want := range []string{ - "https://places.googleapis.com/v1/places/ChIJ_x/photos/AT_abc/media", - "maxWidthPx=400", - "key=test-key", - } { - if !strings.Contains(got, want) { - t.Errorf("URL %q missing %q", got, want) - } - } -} - -// TestPhotoMediaURLRejectsLegacyReference pins that an opaque legacy photo_reference -// cannot be passed to the New media endpoint. Cached legacy references are not -// convertible, which is why photos_client.go routes on the "places/" prefix. -func TestPhotoMediaURLRejectsLegacyReference(t *testing.T) { - c := New("test-key") - if _, err := c.PhotoMediaURL("ATtYBwLQ_legacy_opaque_ref", 400); err == nil { - t.Error("PhotoMediaURL accepted a legacy photo_reference, want error") - } -} - -func TestFetchPhotoFollowsRedirectAndReturnsBytes(t *testing.T) { - image := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - w.Header().Set("Content-Type", "image/jpeg") - _, _ = w.Write([]byte{0xFF, 0xD8, 0xFF, 0xE0}) - })) - defer image.Close() - - api := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - http.Redirect(w, r, image.URL, http.StatusFound) - })) - defer api.Close() - - c := New("test-key", WithBaseURL(api.URL)) - data, contentType, err := c.FetchPhoto(context.Background(), "places/ChIJ_x/photos/AT_abc", 400) - if err != nil { - t.Fatalf("FetchPhoto error: %v", err) - } - if contentType != "image/jpeg" { - t.Errorf("contentType = %q, want image/jpeg", contentType) - } - if len(data) != 4 { - t.Errorf("got %d bytes, want 4", len(data)) - } -} -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `go test ./iowrappers/placesv1/ -run 'TestPhoto|TestFetchPhoto' -v` - -Expected: FAIL with `c.PhotoMediaURL undefined`. - -- [ ] **Step 3: Write `photo.go`** - -```go -package placesv1 - -import ( - "context" - "fmt" - "io" - "net/http" - "net/url" - "strings" -) - -// PhotoNamePrefix marks a New API photo resource name. Legacy photo_reference strings -// are opaque and have no prefix, which makes this a reliable discriminator for cached -// records written by either API. -const PhotoNamePrefix = "places/" - -// PhotoMediaURL builds the media URL for a photo resource name obtained from a search -// or details response, e.g. "places/{placeID}/photos/{photoResource}". -// -// A legacy photo_reference is NOT convertible to this form — the only way to get a -// usable reference for a place cached under the legacy API is to re-fetch the place. -func (c *Client) PhotoMediaURL(photoName string, maxWidthPx int) (string, error) { - if !strings.HasPrefix(photoName, PhotoNamePrefix) { - return "", fmt.Errorf("placesv1: %q is not a photo resource name (want %s...); legacy photo_reference values are not convertible", photoName, PhotoNamePrefix) - } - if maxWidthPx < 1 || maxWidthPx > 4800 { - return "", fmt.Errorf("placesv1: maxWidthPx must be 1..4800, got %d", maxWidthPx) - } - q := url.Values{} - q.Set("maxWidthPx", fmt.Sprint(maxWidthPx)) - q.Set("key", c.apiKey) - return fmt.Sprintf("%s/v1/%s/media?%s", c.baseURL, photoName, q.Encode()), nil -} - -// FetchPhoto downloads the image bytes. The endpoint answers with an HTTP redirect to -// the image by default, which http.Client follows. -func (c *Client) FetchPhoto(ctx context.Context, photoName string, maxWidthPx int) ([]byte, string, error) { - mediaURL, err := c.PhotoMediaURL(photoName, maxWidthPx) - if err != nil { - return nil, "", err - } - req, err := http.NewRequestWithContext(ctx, http.MethodGet, mediaURL, nil) - if err != nil { - return nil, "", fmt.Errorf("building photo request: %w", err) - } - resp, err := c.http.Do(req) - if err != nil { - return nil, "", fmt.Errorf("fetching photo %s: %w", photoName, err) - } - defer func() { _ = resp.Body.Close() }() - - if resp.StatusCode != http.StatusOK { - raw, _ := io.ReadAll(resp.Body) - return nil, "", &APIError{Code: resp.StatusCode, Status: resp.Status, Message: string(raw)} - } - data, err := io.ReadAll(resp.Body) - if err != nil { - return nil, "", fmt.Errorf("reading photo %s: %w", photoName, err) - } - return data, resp.Header.Get("Content-Type"), nil -} -``` - -- [ ] **Step 4: Route by reference format in `photos_client.go`** - -`MapsPhotoClient.placeImage` (`iowrappers/photos_client.go:183`) currently always calls the legacy `client.PlacePhoto`. Route on the prefix instead, because the details keyspace holds both formats during and after cutover — brand/keyword places stay on the legacy path in this PR: - -```go -func (c *MapsPhotoClient) placeImage(ctx context.Context, ref string) (image.Image, error) { - // Acquire semaphore for API rate limiting - c.mapsClient.apiSemaphore <- struct{}{} - defer func() { <-c.mapsClient.apiSemaphore }() - - // A "places/..." reference is a Places API (New) resource name and must go to the - // New media endpoint. Legacy opaque photo_reference values stay on the SDK. Both - // formats coexist: brand/keyword searches still write legacy references. - if strings.HasPrefix(ref, placesv1.PhotoNamePrefix) { - data, contentType, err := c.placesV1.FetchPhoto(ctx, ref, 400) - if err != nil { - return nil, err - } - Logger.Debugf("photo response content type is: %s", contentType) - switch contentType { - case "image/png": - return png.Decode(bytes.NewReader(data)) - case "image/jpeg": - return jpeg.Decode(bytes.NewReader(data)) - default: - return nil, fmt.Errorf(UnknownImageFormat+": %s", contentType) - } - } - - resp, err := c.mapsClient.client.PlacePhoto(ctx, &maps.PlacePhotoRequest{PhotoReference: ref, MaxWidth: 400}) - if err != nil { - return nil, err - } - Logger.Debugf("photo response content type is: %s", resp.ContentType) - switch resp.ContentType { - case "image/png": - return png.Decode(resp.Data) - case "image/jpeg": - return resp.Image() - default: - return nil, fmt.Errorf(UnknownImageFormat+": %s", resp.ContentType) - } -} -``` - -Add a `placesV1 *placesv1.Client` field to `MapsPhotoClient` and initialize it in `CreatePhotoClient` (`iowrappers/photos_client.go:55`) from the same API key. - -Also update the stale-reference recovery branch in `GetPhotoURL` (`:131-160`): when the active version is v2, re-fetching a place's photo must come from a New API lookup rather than `PlaceDetailedSearch`. The simplest correct behavior for this PR is to skip recovery on v2 records and let the next cache refresh repopulate: - -```go - if strings.HasPrefix(err.Error(), UnknownImageFormat) { - if strings.HasPrefix(photoRef, placesv1.PhotoNamePrefix) { - // v2 records carry a resource name that cannot be repaired by a legacy - // Place Details call. Let the 14-day cache refresh replace it. - return "", fmt.Errorf("stale Places API (New) photo reference for place %s: %w", placeId, err) - } - // ... existing legacy recovery path unchanged ... - } -``` - -- [ ] **Step 5: Run tests to verify they pass** - -Run: `go build -v . && go test ./iowrappers/... -v 2>&1 | tail -25` - -Expected: PASS, including the existing `photos_client` tests on the legacy branch. - -- [ ] **Step 6: Commit** - -```bash -git add iowrappers/placesv1/photo.go iowrappers/placesv1/photo_test.go iowrappers/photos_client.go -git commit -m "feat: fetch photos from the Places API (New) media endpoint - -Route on the reference format: 'places/{id}/photos/{ref}' resource names -go to /v1/{name}/media, opaque legacy photo_reference values stay on the -SDK. Both formats coexist because brand/keyword search is still legacy." -``` - ---- - -### Task 5: Measure coverage, then cut over - -The one hard tradeoff in this design is the 20-result cap. Measure it on production data before flipping the flag. Do not skip this task, and do not tune `TypeGroups` by guesswork. - -**Files:** -- Create: `iowrappers/coverage_compare_test.go` (build-tagged, opt-in) - -- [ ] **Step 1: Write the comparison harness** - -Create `iowrappers/coverage_compare_test.go`. The build tag keeps it out of CI, since it makes real billed API calls: - -```go -//go:build coverage_compare - -package iowrappers - -import ( - "context" - "os" - "testing" - - "github.com/weihesdlegend/Vacation-planner/POI" -) - -// TestCompareLegacyVsNewCoverage reports how many distinct places each API returns for -// the same request, so the 20-result searchNearby cap can be judged on real data rather -// than assumed acceptable. -// -// Run with: -// GOOGLE_MAPS_API_KEY=... go test -tags coverage_compare ./iowrappers/ \ -// -run TestCompareLegacyVsNewCoverage -v -func TestCompareLegacyVsNewCoverage(t *testing.T) { - apiKey := os.Getenv("GOOGLE_MAPS_API_KEY") - if apiKey == "" { - t.Skip("GOOGLE_MAPS_API_KEY not set") - } - if err := CreateLogger(); err != nil { - t.Fatalf("CreateLogger: %v", err) - } - mapsClient := CreateMapsClient(apiKey) - newClient := NewPlacesV1SearchClient(apiKey, mapsClient) - - locations := map[string]POI.Location{ - "los altos (dense suburb)": {Latitude: 37.38006, Longitude: -122.11612, City: "Los Altos", AdminAreaLevelOne: "CA", Country: "United States"}, - "manhattan (very dense)": {Latitude: 40.7580, Longitude: -73.9855, City: "New York", AdminAreaLevelOne: "NY", Country: "United States"}, - "bozeman (sparse)": {Latitude: 45.6796, Longitude: -111.0471, City: "Bozeman", AdminAreaLevelOne: "MT", Country: "United States"}, - } - categories := []POI.PlaceCategory{POI.PlaceCategoryEatery, POI.PlaceCategoryVisit, POI.PlaceCategoryShopping} - - for name, loc := range locations { - for _, cat := range categories { - legacyReq := &PlaceSearchRequest{ - Location: loc, PlaceCat: cat, Radius: ColdStartSearchRadius, - MinNumResults: 40, PriceLevel: POI.PriceLevelDefault, - BusinessStatus: POI.Operational, AllPriceLevels: cat == POI.PlaceCategoryEatery, - } - legacy, err := mapsClient.NearbySearch(context.Background(), legacyReq) - if err != nil { - t.Errorf("%s/%s legacy: %v", name, cat, err) - continue - } - - singleReq := *legacyReq - newClient.TypeGroups = SingleGroupTypes - single, err := newClient.NearbySearch(context.Background(), &singleReq) - if err != nil { - t.Errorf("%s/%s new(single): %v", name, cat, err) - continue - } - - perTypeReq := *legacyReq - newClient.TypeGroups = PerTypeGroups - perType, err := newClient.NearbySearch(context.Background(), &perTypeReq) - if err != nil { - t.Errorf("%s/%s new(per-type): %v", name, cat, err) - continue - } - - // After ReclassifyForCategory, which is what actually reaches the response. - t.Logf("%-26s %-9s legacy=%3d (kept %3d) new-single=%3d new-per-type=%3d", - name, cat, len(legacy), countKept(legacy, cat), len(single), len(perType)) - } - } -} - -func countKept(places []POI.Place, cat POI.PlaceCategory) int { - kept := 0 - for _, p := range places { - if _, keep := POI.ReclassifyForCategory(p, cat); keep { - kept++ - } - } - return kept -} -``` - -- [ ] **Step 2: Run the comparison and record the numbers** - -Run: - -```bash -GOOGLE_MAPS_API_KEY=$GOOGLE_MAPS_API_KEY go test -tags coverage_compare ./iowrappers/ \ - -run TestCompareLegacyVsNewCoverage -v 2>&1 | tee /tmp/coverage-compare.txt -``` - -Compare `legacy (kept N)` — the count that actually survives to the response today — against `new-single`. The kept count is the honest baseline, because the legacy raw count includes results `ReclassifyForCategory` discards. - -- [ ] **Step 3: Choose the fan-out and record why** - -Decision rule, to be written into the PR description with the measured numbers: - -- If `new-single` >= the legacy kept count for every location and category, keep `SingleGroupTypes`. -- If any sparse or dense case regresses materially, set the default to `PerTypeGroups` for the affected categories. Encode it explicitly rather than leaving the default implicit: - -```go -// TypeGroupsForCategory splits Eatery across per-type calls because a single -// 20-result searchNearby underperformed the legacy kept count in dense areas -// (see docs/superpowers/plans/ measurements, 2026-07-29). Other categories fit in one call. -func TypeGroupsForCategory(cat POI.PlaceCategory) [][]POI.LocationType { - if cat == POI.PlaceCategoryEatery { - return PerTypeGroups(cat) - } - return SingleGroupTypes(cat) -} -``` - -- [ ] **Step 4: Commit the harness and the decision** - -```bash -git add iowrappers/coverage_compare_test.go iowrappers/places_v1_search_client.go -git commit -m "test: add legacy vs new coverage comparison harness - -searchNearby caps at 20 results with no pagination, so the fan-out choice -has to be measured against production data rather than assumed. Build-tagged -out of CI because it makes real billed API calls." -``` - -- [ ] **Step 5: Cut over in staging, then production** - -```bash -# 1. Enable on a staging/review app first. -heroku config:set PLACES_API_VERSION=new -a - -# 2. Exercise a cold search per category and confirm v2 keys appear. -redis-cli --scan --pattern 'v2:placeIDs:*' | head - -# 3. Confirm correct typing — the whole point of the migration. -# No record in a v2 eatery bucket should have a lodging primary type. -curl -s -H "Authorization: Bearer $ADMIN_JWT" \ - "https:///v1/migrate/reclassify-buckets?category=Eatery" | jq '.report.misclassified' -# Expected: 0 - -# 4. Production. -heroku config:set PLACES_API_VERSION=new -a best-vacation-planner - -# Rollback at any point, no deletes, legacy cache still warm: -heroku config:set PLACES_API_VERSION=legacy -a best-vacation-planner -``` - ---- - -### Task 6: Re-add `fast_food_restaurant` and `food_court` - -Only after Task 5's cutover is stable in production. This is the original intent of commit `8644199`, now actually achievable. - -**Files:** -- Modify: `POI/categories.go` -- Test: `test/place_category_test.go` - -- [ ] **Step 1: Write the failing test** - -In `test/place_category_test.go`, extend the Eatery entry of `TestGetPlaceTypesByCategory` and add: - -```go -// TestFastFoodTypesRoundTrip pins that the Places API (New) Table A eatery types are -// mapped in BOTH directions. Commit 8644199 added them to GetPlaceTypes only; the -// GetPlaceCategory default silently absorbed them into Eatery and the round-trip guard -// could not fail. Both directions must be explicit now. -func TestFastFoodTypesRoundTrip(t *testing.T) { - for _, placeType := range []POI.LocationType{POI.LocationTypeFastFood, POI.LocationTypeFoodCourt} { - got, ok := POI.GetPlaceCategory(placeType) - if !ok { - t.Errorf("GetPlaceCategory(%q) returned ok=false, want Eatery", placeType) - continue - } - if got != POI.PlaceCategoryEatery { - t.Errorf("GetPlaceCategory(%q) = %q, want Eatery", placeType, got) - } - } - types := POI.GetPlaceTypes(POI.PlaceCategoryEatery) - for _, want := range []POI.LocationType{POI.LocationTypeFastFood, POI.LocationTypeFoodCourt} { - found := false - for _, t2 := range types { - if t2 == want { - found = true - } - } - if !found { - t.Errorf("GetPlaceTypes(Eatery) = %v, missing %q", types, want) - } - } -} -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `go test ./test/ -run TestFastFoodTypesRoundTrip -v` - -Expected: FAIL with `undefined: POI.LocationTypeFastFood`. - -- [ ] **Step 3: Add the constants and both mappings** - -In `POI/categories.go`, restore the constants: - -```go - // LocationTypeFastFood and LocationTypeFoodCourt are Places API (New) Table A types. - // They only work when PLACES_API_VERSION=new: the legacy Nearby Search does not - // define them, ignores the ?type= filter rather than erroring, and its response - // types[] never contains them, so nothing can classify a place as either one. - LocationTypeFastFood = LocationType("fast_food_restaurant") - LocationTypeFoodCourt = LocationType("food_court") -``` - -Add them to **both** functions — this is the invariant that was violated the first time: - -```go - // in GetPlaceCategory - case LocationTypeCafe, LocationTypeRestaurant, LocationTypeBar, LocationTypeBakery, - LocationTypeMealTakeaway, LocationTypeFastFood, LocationTypeFoodCourt: - return PlaceCategoryEatery, true - - // in GetPlaceTypes - case PlaceCategoryEatery: - placeTypes = append(placeTypes, - []LocationType{LocationTypeCafe, LocationTypeRestaurant, LocationTypeBar, - LocationTypeBakery, LocationTypeMealTakeaway, LocationTypeFastFood, - LocationTypeFoodCourt}...) -``` - -- [ ] **Step 4: Guard the legacy path against them** - -The legacy `CreateMapSearchRequest` validation added in the prior PR now rejects these two types, which is correct — but it would log an error on every legacy search. Make the legacy client skip them quietly instead, in `extensiveNearbySearch` where `placeTypes` is built: - -```go - placeTypes := POI.GetPlaceTypes(request.PlaceCat) // get place types in a category - // Drop types the legacy Places API does not define. They are searched only when - // PLACES_API_VERSION=new; sending them here would spend a call whose type filter - // Google silently ignores. - placeTypes = Filter(placeTypes, func(t POI.LocationType) bool { - if t == POI.LocationTypeAny { - return true - } - _, err := maps.ParsePlaceType(string(t)) - return err == nil - }) -``` - -- [ ] **Step 5: Run the full suite** - -Run: `go build -v . && go test ./... 2>&1 | tail -20` - -Expected: PASS, including `TestPlaceCategoryRoundTrip` and the legacy `CreateMapSearchRequest` tests. - -- [ ] **Step 6: Verify against the live New API** - -```bash -curl -s -X POST 'https://places.googleapis.com/v1/places:searchNearby' \ - -H "X-Goog-Api-Key: $GOOGLE_MAPS_API_KEY" \ - -H 'X-Goog-FieldMask: places.displayName,places.primaryType,places.types' \ - -H 'Content-Type: application/json' \ - -d '{"includedPrimaryTypes":["fast_food_restaurant","food_court"],"maxResultCount":20, - "rankPreference":"DISTANCE", - "locationRestriction":{"circle":{"center":{"latitude":37.38006,"longitude":-122.11612},"radius":8000}}}' \ - | jq '.places[] | {name: .displayName.text, primaryType}' -``` - -Expected: every `primaryType` is `fast_food_restaurant` or `food_court`, and no hotels appear. That is the concrete difference from the legacy behavior that started this work. - -- [ ] **Step 7: Commit** - -```bash -git add POI/categories.go iowrappers/nearby_search.go test/place_category_test.go -git commit -m "feat: search fast_food_restaurant and food_court on the New API - -These Table A types are filtered server-side by includedPrimaryTypes, so -they now return correctly typed places instead of prominence-ranked -establishments. Mapped in both GetPlaceTypes and GetPlaceCategory, and -filtered out of the legacy path where they are undefined." -``` - ---- - -## Verification checklist - -- [ ] `go build -v .` and `go test -v ./...` pass with `PLACES_API_VERSION` unset (legacy path untouched). -- [ ] `go test -v ./...` passes with `PLACES_API_VERSION=new`. -- [ ] `grep -rn 'places.googleapis.com' --include='*.go' iowrappers/ | grep -v _test` shows requests only from `iowrappers/placesv1`. -- [ ] Coverage comparison numbers recorded in the PR description, with the `TypeGroups` choice justified by them. -- [ ] After staging cutover, `reclassify-buckets?category=Eatery` reports `misclassified: 0` against v2 buckets. -- [ ] A saved trip plan created before cutover still renders (exercises `getPlaceAnyVersion`). -- [ ] Photos load for both a v2 place and a legacy brand-search place. -- [ ] Rollback tested: set `PLACES_API_VERSION=legacy`, confirm legacy results still serve from the warm legacy cache. - -## Deliberately out of scope - -- **Brand/keyword search.** `searchNearby` has no keyword parameter; this needs `places:searchText` with `locationBias`, plus `MatchesBrandName`/`StrictNameMatch` re-tested against relevance-ranked results. `PlacesV1SearchClient.NearbySearch` delegates keyword requests to the legacy client until then. -- **Geocoding and ReverseGeocode.** `/maps/api/geocode/json` is the Geocoding API, not Places, and is not deprecated. It stays on `googlemaps.github.io/maps` v1.7.0. -- **Place Details.** Once search returns the full field set, the only remaining legacy Place Details caller is the stale-photo recovery path and the `data_migrations.go` backfills. Retire those separately. -- **Removing `googlemaps.github.io/maps`.** Cannot happen while Geocoding and brand search remain on it. - -## Risk notes - -- **Billing.** The New API bills per SKU tier by field mask. `SearchNearbyFieldMask` requests Enterprise-tier fields (`regularOpeningHours`, `editorialSummary`). Per-search cost may rise even as call count falls sharply — check the first days of billing after cutover rather than assuming the call-count reduction dominates. -- **The 20-result cap is the one-way door in this design.** Task 5 exists specifically to size it. If coverage proves unacceptable even with `PerTypeGroups`, the fallback is `places:searchText` per type, which paginates to 60 — a larger change that would supersede Task 3's client. -- **Legacy is not dead, but it is frozen.** Legacy Places became unavailable to Cloud projects created after 2025-03-01 and receives no fixes; Google has announced no turn-down date and promises 12 months' notice. The concrete risk is that recreating or swapping the GCP project behind `GOOGLE_MAPS_API_KEY` would break the legacy path outright — which is also why the brand-search path should not stay on it indefinitely. \ No newline at end of file diff --git a/docs/superpowers/plans/2026-07-30-followups-from-place-type-fixes.md b/docs/superpowers/plans/2026-07-30-followups-from-place-type-fixes.md deleted file mode 100644 index 7a101581..00000000 --- a/docs/superpowers/plans/2026-07-30-followups-from-place-type-fixes.md +++ /dev/null @@ -1,70 +0,0 @@ -# Follow-ups from the eatery place-type misclassification PR - -Carried forward from the review of `fix/eatery-place-type-misclassification` (plan: `2026-07-29-eatery-place-type-fixes.md`). Everything here was reviewed, triaged, and deliberately deferred — none of it blocked that merge. Ordered by value. - -## Important — deferred by explicit decision - -### 1. `POI.AllPlaceCategories` — make the guard cover the invariant, not today's categories - -**This is the highest-value item on the list.** The original incident survived because a guard test could not fail. Both current guards enumerate categories by hand — `test/place_category_test.go:41-44` and `iowrappers/nearby_search_validation_test.go:44-47` — so they protect the five categories that exist today rather than the invariant. - -The final reviewer demonstrated this empirically: adding a `PlaceCategoryNightlife` with `GetPlaceTypes` returning `{"karaoke", "pub"}` (neither exists in the v1.7.0 SDK — the same species of mistake as `fast_food_restaurant`) plus a matching `GetPlaceCategory` case produced a **fully green suite**, build and all 6 packages. The incident is reproducible verbatim for any newly added category. - -Fix: add `POI.AllPlaceCategories` and drive `ParsePlaceCategory` (`POI/categories.go:113-120`), `TestPlaceCategoryRoundTrip`, `TestCreateMapSearchRequestAcceptsKnownPlaceTypes`, `TestEncodeNearbySearchRedisKeyDistinct`, and the migration handler off it. A new category then lands inside every guard automatically. This converts "we fixed this bug" into "this bug shape cannot return." - -### 2. `getNearbyPlacesByBrand` still truncates by prominence - -`planner/planner.go:1298-1302` carries the identical false premise that the distance-sort task refuted for the category handler: - -```go -places = iowrappers.Filter(places, func(place POI.Place) bool { return !place.KnownClosedOnDay(day) }) -// Redis results are sorted by distance ascending; keep the nearest ones -if len(places) > limit { places = places[:limit] } -``` - -True on the cache path, false on the fresh path, where `PoiSearcher.NearbySearch` returns only `newPlaces` in Google prominence order (`iowrappers/poi_searcher.go:203-213`). A cold brand search can drop a 300m Dunkin' in favour of a 5km one. - -Impact is ordering-only (brand searches use a single `LocationTypeAny` type, so no whole place types are lost), which is why it was deferred — but the repo now has one handler sorted and its sibling unsorted, carrying a comment this work explicitly falsified. Fix is one line: `iowrappers.SortPlacesByDistance(places, req.Location.Latitude, req.Location.Longitude)` before the truncation. - -## Minor — migration robustness - -### 3. Removals are not pipelined - -`iowrappers/data_migrations.go:316` — reads were pipelined into batches of 100 but `ZRem` is still one round trip per removed member. The stated rationale for pipelining (a serial N+1 cannot finish inside Heroku's hard 30s H12) now covers only the read half. Irrelevant for the incident's ~17 rows; a bulk `apply=true` with thousands of hits re-enters the same ceiling. - -### 4. Bucket sizes are only delivered if the run completes - -`BucketSizes` / `TotalMembers` are measured up front (`data_migrations.go:270-277`) but serialized only in the terminal `ctx.JSON` (`planner/planner.go:303`). An H12 severs the request, so the operator who most needs the scale number is exactly the one who never receives it — `partial_report` covers returned errors, not a router timeout. A size-only mode (`?sizes=true`, returning right after the `ZCARD` loop) would make the property unconditional. - -### 5. `getPlace` failure is indistinguishable from "no record exists" - -`iowrappers/data_migrations.go:295-299`. A mid-run Redis fault silently skips every remaining member and returns a report reading `Misclassified: 0`, which an operator would reasonably read as "buckets are clean." Fail-safe in direction (nothing is deleted) but misleading. Distinguish `redis.Nil` from transport errors, or add a skipped/error count to the report. - -## Minor — comment and doc precision - -These matter more than usual: the write rule and the cleanup rule now **deliberately disagree**, and comments are most of what holds them apart. See "residual risk" below. - -- `iowrappers/data_migrations.go:228-229` — the summary sentence still reads "removes places whose PRIMARY Google type does not belong to `cat`," which describes the *discarded* broad rule. Only the paragraph beneath it is accurate. -- `iowrappers/data_migrations.go:248-250` — describes `ReclassifyForCategory` as keeping "only when its primary type is one of the category's five search types." Omits its keep-on-no-`Types` branch (`POI/categories.go:161-163`), and "five" is Eatery-specific (Lodging has one). -- `test/redis_client_mocks/bucket_cleanup_test.go:101-102` — stale pre-existing comment still says untyped records are kept "matching `ReclassifyForCategory`'s keep-on-unknown rule," contradicting the deliberate-divergence doc added alongside it. -- `planner/planner.go:280-281` — the handler doc comment repeats the discarded rule in unqualified form; it is now the only place stating it that way. -- `2026-07-29-eatery-place-type-fixes.md:118` — the replacement grep invariant checks only `POI/categories.go` and only the `= LocationType("…")` declaration form. `TestPlaceCategoryRoundTrip` is the real guard, so this is cosmetic. - -## Minor — pre-existing, untouched - -- `iowrappers/nearby_search.go:196` — `maxRetries` equals the *total* category type count rather than the count actually attempted in a round, so a round where every active type fails but one sibling is skipped never reaches the cap. Bounded by `GoogleMapsSearchCallMaxCount = 5`, so no unbounded loop. Per-type failure tracking was explicitly ruled out of scope. -- `iowrappers/nearby_search.go:222` — the error log says "nearby search … failed" for a validation rejection where no search was attempted. Cheap fix, and worth doing because "fail loudly" is that code's whole promise. -- Malformed format verbs `%!s()` at `planner/users.go:212,236` and `iowrappers/redis_client.go:497`. -- `iowrappers/data_migrations.go` hosts `SetPlace` / `AddGeoLocation`, which are generic `RedisClient` concerns; `redis_client.go` is their natural home. `AddGeoLocation` also widens the production API with a geo write that bypasses the type validation this PR added, and has no production caller. - -## Residual risk to keep in mind - -**The write rule and the cleanup rule now legitimately disagree.** The write path (`iowrappers/redis_client.go:225-245`) keys on the stamped `LocationType`; the cleanup rule keys on Google's *primary* type and deliberately keeps unmapped primaries (`meal_delivery`, `night_club`, empty `Types`) because the write path would legitimately place them there. A future refactor that "unifies" the two would reintroduce the incident. - -The dangerous direction is test-pinned: reverting the migration to `ReclassifyForCategory` fails `TestRemoveMisclassifiedPlacesPrimaryTypeTruthTable`. What is *not* pinned is someone changing `ReclassifyForCategory` itself, or collapsing both into a single shared helper. That is why the comment-precision items above are load-bearing rather than cosmetic. - -## Context worth not relearning - -- `POI.ReclassifyForCategory` has exactly **one** production caller, `planner/planner.go:1439` (the merchant endpoint). The trip-planning path — `planner/solver.go:532` → `matching.NearbySearchForCategory` → `matching.CreatePlace` — reads the same `placeIDs:eatery:level*` buckets and never reclassifies. Anything reasoning about "what the buckets contain" must account for both readers. -- `meal_delivery`, `night_club`, `liquor_store`, `convenience_store` are all legal legacy Places types (`maps@v1.7.0/types.go:257,264,253,227`) that Google routinely lists first in `types[]`. -- Legacy Nearby Search answers an unknown `?type=` by **ignoring the filter**, not by erroring. Places API (New) `searchNearby` rejects it with `INVALID_ARGUMENT` — see `2026-07-29-places-api-new-migration.md`.