feat: unblock geo-enrichment (coords) + exact/estimate EPC with manual address override - #1
Conversation
…usters Zoopla/OpenRent carry true per-property lat/lng in rawJson but they were never promoted to the listings.lat/lng columns, leaving ~80% of clusters coordinate-less and starving enrich-crime/flood/amenities/council-tax (they no-op without coords). This one-off promotes rawJson coords -> listings -> property_clusters and re-fires the geo enrichments. No geocoding (the coords are ours; scraped postcodes are outcodes only).
…full postcode
Scraped postcodes are outcodes only and EPC certs carry no coordinates,
so the old code stamped one arbitrary district cert on every cluster.
Now: reverse-geocode the (newly-populated) cluster coords to a full unit
postcode, exact-match a cert by house number + street when the address
has one, else summarise the postcode into a labelled estimate (modal
rating + range), else blank. Guards bedroom counts ('2 Bedroom Flat')
from being read as house numbers. Backward-compatible epc blob.
Adds property_clusters.user_address (migration 0014) + setClusterAddress server fn: the user pins the exact door (read off the photos vs Google Maps), which re-fires enrich-epc to resolve a precise certificate. The listing-detail PriceCard and mobile header expose a 'Fix address for exact EPC' link -> a 'Pin the exact address' dialog. Review cards show '~C est' and listing detail shows 'EPC area estimate ~C, range C-E, N homes' so estimates never read as fake-precise.
ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Free Run ID: 📒 Files selected for processing (15)
WalkthroughThis PR adds a postcode-and-address-driven EPC enrichment workflow, user address pinning capability, and estimate markers for postcode-level EPC data. Schema adds ChangesCore EPC Address Override & Enrichment
UI: EPC Estimate Marker & Address Override
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Note 🎁 Summarized by CodeRabbit FreeYour organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login. Comment |
…islabelled stops Two compounding bugs let listings far from any station into the queue: 1. classifyKind mapped Google's generic 'transit_station' type to 'rail'. In residential London that type is mostly bus-stop clusters, so bus stops were stamped kind='rail' at ~0.03mi with no walkMinutes — and the transport filter's straight-line heuristic then read 'rail ~0.6 min away', passing the 15-min rule. Treat a bare transit_station as unclassified (dropped) instead. 2. clusterPassesSearch judged tube/rail targets on that nearbyTransit data. Switch station kinds to the Google Routes walk times in stationRoutes — the same source the review card shows (e.g. Bush Hill Park 19 min) — so a cluster whose nearest real station is over the limit is now dropped. Bus/tram keep nearbyTransit; targets with no data stay pending. Fix #1 needs a nearby-transit re-enrichment to clean existing rows; fix #2 takes effect immediately for Rightmove-sourced clusters (stationRoutes).
…islabelled stops Two compounding bugs let listings far from any station into the queue: 1. classifyKind mapped Google's generic 'transit_station' type to 'rail'. In residential London that type is mostly bus-stop clusters, so bus stops were stamped kind='rail' at ~0.03mi with no walkMinutes — and the transport filter's straight-line heuristic then read 'rail ~0.6 min away', passing the 15-min rule. Treat a bare transit_station as unclassified (dropped) instead. 2. clusterPassesSearch judged tube/rail targets on that nearbyTransit data. Switch station kinds to the Google Routes walk times in stationRoutes — the same source the review card shows (e.g. Bush Hill Park 19 min) — so a cluster whose nearest real station is over the limit is now dropped. Bus/tram keep nearbyTransit; targets with no data stay pending. Fix #1 needs a nearby-transit re-enrichment to clean existing rows; fix #2 takes effect immediately for Rightmove-sourced clusters (stationRoutes).
…islabelled stops Two compounding bugs let listings far from any station into the queue: 1. classifyKind mapped Google's generic 'transit_station' type to 'rail'. In residential London that type is mostly bus-stop clusters, so bus stops were stamped kind='rail' at ~0.03mi with no walkMinutes — and the transport filter's straight-line heuristic then read 'rail ~0.6 min away', passing the 15-min rule. Treat a bare transit_station as unclassified (dropped) instead. 2. clusterPassesSearch judged tube/rail targets on that nearbyTransit data. Switch station kinds to the Google Routes walk times in stationRoutes — the same source the review card shows (e.g. Bush Hill Park 19 min) — so a cluster whose nearest real station is over the limit is now dropped. Bus/tram keep nearbyTransit; targets with no data stay pending. Fix #1 needs a nearby-transit re-enrichment to clean existing rows; fix #2 takes effect immediately for Rightmove-sourced clusters (stationRoutes).
Why
A read of prod data showed the enrichment pipeline — gaff's differentiator — was mostly empty: only ~18% of clusters had crime/flood/amenities/council-tax, and EPC was 0/102. Root causes (both confirmed against prod, not guessed):
lat/lnginrawJson, but they were never promoted to thelistings.lat/lngcolumns the enrichments read — so the geo tasks no-op'd for ~80% of clusters. (Geocoding from postcode doesn't help: scraped postcodes are outcodes only, e.g. "N11".)What
fix(enrich)— backfill strandedrawJsoncoords →listings→property_clusters, then re-fire the geo enrichments. (Already run against prod: geo-enrichment 18% → ~95%.)feat(enrich-epc)— reverse-geocode the now-populated coords to a full unit postcode, exact-match a certificate by house number + street when the address has one, else a labelled postcode-level estimate (modal rating + range), else blank. Guards bedroom counts ("2 Bedroom Flat") from being mistaken for house numbers.feat(epc)— manual address override:property_clusters.user_address(migration0014) +setClusterAddressserver fn. The user pins the exact door (read off the photos vs Google Maps) → re-resolves EPC precisely. UI: a "Fix address for exact EPC" link → "Pin the exact address" dialog; cards show~C est, listing detail showsEPC · area estimate · ~C · range C–E · N homes.Verified
Driven in-app against prod data (logged in): geo-enrichment card populated, EPC renders
~D · area estimate · range C–E · 18 homesfor a sample N14 listing, Fix-address dialog opens. Resolver across all clusters: 88 estimate / 1 exact / 1 none of 90.bun run buildclean, 154 tests pass.Done to prod DB already this session: migration
0014applied, coords backfilled, EPC populated, geo-enrichment populated. Still needed: deploy this branch so the Worker serves the new code — until then the live site won't show the EPC display / dialog, and new scrapes won't self-promote coords or run the newenrich-epc. (The coords forward-path itself was already inmainvia5c7fd543; it just needs deploying.)🤖 Generated with Claude Code
Summary by CodeRabbit
Release Notes
New Features
~indicator and show min/max rating ranges.Improvements