Fix DOM-extracted extended reviews

[kostaspt]

When the review RPC is blocked and the scraper falls back to DOM extraction, every entry written to user_reviews_extended has Rating=0, review_id="" and source="", only the name, text, images and the relative When string have data. So the extended reviews can't be used for anything rating-based and can't be joined against user_reviews.

Symptom

Browser-based RPC fetch failed: fetch error: HTTP 403, trying HTTP
RPC fetch failed, will try DOM extraction: .../maps/rpc/listugcposts?...: unexpected status code: 403
Attempting DOM-based review extraction
DOM extraction completed: 10 reviews found

All 10 entries per place land with those fields zeroed. The counts look healthy, which makes the degradation easy to miss. Reproduced against v1.17.2-1419d5d (digest sha256:33c1ec397acbb3038515b5854cb8cea22ab7131a9ecb4b14995e5e982144e755), two real places, -depth 1 -lang en -extra-reviews.

Root cause

The rating is read using a float64 type assertion (reviews.go:635 before this PR):

if v, ok := reviewMap["rating"].(float64); ok {
    review.Rating = int(v)
}

but playwright-go returns integral JS numbers as Go int (parseValue, js_handle.go:104-113 in v0.6100.0):

if v, ok := vMap["n"]; ok {
    if math.Ceil(v.(float64))-v.(float64) == 0 {
        return int(v.(float64))
    }
    return v.(float64)
}

The extraction script Math.rounds the rating before pushing it, so the value's always an integer and the assertion fails 100% of the time. review_id and source just never got extracted on this path.

What this PR changes

• The rating decode now accepts int and float64. The JS side is untouched.

• review_id is read from the card's data-review-id attribute and carried through to Review.ReviewID.

• With ids populated, extended entries are deduplicated against user_reviews: on two live test places every primary review reappeared in the extended set (matched heuristically before the fix, confirmed by id identity after, 8/8 per place). Dropping those leaves the genuinely new reviews (10 -> 2 per place). Empty ids never match. Note this changes output for anyone summing both columns (18 -> 10 per place).

• Decoded ratings are bounded to 1-5, anything out of range reads as unread (0).

• If DOM and primary review IDs ever stop overlapping entirely, the dedup logs one line instead of failing silently.

• Tests cover the decode edges, the converter, the dedup,.

• The switch to index-based loops isn't a style choice, gocritic's rangeValCopy trips once the struct passes 128 bytes.

Known issues NOT fixed here

• The listugcposts 403 itself: this improves the fallback's output, it does not restore the RPC path. Diagnosis is in progress, will follow separately.
• AuthorURL is extracted by the DOM script but dropped by the converter.
• Extended descriptions stay truncated: the cross-attempt dedup keeps the first (collapsed) copy of a body instead of the longest.

Verification

make lint, go build ./..., go test -count=1 ./..., gofmt -s -l gmaps/ all clean. Live run against the pinned image with the patched binary, same places and flags: extended entries went from all-zeroed to ratings 1-5 (observed 5, 2, 5, 5, read, not defaulted), non-empty ids, relative times verbatim. Primary reviews unchanged. posted_at_unix_micros / published_at stay zero on purpose, the panel carries no absolute timestamp and nothing is fabricated.

Related

• x #315 reported the same failure mode, this fixes the degraded fallback output, not the 403 itself.
• gmaps: Add extended review fields (source, language, timestamps, reply) #272 introduced the extended review fields this PR populates.

[Copilot]

Pull request overview

This PR improves the correctness and usability of the DOM-extraction fallback for extended reviews when the review RPC path is blocked (e.g., HTTP 403). It ensures key identity and rating fields are populated so extended reviews can be joined/deduped against primary reviews instead of silently degrading with “healthy-looking” counts but unusable data.

Changes:

• Fix DOM rating decoding by accepting both int and float64 from Playwright and clamping decoded ratings to the 1–5 range.
• Extract and propagate DOM review_id into Review.ReviewID, enabling reliable identity matching.
• Deduplicate DOM-extracted extended reviews against user_reviews by ReviewID, with diagnostic logging and unit tests covering decode + dedupe behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
gmaps/reviews.go	Adds review_id extraction, robust rating decode, and helper logic to parse and dedupe DOM reviews.
gmaps/place.go	Dedupes DOM-derived extended reviews against primary reviews before appending to UserReviewsExtended, with logging.
gmaps/dom_reviews_test.go	Adds targeted unit tests for DOM decode, conversion, and deduplication behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.