Two-stage, fully local, open-weights image geolocation for Google Street View panoramas (the GeoGuessr setting). Zero recurring API cost; tuned for a dual RTX 4090 workstation.
- Stage 1 — a
SigLIP2-SO400Mvision backbone feeding an autoregressive hierarchical S2-cell classifier (levels 3/6/9/12), with a ProtoNet content-aware selector that picks the final cell by image-feature similarity. - Stage 2 (optional) — a Gemma-4-26B vision-language layer that reads on-image text (OCR → VLM → geocode) to pinpoint within Stage 1's region.
Occlusion attribution — slide an occluder across each view, measure how much it shifts the ProtoNet match, and project the result back onto the equirectangular pano. Brighter red = the model relied on that region more. Rural scenes lean on the roadway, vegetation, and horizon rather than any single landmark.
The training corpus — 3.80M panoramas, coloured by capture year; teal = no date recorded. Regenerate with
scripts/render_density_map.py.
Deployed model (stage1_v3_long/epoch_04 + ProtoNet-select,
| median | mean | < 1 km | < 25 km | < 200 km | < 500 km | < 750 km |
|---|---|---|---|---|---|---|
| 24.9 km | 82.8 km | 4.4% | 50.1% | 90.2% | 98.1% | 99.1% |
Across three training generations the validation median plateaued near 78 km, and fine-grained accuracy stalled (within-25 km ≈ 22%) despite more data and training. A short battery of diagnostics showed the bottleneck was selection, not the model: an oracle that perfectly classifies to the finest cell hits a 5.4 km median, and the true L9 cell is in the model's top-50 candidates 93.5% of the time (given correct coarse context). The heads find the right cell; the model's density-biased probability can't rank it first.
Replacing that ranker with ProtoNet feature-similarity over the top-K candidate cells — picking the cell whose stored prototypes best match the query image — recovers most of the lost accuracy, with no retraining:
| median | mean | within 25 km | |
|---|---|---|---|
| greedy (probability top-1) | ~70 km | ~110 km | ~20% |
|
ProtoNet-select ( |
24.9 km | 82.8 km | 50.1% |
It also simplified the system: every candidate-masking "cascade mode" hurt once
the selector was active, so Stage 1 is a single path (raw logits → select). The
full study, including the per-country breakdown, is in
paper/geoai_paper.md.
The remaining error lives between 5 and 100 km, and five separate post-hoc fixes failed to touch it — a learned reranker (25.3 → 41.3 km on a genuinely held-out split), geometric verification (+6 points / 5000), metric reshaping (spread 4.5× wider, retrieval worse), aerial fusion, and offset regression on frozen features. All five tried to recover information downstream of the encoder.
Measuring the loss itself explained why. scripts/sigma_analysis.py evaluates the
haversine-smoothed soft target against the real cell layout:
| level | cell spacing | σ | ratio | cells sharing target | P(true cell) | contrast @25 km |
|---|---|---|---|---|---|---|
| 6 | 101.8 km | 500 | 4.9× | 110 | 0.020 | — |
| 9 | 12.9 km | 100 | 7.8× | 364 | 0.007 | 1.03 |
| 12 | 1.9 km | 20 | 10.5× | 223 | 0.011 | 1.04 @5 km |
At L9 the correct cell receives 0.7% of the target mass, and a cell 25 km wrong receives 97% of what the correct one does. The information was never absent from the images — the objective simply never asked for it.
A 5-arm controlled A/B (110k subset, identical warm start, seed 42) confirms it:
| arm | median | L9 top-1 | L6 top-1 |
|---|---|---|---|
| baseline (σ = 100/20) | 42.0 km | 3.7% | 0.230 |
| σ = 10.3/1.5 | 26.6 km | 25.6% | 0.156 |
| σ + offset head | 26.9 km | 25.8% | 0.160 |
| σ + L6 σ = 81.4 (ep1) | 30.1 km | 22.7% | 0.516 |
−37% median and 6.9× fine-cell accuracy from changing two numbers. The baseline moved only 43.3 → 42.0 km across a full extra epoch — stuck, not slow. The offset head adds nothing once σ is fixed, so it stays off; fixing L6's σ too recovers coarse accuracy at no cost. Absolute values here are not comparable to the deployed 24.9 km — a 110k-subset fine-tune with a fresh optimizer perturbs a converged model, which is why the baseline arm scores worse than production. Only the between-arm comparison is meaningful.
Ceiling: perfect L9 classification is 5.96 km. L12 would go finer, but 64.6% of
its cells hold a single pano, and scripts/project_l12_coverage.py shows the
in-flight scrape moves coverage only 20% → 24.9% — new panos open new cells
faster than they fill existing ones. Getting below ~6 km needs regression, not a
finer grid.
| Component | Choice |
|---|---|
| Backbone |
google/siglip2-so400m-patch14-384, full-finetune |
| View pooling | Concat across 4 perspective crops (90° FOV at headings 0/90/180/270) |
| Head | Autoregressive over levels L3/6/9/12 — each conditions on a 256-d embedding of the previous (GeoToken-style) |
| Loss | Haversine-smoothed soft-CE per level + auxiliary country CE + cross-country border penalty |
| Selection | ProtoNet L9 retrieval similarity over the top-$K$ candidate cells (1.94M prototypes, 119,026 cells) |
| Compute | 2× RTX 4090, bf16, torch.compile, gradient checkpointing |
S2 cell sizes (avg edge): L3 ~1150 km, L6 ~144 km, L9 ~18 km, L12 ~2.3 km. See
TECHNICALS.md for a full code walkthrough and
paper/geoai_paper.md for the technical report.
For text-rich panoramas, refined mode runs: Surya OCR → language ID → NLLB
translation → Gemma 4 26B (via an OpenAI-compatible LM Studio endpoint) →
Nominatim geocode → country-sanity check. It overrides Stage 1 only when it finds a
specific OSM point feature near Stage 1's guess; otherwise it defers. Its chief
use is visually-homogeneous countries (e.g. Japan, where pixels alone are
ambiguous and the disambiguating signal is on-image text).
gemma mode (geoai/stage2/gemma_refine.py) is a leaner alternative: one
multimodal call against any OpenAI-compatible endpoint (point
GEOAI_VLLM_URL/GEOAI_VLLM_MODEL at e.g. a local vLLM). It renders 8
overlapping 896px perspective tiles — feeding the raw equirect is useless,
since the server downscales it and every sign becomes unreadable — and asks
for readable text plus one geocodable place. Stage 1 keeps the country
(its country head is ~96%; the VLM is worse), so the refiner may only move the
pin within it.
Two guards matter, both learned the hard way: a hit that is an area
centroid (place=city, boundary=administrative) is rejected when ProtoNet
similarity ≥ 0.95, because overwriting a precise retrieval pin with a city
centroid measurably hurts (1.7 km → 18.2 km on one Okinawa pano); and point
features are capped at 75 km of drift to reject same-name-wrong-branch matches.
Measured on 120 Japan test panos: it commits on ~1% and leaves the median unchanged, because ~90% of those panos contain no readable text at all. It is a precision tool for text-rich/urban locations, not a fix for the rural ambiguity tail — which is why it should be gated by routing, below.
auto runs Stage 1, then escalates to a VLM refiner only when retrieval
looks unsure, using ProtoNet's top cosine similarity as the router signal
(≈0.94 in data-starved PHL/IDN vs ≈0.98 in confident-but-wrong JPN). Above
GEOAI_ROUTE_SIM_THRESHOLD (default 0.965) Stage 1 has matched a stored pano
and is already precise, so escalating can only add latency and risk a
regression. The decision is recorded on the response (route_sim,
route_escalated, route_reason) so it is auditable.
The ProtoNet index doubles as a cache of ~1.9M encoded panos, so several upgrades train without a GPU or any re-encoding:
# Learned reranker over the top-K L9 candidates, with a distance-aware
# listwise loss. Today's selector is an unlearned argmax over cosine
# similarity; measured headroom on our own data is median 17.9 km → 5.0 km
# (oracle best-in-shortlist), i.e. the right cell is nearly always present.
.venv/bin/python scripts/build_rerank_dataset.py --n-queries 30000
.venv/bin/python scripts/train_reranker.py
# Semivariogram hard negatives → per-country sampling weights for training.
# Also prints the spatial-autocorrelation curve; ours is cleanly monotonic
# (γ: 0.012 at <1 km → 0.999 beyond 8000 km).
.venv/bin/python scripts/build_semivariogram_hardnegs.py
# → --hardneg-weights-path /data/geolocation/ops/hardneg_weights.json
# GRPO groundwork: sample K completions per pano from a running VLM and score
# them with a smooth geo-similarity reward (abstention earns partial credit).
.venv/bin/python scripts/build_grpo_traces.py --country JPN --n 40 --k 4Optional web reverse-image-search refinement lives in
geoai/stage2/websearch.py; it stays disabled unless a provider key is set.
# What can the smoothing kernel actually distinguish? Prints, per level, the
# effective number of cells sharing the target and the contrast between the
# correct cell and one N km away. Contrast 1.0 = the loss cannot tell them apart.
.venv/bin/python scripts/sigma_analysis.py --level 9
# Will more scraping make L12 usable? (Answer: no — see above.)
.venv/bin/python scripts/project_l12_coverage.py
# Can the features locate a pano WITHIN its L9 cell? --split-by cell holds out
# whole cells; a pano-level split leaks, since our split has no spatial
# separation (median 805 m to the nearest train pano).
.venv/bin/python scripts/test_offset_head.py --split-by cellscripts/sigma_harness.py trains only the heads on cached frozen features. It
answers decoding questions cheaply, but cannot rank σ schedules — σ works by
shaping what the backbone encodes, and freezing the backbone removes that
mechanism. It scored three schedules at 122.6 / 127.0 / 122.6 km that a real run
separates by 37%.
Python 3.11 is required (Unsloth's Gemma 4 MoE support, for Stage 2).
If the system Python is newer than 3.11, uv can supply the right interpreter
without touching system packages (uv python install 3.11).
python3.11 -m venv .venv
.venv/bin/pip install -e .
# torch from PyTorch's CUDA wheel index:
.venv/bin/pip install --index-url https://download.pytorch.org/whl/cu128 torch torchvision
.venv/bin/python -c "import geoai.config; print(geoai.config.DATA_ROOT)" # /data/geolocationA run.py wrapper covers the same commands without the venv path
(python run.py serve|train|scrape|help).
# Data pipeline (resumable; reads/writes /data/geolocation)
.venv/bin/geoai-catalog # CSV + GADM + S2 + climate + pop → SQLite
.venv/bin/geoai-render-crops # equirect → 4× 384px perspective JPEGs
.venv/bin/geoai-split # deterministic location-level held-out split
.venv/bin/geoai-cell-stats --out-path .../cells_v3.parquet
# Train Stage 1 (dual-GPU)
.venv/bin/accelerate launch --num_processes 2 .venv/bin/geoai-train-stage1 \
--cells-parquet .../cells_v3.parquet --country-loss-weight 0.3 \
--border-lambda 0.5 --compile \
--sigma-l6 81.4 --sigma-l9 10.3 --sigma-l12 1.5 # geometry-derived; see above
# Build the ProtoNet L9 index (supports --shard/--num-shards for dual-GPU)
.venv/bin/python scripts/build_protonet_index.py \
--ckpt .../stage1_v3_long/epoch_04 --out .../epoch_04/protonet_l9.pt \
--cells-parquet .../cells_v3.parquet --max-per-cell 50
# Serve (FastAPI; ProtoNet-select on by default; auto-loads <ckpt>/protonet_l9.pt)
.venv/bin/geoai-serve --ckpt .../stage1_v3_long/epoch_04 \
--cells-parquet .../cells_v3.parquet --device cuda:0The userscript (script.js, Tampermonkey) plays live on geoguessr.com against the
server, with two modes: ProtoNet (Stage 1) and + Stage 2 (OCR/VLM).
geoai/
config.py data paths, raster/GADM lookups, env overrides
scraper/ Street View scraper (parked, functional)
processing/ catalog + crop renderer + reverse geocoding
stage1/ model, loss, dataset, predict (ProtoNet-select), protonet, eval
stage2/ OCR (extract) + pinpoint (Gemma) + geocode + refine
serve/ FastAPI inference server
scripts/ ProtoNet index build (+ dual-GPU shard/merge)
paper/ technical report (Markdown + figures)
TECHNICALS.md code/architecture walkthrough + gotchas
- Open weights only — no proprietary models at train or inference.
- Data lives on
/data/geolocation/, never in the repo; read in place viageoai.config(override withGEOAI_DATA_ROOT). - Resumable by construction — catalog, crop render, and scraper dedupe so kill/restart is always safe.
Non-obvious gotchas (S2 cell IDs stored as TEXT, SigLIP 0.5/0.5 normalization,
GHS-POP Mollweide reprojection, location-level splits, …) are documented in
TECHNICALS.md.