Add ISTAT comune-coordinates validation as a second data source (v0.4.0)

New geocoding.py module fetches and caches an ISTAT-derived comune->(lat, lon)
reference (opendatasicilia/comuni-italiani, 97.5% match against MIMIT). In
query_stations: stations whose stored coord deviates >30km from their declared
comune's true location are now flagged coordinate_suspect (works for single-
station comuni that the centroid heuristic could not reach), and --near
additionally excludes stations whose declared comune is outside radius_km+30km
of the query point. The original "Agip Tankstelle Rasen" case no longer
appears in --near 46.498,11.354 --radius 6. --no-comune-validate disables.

Author: galjos

CHANGELOG.md

@@ -4,6 +4,31 @@ All notable changes to this project are documented here. The format is based on
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project aims
 to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.4.0] - 2026-05-30
+
+### Added
+- **Second data source: Italian comune coordinates.** New `geocoding.py` module
+  fetches and caches an ISTAT-derived comune→(lat, lon) reference
+  (opendatasicilia/comuni-italiani) with 30-day cache and graceful fallback if
+  the upstream is unreachable. Achieves 97.5% comune-name match against MIMIT.
+- Comune validation in `query_stations`:
+  - flags `coordinate_suspect` when a station's stored coord is more than 30 km
+    from its declared comune's **true** location — this catches single-station
+    comuni (the original Rasen case) that the self-contained centroid method
+    provably could not.
+  - in `--near`, also excludes stations whose declared comune is outside
+    `radius_km + 30 km` of the query point; "Agip Tankstelle Rasen" no longer
+    appears in `--near 46.498,11.354 --radius 6`.
+- `--no-comune-validate` flag to skip the second source (offline/perf).
+
+### Known limitations
+- 2.5% of MIMIT comune strings don't match the reference (mostly garbage in the
+  comune field, recent comune mergers, minor spelling). The self-contained
+  centroid heuristic still applies to those.
+- The upstream comune dataset has no explicit LICENSE in its repo, though the
+  underlying data is ISTAT-derived. `pitstop` runtime-fetches only and credits
+  the source — if pitstop ever goes public, prefer a Wikidata-bundled file.
+
 ## [0.3.0] - 2026-05-29
 
 ### Added
@@ -87,6 +112,7 @@ Initial release.
 - `--fuel` is a substring match, so `Gasolio` also matches variants such as
   `Gasolio Alpino`.
 
+[0.4.0]: https://github.com/galjos/pitstop/releases/tag/v0.4.0
 [0.3.0]: https://github.com/galjos/pitstop/releases/tag/v0.3.0
 [0.2.0]: https://github.com/galjos/pitstop/releases/tag/v0.2.0
 [0.1.1]: https://github.com/galjos/pitstop/releases/tag/v0.1.1

README.md

@@ -16,7 +16,7 @@ Unofficial community project. Not affiliated with or endorsed by MIMIT. Fuel dat
 - **Known caveats:**
   - Some operators report placeholder values (e.g. `1.000`); use `--min-price` (e.g. `1.2`) to drop them.
   - Some price records are **stale** (a few were last updated years ago); use `--fresh-within-days` and check the `UPDATED` column / `updated` field.
-  - Some stations are **mis-geocoded** in the registry. v0.3.0 annotates stations >30 km from their comune's median (or outside Italy) as `coordinate_suspect` (`*` in the table), and `--near` skips coordinates outside the Italy bounding box. Single-station comuni cannot be validated this way and need a second source (planned).
+  - Some stations are **mis-geocoded** in the registry. As of v0.4.0 `pitstop` joins a second data source (ISTAT-derived comune coordinates from [opendatasicilia/comuni-italiani](https://github.com/opendatasicilia/comuni-italiani), 97.5% match) to validate each station's coordinate against its declared comune's *true* location. Stations >30 km off are flagged `coordinate_suspect` (`*` in the table), and `--near` excludes stations whose declared comune is geographically too far from the query point — even single-station comuni like RASUN-ANTERSELVA. Pass `--no-comune-validate` to disable.
 
 ## Install
 
@@ -87,13 +87,17 @@ pytest -q
 
 ## Status & roadmap
 
-v0.3.0 — fuel-price core (registry+price join, filters, proximity, cheapest, `--min-price` floor, `--fresh-within-days` freshness filter, `coordinate_suspect` geocoding signal, JSON), an MCP server, a Claude skill, tests, and CI.
+v0.4.0 — fuel-price core (registry+price join, filters, proximity, cheapest, `--min-price` floor, `--fresh-within-days` freshness filter, ISTAT-derived comune-coordinate validation, JSON), an MCP server, a Claude skill, tests, and CI.
 
 Planned, roughly in order:
-- a **second data source** of Italian comune coordinates (e.g. ISTAT) to validate single-station-comune coordinates that the self-contained method can't reach;
 - **EV charging** (locations via Open Charge Map; prices via the AFIR National Access Point / DATEX II as that data matures);
 - additional countries behind a per-country source adapter (e.g. Germany Tankerkönig, France/Spain official feeds).
 
+## Data sources & attributions
+
+- Fuel stations and prices: **MIMIT Osservaprezzi Carburanti** open data.
+- Comune coordinates (validation): **opendatasicilia/comuni-italiani** (ISTAT-derived).
+
 ## Links
 
 - MIMIT fuel open data: https://www.mimit.gov.it/it/open-data/elenco-dataset/carburanti-prezzi-praticati-e-anagrafica-degli-impianti

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "pitstop"
-version = "0.3.0"
+version = "0.4.0"
 description = "Unofficial JSON-first CLI for Italian fuel-station prices (MIMIT Osservaprezzi Carburanti open data)."
 readme = "README.md"
 requires-python = ">=3.10"

skills/pitstop/SKILL.md

@@ -47,4 +47,4 @@ Exit codes: `0` ok, `1` runtime error (e.g. network), `2` usage error.
 - **`--fuel` is a substring match** — `Gasolio` also matches `Gasolio Premium`, `Gasolio Oro Diesel`, etc. Use `pitstop fuels` to pick exact names; inspect the `fuel` field in results.
 - **Placeholder prices** — some operators report junk values like `1.000`; with `--cheapest` these can rank first. Pass `--min-price 1.2` (petrol/diesel) to drop them; use a lower floor for cheaper fuels like GPL. Still sanity-check the lowest result.
 - **Stale prices** — some records were last updated years ago and may be far below the current market. Pass `--fresh-within-days 90` and check the `updated` field before reporting a price as current.
-- **Mis-geocoded stations** — a few registry coordinates are wrong. Results carry a `coordinate_suspect: true` flag (table marker `*`) when a station is far from its comune's other stations or outside Italy. When you see it, treat the result as low-confidence and double-check the `comune`/`address`. **Single-station comuni cannot be flagged** with the current method, so for `--near` results in remote/small comuni, always sanity-check the comune name against the user's query.
+- **Mis-geocoded stations** — some registry coordinates are wrong. `pitstop` now joins an ISTAT-derived comune-coordinates reference (97.5% coverage): a `coordinate_suspect: true` flag (table marker `*`) appears when a station's coord is far from its declared comune's true location, and `--near` excludes stations whose declared comune is geographically too far from the query point. Even single-station comuni are now caught. Still: when you see `coordinate_suspect`, treat as low-confidence and surface the `address`/`comune` to the user.

src/pitstop/cli.py

@@ -63,6 +63,8 @@ def _build_parser() -> argparse.ArgumentParser:
                           help="drop prices below this floor (e.g. 1.2 to skip placeholder values); 0 = off")
     stations.add_argument("--fresh-within-days", dest="fresh_days", type=int, default=0,
                           help="drop prices last updated more than N days ago; 0 = off")
+    stations.add_argument("--no-comune-validate", dest="validate_comune", action="store_false",
+                          help="skip validating coordinates against the comune-coords reference")
     stations.add_argument("--limit", type=int, default=20, help="max stations; 0 = no limit")
     stations.set_defaults(func=_cmd_stations)
 
@@ -121,6 +123,7 @@ def _cmd_stations(args) -> int:
         cheapest=args.cheapest,
         min_price=args.min_price,
         max_age_days=args.fresh_days,
+        validate_comune=args.validate_comune,
         limit=args.limit,
     )
 

src/pitstop/core.py

@@ -284,11 +284,19 @@ def query_stations(
     min_price: float = 0.0,
     max_age_days: int = 0,
     limit: int = 20,
+    validate_comune: bool = True,
+    comune_coords: dict[str, tuple[float, float]] | None = None,
 ) -> list[Station]:
     """Filter, sort, and limit stations. Mutates the dataset's Station objects
     (narrows prices, sets distance_km), so pass a freshly loaded Dataset."""
     today = date.today() if max_age_days > 0 else None
     centroids = comune_centroids(ds)
+    if validate_comune and comune_coords is None:
+        from . import geocoding
+        comune_coords = geocoding.load_comune_coords()
+    elif comune_coords is None:
+        comune_coords = {}
+
     out: list[Station] = []
     for st in ds.stations.values():
         if comune and st.comune.casefold() != comune.strip().casefold():
@@ -305,11 +313,15 @@ def query_stations(
             continue
         st.prices = prices
 
-        # Flag coordinates that are implausible or far from the station's
-        # comune-cluster. Single-station comuni cannot be validated this way
-        # and remain unflagged — a known limitation requiring a second source.
+        # Flag coordinates that are implausible or far from where they should be.
+        # Prefer the true ISTAT-derived comune coord when available (works for
+        # single-station comuni); fall back to the in-data comune centroid.
+        true_coord = comune_coords.get(st.comune.upper()) if comune_coords else None
         if not in_italy(st.lat, st.lon):
             st.coordinate_suspect = True
+        elif true_coord is not None:
+            if haversine_km(true_coord[0], true_coord[1], st.lat, st.lon) > SUSPECT_DISTANCE_KM:
+                st.coordinate_suspect = True
         else:
             c = centroids.get(st.comune.upper())
             if c is not None and haversine_km(c[0], c[1], st.lat, st.lon) > SUSPECT_DISTANCE_KM:
@@ -318,6 +330,13 @@ def query_stations(
         if near is not None:
             if not in_italy(st.lat, st.lon):
                 continue  # invalid coords cannot be reliably near anything
+            # Reject stations whose declared comune is geographically too far
+            # from the query point — even if the stored coordinate happens to
+            # land close (the Rasen case). Tolerance pads for large comuni.
+            if true_coord is not None:
+                comune_dist = haversine_km(near[0], near[1], true_coord[0], true_coord[1])
+                if comune_dist > radius_km + 30.0:
+                    continue
             d = haversine_km(near[0], near[1], st.lat, st.lon)
             if d > radius_km:
                 continue

src/pitstop/geocoding.py

@@ -0,0 +1,89 @@
+"""Second data source: authoritative Italian comune coordinates, used to
+validate MIMIT station coordinates. Self-contained centroid heuristics in
+`core` cannot catch mis-geocoded stations in single-station comuni (e.g.
+RASUN-ANTERSELVA), so a true comune→(lat, lon) reference is required.
+
+Source: opendatasicilia/comuni-italiani `main.csv`, derived from ISTAT.
+Runtime fetch + local cache; no redistribution."""
+
+from __future__ import annotations
+
+import csv
+import os
+import sys
+import time
+import urllib.error
+import urllib.request
+from pathlib import Path
+
+COMUNI_URL = (
+    "https://raw.githubusercontent.com/opendatasicilia/comuni-italiani/main/dati/main.csv"
+)
+COMUNI_SOURCE_NAME = "opendatasicilia/comuni-italiani (ISTAT-derived)"
+
+DEFAULT_COMUNI_MAX_AGE = 30 * 24 * 60 * 60  # 30 days; comuni change rarely
+DEFAULT_TIMEOUT = 180
+
+
+def _cache_dir() -> Path:
+    base = os.environ.get("XDG_CACHE_HOME") or str(Path.home() / ".cache")
+    d = Path(base) / "pitstop"
+    d.mkdir(parents=True, exist_ok=True)
+    return d
+
+
+def normalize_comune(name: str) -> str:
+    """Uppercase, trim, collapse internal whitespace. MIMIT uses uppercase
+    names; opendatasicilia uses capitalized — uppercase makes both match."""
+    return " ".join(name.strip().upper().split())
+
+
+def _cached_path(refresh: bool, max_age: int, timeout: int) -> Path | None:
+    path = _cache_dir() / "comuni_main.csv"
+    if not refresh and path.exists():
+        if max_age <= 0 or (time.time() - path.stat().st_mtime) < max_age:
+            return path
+    try:
+        req = urllib.request.Request(COMUNI_URL, headers={"User-Agent": "pitstop"})
+        with urllib.request.urlopen(req, timeout=timeout) as resp:
+            data = resp.read()
+    except (urllib.error.URLError, OSError) as e:
+        # Graceful fallback: if we cannot fetch, return any stale cache or None.
+        print(f"pitstop: could not fetch comune coordinates ({e}); "
+              f"falling back to self-contained heuristics", file=sys.stderr)
+        return path if path.exists() else None
+    tmp = path.with_suffix(path.suffix + ".tmp")
+    tmp.write_bytes(data)
+    tmp.replace(path)
+    return path
+
+
+def load_comune_coords(
+    *,
+    refresh: bool = False,
+    max_age: int = DEFAULT_COMUNI_MAX_AGE,
+    timeout: int = DEFAULT_TIMEOUT,
+) -> dict[str, tuple[float, float]]:
+    """Return {normalized_comune_name: (lat, lon)}. Empty dict on fetch failure
+    with no cache, so callers should treat it as best-effort."""
+    path = _cached_path(refresh, max_age, timeout)
+    if path is None:
+        return {}
+    return _parse_comuni(path)
+
+
+def _parse_comuni(path: Path) -> dict[str, tuple[float, float]]:
+    out: dict[str, tuple[float, float]] = {}
+    with path.open("r", encoding="utf-8", errors="replace", newline="") as f:
+        reader = csv.DictReader(f)
+        for row in reader:
+            name = row.get("comune", "").strip()
+            if not name:
+                continue
+            try:
+                lat = float(row["lat"])
+                lon = float(row["long"])
+            except (KeyError, ValueError, TypeError):
+                continue
+            out[normalize_comune(name)] = (lat, lon)
+    return out

src/pitstop/version.py

@@ -1 +1 @@
-__version__ = "0.3.0"
+__version__ = "0.4.0"

tests/test_core.py

@@ -172,6 +172,39 @@ def st(sid, lat, lon):
     assert flagged == {"1": False, "2": False, "3": False, "4": True}
 
 
+def test_query_stations_uses_true_comune_coord_to_flag_single_station_comune():
+    # Single-station comune: centroid heuristic can't help; only the true
+    # comune coord (from the second source) can flag it.
+    bad = core.Station("X", "", "", "", "Rasen", "", "RASUN-ANTERSELVA", "BZ",
+                       46.4545, 11.3188,  # registry-stored (wrong)
+                       [core.Price("Gasolio", 2.115, True, "2026-05-27T00:00:00")])
+    ds = core.Dataset(stations={"X": bad}, registry_date="2026-05-28", price_date="2026-05-28")
+    coords = {"RASUN-ANTERSELVA": (46.839, 12.112)}  # true coord
+    out = core.query_stations(ds, comune_coords=coords)
+    assert out[0].coordinate_suspect is True
+    # Without comune validation, single-station comune isn't flagged.
+    bad.coordinate_suspect = False
+    out2 = core.query_stations(ds, validate_comune=False, comune_coords={})
+    assert out2[0].coordinate_suspect is False
+
+
+def test_query_stations_near_rejects_far_comune():
+    # The Rasen case: stored coord lands ~5km from Bolzano so it passes the
+    # radius check, but the declared comune's true location is ~50km away ->
+    # excluded by the comune sanity check.
+    rasen = core.Station("R", "", "", "", "Rasen", "", "RASUN-ANTERSELVA", "BZ",
+                         46.4545, 11.3188,
+                         [core.Price("Gasolio", 1.749, True, "2026-05-27T00:00:00")])
+    bolzano = core.Station("B", "", "", "", "BZ Station", "", "BOLZANO", "BZ",
+                           46.498, 11.354,
+                           [core.Price("Gasolio", 2.0, True, "2026-05-27T00:00:00")])
+    ds = core.Dataset(stations={"R": rasen, "B": bolzano},
+                      registry_date="2026-05-28", price_date="2026-05-28")
+    coords = {"RASUN-ANTERSELVA": (46.839, 12.112), "BOLZANO": (46.498, 11.354)}
+    out = core.query_stations(ds, near=(46.498, 11.354), radius_km=6, comune_coords=coords)
+    assert [s.id for s in out] == ["B"]
+
+
 def test_query_stations_skips_invalid_coords_for_near():
     bad = core.Station("X", "", "", "", "S", "", "NOWHERE", "ZZ", 0.0, 0.0,
                        [core.Price("Benzina", 2.0, True, "2026-05-27T00:00:00")])