agent-engineering-studio · gzileni · Sep 5, 2026 · Sep 5, 2026
diff --git a/.env.example b/.env.example
@@ -174,9 +174,9 @@ LLM__LLAMACPP_BASE_URL=http://127.0.0.1:8091
 #LIMEN_FWI_AOI=
 #LIMEN_FWI_DAYS=60
 
-# Backtest incendio contro i perimetri di area bruciata EFFIS. Nota: l'endpoint
-# pubblico EFFIS risponde 403 senza accreditamento, quindi senza ingest il
-# report esce vuoto e lo dichiara.
+# Backtest incendio contro i perimetri di area bruciata EFFIS. Il servizio è
+# pubblico e non richiede credenziali: se il report esce vuoto va eseguito
+# l'ingest, non cercato un accreditamento.
 #LIMEN_BACKTEST_WILDFIRE_AOI=
 #LIMEN_BACKTEST_WILDFIRE_START=
 #LIMEN_BACKTEST_WILDFIRE_END=

diff --git a/CLAUDE.md b/CLAUDE.md
@@ -90,6 +90,8 @@
 | Geodata image carries no data | The Docker image bundles Python + GDAL/PROJ + tippecanoe + the code. Datasets land in the named PostGIS volume at first `limen geodata init`. Verify with `docker image inspect` — image size ≪ data size. |
 | Geodata never in critical path | The operational API reads pre-computed numeric per-cell features. `limen geodata export-features` ships those across with one upsert per cell. The MCP server is for agents; nothing in the hourly scoring path waits on it. |
 | Geodata is self-contained | `geodata/` is a uv workspace member designed to be extracted into a standalone repo with one `mv`. Nothing in `geodata.*` imports from `limen.*`; Prompt-2 parsers are duplicated in `geodata/parsers.py`. |
+| EFFIS is MapServer, not GeoServer | `maps.effis.emergency.copernicus.eu/gwis` is a **MapServer** CGI, and every GeoServer habit fails differently: `/gwis/ows` hangs (use `/gwis`), a wrong `typeNames` closes the connection mid-response instead of 404ing, `CQL_FILTER` is silently ignored, and `bbox` + `filter` together kill the connection. The layer is `nrt.ba.poly` with `initialdate`/`area`; dates are filtered **client-side** after a bbox request, which is cheap because one AOI's whole history is a single response (Basilicata: 515 perimeters 2012-2026, 305 KB). A sync that returns zero is a client bug, not missing credentials — that mistake cost this project a year of an empty `fire_perimeters` and three issues blocked on an accreditation nobody needed. |
+| A hit rate without a base rate is unreadable | `backtest-wildfire` reports both, plus their ratio. A model that calls every summer day dangerous scores a perfect hit rate while discriminating nothing, and the base rate is what exposes it. Measure it over the **fire season only**, never the spin-up days: including them dropped Basilicata's 2025 base from 43 % to 35 % and inflated the discrimination from 1.82 to 2.27 for free. |
 | Geodata URLs are official | The manifest schema refuses any URL outside `https://idrogeo.isprambiente.it/` by construction. To add a dataset, edit `datasets.yaml` (single source of truth) — no code change required. |
 | MCP refresh is admin-only | The `refresh` tool requires `MCP_ADMIN_TOKEN`. Env var **unset** = refresh disabled (fail-closed). |
 | `quality-local` never on a synchronous path | Colibrì/GLM-5.2 generate in *tens of minutes* (measured: 40 s for 3 tokens). Every `LLM__MODELS__*` role is synchronous — HTTP request, MCP tool call, or a scheduler tick shorter than the response — so mapping one there stalls the hourly sweep silently instead of just being slow. Enforced by `SLOW_GENERATION_MODELS` + a `model_validator` on `LLMSettings`: the process **refuses to start**. See `docs/inference.md`. |
@@ -286,9 +288,11 @@ extension points already:
   `landuse_code` (CORINE) and `slope_deg` (DEM) are unpopulated in most
   deployments, so the terrain term is constant and the wildfire map is
   effectively the FWI map — which is what EFFIS publishes, but not yet
-  territory-modulated risk. And the EFFIS perimeter endpoint answers **403**
-  without accreditation, so `backtest-wildfire` has no ground truth until
-  someone ingests it; the report says so instead of reporting zeros.
+  territory-modulated risk. The EFFIS perimeter service, on the other hand,
+  **needs no credentials** — the earlier conclusion that it required CEMS
+  accreditation was wrong, and the fix was in our own client. Measured on
+  Basilicata against real perimeters: hit rate 79 % against a 43 % base rate
+  in 2025, 100 % against 28 % in 2024.
 - Flood (#63-#64) still has no YAML and no engine.
 - V2 ML scoring engine (drop-in replacement of
   `MultiFactorScoringEngine` consuming the same `CellFeatureBundle`).

diff --git a/docs/HANDOFF.md b/docs/HANDOFF.md
@@ -97,10 +97,12 @@ Versione: implementazione completa, in fase di test. Feature mergiate di recente
      lo dichiara. `limen fwi-backfill` ricostruisce la storia dall'archivio
      ERA5 e va eseguito prima di mettere la mappa davanti a un operatore.
 
-  **Backtest senza ground truth**: l'endpoint pubblico EFFIS risponde **403**
-  senza accreditamento, quindi `fire_perimeters` resta vuota e il report lo
-  dichiara invece di stampare zeri. La logica di valutazione è verificata con
-  perimetri sintetici in `tests/integration/test_backtest_wildfire.py`.
+  **Backtest su ground truth vero**: EFFIS è pubblico e non richiede
+  credenziali. Per un anno il client ha creduto il contrario perché parlava
+  dialetto GeoServer a un servizio MapServer — vedi l'invariante «EFFIS is
+  MapServer» in CLAUDE.md. Misurato su Basilicata con perimetri reali: hit
+  rate **79 %** contro un tasso di base del **43 %** nel 2025 (1.82 volte),
+  **100 %** contro **28 %** nel 2024 (3.61 volte), preavviso medio ~55 h.
 
 - **Fase 1 hazard-agnostic COMPLETA** (2026-09-05, epic #57, PR #88 #89 #91 #92
   #93 + #94). L'architettura è multi-rischio nella struttura: `hazard_type`

diff --git a/src/limen/cli/backtest_wildfire.py b/src/limen/cli/backtest_wildfire.py
@@ -12,9 +12,15 @@
 exactly as the operational step does. Forcing the two through one function
 would make both harder to read than either is alone.
 
-Truth is ``fire_perimeters`` — the EFFIS burnt areas Limen already ingests.
-The public EFFIS endpoint now answers 403 without accreditation, so an
-un-ingested deployment gets an honest empty report rather than a crash.
+Truth is ``fire_perimeters`` — the EFFIS burnt areas Limen ingests. The
+service is public and needs no credentials; an un-ingested deployment still
+gets an honest empty report rather than a crash.
+
+The report pairs the hit rate with the **base rate**: the share of days a
+burnt cell was in alert anyway. Without it the hit rate is unreadable,
+because a model that calls every summer day dangerous maximises it while
+discriminating nothing. Measured on Basilicata against real perimeters: 79 %
+against a 43 % base in 2025, 100 % against 28 % in 2024.
 
 Env knobs:
 
@@ -88,6 +94,12 @@ class WildfireBacktestMetrics:
     hit_rate: float
     far: float
     mean_lead_hours: float
+    #: Quota di giornate della finestra in cui una cella poi bruciata era
+    #: comunque in allerta. È il metro dell'hit rate: un modello che dice
+    #: "pericolo alto" tutti i giorni d'estate ottiene un hit rate altissimo
+    #: senza discriminare nulla, e senza questo numero il report lo
+    #: presenterebbe come bravura.
+    base_rate: float
     report_path: Path | None
 
 
@@ -218,6 +230,7 @@ def evaluate(
     days: list[date],
     thresholds: WildfireThresholds,
     alert_level: RiskLevel,
+    season: tuple[date, date] | None = None,
 ) -> WildfireBacktestMetrics:
     """Score every burnt cell across the window and measure the warning.
 
@@ -233,6 +246,27 @@ def evaluate(
     horizon_days = int(_LEAD_MAX_HOURS // 24)
     scored_days = set(days)
 
+    def alerted(cell_id: str, sf: StaticFactors, fw: FireWeatherState, day: date) -> bool:
+        scored = engine.score(
+            CellFeatureBundle(
+                aoi_id=aoi_id,
+                cell_id=cell_id,
+                static=sf,
+                dynamic=DynamicInputs(
+                    valuation_time=datetime.combine(day, time(12), UTC),
+                    fire_weather=fw,
+                ),
+            )
+        )
+        return _at_least(scored.level, alert_level)
+
+    # Il tasso di base si misura sulla **stagione degli incendi**, non sui
+    # giorni di spin-up che la precedono: quelli sono primavera, il pericolo è
+    # basso, e includerli abbasserebbe il metro gonfiando la discriminazione.
+    first, last = season or (days[0], days[-1])
+    base_days = 0
+    base_alerted = 0
+
     hits = 0
     misses = 0
     warned_cells = 0
@@ -251,23 +285,24 @@ def evaluate(
             for d in range(horizon_days, -1, -1)
             if (fire_date - timedelta(days=d)) in scored_days
         ]
+        # Il metro: quanto spesso questa cella sarebbe stata in allerta in un
+        # giorno qualunque della finestra, non solo prima del suo incendio.
+        for day in days:
+            if not first <= day <= last:
+                continue
+            fw = chain.get(day)
+            if fw is None:
+                continue
+            base_days += 1
+            if alerted(cell_id, sf, fw, day):
+                base_alerted += 1
+
         earliest: date | None = None
         for day in window:
             fw = chain.get(day)
             if fw is None:
                 continue
-            scored = engine.score(
-                CellFeatureBundle(
-                    aoi_id=aoi_id,
-                    cell_id=cell_id,
-                    static=sf,
-                    dynamic=DynamicInputs(
-                        valuation_time=datetime.combine(day, time(12), UTC),
-                        fire_weather=fw,
-                    ),
-                )
-            )
-            if _at_least(scored.level, alert_level):
+            if alerted(cell_id, sf, fw, day):
                 earliest = day
                 break
         if earliest is None:
@@ -295,10 +330,16 @@ def evaluate(
         hit_rate=hit_rate,
         far=far,
         mean_lead_hours=mean_lead,
+        base_rate=base_alerted / base_days if base_days else 0.0,
         report_path=None,
     )
 
 
+def _lift(metrics: WildfireBacktestMetrics) -> float:
+    """Quante volte l'hit rate supera il tasso di base. 0 se non misurabile."""
+    return metrics.hit_rate / metrics.base_rate if metrics.base_rate > 0 else 0.0
+
+
 def write_report(
     metrics: WildfireBacktestMetrics, *, start: date, end: date, alert_level: RiskLevel
 ) -> Path:
@@ -320,9 +361,16 @@ def write_report(
         "## Metriche §2.5",
         "",
         f"- **Hit rate**: {metrics.hit_rate:.2%}",
+        f"- **Tasso di base**: {metrics.base_rate:.2%} "
+        f"(quota di giornate in allerta a prescindere dagli incendi)",
+        f"- **Discriminazione**: {_lift(metrics):.2f} volte il tasso di base",
         "- **FAR**: non misurato (vedi nota)",
         f"- **Preavviso medio**: {metrics.mean_lead_hours:.1f} h",
         "",
+        "> L'hit rate da solo non dice nulla: un modello che dichiara pericolo",
+        "> alto tutti i giorni d'estate lo massimizza senza discriminare. Il",
+        "> confronto con il tasso di base è ciò che separa la bravura dalla",
+        "> frequenza — sotto 1.0 il modello è peggio del caso.",
         "> Il replay guarda solo le celle che hanno bruciato, e solo",
         f"> l'orizzonte di {_LEAD_MAX_HOURS:.0f} h prima di ciascun incendio. Hit rate e",
         "> preavviso sono quindi misurati; il **FAR no**: un falso allarme è",
@@ -334,9 +382,10 @@ def write_report(
     if metrics.truth_fires == 0:
         lines.insert(
             5,
-            "> **Nessun perimetro EFFIS nella finestra.** L'endpoint pubblico "
-            "risponde 403 senza accreditamento: senza ingest non c'è ground "
-            "truth e le metriche sotto sono vuote per costruzione, non zero.",
+            "> **Nessun perimetro EFFIS nella finestra.** Il servizio è "
+            "pubblico e non richiede credenziali: se `fire_perimeters` è vuota "
+            "va eseguito l'ingest. Le metriche sotto sono vuote per "
+            "costruzione, non zero.",
         )
     out.write_text("\n".join(lines) + "\n", encoding="utf-8")
     return out
@@ -364,6 +413,7 @@ async def backtest_aoi(
             hit_rate=0.0,
             far=0.0,
             mean_lead_hours=0.0,
+            base_rate=0.0,
             report_path=None,
         )
         path = write_report(metrics, start=start, end=end, alert_level=alert_level)
@@ -409,6 +459,7 @@ async def backtest_aoi(
         days=days,
         thresholds=thresholds,
         alert_level=alert_level,
+        season=(start, end),
     )
     path = write_report(metrics, start=start, end=end, alert_level=alert_level)
     log.info(
@@ -418,6 +469,8 @@ async def backtest_aoi(
         hit_rate=round(metrics.hit_rate, 4),
         far=round(metrics.far, 4),
         mean_lead_hours=round(metrics.mean_lead_hours, 1),
+        base_rate=round(metrics.base_rate, 4),
+        lift=round(_lift(metrics), 2),
         report=str(path),
     )
     return replace(metrics, report_path=path)

diff --git a/src/limen/integrations/effis/fire_client.py b/src/limen/integrations/effis/fire_client.py
@@ -41,8 +41,26 @@
 
 # Default WFS endpoint and typeName. Override at construction time if
 # Copernicus moves endpoints.
-DEFAULT_WFS_URL = "https://maps.effis.emergency.copernicus.eu/gwis/ows"
-DEFAULT_TYPENAME = "effis:ba.fires"
+#
+# The endpoint is **MapServer**, not GeoServer, and that shapes every choice
+# below. Verified against the live service:
+#
+# * the base path is ``/gwis``, not ``/gwis/ows`` — the latter hangs;
+# * the layer is ``nrt.ba.poly``. ``effis:ba.fires`` does not exist, and a
+#   request for it makes MapServer close the connection mid-response, which
+#   surfaces as a transport error rather than a 404;
+# * ``CQL_FILTER`` is a GeoServer vendor parameter. MapServer ignores it, so
+#   a date range expressed that way silently filtered nothing;
+# * ``bbox`` and ``filter`` are mutually exclusive in WFS (and combining them
+#   here kills the connection), so the bbox goes on the wire and the dates are
+#   applied locally. Cheap in practice: one AOI's whole history is a single
+#   response — Basilicata is 515 perimeters across 2012-2026, 305 KB.
+DEFAULT_WFS_URL = "https://maps.effis.emergency.copernicus.eu/gwis"
+DEFAULT_TYPENAME = "nrt.ba.poly"
+
+#: Server-side cap. Above the largest per-AOI history we measured, so the
+#: local date filter never sees a truncated set.
+DEFAULT_MAX_FEATURES = 5000
 
 _DEGRADATION_EXC: tuple[type[BaseException], ...] = (
     httpx.HTTPError,
@@ -52,6 +70,30 @@
 )
 
 
+def _feature_date(feat: dict[str, Any]) -> date | None:
+    """The day a fire started, from whichever field this layer carries."""
+    props = feat.get("properties") or {}
+    for key in ("initialdate", "firedate", "FIREDATE", "fire_date"):
+        raw = props.get(key)
+        if raw:
+            try:
+                return date.fromisoformat(str(raw)[:10])
+            except ValueError:
+                return None
+    return None
+
+
+def _within(feat: dict[str, Any], start: date, end: date) -> bool:
+    """Keep a feature whose start date falls in the window.
+
+    A feature with no readable date is **kept**: the sync stores it with a
+    NULL date, which a backtest ignores, and dropping it here would lose a
+    perimeter that is still real.
+    """
+    when = _feature_date(feat)
+    return when is None or start <= when <= end
+
+
 class EffisHttpClient:
     """Concrete :class:`EffisClient` Protocol implementation."""
 
@@ -61,10 +103,12 @@ def __init__(
         wfs_url: str = DEFAULT_WFS_URL,
         typename: str = DEFAULT_TYPENAME,
         http_client: httpx.AsyncClient | None = None,
+        max_features: int = DEFAULT_MAX_FEATURES,
     ) -> None:
         self._wfs_url = wfs_url
         self._typename = typename
         self._http = http_client
+        self._max_features = max_features
 
     async def _client(self) -> httpx.AsyncClient:
         return self._http if self._http is not None else await SharedHttpClient.get()
@@ -82,15 +126,15 @@ async def fetch_perimeters(
         """
         params: dict[str, Any] = {
             "service": "WFS",
-            "version": "2.0.0",
+            "version": "1.1.0",
             "request": "GetFeature",
-            "typeNames": self._typename,
-            "outputFormat": "application/json",
-            "srsName": "EPSG:4326",
-            "bbox": f"{bbox[0]},{bbox[1]},{bbox[2]},{bbox[3]},EPSG:4326",
-            "CQL_FILTER": (
-                f"firedate >= '{start.isoformat()}' AND firedate <= '{end.isoformat()}'"
-            ),
+            # Lowercase keys: MapServer's WFS 1.1.0 does not accept the
+            # camelCase spellings GeoServer tolerates.
+            "typename": self._typename,
+            "outputformat": "geojson",
+            "srsname": "EPSG:4326",
+            "bbox": f"{bbox[0]},{bbox[1]},{bbox[2]},{bbox[3]}",
+            "maxfeatures": str(self._max_features),
         }
         log.info(
             "effis.perimeters.fetch",
@@ -120,8 +164,23 @@ async def fetch_perimeters(
             log.warning("effis.perimeters.bad_payload", status=resp.status_code)
             return []
         features = list(payload.get("features") or [])
-        log.info("effis.perimeters.fetched", count=len(features))
-        return features
+        if len(features) >= self._max_features:
+            # The local date filter below would then be working on a truncated
+            # set, and the gap would look like "no fires that summer".
+            log.warning(
+                "effis.perimeters.truncated",
+                count=len(features),
+                max_features=self._max_features,
+            )
+        in_window = [f for f in features if _within(f, start, end)]
+        log.info(
+            "effis.perimeters.fetched",
+            fetched=len(features),
+            in_window=len(in_window),
+            start=start.isoformat(),
+            end=end.isoformat(),
+        )
+        return in_window
 
     async def fetch_perimeters_bulk(
         self,

diff --git a/src/limen/integrations/effis/sync_job.py b/src/limen/integrations/effis/sync_job.py
@@ -61,16 +61,32 @@ def _parse_feature(feat: dict[str, Any]) -> FirePerimeter | None:
         log.warning("effis.feature.skip", reason="non-polygon geometry", feat_id=feat_id)
         return None
 
-    fire_date_str = props.get("firedate") or props.get("FIREDATE") or props.get("fire_date")
+    # `initialdate` è il campo del layer `nrt.ba.poly`; gli altri restano per
+    # il fallback bulk, che è un altro dataset con un'altra intestazione.
+    fire_date_str = (
+        props.get("initialdate")
+        or props.get("firedate")
+        or props.get("FIREDATE")
+        or props.get("fire_date")
+    )
     fire_date: date | None = None
     if fire_date_str:
         try:
             fire_date = date.fromisoformat(str(fire_date_str)[:10])
         except ValueError:
             log.warning("effis.feature.bad_date", value=fire_date_str, feat_id=feat_id)
 
-    area_ha_raw = props.get("area_ha") or props.get("AREA_HA") or props.get("AREA")
-    area_ha = float(area_ha_raw) if area_ha_raw is not None else None
+    # `area` sul WFS, già in ettari; `area_ha` sui file bulk. Arriva come
+    # stringa, quindi la conversione è esplicita e tollera un valore illeggibile.
+    area_raw = (
+        props.get("area") or props.get("area_ha") or props.get("AREA_HA") or props.get("AREA")
+    )
+    area_ha: float | None = None
+    if area_raw is not None:
+        try:
+            area_ha = float(area_raw)
+        except (TypeError, ValueError):
+            log.warning("effis.feature.bad_area", value=area_raw, feat_id=feat_id)
 
     return FirePerimeter(
         id=str(feat_id),