How to integrate ERA LTE data with the Climate Toolkit

[Rwema25]

Background

ERA Long-Term Experiments (LTEs) provide site-level agricultural trial records with geolocation, season definitions, rainfall totals, crop names, and yield outcomes. These datasets are a strong candidate for validation and benchmarking of the Climate Toolkit’s seasonal climate summaries.

Motivation

Integrating ERA LTE data would enable:

• site-specific climate summary generation with the toolkit
• comparison of toolkit-derived season totals against ERA-reported values
• exploration of how climate statistics can help explain crop yield outcomes
• a practical workflow for connecting LTE metadata to toolkit analytics

Proposed integration approach

1. Extract LTE site/year metadata:

   • Site.LatD, Site.LonD
   • Time.Start.Year, Time.End.Year
   • Site.Start.S1, Site.End.S1
   • Site.Start.S2, Site.End.S2 (if present)
   • crop name, yield, and experiment identifiers

2. Convert LTE season windows into toolkit fixed-season syntax:

   • single season: MM-DD:MM-DD
   • two seasons: MM-DD:MM-DD,MM-DD:MM-DD
   • year-crossing season: MM-DD:MM-DD where end date belongs to next year

3. Run climate statistics for each ERA LTE site/year using fixed seasons:

   • location = latitude,longitude
   • years = experiment start/end
   • fixed season = LTE season window
   • source = appropriate climate source for the site

4. Collect toolkit outputs such as:

   • seasonal rainfall total
   • rainy day count
   • dry day count
   • water balance / ET0 metrics
   • VPD/humidity-derived summaries if humidity data are available

5. Compare results:

   • LTE-reported rainfall vs toolkit seasonal rainfall
   • LTE season dates vs toolkit season window interpretation
   • climate-derived metrics vs yield outcomes

Data fields of interest

From LTE datasets, the most useful fields are:

• Site.Out

  • Site.LatD, Site.LonD
  • Site.Start.S1, Site.End.S1
  • Site.Start.S2, Site.End.S2
  • Site.Rain.Seasons

• Time.Out

  • Time.Start.Year, Time.End.Year
  • Time.Season

• Climate.Out

  • climate normals by site/year

• Data.Out

  • experiment records with crop/yield outcomes

Why this is feasible

The toolkit already supports fixed-season climate summaries and seasonal metrics, so LTE date metadata can be translated into direct toolkit inputs without major code changes.

Suggested next steps

• explore the ERA LTE repository and export core site/year metadata
• prototype a script that reads LTE rows and constructs toolkit fixed-season calls
• validate with a few representative sites by comparing reported rainfall and season windows
• assess whether additional data preprocessing or source selection is needed

Outcome

This integration should demonstrate that the Climate Toolkit can:

• reproduce ERA LTE seasonal climate statistics
• provide a climate-derived explanatory layer for yield analysis
• support cross-site comparisons between LTE reports and toolkit-generated metrics

Further explanation :

Suggested workflow

1. Get the ERA LTE metadata

From the ERA dataset, the key fields you need are:

• Site.LatD, Site.LonD
• Time.Start.Year, Time.End.Year
• Site.Start.S1, Site.End.S1 (and Site.Start.S2 / Site.End.S2 if two seasons)
• Time.Season or crop-specific experiment year
• crop name / yield

This gives you:

• location
• year(s)
• season window(s)
• crop / yield for later comparison

---

2. Convert ERA season dates to toolkit fixed-season format

The toolkit uses fixed-season windows like:

• "03-01:05-31" for March 1 to May 31
• "10-01:12-15" for October 1 to December 15
• "11-01:02-28" for a season crossing calendar year

So map:

• Site.Start.S1 / Site.End.S1 → MM-DD:MM-DD
• if there is a second season, use "S1,S2" format

---

3. Use the toolkit to generate climate statistics for each site/year

The best module is climate_toolkit.climate_statistics.statistics.

If you want a command-line path, use:

python -m climate_tookit.climate_statistics.statistics \
  --location="-1.286,36.817" \
  --start-year=2018 \
  --end-year=2020 \
  --fixed-season "03-01:05-31" \
  --source era_5 \
  --format pandas

For ERA LTE, change:

• -1.286,36.817 → site latitude,longitude
• --start-year / --end-year → experiment years
• --fixed-season → season window from ERA metadata
• --source → best available data source for the site

If the site has two seasons, use:

--fixed-season "03-01:05-31,10-01:12-15"

If you want to use the toolkit from Python instead:

from climate_tookit.climate_statistics import statistics

# if there is a Python API wrapper available, or use module CLI via subprocess

---

4. Use the toolkit outputs to compute the metrics you need

The toolkit can give you season-based summaries such as:

• total rainfall
• rainy days
• dry days
• water balance
• VPD / humidity-derived metrics
• seasonal climate statistics

This matches your target:

• compare ERA reported total rainfall vs toolkit-calculated total rainfall
• compare season length / rainfall season definition
• compare weather-derived climate statistics against yield outcomes

---

5. Build a comparison dataset

For each ERA site-year:

• ERA reported variables:
  • crop yield
  • total rainfall
  • season start/end
  • crop name
• Toolkit-derived variables:
  • seasonal rainfall total
  • rainy day count
  • dry day count
  • mean temperature, max/min, water balance
  • VPD/humidity metrics if available

Then you can analyze:

• How close toolkit rainfall totals are to ERA totals
• whether tool-based climate summaries explain yield variation
• whether the toolkit can be used as an explanatory input layer for yield modeling

---

Practical integration advice

If ERA LTE data are currently in R

• You can still export the key fields as CSV/Parquet from R
• Then read them in Python
• For each ERA row, call the toolkit or the CLI with the site-season inputs

If you want a more automated pipeline

Create a small integration script that:

1. reads ERA LTE site/year records
2. builds location, start_year, end_year, fixed_season
3. runs the toolkit stats module
4. saves climate metrics to a table
5. merges the result with ERA yield data

---

What to pay attention to

• ERA Site.Start.S1 and Site.End.S1 may be seasonal dates, not full-year dates.
• Use the same fixed-season window for every year of that site if the ERA dataset records a stable season.
• If a season crosses year-end, the toolkit supports that format.
• Choose the climate source carefully: era_5, agera_5, chirps_v2, etc. are available depending on the data needed.

---

Conclusion

This is a very reasonable integration task and should be done as a workflow/project, not as a toolkit bug fix.

You can integrate ERA LTE by:

• extracting lat/lon and season windows
• passing them into climate_toolkit.climate_statistics with --fixed-season
• generating climate statistics per ERA site-year
• comparing toolkit outputs to ERA reported rainfall and yield

[Sammyjoseph999]

Nice write-up @Rwema25 — I checked the plan against the current CLI/API and it's feasible as described. A few grounding points and a runnable starting script.

Verified against the code

• climate-toolkit-stats (climate_tookit.climate_statistics.statistics) supports --fixed-season MM-DD:MM-DD[,MM-DD:MM-DD] with single, two-season, and year-crossing windows — exactly the mapping you need. --format {json,pandas} and --source era_5/agera_5/... are all valid.
• You don't need subprocess/CLI for the batch loop. There's a public Python API:
  from climate_tookit.climate_statistics import analyze_climate_statistics, with signature
  analyze_climate_statistics(location_coord, start_year, end_year, source, fixed_season=None, precip_source=None, temp_source=None, ee_project_id=None, ...) returning the same dict the CLI emits. For looping over many ERA site-years this is cleaner and faster than shelling out.
• Auth: EE-backed sources need GCP_PROJECT_ID in the environment (or pass ee_project_id=). For historical LTE sites, agera_5 (or --source paired --precip-source chirps_v3_daily_rnl --temp-source agera_5) is the recommended default.

Prototype adapter (LTE row → toolkit → comparison table)

import json
from pathlib import Path
import pandas as pd
from climate_tookit.climate_statistics import analyze_climate_statistics

def to_mmdd(value):
    """Accept 'MM-DD', 'YYYY-MM-DD', or any pandas-parseable date -> 'MM-DD'."""
    s = str(value).strip()
    if len(s) == 5 and s[2] == "-":
        return s
    return pd.to_datetime(s).strftime("%m-%d")

def fixed_season_from_lte(row):
    """Build the toolkit fixed-season string from ERA S1 (+ optional S2)."""
    windows = [f"{to_mmdd(row['Site.Start.S1'])}:{to_mmdd(row['Site.End.S1'])}"]
    if pd.notna(row.get("Site.Start.S2")) and pd.notna(row.get("Site.End.S2")):
        windows.append(f"{to_mmdd(row['Site.Start.S2'])}:{to_mmdd(row['Site.End.S2'])}")
    return ",".join(windows)

def run(lte_csv, out_dir="outputs/era_lte", source="agera_5"):
    out = Path(out_dir); out.mkdir(parents=True, exist_ok=True)
    lte = pd.read_csv(lte_csv)
    summary_rows = []
    for i, row in lte.iterrows():
        try:
            result = analyze_climate_statistics(
                location_coord=(float(row["Site.LatD"]), float(row["Site.LonD"])),
                start_year=int(row["Time.Start.Year"]),
                end_year=int(row["Time.End.Year"]),
                source=source,
                fixed_season=fixed_season_from_lte(row),
                verbose=False,
            )
        except Exception as e:
            print(f"[skip] row {i}: {e}")
            continue
        (out / f"stats_{i}.json").write_text(json.dumps(result, default=str, indent=2))
        summary_rows.append({
            "row": i,
            "lat": row["Site.LatD"], "lon": row["Site.LonD"],
            "years": f"{row['Time.Start.Year']}-{row['Time.End.Year']}",
            "fixed_season": fixed_season_from_lte(row),
            "era_reported_rain": row.get("Site.Rain.Seasons"),
            # Map toolkit seasonal totals here after inspecting one stats_*.json:
            # the per-season summary view holds seasonal rainfall / rainy / dry counts.
        })
    pd.DataFrame(summary_rows).to_csv(out / "era_vs_toolkit.csv", index=False)
    print(f"wrote {out/'era_vs_toolkit.csv'}")

if __name__ == "__main__":
    import sys
    run(sys.argv[1])

One thing to confirm before wiring the comparison column

The stats result has three views (raw / overall / per-season). Inspect a single stats_*.json to pin the exact key holding seasonal rainfall total for your chosen source, then fill in the toolkit_rain_total mapping — it's the only source-dependent bit. Once that's set, merging with ERA yield columns for the explanatory analysis is a straight pandas join.

This sits well as a standalone workflow/repo script (as you noted, not a core-package change). Happy to help firm up the exact output-key mapping and a couple of validation sites if you share a small LTE export.

[Sammyjoseph999]

Followed up on the output-key mapping I flagged. I ran analyze_climate_statistics and inspected the real result, so the comparison columns can now be wired precisely:

• Seasonal metrics are in season_statistics — a list with one entry per (year, season). season_number matches the order of the fixed-season windows (S1 → 1, S2 → 2).
• Seasonal rainfall total = season_statistics[i]["precipitation"]["total_mm"]. That block also has rainy_days, max_daily, intensity; ["temperature"] has mean_tavg/mean_tmax/mean_tmin; ["water_balance"] has NDWS, WRSI, total_balance, etc. Each entry also carries year, onset, cessation, length_days.

Finished adapter (ERA LTE → toolkit seasonal comparison)

"""ERA LTE -> Climate Toolkit seasonal-stats comparison.

For each ERA site-year, builds the toolkit fixed-season windows from the ERA
season dates, runs analyze_climate_statistics, and emits a tidy table pairing
ERA-reported values with toolkit-derived per-season metrics.

    set GCP_PROJECT_ID=<your-ee-project>
    python era_lte_integration.py lte.csv
"""
import sys
import pandas as pd
from climate_tookit.climate_statistics import analyze_climate_statistics


def to_mmdd(value):
    """'MM-DD', 'YYYY-MM-DD', or any pandas-parseable date -> 'MM-DD'."""
    s = str(value).strip()
    return s if (len(s) == 5 and s[2] == "-") else pd.to_datetime(s).strftime("%m-%d")


def season_windows(row):
    """ERA S1 (+ optional S2) -> ['MM-DD:MM-DD', ...]; index == season_number - 1."""
    w = [f"{to_mmdd(row['Site.Start.S1'])}:{to_mmdd(row['Site.End.S1'])}"]
    if pd.notna(row.get("Site.Start.S2")) and pd.notna(row.get("Site.End.S2")):
        w.append(f"{to_mmdd(row['Site.Start.S2'])}:{to_mmdd(row['Site.End.S2'])}")
    return w


def toolkit_seasonal_rows(result):
    """Flatten season_statistics -> per (year, season_number) toolkit metrics."""
    rows = []
    for s in result.get("season_statistics", []):
        p, t, wb = s.get("precipitation") or {}, s.get("temperature") or {}, s.get("water_balance") or {}
        length, rainy = s.get("length_days"), (p.get("rainy_days"))
        rows.append({
            "year": s.get("year"),
            "season_number": s.get("season_number"),
            "onset": s.get("onset"),
            "cessation": s.get("cessation"),
            "tk_rain_total_mm": p.get("total_mm"),
            "tk_rainy_days": rainy,
            "tk_dry_days": (length - rainy) if (length is not None and rainy is not None) else None,
            "tk_mean_tavg_c": t.get("mean_tavg"),
            "tk_NDWS": wb.get("NDWS"),
            "tk_WRSI": wb.get("WRSI"),
        })
    return rows


def run(lte_csv, source="agera_5", out_csv="era_vs_toolkit.csv"):
    lte = pd.read_csv(lte_csv)
    out = []
    for i, row in lte.iterrows():
        try:
            result = analyze_climate_statistics(
                location_coord=(float(row["Site.LatD"]), float(row["Site.LonD"])),
                start_year=int(row["Time.Start.Year"]),
                end_year=int(row["Time.End.Year"]),
                source=source,
                fixed_season=",".join(season_windows(row)),
                verbose=False,
            )
        except Exception as e:
            print(f"[skip] row {i}: {e}")
            continue
        for tk in toolkit_seasonal_rows(result):
            out.append({
                "lte_row": i,
                "lat": row["Site.LatD"], "lon": row["Site.LonD"],
                # adjust these to your ERA export's actual column names:
                "crop": row.get("Product"),
                "era_yield": row.get("Yield"),
                "era_reported_rain": row.get("Site.Rain.Seasons"),
                **tk,
            })
    df = pd.DataFrame(out)
    df.to_csv(out_csv, index=False)
    print(f"wrote {out_csv}  ({len(df)} season-rows from {len(lte)} ERA rows)")
    return df


if __name__ == "__main__":
    run(sys.argv[1])

Output is one row per ERA site-year-season, pairing era_reported_rain / era_yield with tk_rain_total_mm, tk_rainy_days, tk_dry_days, tk_mean_tavg_c, etc. — so the rainfall-vs-rainfall check and the climate-vs-yield exploration are both a straight pandas step from here.

Two things to adjust to your actual ERA export: the crop/yield column names (Product/Yield are placeholders), and how Site.Rain.Seasons is reported (single value vs per-season) so it lines up with season_number. Share a small LTE sample and I'll pin those exactly and add a couple of validation sites.

Verified the toolkit side against a live run (agera_5, fixed two-season windows) — season_statistics[i]["precipitation"]["total_mm"] is confirmed.

[Sammyjoseph999]

Looked at the actual ERA LTEs repo (ERAgriculture/LTEs) to ground the approach. Two things shape the design:

1. The LTE registry ships as CSV — data/unique_ltes.csv (LTE.ID, Site.ID, Year.start, Year.end, Latitude, Longitude). No R needed for site/period metadata.
2. The full season windows + crop + yield live in the R .RData (industrious_elephant_* → Site.Out / Time.Out / Data.Out) on s3://digital-atlas/era/data/, and ERA already has its own climate layer (POWER.CHIRPS.parquet). So the toolkit's role is a richer climate layer (AgERA5/GEE, seasonal onset/cessation, water balance, SPEI) keyed to each LTE's seasons.

Recommended architecture: loosely-coupled R → CSV → Python bridge

Each tool stays native, joined by a flat file. Beats reticulate (couples runtimes) and Python-reads-.RData (fragile). Two tiers:

Tier 1 — pure Python, runnable now on the repo CSV (no R)

Runs the toolkit with auto season detection for every LTE site in unique_ltes.csv. I ran the data handling against the real file: 358 unique site-periods after parsing/dedup.

"""Tier 1: run the Climate Toolkit across the ERA LTE registry (no R needed)."""
import argparse
import pandas as pd
from climate_tookit.climate_statistics import analyze_climate_statistics

def _year(v):
    try: return int(float(str(v).strip()[:4]))
    except (ValueError, TypeError): return None

def load_sites(csv_path):
    df = pd.read_csv(csv_path).rename(columns={"Latitude": "lat", "Longitude": "lon"})
    df["start_year"] = df["Year.start"].map(_year)
    df["end_year"] = df["Year.end"].map(_year)
    df = df.dropna(subset=["lat", "lon"])
    df = df[df["start_year"].notna() & df["end_year"].notna()]
    return df.drop_duplicates(subset=["Site.ID","lat","lon","start_year","end_year"]).reset_index(drop=True)

def seasonal_rows(result):
    rows = []
    for s in result.get("season_statistics", []):
        p, t, wb = s.get("precipitation") or {}, s.get("temperature") or {}, s.get("water_balance") or {}
        length, rainy = s.get("length_days"), (s.get("precipitation") or {}).get("rainy_days")
        rows.append({"year": s.get("year"), "season_number": s.get("season_number"),
            "onset": s.get("onset"), "cessation": s.get("cessation"), "length_days": length,
            "tk_rain_total_mm": p.get("total_mm"), "tk_rainy_days": rainy,
            "tk_dry_days": (length - rainy) if (length is not None and rainy is not None) else None,
            "tk_mean_tavg_c": t.get("mean_tavg"), "tk_NDWS": wb.get("NDWS"), "tk_WRSI": wb.get("WRSI")})
    return rows

def run(csv_path, source="agera_5", out_csv="era_lte_climate.csv", limit=None, dry_run=False):
    sites = load_sites(csv_path)
    if limit: sites = sites.head(limit)
    print(f"{len(sites)} unique ERA LTE site-periods")
    if dry_run:
        print(sites[["LTE.ID","Site.ID","lat","lon","start_year","end_year"]].to_string()); return
    out = []
    for _, row in sites.iterrows():
        try:
            result = analyze_climate_statistics(location_coord=(float(row["lat"]), float(row["lon"])),
                start_year=int(row["start_year"]), end_year=int(row["end_year"]), source=source, verbose=False)
        except Exception as e:
            print(f"[skip] {row['LTE.ID']} {row['Site.ID']}: {e}"); continue
        for tk in seasonal_rows(result):
            out.append({"LTE.ID": row["LTE.ID"], "Site.ID": row["Site.ID"], "lat": row["lat"], "lon": row["lon"], **tk})
    pd.DataFrame(out).to_csv(out_csv, index=False)
    print(f"wrote {out_csv} ({len(out)} season-rows)")

if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("csv"); ap.add_argument("--source", default="agera_5")
    ap.add_argument("--out", default="era_lte_climate.csv")
    ap.add_argument("--limit", type=int); ap.add_argument("--dry-run", action="store_true")
    a = ap.parse_args(); run(a.csv, a.source, a.out, a.limit, a.dry_run)

set GCP_PROJECT_ID=<your-ee-project>
python era_lte_tier1.py unique_ltes.csv --dry-run            # list 358 sites, no fetch
python era_lte_tier1.py unique_ltes.csv --limit 5            # smoke on 5 real sites

Tier 2 — the yield comparison (one R export step)

A short R script loads the industrious_elephant .RData and writes a flat per site-year-season CSV; the adapter from my previous comment then produces the toolkit-vs-ERA + climate-vs-yield join (it already reads Site.LatD/Time.Start.Year/Site.Start.S1/...). Template to adapt to the actual ERA object names (see the vignette loader):

library(dplyr)
load("industrious_elephant_2023-2025-01-10.RData")   # from s3://digital-atlas/era/data/ (adjust date)

sites   <- Site.Out %>% select(Site.ID, Site.LatD, Site.LonD,
             Site.Start.S1, Site.End.S1, Site.Start.S2, Site.End.S2, Site.Rain.Seasons)
times   <- Time.Out %>% select(Site.ID, Time.Start.Year, Time.End.Year)
outcomes<- Data.Out %>% filter(Out.SubInd == "Crop Yield") %>%   # adjust outcome label
             select(Site.ID, Product, MeanC, MeanT)              # crop, control/treatment means

export <- sites %>% inner_join(times, by="Site.ID") %>% left_join(outcomes, by="Site.ID")
write.csv(export, "era_export.csv", row.names = FALSE)           # -> feed the Python adapter

Suggested next step

Start with Tier 1 (runs today on the CSV already in the ERA repo) to validate the toolkit across the network and eyeball a few sites, then do Tier 2 for the yield analysis. If you (or @Rwema25) run --limit 5 and share the output — plus confirm the exact Data.Out outcome/crop/yield column names — I'll finalize the Tier-2 join and add a couple of validation sites.