ISTAT: fetch dataflow descriptions from METADATA_API as a resource file

[aborruso]

Problem

ISTAT dataflows expose a title and nothing else. Verified on the full catalogue (4,884 dataflows, dataflow/IT1):

source	dataflows
<common:Description>	26 (0.5%)
LAYOUT_DATAFLOW_KEYWORDS annotation	144 (2.9%)
METADATA_URL annotation	3,962 (81%)

LAYOUT_DATAFLOW_KEYWORDS is sometimes cited as where ISTAT hides descriptions. It is not a usable source: it is present on zero parent dataflows, concentrated in ~40 families (mostly agriculture, 101_*), absent from the entire labour domain, and its content is concatenated keywords rather than prose:

Tipo di coltivazione+Strutturale, annuale, anno+Nazionale, Italia,area, aree,
ripartizione, ripartizioni, ripartizionale,regione, regioni, regionale…

Real descriptions exist, in a third source

ISTAT publishes prose in its reference metadata repository, reachable as JSON:

https://esploradati.istat.it/METADATA_API/api/getMetadata
    ?metadataSetId=MDS_DIFF_REPORT&reportId={df_structure_id}

JSON path:
.data.metadataSets[0].reports[0].attributeSet.reportedAttributes[0].attributeSet.reportedAttributes[0].texts.it

Sample response:

Attività notarile: La rilevazione sull'attività notarile rileva il numero e le caratteristiche degli atti notarili e delle convenzioni in essi contenute. L'indagine è condotta ogni anno, in collaborazione con l'Ufficio centrale archivi notarili e con il Consiglio nazionale del notariato, presso tutti i notai in esercizio…

The key is df_structure_id, which the catalogue already carries. One report serves every dataflow sharing that DSD.

Coverage and cost

Measured by joining a prior extraction (760 reports, 2025-11-17) against the current catalogue:

dataflows in catalogue	4,879
df_structure_id present in the extraction	4,333 (89%)
non-empty prose description	3,360 (69%)

Cost is 760 requests, not 4,879 — the mapping is many-to-one. Endpoint re-tested today: HTTP 200, ~0.15 s per call.

Important: the mapping is not 1:1

The description lives at survey (DSD) level, not dataflow level. All dataflows of one survey receive the same text. On the four dataflows titled Sesso, età:

df_id	DSD	description
85_83_DF_DCSA_VOLON_1	DCSA_VOLON	Multiscopo — aspetti della vita quotidiana
151_877_DF_DCCV_DISOCCUPTMENS1_1	DCCV_DISOCCUPTMENS1	Rilevazione forze di lavoro
150_875_DF_DCCV_OCCUPATIMENS1_2	DCCV_OCCUPATIMENS1	Rilevazione forze di lavoro
150_1170_DF_..._UNT2020_2	DCCV_OCCUPATIMENS1_UNT2020	Rilevazione forze di lavoro

It separates volunteering from labour — useful for topical retrieval. It does not separate employed from unemployed from employed-until-2020.

Worse for the discontinued-series case: the description of the discontinued dataflow never mentions 2020, "regolamento" or "precedente", while the current one discusses Regulation (EU) 2019/1700 and the recalculation through December 2020. Weighting descriptions would pull retrieval toward the wrong dataflow, not away from it.

So this is complementary to, not a replacement for, the parent-title work: descriptions answer which survey, parent titles answer which cut and which vintage.

Proposal

Generate the descriptions offline as a committed resource file, following the existing data/constraints/ + .github/workflows/constraints-archive.yml pattern: a scheduled job, keyed on df_structure_id, incremental with a budget and a status file.

Prior art proving the recipe works end to end: script/jiku_istat/jiku_istat.sh in the author's apiRestIstat repo.

Open questions

• Pacing. METADATA_API is a different path on the same host as the SDMX endpoint, which enforces a strict rate limit and has produced 1–2 day IP blocks. Whether it shares that budget is unverified — measure conservatively before choosing a pause, do not assume the ~0.15 s response time means it is safe to hammer.
• Descriptions are HTML (<strong>, <p>) and Italian-only in this path; check whether an en text exists for the same report.
• Refresh cadence: survey metadata changes rarely, so a weekly or monthly run is likely enough.
• Should the resource ship inside the package, or stay a repo data file consumed opportunistically?

[aborruso]

Companion issues: #52 (parent title — the disambiguation axis this one does not solve) and #54 (Eurostat).

[aborruso]

Correction to the JSON path in the issue body — it returns the wrong field on some DSDs.

The path recorded above is positional:

.data.metadataSets[0].reports[0].attributeSet.reportedAttributes[0].attributeSet.reportedAttributes[0].texts.it

The report is a set of named attributes whose order is not fixed — observed ids: SOURCE_ITEMS, CHARACTERISTICS_ITEMS, MANIPULATION_ITEMS. Indexing into position 0 does not fail on a mismatch, it silently returns whatever happens to sit there.

The prose lives in SOURCE_ITEMS → DATA_SOURCE:

.data.metadataSets[0].reports[0].attributeSet.reportedAttributes[]
  | select(.id=="SOURCE_ITEMS") | .attributeSet.reportedAttributes[]
  | select(.id=="DATA_SOURCE") | .texts.it

Two consequences for the plan:

1. Select by id, never by index, or the archive fills with the wrong field for an unknown share of DSDs.
2. The field is DATA_SOURCE — "source", not "description". It usually opens with the survey name in <strong> followed by prose, but not always. On DCCV_DETENUTI it holds only Ministero della Giustizia - Dipartimento dell'amministrazione penitenziaria — the issuing body, no prose. So the measured 69% coverage overstates the share of DSDs carrying real descriptive text: it was computed with the positional path and counts non-empty strings. Non-empty is not the same as descriptive; the verification phase has to re-measure.

Verified live today on DCIS_NATI2, DCCV_TAXOCCU1, DCSC_TUR, DCSA_VOLON, DCCV_DISOCCUPTMENS1, DICA_AINDP, DCIS_ACQCITIZ, DCCV_TURISTI, DCCV_DETENUTI — all HTTP 200.

[aborruso]

The SDMX catalogue already declares the metadata endpoint — no need to guess it.

Dumping the raw dataflow (dataflow/IT1/25_74?references=none&detail=full) shows the METADATA_URL annotation — present on 3,962 dataflows (81%) — carries an it/en URL whose query string names both the metadata set and the report:

https://esploradati.istat.it/RefMeta/template/GenericMetadataTemplate.html?nodeId=DW&metadataSetId=MS_ISTAT_TOPMETA2&reportId=DCIS_NATI2&lang=it&BaseUrlMDA=https://esploradati.istat.it/METADATA_API

So ISTAT points at METADATA_API from inside SDMX itself. Verified today: metadataSetId=MS_ISTAT_TOPMETA2 returns the same DATA_SOURCE prose as the MDS_DIFF_REPORT set this issue has been using.

Consequences for the plan:

• Parse reportId from the annotation rather than assuming it equals df_structure_id. On 25_74 the two coincide, but that assumption was never tested at scale and would fail silently wherever it does not hold.
• Prefer MS_ISTAT_TOPMETA2, the set ISTAT itself references, absent a measured reason to prefer MDS_DIFF_REPORT.
• The 81% annotation coverage becomes a measured upper bound on reachable reports, to reconcile with the re-measured prose coverage.
• Caveat: the package does not parse annotations today (discovery.py reads Name only), so obtaining reportId per dataflow means either extending the catalogue parser or a separate detail=full fetch in the script.

[aborruso]

The 26 <common:Description> are not descriptions — they are release notes. Dataflow-level description coverage is zero, not 0.5%.

Downloaded the full catalogue raw (dataflow/IT1?references=none&detail=full, 13 MB, 4,884 dataflows) and read the field contents rather than just counting occurrences:

field	dataflows	what it actually holds
<common:Description>	25	release/discontinuity notices
annotation DATAFLOW_NOTES	136	methodological notes about the specific cut
annotation LAYOUT_DATAFLOW_KEYWORDS	144	concatenated keywords
annotation METADATA_URL	3,962	link to the metadata report

Verbatim samples from <common:Description>:

Con il rilascio dei dati relativi al 2024, si rendono disponibili i dati revisionati per gli anni 2019-2023.

Da gennaio 2025, i dati degli alloggi in affitto includono anche quelli gestiti in forma non imprenditoriale (locazioni brevi o per finalità turistiche).

So ISTAT uses the standard Description element as a notices field. No dataflow in the catalogue carries a description of its own content.

DATAFLOW_NOTES is the only genuinely dataflow-level prose — and it is good ("I dati PROVINCIALI non sono perfettamente confrontabili con quelli delle analoghe tavole diffuse: prima del 2017, a seguito dell'abolizione delle province…") — but it covers 2.8% and is a non-standard annotation, so no generic SDMX client reads it.

Consequence for this issue: the survey-level text from METADATA_API remains the only prose available at scale, and there is no dataflow-level alternative to weigh it against. It also means that if we ever ask ISTAT to populate <common:Description>, the request must target the DataStructure — on the dataflow that element already has an established, different use.

[aborruso]

A request has been sent to ISTAT support (2026-07-20), asking them to expose the survey descriptions through SDMX by populating <common:Description> on the DataStructure.

Why the DSD rather than the dataflow: reportId matches the DSD, and the 10 dataflows sharing DCIS_NATI2 all resolve to one report. Placing it on the dataflow would repeat the same text 10 times; on the DSD it is written once, ~760 values instead of 4,879, and any client can resolve dataflow → DSD itself. Verified that datastructure/IT1/DCIS_NATI2 currently exposes <common:Name> and no <common:Description> — the standard element is there, simply unpopulated.

This blocks nothing, but it should be checked before implementing. If ISTAT populates the element, most of this issue — 760 HTTP requests, a scheduled workflow, a committed resource file — becomes unnecessary, and the work reduces to reading a field the catalogue already returns.

Suggested handling: revisit around end of August 2026. If there is no answer by then, proceed with the plan as written, starting with the blocking pacing measurement.

[aborruso]

Shipped in v0.20.0 (2026-07-24). src/opensdmx/data/descriptions/istat.parquet ships inside the wheel — 3,949 dataflows, 618 distinct reports, ~81% of the catalogue — harvested via the METADATA_URL annotation → METADATA_API (DATA_SOURCE field), plus the siqual_id bridge to ISTAT SIQual.

embed.py reads it locally and makes no metadata request at embedding time. Refreshed monthly by .github/workflows/descriptions-archive.yml; last run 2026-08-01, success.

Design: docs/descriptions.md. Measured effect on retrieval: docs/search.md.