Eurostat: $DV_ bookmark entries inflate the catalogue, and detail=allstubs strips the Description

[aborruso]

Two independent problems in the Eurostat catalogue, sharing one root cause: portals.json sets dataflow_params = {"detail": "allstubs", "references": "none"}, so everything except Name is discarded at the wire.

They are separately shippable and have very different costs — part 1 needs no extra fetch at all.

---

Part 1 — $DV_ entries inflate the catalogue

546 of 8,246 Eurostat catalogue entries carry a $DV_ suffix in their ID. Sampling ISOC_CI_AC_I$DV_645 at detail=full:

DISSEMINATION_OBJECT_TYPE     = EXTRACTION      (PRC_HICP_MIDX has DATASET)
DISSEMINATION_EXTRACTION_TYPE = BOOKMARK
DISSEMINATION_PARENT_ID       = ISOC_CI_AC_I
<Ref id="ISOC_CI_AC_I">                         same DSD as the parent
no <c:Description>                              only the parent has one

These are saved extractions/bookmarks, not distinct datasets, and they duplicate their parent's title. Individuals - internet activities appears 6 times, Business demography by size class and NACE Rev. 2 activity 5 times.

Effect on title ambiguity:

catalogue	dataflows	ambiguous titles
as-is	8,246	1,120 (14%)
excluding $DV_	7,700	148 (1.9%)

So Eurostat's apparent title problem is almost entirely these entries — the underlying catalogue is healthy. This fix needs no detail=full: the $DV_ pattern is already visible in the cached df_id.

Evidence level: 546 identified by ID pattern; one entry verified via annotations.

Open question, worth resolving before implementing: is $DV_ visibility a consequence of detail=allstubs, or does the API expose a parameter to exclude extractions server-side? The latter would be the clean fix. Filtering on DISSEMINATION_OBJECT_TYPE is the robust client-side option but requires detail=full.

---

Part 2 — the dataflow Description is stripped

PRC_HICP_MIDX, references=none:

	payload	<c:Description>
detail=full	7,276 B	present, EN/DE/FR
detail=allstubs	995 B	absent

The text carries information that is not in the title:

The Harmonised Index of Consumer Prices (HICP) measures price changes of various consumer goods and services over time (inflation). […] Please note that this dataset is discontinued and replaced by dataset prc_hicp_minr

This is content for retrieval, not a fix for ambiguity — that is Part 1.

Second, provider-independent cause: even with allstubs removed, the field would still be ignored. df_description is populated by get_name_by_lang() (discovery.py:129), which reads Name — the title. The only Description reader in the package is categories.py:92, for category schemes. Both changes are needed.

Bonus: this same fetch unblocks #9. That issue is trying to infer the ESMS metadata URL from the dataflow ID and finding no reliable pattern. There is no need to guess — detail=full includes it verbatim:

<c:AnnotationType>ESMS_HTML</c:AnnotationType>
<c:AnnotationURL>https://ec.europa.eu/eurostat/cache/metadata/en/prc_hicp_esms.htm</c:AnnotationURL>

Open questions

• Cost of detail=full at catalogue scale is unmeasured: 7.3× payload on a single dataflow, unknown on 8,246. Measure before choosing lazy-per-dataflow vs bulk at catalogue build.
• Descriptions are HTML and need stripping before indexing.
• Comext shares the same dataflow_params; it has only 9 dataflows, so likely moot, but worth confirming.

---

Scope note

The ISTAT catalogue needs neither of these — no $DV_ equivalent, and Description is present on 26 of 4,884 dataflows. Its descriptions come from a different endpoint entirely. Each catalogue imposes its own strategy, which argues for per-provider hooks rather than branching in shared code — see #41.

[aborruso]

Companion issues: #52 and #53 (ISTAT — needs neither of these fixes, its descriptions come from a separate endpoint). Part 2 here also unblocks #9.

[aborruso]

Measured the whole ESTAT catalogue on 2026-08-28 (curl, two full fetches). Three premises of this issue change, and a fourth consequence of allstubs shows up that is not recorded here.

1. The open question of Part 1 is closed: no server-side filter

The $DV_ entries are not a consequence of detail=allstubs — they are present, in the same number, at both detail levels:

	detail=allstubs	detail=full
dataflows	8,152	8,152
ids with $DV_	548	548

allstubs does not create or hide them; it only strips the annotations that explain what they are.

And the API exposes no parameter to exclude extractions server-side. The documented structure-query parameters are detail, references, format, lang, compressed — nothing filters by dissemination object type (structure queries). So the filter has to be client-side.

The id pattern is a safe proxy — verified across all 8,152 entries, not sampled:

DISSEMINATION_OBJECT_TYPE:      DATASET 7604 | EXTRACTION 548
DISSEMINATION_EXTRACTION_TYPE:  (none)  7604 | BOOKMARK    548
ids containing $DV_:            548, all EXTRACTION
no $DV_ but not DATASET:        0
other ids containing $:         0

Exact bijection in both directions. Filtering on the string gives the same result as filtering on the annotation, at zero network cost. Parent always exists: 548/548. Title identical to the parent: 547/548.

2. What a $DV_ entry actually is — and it does serve data

It is a saved Data Browser view: a stored selection over the parent's dimensions plus a table layout. LFST_HHEREDCH$DV_1343 carries DISSEMINATION_PRESENTATIONS with preselection (freq=A, sex=T, age=Y18-24, …) and layoutAxes (time on y, geo on x).

But it is not an empty shell — the data endpoint resolves it:

data/LFST_HHEREDCH$DV_1343  → HTTP 200, 801 KB, 8,136 rows
data/LFST_HHEREDCH          → HTTP 200, 102 MB

The subset is a genuine sub-cube: freq fixed to A, age to Y20-49, n_child to TOTAL, free on sex (3), agechild (2), isced11 (4), geo (36), time. Note the served filter is not the display preselection quoted above — extraction filter and presentation layout are two distinct things inside the same entry.

Consequence for the design: these ids should be hidden from the catalogue (search, tree, counts) but stay resolvable when a user pastes one from a Data Browser link. They are not resolvable today — opensdmx get 'LFST_HHEREDCH$DV_1343' returns HTTP 404. See point 4.

3. Part 2 is not the expensive half — but correct the numbers

The issue extrapolated 7.3× from a single dataflow. At catalogue scale it is 5.25×, and compression changes the picture:

	uncompressed	compressed=true
allstubs	7.08 MB / 23 s	0.93 MB
full	37.2 MB / 114 s	1.92 MB / 6.8 s

Apples to apples, full compressed costs ~2× allstubs compressed — not less. What makes it acceptable is the absolute: 1.9 MB and 6.8 s, once per catalogue build, against 7.1 MB and 23 s for what we actually pay today, since we do not compress at all.

Two caveats on compression, both measured:

• compressed=true returns a gzip file (Content-Disposition: ALL_DATAFLOWS.xml.gz) with no Content-Encoding header, so httpx does not decompress it. Explicit gunzip is required.
• Standard HTTP negotiation is unreliable here: with Accept-Encoding: gzip the server returns gzip bytes but still omits Content-Encoding (non-conformant); with httpx's default gzip, deflate, br, zstd it returns plain XML. So today we get no compression, and we must not rely on the header path.

What detail=full contains, on 8,152 dataflows: <c:Description> on 3,993 (49%), ESMS_HTML annotation on 7,812 (96%). So this single fetch covers Part 2 and #9.

4. Fourth consequence of allstubs, not recorded in this issue: no DSD reference

detail=allstubs → 0  <s:Structure> references
detail=full     → 8,152

df_structure_id is null for every Eurostat dataflow. It works only because load_dataset falls back to the dataflow id (discovery.py:584, structure_id = match_row["df_structure_id"] or match_row["df_id"]), and for real datasets the two coincide. Verified on the full payload:

DATASET   with ref != df_id:  0     (out of 7,604)
EXTRACTION with ref != df_id: 548   (all point at the parent DSD)

So populating df_structure_id from detail=full is a no-op for real datasets and the fix for the 404 above. Note the dims cache is keyed on structure_id, so cached entries for bookmarks become orphaned — a one-off cost, not a blocker.

5. Comext: decide explicitly

Comext has 11 dataflows (not 9), no $DV_, no <c:Description>, but ESMS_HTML and a DSD reference on all 11. It honours compressed=true the same way. detail=full costs 23 KB there, so compression is pointless but harmless — the decision is only whether to move it to full for the ESMS URLs and the DSD reference.

Effect on scope

Part 1 stays shippable on its own, and its open question is now answered. Part 2's cost objection dissolves, provided the gunzip path is added. The DSD reference is a third reason to move to detail=full, and it is what makes $DV_ ids resolvable instead of merely hidden.

[aborruso]

Correction to my previous comment, and one more measurement on whether the $DV_ entries are worth keeping.

Description count was wrong. I counted <c:Description> elements, and each dataflow carries one per language (en/de/fr). Per dataflow:

	dataflows with <c:Description>
whole catalogue	1,331 of 8,152 (16%)
real datasets only	1,293 of 7,604 (17%)
$DV_ bookmarks	38 of 548 (7%)

Not 3,993/49%. ESMS_HTML was right — 7,812 of 8,152 (96%), one annotation per dataflow.

This weakens part 2's retrieval argument: the description covers a sixth of the catalogue, not half. It does not change the case for detail=full, which is carried by ESMS_HTML (96%, closes #9) and by the DSD reference (100%, absent under allstubs).

Could the bookmarks be kept if they were well described? They are not described:

• 510 of 548 have no <c:Description> at all
• 547 of 548 repeat the parent's title verbatim
• all 548 name their stored view Default presentation — there is no per-bookmark label anywhere in the payload

So nothing in the SDMX payload says what any of them selects. The filter they apply is only readable by parsing the DISSEMINATION_PRESENTATIONS JSON dimension by dimension, and even then it names codes, not a human-readable summary. Keeping them listed would mean showing the user 548 entries whose only distinguishing text is an id suffix.

[aborruso]

detail=full also carries per-dataflow coverage — add it to part 4

Moving this here from #81, which is being closed: the same detail=full fetch already planned for descriptions carries dataset size and time coverage as annotations, at zero extra network cost.

Present on the Eurostat catalogue (measured 2026-08-28):

annotation	coverage
OBS_COUNT	7,604 of 7,604 real dataflows
OBS_PERIOD_OVERALL_OLDEST	7,602
OBS_PERIOD_OVERALL_LATEST	7,602

The value sits in <c:AnnotationTitle>, the key in <c:AnnotationType> — same shape as the ESMS URL already noted above for #9. Verified on PRC_HICP_MIDX:

OBS_COUNT                  7699058
OBS_PERIOD_OVERALL_OLDEST  1996-01
OBS_PERIOD_OVERALL_LATEST  2025-12

Worth caching alongside the description: it gives search/tree a size and a time range for every dataset, which today a user can only discover by downloading it. OBS_PERIOD_OVERALL_LATEST also answers "how fresh is this dataflow" without a data call.

One caveat: these values move. #81 recorded 7,589,850 / 2025-09 for this same dataflow; today it reads 7,699,058 / 2025-12. So they are cache-with-TTL material, not facts to freeze — and the TTL should probably match whatever we pick for the description cache.