Add dataset-header validation & auto-fixing tool - #331
Merged
benceszirbik merged 3 commits intoJul 15, 2026
Conversation
Introduces a reusable, branch-agnostic validator for DCAT dataset headers, encoding the rules from Svein's PR #322 review as a single source of truth (buildScripts/dataset_header/). Two tiers: - Tier A (structural / fixed-value): required properties, fixed values, xml:lang tags, forbidden properties, disallowed namespaces, UUID and UTC-Z datetime formats, TSO-/RCC- publisher naming. Auto-fixable. - Tier B (reference-data membership): conformsTo / isVersionOf / spatial / wasGeneratedBy / publisher-party validated against the referenceData schemes. Report-only, blocked on extending the schemes. Validation is semantic (rdflib); schemes are loaded from Instance/referenceData so the same tool serves any NCP version/branch. Ships: - CLI: buildScripts/validate_dataset_header.py (text + JSON report) - pytest: tests/test_dataset_header.py (report-only via xfail; an ENFORCE flag flips Tier-A to blocking once the fixer cleans the branch) - deps: rdflib, pyshacl added to pyproject On cgmes-3.0_ncp-2.5_tc-2.0 the NetworkCode cimxml scope reports 329 Tier-A (146 auto-fixable) and 429 Tier-B findings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
buildScripts/fix_dataset_header.py + dataset_header/fixer.py apply the deterministic (Tier-A) fixes from the same rule set the validator uses, via formatting-preserving text edits (idempotent, clean diffs): - drop disallowed namespaces (eumd/euvoc) and their usages - fix the dcterms trailing-# namespace and doubled-quote well-formedness - remove forbidden properties and the now-unused dcatcim namespace - force fixed values (type, accessRights, license, rights, rightsHolder) - add xml:lang="en" to description/versionNotes - normalize publisher to TSO-/RCC- for known parties - insert missing fixed-value properties + a placeholder adms:versionNotes Dry-run by default; --apply writes. Non-deterministic gaps (real dates, titles, spatial, unknown-party publishers) and all Tier-B values are left untouched and reported by the validator. Verified end-to-end on the NetworkCode cimxml scope: auto-fixable Tier-A 329 -> 0, all 68 files well-formed; remaining 60 Tier-A are genuine data/scheme gaps. Adds fixer unit tests. The pytest integration stays report-only (xfail); flip the fixable-Tier-A gate to blocking on a branch once its headers have been fixed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dataset-header validation & fixing tool
A reusable, branch/version-agnostic tool that checks — and auto-fixes — the DCAT dataset headers of instance files against the rules from Svein's PR #322 review. Motivation: the header standardization done by hand in #322 does not carry across version branches (this branch,
cgmes-3.0_ncp-2.5_tc-2.0, has the old headers again), and hand-fixing every branch does not scale.Instead of Svein's "write a validator, then AI-generate a separate fixer" (which drift apart), the validator and fixer share one rule set (
buildScripts/dataset_header/rules.py). Two modes of the same rules.Two tiers
xml:langtags, forbidden properties, disallowed namespaces (eumd/euvoc), UUID + UTC-Zdatetime formats,TSO-/RCC-publisher naming.conformsTo/isVersionOf/spatial/wasGeneratedBy/ publisher-party validated againstInstance/referenceData. New scheme entries become valid automatically — this is exactly the parked Complete dcat:Dataset headers in all NetworkCode cimxml files #322 reference-data work.What's here
buildScripts/validate_dataset_header.py— rdflib-based validator (text + JSON report,--strictfor CI).buildScripts/fix_dataset_header.py— formatting-preserving, idempotent auto-fixer for the Tier-A subset (dry-run by default,--applyto write).buildScripts/dataset_header/— shared package (rules,schemes,discovery,fixer,report) + README.tests/test_dataset_header.py— report-only integration (xfail) + fast unit tests for the rule and fixer logic.pyproject.toml— declaresrdflibandpyshacl(previously only transitive).No instance files are modified in this PR — it is the tool only.
Verified end-to-end (NetworkCode cimxml scope, 68 files)
fix_dataset_header.py --apply: auto-fixable Tier-A 329 → 0, all 68 files well-formed; the remaining 60 Tier-A are genuine data/scheme gaps (missing real dates/titles/spatial, unknown-party publishers) that are correctly left for humans.Roadmap
pyshaclas an independent cross-check of the Python rules (Svein's "check the SHACL is correct").🤖 Generated with Claude Code