create running version with final data dic v1

[aslgraefe]

running v1 version

[aslgraefe]

Refactor: Single Source of Truth, Simplify Mappings, Fix Structural Issues

Context

We successfully produced a validated Phenopacket from the CIEINR pipeline, but the process exposed significant architectural debt. The current setup has too many manual steps, redundant configuration, scattered sources of truth, and fragile workarounds. This issue tracks a systematic cleanup.

---

1. Single Source of Truth for Code Systems

Problem: Code system metadata is defined in three places:

• cieinr_code_systems.yaml (LinkML schema) — versions, URLs, descriptions
• resources.py (Python @dataclass) — same info, manually duplicated
• rarelink's built-in CodeSystemsContainer (rd-cdm) — yet another copy with potentially stale versions

Fix:

• Generate resources.py from the LinkML YAML automatically (write a small script or use gen-python)
• The LinkML YAML becomes the single source; the Python dataclass is a build artifact
• Add a CI check or Makefile target: make codegen-codesystems
• Consider contributing CIEINR-specific code systems (VO, IUIS, BIOPORTAL) upstream to rd-cdm so _FIELD_TO_PREFIXES in rarelink recognises them natively

---

2. Generate Python Enums from LinkML Schemas

Problem: combined.py references enum classes like cieinr.datamodel.linkml_schemas.infections_initial_form.TypeOfInfectionEnum — but no .py files exist. Every run produces ~19 Failed to import enum class errors. They're non-fatal but noisy and mean no local value validation occurs.

Fix:

• Rename LinkML YAML files to drop numeric prefixes (03_infections_initial_form.yaml → infections_initial_form.yaml) so generated Python modules are importable
• Add a codegen step: gen-python <schema>.yaml > <schema>.py for each schema
• Add to Makefile / CI: make codegen-schemas
• Validate that enum_classes paths in combined.py match the generated module names

---

3. Simplify combined.py — It's Too Verbose

Problem: combined.py is ~250 lines of deeply nested dicts with massive repetition. Every instrument block repeats the same structural pattern. The ontology_routing config, enum_classes, and mapping_dicts are all manually maintained.

Fix:

• Extract a builder/helper function per block type:

  def phenotypic_feature_config(instrument, mapping_block, *, multi_onset=False, enum_module=None):
      ...

• Auto-derive enum_classes paths from instrument name + a naming convention instead of hardcoding each one
• Consider generating the entire mapping config from a declarative YAML that mirrors the LinkML schema structure — the mapping metadata (which field is the type field, which is severity, etc.) could live in the LinkML schema as annotations
• Move ontology_routing.scan_fields derivation to code: scan the LinkML schema for fields whose range is an ontology-backed enum, instead of listing every SNOMED field manually

---

4. Fix the genetic_information Non-Repeating ↔ Repeating Mismatch

Problem: genetic_information is non-repeating in REDCap (top-level fields) but rarelink's variation_descriptor_mapper.py hardcodes:

repeated_elements = data.get("repeated_elements", [])
genetic_data = element.get("genetic_findings")  # hardcoded key

Current workaround: post-process in redcap_to_linkml_script.py to wrap top-level data into repeated_elements with key "genetic_findings".

Fix (rarelink-side):

• Open RareLink issue: variation_descriptor_mapper should support top-level instruments, not just repeated elements
• The mapper should read instrument_name from config and check both top-level and repeated_elements
• Remove the hardcoded "genetic_findings" key — use the instrument name from config

Fix (cieinr-side, interim):

• Document the workaround in the script with a # TODO: remove when rarelink supports top-level genetics
• Or make genetic_information a repeating instrument in REDCap (it conceptually is — multiple variants per patient)

---

5. Fix the computed_dob Workaround

Problem: REDCap stores birth as separate birth_month + birth_year fields. The INDIVIDUAL_BLOCK references "date_of_birth_field": "computed_dob" but nothing in the pipeline creates this field. Current workaround: post-processing in the conversion script.

Fix:

• Add a computed_fields mechanism to the redcap_to_linkml registry:

  "patient_demographics_initial_form": {
      "mapper": map_demographics,
      "is_repeating": False,
      "computed_fields": {
          "computed_dob": lambda rec: f"{rec['birth_year']}-{rec['birth_month'].zfill(2)}-01"
              if rec.get('birth_year') and rec.get('birth_month') else None,
      },
  }

• Make redcap_to_linkml() apply computed fields automatically after mapping
• Generalise this for any month+year → date conversion (immunostatus date, genetic evaluation date, CBC date, etc.)

---

6. Fix Ontology-Routed Diseases Not Reaching the Phenopacket

Problem: The ontology routing adapter correctly identifies MONDO-coded infection instances and routes them to diseases. But _inject_routed_disease() in rarelink places sentinel values at the data root, while the DiseaseMapper navigates INTO the instrument section (patient_demographics_initial_form), so it never finds them. Result: "No diagnosis ID found in element".

Fix (rarelink-side):

• Open RareLink issue: _inject_routed_disease should inject sentinels into the instrument section, or the routed processor should not inherit the original instrument_name
• Alternative: create a lightweight "routed disease" processor with no instrument_name so the mapper reads from the root

---

7. Consolidate Month+Year Date Fields

Problem: Multiple instruments store dates as separate month/year fields: birth_month/birth_year, geneticevaluationmonth/geneticevaluationyear, cbcmonth/cbcyear, cd3_tmonth/cd3_tyear, onsetmonth/yearonset, etc. Each needs a different manual transformation.

Fix:

• Create a generic combine_month_year(record, month_field, year_field, day=1) utility in cieinr.utils
• Define a registry of all month+year field pairs per instrument
• Apply automatically during redcap_to_linkml conversion as a post-processing step
• This also enables onset age calculations that currently fail with [_extract_onset] No onset date or code found

---

8. Leverage LinkML More Deeply

Problem: We have LinkML schemas but use them mostly as documentation. The mapping config in combined.py manually lists every field, every enum class path, and every scan field — information that already exists in the LinkML YAML.

Improvements:

• Use LinkML slot annotations to mark phenopacket-relevant metadata (e.g., annotations: { phenopacket_role: "type_field" })
• Generate combined.py (or its equivalent) FROM the annotated LinkML schemas
• Use linkml-runtime SchemaView to introspect schemas at runtime and auto-discover enum classes, field types, and ontology ranges
• This eliminates the need to manually maintain parallel configs

---

9. Clean Up the redcap_to_linkml Registry

Problem: The registry mixes concerns:

• is_repeating — structural concern
• output_key — naming convention for downstream consumers
• mapper — the actual function

Some instruments need special handling (genetic_information wrapping, computed_dob injection) that lives in an ad-hoc post-processing script.

Fix:

• Add computed_fields support (see item 5)
• Add post_transform hooks per instrument for structural reshaping
• Remove output_key — use instrument name consistently, fix rarelink to not hardcode "genetic_findings"
• Document the registry format with a clear schema

---

10. Upstream Fixes Needed in RareLink

Summary of issues to open on BIH-CEI/rarelink:

Issue	Description
variation_descriptor_mapper top-level support	Mapper only reads from repeated_elements, can't handle non-repeating instruments
Hardcoded "genetic_findings" key	Should use instrument name from config
_inject_routed_disease sentinel placement	Sentinels placed at root but DiseaseMapper reads from instrument section
_FIELD_TO_PREFIXES missing VO	Vaccine Ontology prefix not in the default mapping
CodeSystemsContainer missing vo, eco fields	rd-cdm container doesn't have all code systems CIEINR needs
MetadataMapper version overlay	Latest version map may override correct CIEINR-specific versions

---

Priority Order

1. Generate Python from LinkML (items 2, 1) — eliminates 19 import errors, establishes codegen pipeline
2. Month+year date consolidation (items 5, 7) — fixes DOB and enables onset calculations across all instruments
3. Simplify combined.py (item 3) — reduces maintenance burden dramatically
4. Open rarelink upstream issues (items 4, 6, 10) — unblock proper genetics and routed disease support
5. Leverage LinkML annotations (item 8) — long-term, eliminates manual mapping config entirely

---

Definition of Done

• make codegen generates all Python from LinkML (code_systems + enums)
• Zero Failed to import enum class errors during export
• No post-processing script needed — redcap_to_linkml handles computed fields and structural wrapping
• combined.py is under 100 lines
• All upstream rarelink issues filed with reproducible examples
• DOB and all month+year dates produce correct ISO ages
• Ontology-routed MONDO diseases appear in the phenopacket diseases block