From 986991d3f2fbc9ec65d9832e5120d313f0db71e7 Mon Sep 17 00:00:00 2001 From: graefea Date: Fri, 12 Dec 2025 18:57:38 +0100 Subject: [PATCH 1/8] update_changelog --- docs/6_changelog.rst | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/6_changelog.rst b/docs/6_changelog.rst index ee7710c43..794ea4f77 100755 --- a/docs/6_changelog.rst +++ b/docs/6_changelog.rst @@ -5,7 +5,12 @@ Changelog v2.0.5 (2025-11-19) --------------------- - +- Fixed a few of bugs +- enhanced versioning and refactored repo folders +- improved the functionality of MedicalActions for Phenopackets export +- included MaXo as a novel coding system into the LinkML schema, Python classes, + REDCap data dictionary and for Phenopackets export. +- See release notes here for more detail: https://github.com/BIH-CEI/rarelink/releases/tag/v2.0.5 v2.0.4 (2025-09-25) --------------------- From 88cb091e821c3e527ad3a75fd515cc377acdd78f Mon Sep 17 00:00:00 2001 From: graefea Date: Mon, 15 Dec 2025 09:38:05 +0100 Subject: [PATCH 2/8] changelog date update --- docs/6_changelog.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/6_changelog.rst b/docs/6_changelog.rst index 794ea4f77..945e2534e 100755 --- a/docs/6_changelog.rst +++ b/docs/6_changelog.rst @@ -3,7 +3,7 @@ Changelog =========== -v2.0.5 (2025-11-19) +v2.0.5 (2025-12-12) --------------------- - Fixed a few of bugs - enhanced versioning and refactored repo folders From d9ced1c96ff9f4cdd6f8e14b6aad3afeeb86560e Mon Sep 17 00:00:00 2001 From: graefea Date: Mon, 15 Dec 2025 09:41:36 +0100 Subject: [PATCH 3/8] fixed index --- docs/index.rst | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/index.rst b/docs/index.rst index 0c226cffe..4a8d307de 100755 --- a/docs/index.rst +++ b/docs/index.rst @@ -22,9 +22,9 @@ ________________________________________________________________________________ .. figure:: _static/res/rarelink_overview.png Overview of the RareLink framework integrated with a local REDCap instance. - The framework comprises the RareLink instruments based on the RD-CDM, the - RareLink API, the toFHIR & CDIS module for export and import to HL7 FHIR IPS - and Genomic Reporting Resources, and the RareLink-Phenopackets module for + The framework comprises the RareLink instruments based on the `RD-CDM `_, + the RareLink API, the toFHIR & CDIS module for export and import to HL7 FHIR + IPS and Genomic Reporting Resources, and the RareLink-Phenopackets module for direct export to GA4GH Phenopackets. Additionally, the LinkML-based import mapper facilitates data import from tabular databases into REDCap, while the Manual Data Capture Guide assists with the manual data capture of @@ -69,7 +69,7 @@ Sections 8) :doc:`8_glossary` 9) :doc:`9_acknowledgements` 10) :doc:`10_license` -11) :doc:`11_download_manifest` +11) :doc:`12_contact` ________________________________________________________________________________ @@ -124,7 +124,6 @@ Additional Information - :doc:`8_glossary` - :doc:`9_acknowledgements` - :doc:`10_license` -- :doc:`11_download_manifest` - :doc:`12_contact` ________________________________________________________________________________ From 16b1f8e39a9297c537af5ae0ba78461426ce720b Mon Sep 17 00:00:00 2001 From: graefea Date: Mon, 15 Dec 2025 09:41:58 +0100 Subject: [PATCH 4/8] fixed index docs --- docs/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/index.rst b/docs/index.rst index 4a8d307de..8856c9c35 100755 --- a/docs/index.rst +++ b/docs/index.rst @@ -22,7 +22,7 @@ ________________________________________________________________________________ .. figure:: _static/res/rarelink_overview.png Overview of the RareLink framework integrated with a local REDCap instance. - The framework comprises the RareLink instruments based on the `RD-CDM `_, + The framework comprises the RareLink instruments based on the `RD-CDM `_, the RareLink API, the toFHIR & CDIS module for export and import to HL7 FHIR IPS and Genomic Reporting Resources, and the RareLink-Phenopackets module for direct export to GA4GH Phenopackets. Additionally, the LinkML-based import From 789e76e1bbdc2359b83bc983f27d0e8a737e2f0f Mon Sep 17 00:00:00 2001 From: Adam SL Graefe <109136019+aslgraefe@users.noreply.github.com> Date: Sun, 29 Mar 2026 21:13:36 +0200 Subject: [PATCH 5/8] Chore/release v2.0.6 (#211) * rmv ds_store * updated eval cohort ppkts * fixed camelCase in write ppkts * issue 205, imrpove ppkts export pipeline, validation, fixed bug * clean up code * clean up * updated rarelink figure in docs * added variantValidator for manual hgvs curation * added phenopacket adapters to docs * updated index figure subheading * added ontology_routing adapter * test for adapter * updated phenopackets export adn validation with new functions and cli overview * added chaching * updated toml to 2.0.6 and clean up * updated codes according to rd-cdmv2.0.3 (specifically update of sex of birth to loinc_76689_9 in all files) and version updates * improved warnings output for pptks pipeline * removed rd-cdm in this version as it was unused * improved caching ppkts for label fetching util * grouped linkml validate data function * fixed updaet in kafka docker versioning to tofhir pipeline * improved packaging and readme * updated phenopackets tests * updated versions according to rd-cdm 2.0.3 * improved installation docs with updates * improved pytest * added changelog for v2.0.6 * updated eval cohort --- .DS_Store | Bin 10244 -> 0 bytes .gitignore | 7 +- README.md | 37 +- docs/.DS_Store | Bin 8196 -> 0 bytes .../2_1_rarelink_overview.rst | 12 +- .../2_rarelink_framework/2_2_rarelink_cdm.rst | 2 +- .../3_1_setup_rarelink_framework.rst | 55 +- docs/4_user_guide/4_1_manual_data_capture.rst | 3 +- docs/4_user_guide/4_2_import_mapper.rst | 10 +- docs/4_user_guide/4_3_phenopackets.rst | 314 +- docs/6_changelog.rst | 68 + .../2_2_rarelink_cdm.html | 4 +- .../html/4_user_guide/4_2_import_mapper.html | 10 +- .../rarelink_cdm_datadictionary - v2_0_5.csv | 2 +- .../2_2_rarelink_cdm.rst.txt | 2 +- .../4_user_guide/4_2_import_mapper.rst.txt | 10 +- .../rarelink_cdm_datadictionary - v2_0_5.csv | 2 +- .../import_mapper_tsvs/example_1_sssom.tsv | 10 +- .../rarelink_cdm_sssom_template.tsv | 2 +- docs/_build/html/searchindex.js | 2 +- .../rarelink_cdm_datadictionary - v2_0_6.csv | 4 +- .../import_mapper_tsvs/example_1_sssom.tsv | 10 +- .../rarelink_cdm_sssom_template.tsv | 2 +- docs/_static/res/rarelink_overview.svg | 26571 ---------------- docs/conf.py | 10 +- docs/index.rst | 6 +- pyproject.toml | 64 +- pytest.ini | 1 + res/.DS_Store | Bin 6148 -> 0 bytes res/evaluation_cohort/phenopackets/1.json | 235 +- res/evaluation_cohort/phenopackets/10.json | 109 +- res/evaluation_cohort/phenopackets/2.json | 104 +- res/evaluation_cohort/phenopackets/3.json | 85 +- res/evaluation_cohort/phenopackets/4.json | 87 +- res/evaluation_cohort/phenopackets/5.json | 91 +- res/evaluation_cohort/phenopackets/6.json | 87 +- res/evaluation_cohort/phenopackets/7.json | 87 +- res/evaluation_cohort/phenopackets/8.json | 87 +- res/evaluation_cohort/phenopackets/9.json | 101 +- .../phenopackets/failures.json | 42 - ...evaluation_cohort_rarelink-cdm-linkml.json | 20 +- .../redcap/evaluation_cohort_redcap.json | 276 +- src/.DS_Store | Bin 6148 -> 0 bytes src/rarelink/.DS_Store | Bin 6148 -> 0 bytes src/rarelink/cli/phenopackets/export.py | 516 +- src/rarelink/cli/redcap/download_records.py | 9 +- src/rarelink/cli/redcap/upload_records.py | 5 +- src/rarelink/phenopackets/adapter/__init__.py | 13 +- .../phenopackets/adapter/multi_onset.py | 16 - .../adapter/ontology_routing_adapter.py | 261 + src/rarelink/phenopackets/create.py | 420 +- .../phenopackets/mappings/base_mapper.py | 21 - .../mappings/interpretation_mapper.py | 20 +- .../mappings/measurement_mapper.py | 7 +- .../mappings/medical_action_mapper.py | 9 +- .../phenopackets/mappings/metadata_mapper.py | 19 +- .../mappings/phenotypic_feature_mapper.py | 3 +- .../mappings/variation_descriptor_mapper.py | 13 +- src/rarelink/phenopackets/pipeline.py | 225 +- src/rarelink/phenopackets/validate.py | 226 +- src/rarelink/phenopackets/write.py | 74 +- .../mappings/phenopackets/individual.py | 2 +- .../mappings/phenopackets/resources.py | 18 +- .../redcap/map_2_personal_information.py | 2 +- .../mappings/redcap/template.json | 2 +- .../rarelink_2_personal_information.py | 10 +- .../python_datamodel/rarelink_cdm.py | 10 +- .../python_datamodel/rarelink_code_systems.py | 51 +- .../rarelink_cdm_datadictionary - v2_0_6.csv | 4 +- .../rarelink_2_personal_information.yaml | 10 +- .../rarelink_6_4_family_history.yaml | 2 +- .../rarelink_code_systems.yaml | 16 +- src/rarelink/rd_cdm/__init__.py | 1 - src/rarelink/rd_cdm/cli.py | 13 - src/rarelink/rd_cdm/codegen.py | 303 - src/rarelink/rd_cdm/codegen_cli.py | 24 - src/rarelink/rd_cdm/schema_template.py | 228 - src/rarelink/rd_cdm/sync.py | 64 - src/rarelink/tofhir/.DS_Store | Bin 6148 -> 0 bytes src/rarelink/tofhir/docker-compose.yml | 4 +- src/rarelink/tofhir/mappings/ips.patient.json | 22 +- .../Rarelink_2_personal_information.json | 4 +- src/rarelink/utils/label_fetching.py | 3 +- submodules/tofhir | 1 - tests/.DS_Store | Bin 6148 -> 0 bytes .../test_redcap_to_phenopacket_workflow.py | 233 +- .../adapter/test_ontology_routing_adapter.py | 115 + .../sample_records_rarelink_cdm.json | 6 +- .../sample_records_rarelink_cdm_1.json | 2 +- tests/phenopackets/test_pipeline.py | 85 +- tests/rd_cdm/test_csv_update.py | 93 - tests/rd_cdm/test_ensure_types_and_codegen.py | 49 - .../rd_cdm/test_normalize_and_build_schema.py | 93 - tests/rd_cdm/test_scaffold_and_force.py | 47 - tests/utils/test_processor.py | 4 +- 95 files changed, 3026 insertions(+), 28983 deletions(-) delete mode 100755 .DS_Store delete mode 100755 docs/.DS_Store rename src/rarelink/rarelink_cdm/rarelink_cdm_datadictionary - v2_0_5.csv => docs/_static/rarelink_cdm_datadictionary - v2_0_6.csv (99%) delete mode 100644 docs/_static/res/rarelink_overview.svg delete mode 100755 res/.DS_Store delete mode 100644 res/evaluation_cohort/phenopackets/failures.json delete mode 100755 src/.DS_Store delete mode 100755 src/rarelink/.DS_Store create mode 100644 src/rarelink/phenopackets/adapter/ontology_routing_adapter.py rename docs/_static/rarelink_cdm_datadictionary - v2_0_5.csv => src/rarelink/rarelink_cdm/rarelink_cdm_datadictionary - v2_0_6.csv (99%) delete mode 100644 src/rarelink/rd_cdm/__init__.py delete mode 100644 src/rarelink/rd_cdm/cli.py delete mode 100644 src/rarelink/rd_cdm/codegen.py delete mode 100644 src/rarelink/rd_cdm/codegen_cli.py delete mode 100644 src/rarelink/rd_cdm/schema_template.py delete mode 100644 src/rarelink/rd_cdm/sync.py delete mode 100755 src/rarelink/tofhir/.DS_Store delete mode 160000 submodules/tofhir delete mode 100755 tests/.DS_Store create mode 100644 tests/phenopackets/adapter/test_ontology_routing_adapter.py delete mode 100644 tests/rd_cdm/test_csv_update.py delete mode 100644 tests/rd_cdm/test_ensure_types_and_codegen.py delete mode 100644 tests/rd_cdm/test_normalize_and_build_schema.py delete mode 100644 tests/rd_cdm/test_scaffold_and_force.py diff --git a/.DS_Store b/.DS_Store deleted file mode 100755 index a422ca80f231f03a39a13ad21f46458baef38753..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 10244 zcmeHMUu+ab7@u!@(A_PyTWm|Y4RE1COF87O*p{{!uI;r_5J9e`rGV$%?Y7-;y)CzU zSD+y_5?}bEs8R4i{QoG?pvI7(kQgPtX{y4DQH(zLCqaYyqQP%==i0rt5PTqFvXkt7 zJKvxEW`6UVnVUUA2()FCAR#_N2#*()(mcG)QmCC?GYZEvRVV@DNpdh4N$A5PGdx{| zi-3!Oi-3!Oi-3#3oq_=F*|hLQwsh|<0xkkB0+R^P^}&Z1l_4g_ZD}nXc(aZGC_SjQ z3O-Bp0i}=05R>D!L<<(MLs8nHz>XMThoj!d{6b8Q+tLmvzz!e4G6OrIfR!EJ@3Y(q zgly^FT?AYNW+Fh(?jCY486+Apo#*#zZM)#K4eQ}>JgLV~$3HDzJAUNk$F2Ex<)-$L zp{Z{JE5!2{N(ys%$q>nqxHCH;!k;reoNcH$uzAC|POA;m-ZQLEwdBTL&i1gi%0m>Q zlLRs1qs9p9>-+JMq-L0bz?Z-kc;?J4EEEch3U^0_^wEeJGc!h8EWMM~x^+Dn&ppTD z+ckAGDwVbCDKn;})U*|#DJr_`>eW(;KH8>dQi@^4Mr?y1@Iq9o92;v4)cbwGK+{3L zZ>(ue!0%gI*KqKlz?WC8ZrK*ylN?Uzhbe%#%(3N=6x;R5`w@Ja?2lq3l5uPwb)Hl@ z;r4t`v&?C80{wT#H)qzIV1vy7azZ?t-3hK|!iTS(%A1X(Q%OBFh&tMy0+J?b9^_|d zKwfwoAr}yG@7P#yZBw1!S63IHmEMV@ULF*QA9Y**6;%BWOgARoID zdTMK0rS{&0p3DrVjHpzah$R!5WXx1slgWs7KsEYIb)QL}`VBp8wp)%-Ny-@NHWmG} ztQtmdqff)vQMWP*fTMtLdS3aWrOQ_a>YLYxyEZ>OQG8d4R61W2`;pWsQyb9KbkC@! zm_s{aMnX*~T57O2s~K86sdlGg!)ht?2*N~h#e&L(URl18YhNa=?agTBeyNJ*tsIM& z$l@*q8@E|kXck3!m&f;D_Npbaypvkm(< zVmTuB@Y3fzj+2(kYPbyHu-mHZT!t%f z4St6|;5sL8B3Hzfapl|su992H`MG+onQO7z&gB+4IzJqR3u7fWlZ-4|vbnvp%@=Ry z17g=5F<&mvvuDWNW#x;jYF5=Xw1g&^WK%E4HYbNI;^xuaBCa*c(-=2lPbG0tZkkP> zQ$5IdbstY@D;3(_K5;3JjF;xv)n#Hek6f4L+SOXGpU2rHaaL7a?F|Yzhol0#TI;PB zcnR#vTCrKc*(7=F>iyz+flhFVSaolkH_YRVkobR3)VIl5a-Liy6G+q_$WP=dxeh$i zvjTju90^M4xe6Mf5t^VG)LjtKf0E3`G8VoQY3y;A#()AEL zfnvh?iiXaOv+K;-D?i#BCnKqI+ni~2 zZknXzDow)TW^9=TZkmbwEyhyiZ!wlCZ;N>$u*Z6Q~fE zYhGL5^3k@=Atv3(@owbhB)@*2Z*}Iq=j3EwN4yiK**)^|!*QD5EiM8s0xkkB0=J4l zf$iv{-~T&q{Qv*0zRvY?5pWUsA0hyra3tJ@4bPpkruTt **Note**: You need Python ≥3.10 but <3.13 to be able to use RareLink! - -Next, install rarelink through pypi... - +### Core install ```bash pip install rarelink ``` -... or clone the RareLink repository, navigate to its root directory, and - install RareLink using: +This installs everything needed for REDCap interaction, Phenopacket export, +and the RareLink CLI. It does **not** install Docker, numpy/pandas, or +documentation/testing tools. -```bash -git clone https://github.com/BIH-CEI/rarelink.git -cd rarelink -pip install . -``` +### Optional extras -If you want to install development dependencies (e.g., `pytest`), use: +| Extra | Install command | What it adds | +|-------|----------------|--------------| +| `fhir` | `pip install rarelink[fhir]` | Docker SDK for the toFHIR FHIR export pipeline | +| `data` | `pip install rarelink[data]` | numpy + pandas for tabular data processing | +| `test` | `pip install rarelink[test]` | pytest and pytest-mock | +| `docs` | `pip install rarelink[docs]` | Sphinx and documentation themes | +| `dev` | `pip install rarelink[dev]` | All of the above plus build/twine/jupyter | +For development: ```bash -pip install .[dev] +git clone https://github.com/BIH-CEI/rarelink.git +cd rarelink +pip install -e .[dev] +pytest ``` ### Setting Up the `.env` File diff --git a/docs/.DS_Store b/docs/.DS_Store deleted file mode 100755 index 10f8d71bea9fb3bd6eeb2d2bd0f561e8c5310286..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 8196 zcmeHM%}*0S6n_hawt~t56OJ264ACaE)GwlkT5CLbF;r2brgpovG~MntyDdEsVs9n_ z(UT-zBr$mQ=*f8EFAz^&O}vgf~?T7pTHD-S3Sw0eNt-G?Ct z889F#-QR66twJoFbIKK__UvXiY}rYr49hVP6UzQ^e!ZOExYiwS?g?L+6K(m7UBf6$ zq1`Up?Lr+25;3zR?! z>yBN`7VW%C6^dpaA&>4AZPTeIotkZWR5#5U!r@RjGcvZYa4C8rqQ|1sixGWcIv$Pa zm!~c+E`~#=#wV}LWJ*@qc3u#((TP0R!*TX|Ctt#nkCo}=ECa11D!wLZIi3Bz+=>vL zzrJpk_$Ki6Bf7scN}K>VN6$`QKfYyh45z1xTc8XTzLE3b{U?*umYXympy@atZE@nP z%&sQBEj_1Oq64;h%=W@cGsNCE+xF?3r$O92WjtsLI9Hy;Vc_Z}a**0Z+lSw+d|77U z(1&T-+zBJuIzW$29BO}3KkH_U-2ZQM{{Fu|gQ;GW2b2d6r3bV?DxFHAY1`q}ZzZ{F z>-apuha`e)FiBM~qt|hi^g51m=O2bx))7i{5?nTz#2QQo{vklc{{wtUYa3Uk*a@1B YtDr4TjG?_^y7+F^3oFIcqZ`_, + the RareLink API, the toFHIR & CDIS module for export and import to HL7 FHIR + IPS and Genomic Reporting Resources, and the RareLink-Phenopackets module for direct export to GA4GH Phenopackets. Additionally, the LinkML-based import mapper facilitates data import from tabular databases into REDCap, while the Manual Data Capture Guide assists with the manual data capture of - RD-CDM data in a local REDCap project. Click `here `_ - to see the figure in full size. + RD-CDM data in a local REDCap project. The framework is also configurable + for disease-specific extensions or custom data models. .. note:: You will need your own REDCap instance to deploy the RareLink framework. diff --git a/docs/2_rarelink_framework/2_2_rarelink_cdm.rst b/docs/2_rarelink_framework/2_2_rarelink_cdm.rst index a6d3d3f07..bd4807d16 100755 --- a/docs/2_rarelink_framework/2_2_rarelink_cdm.rst +++ b/docs/2_rarelink_framework/2_2_rarelink_cdm.rst @@ -315,7 +315,7 @@ ______________________________________________________________ - Cardinality: Required (1..1) - Validation: Date format (YYYY-MM-DD) -- 2.2 Sex at Birth (``snomedct_281053000``) +- 2.2 Sex at Birth (``loinc_76689_9``) - Cardinality: Optional - Validation: Dropdown (choices encoded according to :ref:`1_5`) diff --git a/docs/3_installation/3_1_setup_rarelink_framework.rst b/docs/3_installation/3_1_setup_rarelink_framework.rst index f8d8e1952..8e082d737 100755 --- a/docs/3_installation/3_1_setup_rarelink_framework.rst +++ b/docs/3_installation/3_1_setup_rarelink_framework.rst @@ -24,18 +24,34 @@ Follow these steps to set up the project locally and run tests. source .venv/bin/activate # On macOS/Linux .venv\Scripts\activate # On Windows -3. Install dependencies: +3. Install RareLink: .. code-block:: bash - pip install . + pip install rarelink + +This installs everything needed for REDCap interaction, Phenopacket export, +and the RareLink CLI. + +For optional features, install the relevant extras: + +.. code-block:: bash + + pip install rarelink[fhir] # adds Docker SDK for the toFHIR pipeline + pip install rarelink[data] # adds numpy and pandas for tabular data processing + +For development (includes testing, docs, and all optional extras): + +.. code-block:: bash + + pip install -e .[dev] 4. Configure all api keys necessary to use rarelink by running the following command: .. code-block:: bash - rarelink setup keys + rarelink setup keys This command will prompt you to enter the following keys: - BioPortal API key @@ -43,14 +59,14 @@ command: - REDCap URL - REDCap project ID -.. note:: +.. note:: You can create your free BioPortal account here: `BioPortal `_ 5. Run tests: - Use `pytest` to run the test suite. - + .. code-block:: bash + pip install rarelink[test] pytest _____________________________________________________________________________________ @@ -58,7 +74,7 @@ ________________________________________________________________________________ RareLink Framework CLI config ------------------------------------ -Use the following commands to update the framework and its components, view +Use the following commands to update the framework and its components, view its status or reset the framework. See :ref:`2_3` for more information. .. code-block:: bash @@ -91,32 +107,37 @@ ________________________________________________________________________________ Phenopacket Pipeline Configuration ___________________________________ -Via the RareLink CLI type: +The Phenopacket export pipeline requires a BioPortal API key and a +``CREATED_BY`` value, both configured via: .. code-block:: bash - to be implemented + rarelink setup keys -This command guides you through setting up the Phenopacket pipeline for RareLink. -You will be prompted to enter: -- Your location where to store the Phenopackets. +Once configured, export Phenopackets with: + +.. code-block:: bash + + rarelink phenopackets export --help _____________________________________________________________________________________ FHIR Pipeline Configuration ___________________________ +.. note:: + The FHIR pipeline requires the ``fhir`` extra: ``pip install rarelink[fhir]`` + .. code-block:: bash rarelink fhir setup -This command guides you through setting up the FHIR pipeline for RareLink. +This command guides you through setting up the FHIR pipeline for RareLink. You will be prompted to enter: - Your FHIR server URL. - If required, your FHIR server username & password. -.. note:: - All sensitive information will also be stored in the - hidden configuration file. - +.. note:: + All sensitive information will also be stored in the + hidden configuration file. \ No newline at end of file diff --git a/docs/4_user_guide/4_1_manual_data_capture.rst b/docs/4_user_guide/4_1_manual_data_capture.rst index 156f98968..828975288 100755 --- a/docs/4_user_guide/4_1_manual_data_capture.rst +++ b/docs/4_user_guide/4_1_manual_data_capture.rst @@ -354,7 +354,8 @@ Fields: 1. Validate mutations using `ClinVar `_ or `Varsome `_. - 2. Confirm the expression with the `HGVS Validator `_. + 2. Confirm the expression with the `Variant Validator `_ + or the `HGVS Validator `_. 3. If the validation fails, enter the details in the **6.1.6 Genetic Mutation String** field. 4. ``*``: **Are you sure the entered HGVS expression was validated using the variant validator?** - this field is mandatory to ensure the diff --git a/docs/4_user_guide/4_2_import_mapper.rst b/docs/4_user_guide/4_2_import_mapper.rst index 0dc70de95..6ce11f479 100755 --- a/docs/4_user_guide/4_2_import_mapper.rst +++ b/docs/4_user_guide/4_2_import_mapper.rst @@ -187,7 +187,7 @@ The template contains the following columns: RareLink-CDM schema, for example: - ``rarelink_cdm:snomedct_184099003`` for 2.1 Date of birth, or - - ``rarelink_cdm:snomedct_281053000`` for 2.2 Sex at birth. + - ``rarelink_cdm:loinc_76689_9`` for 2.2 Sex at birth. - **predicate_label**: The title of the target element from the schema, i.e. the *name* of the element @@ -198,7 +198,7 @@ The template contains the following columns: The target value in the RareLink-CDM. For literal fields, such as dates, this is the actual value (e.g., "1970-02-15"). For coded fields, such as sex, the object_id is the standardized code from the value set - (e.g., "SNOMEDCT:248152002" for female) or the code from the code system defined + (e.g., "LOINC:76689-9" for female) or the code from the code system defined by the data element (e.g., "HP:0001250" for a phenotype in element 6.2.1). - **object_label**: The human-readable label for the target value either from the @@ -255,7 +255,7 @@ Below you can find a **snippet of the RareLink-CDM SSSOM Template File:** # # === Personal Information (Singular) === "" rarelink_cdm:snomedct_184099003 "2.1 Date of birth" - lexical manual - "" rarelink_cdm:snomedct_281053000 "2.2 Sex at birth" lexical manual + "" rarelink_cdm:loinc_76689_9 "2.2 Sex at birth" lexical manual "" rarelink_cdm:snomedct_1296886006 "2.3 Karyotypic Sex" lexical manual "" rarelink_cdm:snomedct_263493000 "2.4 Gender Identity" lexical manual "" rarelink_cdm:snomedct_370159000 "2.5 Country of birth" lexical manual @@ -447,7 +447,7 @@ schema. subject_id subject_label predicate_id predicate_label object_id object_label match_type mapping_date mapping_tool ADR001 "Patient ADR001" rarelink_cdm:snomedct_422549004 "1.1 Pseudonym" ADR001 ADR001 lexical 2025-01-01 manual ADR001 "Patient ADR001" rarelink_cdm:snomedct_184099003 "2.1 Date of birth" 1970-02-15 - lexical 2025-01-01 manual - ADR001 "Patient ADR001" rarelink_cdm:snomedct_281053000 "2.2 Sex at birth" SNOMEDCT:248152002 Female lexical 2025-01-01 manual + ADR001 "Patient ADR001" rarelink_cdm:loinc_76689_9 "2.2 Sex at birth" LOINC:76689-9 Female lexical 2025-01-01 manual ADR001 "Patient ADR001" rarelink_cdm:snomedct_263493000 "2.4 Gender Identity" SNOMEDCT:446141000124107 Female gender identity lexical 2025-01-01 manual ADR001 "Patient ADR001" rarelink_cdm:snomedct_278844005 "3.1 Vital Status" SNOMEDCT:438949009 Alive lexical 2025-01-01 manual ADR001 "Patient ADR001" rarelink_cdm:snomedct_64572001_mondo "5.1 Disease [MONDO]" MONDO:0012345 Wilson Disease lexical 2025-01-01 manual @@ -475,7 +475,7 @@ schema. # Mappings for a second patient (ADR002) ADR002 "Patient ADR002" rarelink_cdm:snomedct_422549004 "1.1 Pseudonym" ADR002 ADR002 lexical 2025-01-01 manual ADR002 "Patient ADR002" rarelink_cdm:snomedct_184099003 "2.1 Date of birth" 1982-07-30 1982-07-30 lexical 2025-01-01 manual - ADR002 "Patient ADR002" rarelink_cdm:snomedct_281053000 "2.2 Sex at birth" SNOMEDCT:248153007 Male lexical 2025-01-01 manual + ADR002 "Patient ADR002" rarelink_cdm:loinc_76689_9 "2.2 Sex at birth" SNOMEDCT:248153007 Male lexical 2025-01-01 manual ADR002 "Patient ADR002" rarelink_cdm:snomedct_278844005 "3.1 Vital Status" SNOMEDCT:438949009 Alive lexical 2025-01-01 manual ADR002 "Patient ADR002" rarelink_cdm:snomedct_64572001_mondo "5.1 Disease [MONDO]" MONDO:0010526 Fabry Disease lexical 2025-01-01 manual ADR002 "Patient ADR002" rarelink_cdm:loinc_99498_8 "5.2 Verification status" HL7FHIR:differential Differential lexical 2025-01-01 manual diff --git a/docs/4_user_guide/4_3_phenopackets.rst b/docs/4_user_guide/4_3_phenopackets.rst index 6a1726b82..9283e9ed4 100755 --- a/docs/4_user_guide/4_3_phenopackets.rst +++ b/docs/4_user_guide/4_3_phenopackets.rst @@ -21,9 +21,12 @@ ________________________________________________________________________________ - :ref:`phenopacket-validation` - :ref:`rarelink-phenopacket-preconfigurations` - :ref:`phenopackets-other-redcap-data-models` - - :ref:`troubleshooting` + - :ref:`phenopacket-adapters` + - :ref:`multi-onset-adapter` + - :ref:`ontology-routing-adapter` + - :ref:`troubleshooting` _____________________________________________________________________________________ .. _get_started: @@ -686,55 +689,290 @@ placeholders with relevant codes and Phenopacket terms. _____________________________________________________________________________________ -.. _factory-approach: +.. _phenopacket-adapters: -Using the Factory Approach -~~~~~~~~~~~~~~~~~~~~~~~~~~ +Phenopacket Adapters +==================== -For more advanced use cases, you can use the PhenopacketMappingFactory to create -mappings for different data models: +Adapters are preprocessing functions that transform or split data **before** +any mapper runs. They are designed to solve structural challenges that arise +when a single REDCap instrument or data element does not map cleanly to a +single Phenopacket block. All adapters are: -.. code-block:: python +- **Opt-in** — activated by a configuration key; absent means no effect. +- **Mapper-agnostic** — they produce data shaped to the conventions that + existing mappers already understand, so no mapper code changes are needed. +- **Reusable** — they are designed for general use, not tied to any specific + data model. - from rarelink.phenopackets.factory import PhenopacketMappingFactory, get_phenopacket_mappings +The adapters live in ``src/rarelink/phenopackets/adapter/``. - # Get mappings for RareLink CDM (default) - rarelink_mappings = get_phenopacket_mappings() +----- - # Get mappings for a custom data model - custom_mappings = get_phenopacket_mappings(model_name="your_model") +.. _multi-onset-adapter: - # Creating phenopackets with specific model mappings - from rarelink.phenopackets import create_phenopacket +Multi-Onset Adapter +------------------- - # Create phenopacket using the specified mappings - phenopacket = create_phenopacket( - data=record_data, - created_by="Your Name", - mapping_configs=custom_mappings - ) +**Module:** ``rarelink.phenopackets.adapter.multi_onset`` -The factory approach provides additional helper functions for working with different -data models: +**When to use it** -1. **Converting to Multi-Instrument Format** - - .. code-block:: python - - updated_config = PhenopacketMappingFactory.convert_to_multi_instrument_format( - config=mappings, - block_name="phenotypicFeatures" - ) +Use the multi-onset adapter when a single clinical finding has been observed +on **multiple occasions**, each with its own date, and you want to represent +each observation as a separate ``PhenotypicFeature`` entry with its own +``onset`` value. -2. **Merging Configurations** - - .. code-block:: python - - merged_config = PhenopacketMappingFactory.merge_configurations( - base_config=base_mappings, - override_config=custom_overrides - ) +For example, a recurrent respiratory infection may have been recorded on +three separate dates. Without multi-onset, only the first date would be +captured. With multi-onset, three separate ``PhenotypicFeature`` messages +are produced — one per date — all sharing the same type, severity, and +modifiers. + +**Configuration** + +Enable multi-onset in the mapping block for the relevant phenotypicFeatures +instrument:: + + FEATURES_BLOCK = { + "redcap_repeat_instrument": "your_instrument_name", + "type_field": "your_type_field", + "severity_field": "your_severity_field", + + # Enable multi-onset + "multi_onset": True, + "onset_date_fields": [ + "onset_date_1", + "onset_date_2", + "onset_date_3", + # ... add as many as your instrument supports + ], + } + +**What it does** + +For each non-empty date in ``onset_date_fields``: + +1. Creates the base ``PhenotypicFeature`` using the normal mapping function. +2. Deep-copies it. +3. Replaces the ``onset`` field with an ``Age`` element computed from that + date and the subject's date of birth. +4. Appends the copy to the result list. + +If no valid dates are found, the base feature (with whatever onset was +already set) is returned as a single-element list. + +**Data shape expected** + +The adapter reads onset dates directly from the inner instrument dict of +each repeated element:: + + { + "redcap_repeat_instrument": "your_instrument_name", + "redcap_repeat_instance": 1, + "your_instrument_name": { + "your_type_field": "HP:0004469", + "severity_field": "HP:0012826", + "onset_date_1": "2022-02-01", + "onset_date_2": "2023-03-01", + "onset_date_3": "2023-12-01" + } + } + +Produces three separate ``PhenotypicFeature`` messages, each with +``onset.age.iso8601duration`` computed from the respective date and DOB. + +**Notes** + +- All copies share the same type, severity, and modifiers. +- If DOB is unavailable, no age calculation is possible and the base + feature is returned unchanged. +- Combine with :ref:`ontology-routing-adapter` when the same instrument + also mixes HP and MONDO codes. + +----- + +.. _ontology-routing-adapter: + +Ontology Routing Adapter +------------------------- + +**Module:** ``rarelink.phenopackets.adapter.ontology_routing_adapter`` + +**When to use it** + +Use the ontology routing adapter when a **single data element** (or a set +of mutually exclusive sub-fields within one instrument) may be populated +with codes from **different ontologies** depending on the record — and +each ontology should map to a different Phenopacket block. + +The canonical example is a "specific finding" field that stores either an +HPO term (a phenotypic abnormality) or a MONDO term (a disease entity). +Both are clinically valid answers to the same question, but they belong in +different Phenopacket blocks: +.. list-table:: + :widths: 20 20 60 + :header-rows: 1 + + * - Ontology prefix + - Phenopacket block + - Semantic basis + * - ``HP:`` + - ``phenotypicFeatures`` + - HPO terms describe phenotypic abnormalities + (``PhenotypicFeature.type``) + * - ``MONDO:`` + - ``diseases`` + - MONDO terms describe disease entities (``Disease.term``) + * - ``OMIM:`` + - ``diseases`` + - OMIM identifiers describe disease entities + * - ``ORDO:`` + - ``diseases`` + - Orphanet identifiers describe rare disease entities + +These defaults are grounded in the GA4GH Phenopacket v2 schema and are +always active. They can be extended or overridden via the ``rules`` key +(see below). + +**Configuration** + +Add an ``ontology_routing`` key to your ``mapping_configs``:: + + mapping_configs = { + ... + "ontology_routing": { + "enabled": True, + + # Instruments whose repeated elements should be inspected + "instruments": [ + "your_mixed_instrument", + "another_mixed_instrument", + ], + + # Per-instrument: which fields to scan for routable codes. + # These are the type_field_1…N names from your mapping block. + # If omitted, all string fields are scanned (slower, zero-config). + "scan_fields": { + "your_mixed_instrument": [ + "field_holding_hp_or_mondo_1", + "field_holding_hp_or_mondo_2", + ], + "another_mixed_instrument": [ + "field_a", + "field_b", + ], + }, + + # Per-instrument: onset date fields. + # The first non-empty value is used as Disease.onset when a + # MONDO-coded element is routed to the diseases block. + "onset_fields": { + "your_mixed_instrument": [ + "onset_date_1", + "onset_date_2", + ], + "another_mixed_instrument": [ + "condition_onset_date", + ], + }, + + # Optional: override or extend the built-in prefix→block rules. + # Useful if your data model uses a non-standard ontology. + # "rules": { + # "HP": "phenotypicFeatures", + # "MONDO": "diseases", + # "MYCUSTOM": "diseases", + # } + }, + ... + } + +When ``ontology_routing`` is absent from ``mapping_configs``, the adapter +is never called and pipeline behaviour is identical to before. + +**What it does** + +Before any mapper runs, the adapter: + +1. Iterates over ``data["repeated_elements"]``. +2. For each element from a configured instrument, scans the configured + (or all) string fields for a value whose ontology prefix is in the + routing rules. +3. Routes the element: + + - **HP-coded** → placed in ``data["__routed__phenotypicFeatures"]`` + *unchanged*. The ``PhenotypicFeatureMapper`` processes it normally, + including multi-onset if configured. + - **MONDO/OMIM/ORDO-coded** → *normalized* to the ``term_field_1`` / + ``onset_date_field`` convention and placed in + ``data["__routed__diseases"]``. The ``DiseaseMapper`` consumes it + via the same path it uses for all other disease data. + +4. Elements with no routable code (e.g. a SNOMED sub-field that holds only + SNOMED values) are left in the original ``repeated_elements`` stream and + processed normally by whatever mapper is configured for that instrument. + +The original ``repeated_elements`` list is **never mutated**. + +**Data shape expected** + +A typical mixed element:: + + { + "redcap_repeat_instrument": "infections_initial_form", + "redcap_repeat_instance": 2, + "infections_initial_form": { + "type_of_infection": "snomedct_127856007", ← SNOMED category, ignored + "snomedct_127856007": "mondo_0043653", ← MONDO → diseases + "infection_severity": "hp_0012826", + "infection_date": "2023-02-01" + } + } + +The adapter detects ``mondo_0043653`` in ``snomedct_127856007``, routes +the element to ``diseases``, and normalizes it to:: + + { + "term_field_1": "mondo_0043653", + "onset_date_field": "2023-02-01", + "onset_category_field": None, + "excluded_field": None, + "primary_site_field": None, + "__source_instrument": "infections_initial_form", + "__source_instance": 2 + } + +**Validation integration** + +The adapter also provides :func:`check_prefix_placement`, which is called +automatically by ``validate_phenopackets`` after each file is written. It +inspects the serialized Phenopacket JSON and emits **non-fatal soft warnings** +when: + +- An HP: term appears in ``diseases`` (likely a routing error) +- A non-HP: term appears in ``phenotypicFeatures`` (MONDO/OMIM in a + features block) + +These warnings appear in the validation output but do not cause the +phenopacket to fail validation. They are intended to help data curators +catch misconfigured routing rules. + +**Adding the adapter to your mapping config** + +If you are building a custom data model and have instruments with mixed +ontology codes, add the ``ontology_routing`` block alongside your existing +``phenotypicFeatures`` and ``diseases`` configurations. No changes to +the mapper classes or the rest of the pipeline are needed. + +.. tip:: + Combine this adapter with the :ref:`multi-onset-adapter` when MONDO-routed + elements are diseases observed multiple times. Configure ``multi_onset`` + in the base ``phenotypicFeatures`` block; the routing adapter handles the + HP/MONDO split first, and the multi-onset adapter then expands the HP + elements per date. + _____________________________________________________________________________________ .. _troubleshooting: diff --git a/docs/6_changelog.rst b/docs/6_changelog.rst index 945e2534e..0158cdcfa 100755 --- a/docs/6_changelog.rst +++ b/docs/6_changelog.rst @@ -3,6 +3,74 @@ Changelog =========== +v2.0.6 (2026-XX-XX) +--------------------- + +Phenopacket Export Pipeline: +""""""""""""""""""""""""""""" +- Fixed camelCase serialization in Phenopacket JSON output (``preserving_proto_field_name=False``), + producing valid GA4GH Phenopacket v2 JSON (`#205 `_) +- Integrated ``validate.py`` into the export pipeline with a two-tier approach: + Python-native structural checks (required fields, CURIE format, schema version) + plus optional ``phenopacket-tools`` CLI fallback when available on PATH +- Added ontology-prefix placement checks as soft validation warnings + (e.g. HP: terms outside ``phenotypicFeatures``, MONDO: terms outside ``diseases``) +- Improved CLI export with Rich progress bars: sequential two-phase display + (creation bar completes before validation bar appears) +- Added structured warnings system: mapper warnings (e.g. "No diagnosis ID found") + now use ``warnings.warn()`` consistently, are captured per-record with record ID + context, and displayed in the export summary +- Added ``warnings.json`` output file alongside ``failures.json`` for detailed + warning inspection +- Export summary now includes a Warnings column for both creation and validation + stages with deduplicated counts + +RD-CDM v2.0.3 Integration: +"""""""""""""""""""""""""""" +- Updated to `rd-cdm v2.0.3 `_ + with eleven updated ontology/code system versions and the replacement of the + deprecated SNOMED CT Sex at Birth concept with LOINC ``76689-9`` +- Updated the RareLink-CDM REDCap data dictionary and instruments to reflect + rd-cdm v2.0.3 changes +- Updated Phenopacket and FHIR export pipelines to use the revised code systems + and data element definitions + +Ontology Routing Adapter: +"""""""""""""""""""""""""" +- Designed ``ontology_routing_adapter.py`` for use cases where single repeated + elements contain mixed ontology prefixes (e.g. HP and MONDO codes in the same + instrument), enabling automatic routing of HP codes to ``phenotypicFeatures`` + and MONDO/OMIM/ORDO codes to ``diseases`` + (`#207 `_) + +Package & Dependencies: +"""""""""""""""""""""""" +- Reduced core dependencies from ~20 to ~10 by moving test, docs, Docker, numpy, + and pandas to optional extras (``[test]``, ``[docs]``, ``[fhir]``, ``[data]``, + ``[dev]``) +- Removed unused dependencies: ``node``, ``npm``, ``config``, ``schema-automator``, + ``tqdm`` +- Removed the ``rarelink_cdm/rd_cdm/`` codegen subfolder (unused at runtime); + future versions will import directly from the ``rd-cdm`` PyPI package + +Infrastructure: +"""""""""""""""" +- Fixed toFHIR Docker Compose: updated Kafka image references from deprecated + ``bitnami/kafka:latest`` to ``bitnamilegacy/kafka:3.7.0`` following Bitnami's + August 2025 image deprecation + (`#210 `_) +- Added ``.DS_Store`` to ``.gitignore`` + (`#202 `_) + +Documentation & Tests: +""""""""""""""""""""""" +- Updated installation docs to document optional extras + (``pip install rarelink[fhir]``, ``pip install rarelink[data]``, etc.) +- Updated tests to work with ``PipelineResult`` dataclass return type and + corrected mock patch paths +- Removed unused numpy/pandas references from Sphinx ``conf.py`` + + v2.0.5 (2025-12-12) --------------------- - Fixed a few of bugs diff --git a/docs/_build/html/2_rarelink_framework/2_2_rarelink_cdm.html b/docs/_build/html/2_rarelink_framework/2_2_rarelink_cdm.html index c6a011ac0..5ba77d0b8 100644 --- a/docs/_build/html/2_rarelink_framework/2_2_rarelink_cdm.html +++ b/docs/_build/html/2_rarelink_framework/2_2_rarelink_cdm.html @@ -268,7 +268,7 @@

Contents

REDCap Data Dictionary

The RareLink CDM Data Dictionary is available for download as a CSV file:

-

Download: RareLink-CDM Data Dictionary (v2.0.5)

+

Download: RareLink-CDM Data Dictionary (v2.0.5)

… Next steps