Public reproducibility materials for “WHO-anchored clinical-prior guidance for reliable LLM-assisted living guideline drafting: DGQC and an expert-blinded OpenSAGE reanalysis.”
This repository contains aggregate derived data, sensitivity-analysis tables, a revised public supplementary-table set, the scoring-time locked DGQC instrument, figure source data, and scripts for checking the headline results and regenerating publication figures. It does not contain rating-level expert records, expert identity mappings, contact details, private comments, API credentials, evaluated-output PDFs, or internal author-completion files.
analysis/: headline-result and figure-generation scripts.data/figure_source/: aggregate locked tables and figure source data.data/sensitivity/: source and publication-ready sensitivity tables.data/supplementary/: DGQC template, data dictionary, and retrospective disease-anchor audit (Tables S08-S10).figures/submission/: publication figures in SVG format.docs/: author contributions, data dictionary, methods notes, observed reproduction environment, public-release boundaries, and the scoring-time locked DGQC instrument.
Python 3.10 or newer is sufficient for the headline numerical checks:
python analysis/reproduce_core_results.pyThe script writes results/reproduced_summary.json and verifies the main OpenSAGE mean, usable-draft count, critical-defect count, and matched mean difference against the locked values.
The primary and raw-score paired comparisons report two-sided Wilcoxon signed-rank tests using R's continuity-corrected asymptotic normal approximation (exact=FALSE, default correct=TRUE); both yield P=0.00592 in the publication-ready summary.
Figure scripts require R and the packages declared at the top of each .R file. Run scripts from the repository root. Generated files are written below figures/generated*.
Model names are reproduced exactly as recorded in the locked study materials. For each disease in the matched comparison, OpenSAGE and all seven baseline systems received the same disease-specific prompt content, with no model-specific prompt rewriting. The systems differed in whether the OpenSAGE retrieval, reflection, and provenance-preservation workflow was applied.
Table S10 is a retrospective post hoc disease-anchor audit. Five incomplete filename-spelling mappings were repaired and one already-complete STI abbreviation mapping was normalized against the locked dataset. This audit is not evidence of the anchor or source document supplied at generation time.
The DGQC instrument and scoring manual in docs/ are identical to the version locked before formal expert scoring. Domain weights, critical-defect criteria, and the prespecified usability and high-quality thresholds were applied unchanged across all systems; the near-threshold category remained exploratory.
Rating-level expert records and reviewer identifiers are not included because explicit public-disclosure authorization was not available. No reviewer name-to-identifier mapping, pre-unblinding confidential mapping key, expert contact information, free-text review comments, workstation path, or internal QC worksheet is published. The locked post-analysis model-to-blinded-output mapping in public Table T002 is intentionally included to make the aggregate matched comparison auditable; it does not identify reviewers.
These materials support research reproducibility. OpenSAGE outputs are draft research artifacts and must not be used as clinical guidelines or patient-specific medical advice.
Original code in this repository is released under the MIT License. Curated data tables and documentation are released under CC BY 4.0; see DATA_LICENSE.md. Third-party names and cited works remain subject to their respective rights.
Citation metadata are provided in CITATION.cff. Repository: https://github.com/wang177777/OpenSAGE-DGQC