Skip to content

diagnostic-engine.md

Latest commit

 

History

History
155 lines (122 loc) · 6.72 KB

diagnostic-engine.md

File metadata and controls

155 lines (122 loc) · 6.72 KB
Raw

Diagnostic Engine

Last updated: 2026-06-25

Phase 2 converts saved responses and diagnostic records into transparent health signals, evidence-quality findings, decision/dependency/risk context, and rule-based recommendations. The engine does not calculate an averaged program health score.

Scoring Rule Interface

Rules are registered by the scoring_rule value on each question. A rule evaluates one answer and returns one or more RuleResult entries:

RuleResult(
    rule_id="governance.repeated_decision_trigger",
    dimension="governance",
    health_impact="red",
    evidence_impact="partial",
    severity=3,
    reason="The same decision is repeating across executive forums without closure.",
    source_question_ids=["governance.decision_repeats"],
    recommendation_tags=["decision_inventory", "governance_reset"],
)

Unknown rule references fail scoring validation before any dimension results are persisted.

State Definitions

Health states:

  • green: evidence indicates the dimension is controlled.
  • yellow: material risk or incomplete control is present.
  • red: active control failure or high-likelihood value, schedule, cost, or confidence impact is present.
  • insufficient_evidence: required responses or evidence are too weak to support a reliable health signal.

Evidence states:

  • decision_grade: evidence is current, specific, reconciled, and connected to owners, dates, or decisions.
  • partial: evidence exists but is incomplete, stale, inconsistent, or weakly connected to action.
  • not_decision_grade: evidence is absent, contradictory, anecdotal, or disconnected from action.

Aggregation

Dimension aggregation is intentionally rule-based. Material red and yellow rule results contribute aggregate risk points to the dimension:

  • By default, a yellow trigger is 2 risk points and a red trigger is 3 risk points.
  • A single yellow finding normally contributes 2 points; a single red finding normally contributes 3 points.
  • Count-based rules, such as high-severity readiness gap count, can contribute the material count as risk points once the count is above the rule threshold.
  • Admin threshold profiles can raise yellow/red triggers for very large programs where several material findings should be tolerated before the dimension escalates.
  • If required questions are unanswered and no stronger signal exists, the dimension is insufficient evidence.
  • Evidence quality is evaluated independently from health and uses required response completion, rule evidence impacts, and linked evidence metadata.
  • The overall summary lists highest-risk dimensions, weak-evidence dimensions, decisions required, and the strongest action posture. It does not include an overall average score.

Initial Deterministic Rules

The registry covers all current question-set rules. Explicit Phase 2 trigger rules include:

  • Repeated executive decisions without closure.
  • Vendor green status while integrated program milestones are yellow or red.
  • Critical dependencies without accountable owners.
  • Top risks missing triggers or funded mitigations.
  • Technical readiness elements absent from the executive plan.

Evidence Metadata And Uploads

Users can add evidence references with category, title, notes, optional reference text, dimension, and optional question link. Post-MVP upload support adds POST /api/assessments/{assessment_id}/evidence/uploads for approved file types within the configured size limit.

Supported evidence categories include the design examples plus the categories used by the current question set, such as steering materials, decision log, RAID log, dependency tracker, budget forecast, finance forecast, vendor report, release readiness, business case, scope statement, resource plan, risk register, technical plan, and stakeholder feedback.

Upload configuration:

  • PGMHLTH_EVIDENCE_UPLOAD_STORAGE_PATH: source file storage root.
  • PGMHLTH_EVIDENCE_UPLOAD_ALLOWED_TYPES: comma-separated MIME types or file extensions.
  • PGMHLTH_EVIDENCE_UPLOAD_MAX_BYTES: maximum accepted upload size.
  • PGMHLTH_EVIDENCE_TEXT_EXTRACTION_ENABLED: enables text extraction for supported text-like files.

Default supported file types include TXT, Markdown, CSV, JSON, PDF, DOCX, and PPTX. Text extraction currently applies to TXT, Markdown, CSV, and JSON. Source files are stored under the configured upload root; extracted text is stored separately on the evidence record, marked as content_kind=extracted_text, and retrieved through /evidence/{evidence_id}/extracted-text only when extraction succeeds.

Retention and privacy considerations:

  • Uploaded evidence can contain sensitive program data. Store upload roots on encrypted persistent storage for cloud deployments.
  • Deleted evidence records remove the stored source file when the configured path is still under the upload root.
  • Extracted text is searchable application data, not a source file backup. Disable extraction when uploaded files should remain binary-only.
  • Audit events record metadata such as file type, size, category, dimension, and extraction status. They do not record file contents or extracted text.

Dependency Debt

Dependency records include owner, required-by date, critical-path impact, control level, reversibility, escalation trigger, and remediation option. The system derives flags for missing owner, missing required-by date, past due, high impact, low control, and low reversibility. These flags feed health and recommendation context.

Recommendations

Recommendations are deterministic and traceable to source dimensions, rules, dependencies, decisions, or risks. Regeneration reuses existing active recommendations for the same source finding and preserves user-edited title, action, and rationale text unless the user explicitly restores generated text.

LLM-assisted recommendations use source_type=llm_analysis and are merged from validated llm_analyses output only. They are deduplicated by normalized action text, clearly labeled as AI-generated guidance in the UI, and cannot overwrite rule-owned findings or accepted/rejected visible recommendation text. Rule-based recommendations remain authoritative for deterministic health findings.

Recommendation statuses currently include generated, accepted, and rejected. Generated rule recommendations also create traceable remediation candidates.

Audit Events

Diagnostic operations write safe audit metadata for scoring, evidence, dependency, decision, risk, remediation, recommendation generation, LLM analysis generation/reuse, and recommendation review. Metadata includes record IDs, dimensions, categories, status, prompt IDs, prompt versions, model names, field names, and counts. It does not include response text, full prompts, model outputs, or edited recommendation body text.