Skip to content

TRD.md

Latest commit

 

History

History
149 lines (114 loc) · 10.2 KB

TRD.md

File metadata and controls

149 lines (114 loc) · 10.2 KB
Raw

TRD — CT-200 QA Traceability & Test-Case Generation System

1. Stack

  • API: FastAPI + Pydantic v2
  • Relational store: SQLite via SQLAlchemy — document tree, versions, nodes, selections, edges
  • Document store: MongoDB Atlas (free tier) — LLM generations, raw prompts/responses, validation failures
  • LLM: Groq (free tier), model TBD at build time — swappable via a thin provider interface
  • Parser: markdown-it-py token stream (not rendered HTML) — gives block-level structure needed for node extraction
  • Tests: pytest
  • Package/env: uv
  • No Docker, no auth, no frontend — per PRD non-goals.

2. Data model (SQLite / SQLAlchemy)

Document
  id (pk)
  name                     e.g. "ct200_manual"

DocumentVersion
  id (pk)
  document_id (fk)
  version_number            int, 1, 2, ...
  ingested_at
  raw_source                full markdown text, stored for reproducibility

Node
  id (pk)                   stable logical identity — persists across versions
  document_id (fk)

NodeVersion
  id (pk)
  node_id (fk)               -> Node (logical identity)
  document_version_id (fk)   -> DocumentVersion
  heading_text
  level                       int, heading depth
  body_text
  content_hash                sha256(heading_text + body_text)
  parent_node_version_id (fk, nullable)  -> NodeVersion (self-referential, per-version tree)
  order_index                 sibling order within parent
  change_type                 enum: unchanged | text-edited | moved | renamed | split | merged | new | ambiguous
    (relative to the node's prior version, null on first version)

Selection
  id (pk)
  name
  created_at

SelectionItem
  selection_id (fk)
  node_version_id (fk)        pins to an EXACT node+version, not just node

Design note: Node is the durable logical identity (survives across versions); NodeVersion is the per-version snapshot with its own hash and change_type. This is what makes "same logical node across v1→v2" queryable without duplicating identity logic into every endpoint.

3. Node matching strategy (v1 → v2)

Layered matcher, applied per new node in v2 against all v1 nodes in the same parent-region, in this order:

  1. Exact match: content_hash identical → same Node, change_type = unchanged.
  2. Heading + path match: normalized heading text (lowercase, whitespace-collapsed) identical AND same parent-chain path → same Node, change_type = text-edited (body changed, heading/position didn't).
  3. Similarity match: token-overlap or simple ratio (difflib.SequenceMatcher) between old and new body text above a threshold (e.g. 0.6) AND heading similarity above a threshold → same Node, change_type = renamed (heading changed) or moved (parent-chain changed) depending on which shifted.
  4. Split/merge heuristic: one v1 node's content is now covered by >1 v2 nodes (or vice versa) based on partial similarity to multiple candidates → change_type = split / merged, logged with all candidate node IDs — this is inherently uncertain, flagged as such, never auto-resolved silently.
  5. No match above any threshold → change_type = new (v2 side) or the v1 node's Node has no v2 NodeVersion at all → treated as deleted for that version.
  6. Anything the matcher can't confidently place in 1-5 → change_type = ambiguous, surfaced to the API caller rather than guessed at.

Known failure mode (state explicitly in approach doc): a heading rewritten AND moved AND with materially different body text in the same edit will likely be misclassified as new + deleted rather than moved+renamed, because no single signal crosses its threshold. This is a known, accepted limitation — not silently hidden.

4. Parser requirements

  • Use markdown-it-py's token stream, not markdown → HTML → re-parse.
  • Every heading token becomes a NodeVersion candidate; body text = all tokens until the next heading of same-or-shallower level.
  • Must not silently drop content the parser doesn't recognize — anything untyped gets attached to the nearest preceding node as raw body text, never discarded.
  • Must handle (confirm against actual ct200_manual.md before assuming):
    • Duplicate headings at the same level (must produce two distinct Node records with correct parent, not merge them)
    • Irregular heading depth jumps (e.g. H2 → H4 with no H3)
    • Tables, embedded lists, and any embedded HTML inside a section body
  • Minimum 3 unit tests, each targeting one specific irregularity actually found in the manual by manual inspection — not synthetic edge cases invented in the abstract. Document what was found and how (read-through vs. test-driven discovery) in the approach doc.

5. API surface

Browse

  • GET /documents/{doc_id}/sections?version=latest|N — top-level nodes
  • GET /nodes/{node_id}?version=latest|N — node + children + full text + content_hash
  • GET /search?q=...&version=latest|N — heading/text search
  • GET /nodes/{node_id}/changes?from=1&to=2 — change_type + lightweight diff summary between two versions

Selection

  • POST /selections — body: {name, node_version_ids: [...]} → returns selection_id
  • GET /selections/{id} — resolved node text at exact pinned versions

Generation

  • POST /selections/{id}/generate — triggers LLM call, returns generation_id + test cases
    • Idempotency: if a generation already exists for this exact (selection_id, node_version_ids_hash), return the existing one unless ?force=true is passed. Documented and defended in decision log.
  • GET /generations/{id} — test cases + provenance (node versions, prompt version, model, schema version)

Staleness / retrieval

  • GET /generations/{id}/staleness — computed at retrieval time: for each source node in the generation, current node_version vs. the one it was generated from → fresh | text-stale | structure-stale | intent-stale, with a one-line reason
  • GET /generations?selection_id=... or ?node_id=... — retrieval by either axis, staleness inlined in the response, not a separate call

6. LLM generation design

Prompt (paraphrased structure, finalize exact wording during build):

  • System: "You are generating QA test-case drafts for a medical-device requirements section. Output ONLY valid JSON matching the given schema. Each test case must be concrete and executable: specific trigger, specific expected result."
  • User: reconstructed selected node text (headings + body, in document order) + the JSON schema.

Output schema (Pydantic):

class TestCase(BaseModel):
    title: str
    trigger: str          # the specific condition/action
    expected_result: str  # concrete, checkable outcome
    source_node_ids: list[str]

class GenerationResult(BaseModel):
    test_cases: list[TestCase]  # 3-5 items, enforced by validator

Failure handling:

  • Call LLM → attempt GenerationResult.model_validate_json(response).
  • On ValidationError: retry ONCE, with the validation error message appended to the prompt ("your previous response failed validation because: ...").
  • On second failure: do not fabricate a result. Store the raw failure (both attempts) in the NoSQL store, return HTTP 422 with a clear message, and mark the generation as failed — visible via the retrieval API, not swallowed.
  • Every stored generation (success or failure) records: prompt_template_version, schema_version, model_id, raw_response, timestamp.

Duplicate submission policy: idempotent by (selection_id, resolved node_version_ids) — same input always returns the same stored output unless force=true. Rationale: LLM output is inherently non-deterministic; without this, re-hitting an endpoint during testing/demo silently creates drift that's confusing to debug. Documented as a deliberate choice in the decision log, including the tradeoff (loses the ability to "just try again for a better answer" without the explicit flag).

7. Staleness detection logic

At retrieval time, for each node a generation depended on:

  1. Look up the node's current latest NodeVersion (may differ from doc version the selection was pinned to).
  2. If content_hash unchanged since generation → fresh.
  3. If changed and change_type on the current node is moved/split/merged → structure-stale.
  4. If changed and change_type is text-edited/renamed → run the intent-stale check: one LLM call with (old text, new text, the stored test case) asking whether the test case still holds. If the LLM says it still holds → text-stale (surfaced, low severity — cosmetic). If it says it no longer holds → intent-stale (surfaced, high severity).
  5. Explicit stated bias: thresholds and the intent-stale check are tuned to over-flag rather than under-flag — a false "stale" costs a reviewer a glance; a missed stale flag is the actual danger. State this plainly in the approach doc, per traceability research (Jama suspect-link literature; recall valued over precision in traceability recovery research).

Explicit stated limitation: a one-word wording change (e.g. a typo fix) and a changed pressure threshold both currently trigger the same text-edited classification at the matcher level — the intent-stale LLM check is what's supposed to tell them apart, and its accuracy is only as good as that one extra LLM call. This is a known soft spot, named directly in the decision log, not hidden.

8. Non-functional

  • No auth — assignment explicitly out of scope.
  • No auto-regeneration of stale cases — surfacing staleness is the deliverable, not fixing it automatically.
  • Git: real incremental commits per section of build (see execution prompts) — not one final dump.
  • README must cover: setup, env vars (GROQ_API_KEY, MONGODB_URI), how to run tests, and the exact steps to trigger v1→v2 re-ingestion + see a staleness flag appear.

9. Validation pass (post-build)

After the core build passes against the provided CT-200 manuals, re-run ingestion against one real, freely available open-source regulatory-style markdown/text document (e.g. a public device-description excerpt) to check the parser doesn't just work on the assignment's specific formatting. Record what broke, if anything, in the approach doc — this is explicitly a robustness check, not new scope.