Skip to content

Parity execution plan (phased) — for review #44

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
MichaelLeeHobbs wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Parity execution plan (phased) — for review #44

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 108 additions & 0 deletions docs/design/15-parity-execution-plan.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
# 15 — Connector Parity Execution Plan (phased)

> **Status:** Draft for review · **Date:** 2026-07-15 · **Owner:** Michael Hobbs
> Turns the ranked gaps in [`12-connector-parity-gap-analysis.md`](./12-connector-parity-gap-analysis.md) into review-sized phases. Decisions locked in **D-182**.

## 0. Where we are

**Done** (merged): rank **4** (surface hidden runtime options) and rank **9** (SMTP config-driven multi-attachment).
**Decisions** (D-182): DB drivers = pg/mysql2/tedious bundled + oracledb optional; SFTP → File scheme + alias; non-Postgres DB rows = **JSON only** (no `<results>` XML — Mirth channels parsing it must rewrite transformers); JMS **deferred**.

**Remaining doc-12 gaps:** medium — 3, 5, 7, 8, 10; large — 1, 2, 6; deferred — 11; folded — 12 (into File schemes).

## 1. Principles

- **One phase = one reviewable PR** (large phases sub-split). No mega-branch.
- **Every connector gap is proven by a real-message E2E** through the harness (`packages/engine/src/__tests__/support/e2e-harness.ts`) — actually send a message through it — plus unit tests. Infra-needing suites go in the engine integration lane (`*.itest.ts`, docker `--profile test`).
- Each phase updates the **migration matrix** (doc 12 §7) + progress docs (CHANGELOG/DECISIONS/ROADMAP) and ticks its ROADMAP box.
- **Build reusable infra once:** the `${…}` substitution helper (P2) and the generalized cert-resolver (P1) are consumed by later phases — do them early.

## 2. Phases

### P1 — TLS everywhere via the Certificate module (rank 8 + the TCP-TLS deferred from rank 4) · M
Generalize the HTTP cert pattern (already shipped) to the other TLS-capable connectors.
- Make `connector-tls-resolver` connector-agnostic (resolve cert-IDs → PEM for **TCP source/dest** and **DICOM**), and extend the `engine.ts` deploy + `connection-test` resolution seam beyond the current `=== 'HTTP'` gate.
- **TCP TLS:** surface enable + `CertificateSelect` on the TCP source/dest forms (server cert for the listener; CA/client for the dispatcher); server resolves IDs at deploy.
- **DICOM TLS:** wire `dcmjs-dimse` `securityOptions` (key/cert/ca, `requireClientCert`) on the Server/Client in `dcmjs-dimse-adapter.ts`; cert-ID references resolved server-side.
- **Tests:** resolver units (TCP/DICOM); real TLS handshake E2E — TLS TCP source↔dest, DICOM SCU↔SCP over TLS; web form tests. Reuse the certs fixture.
- **Risk:** M (TLS correctness). **Unblocks:** TLS TCP / latin1-secured HL7, secured (TLS) PACS.

### P2 — HTTP dispatcher completeness (rank 5) · M
- Outbound **Basic + Digest** auth (preemptive option); **query-parameters** table + form-urlencoded body; multi-valued **header maps**; a shared **`${…}` substitution helper** (reads message maps) applied across URL/headers/params/body; capture **response headers + status** into `connectorMap`.
- **Files:** `http-dispatcher.ts`, new `substitute.ts` helper, `registry.ts`, `HttpDestinationForm.tsx`, defaults.
- **Tests:** dispatcher units (basic/digest challenge-response, params, subst, response capture); real E2E — HTTP dest → local server asserting the auth header/query it received; web tests.
- **Risk:** L–M. **Unblocks:** the many REST endpoints needing Basic/Digest or query params. **Note:** the `${…}` helper is reused by P6/P7.

### P3 — TCP framing flexibility (rank 3) · M
- Configurable frame **start/end bytes** (hex, multi-byte) replacing the hardcoded VT/FS+CR in `mllp-mode.ts`; a **Basic (non-MLLP) mode** with delimiter/regex batch splitting; `ignoreResponse` / `queueOnResponseTimeout` semantics.
- **Tests:** frame-parser units (custom bytes, split packets, multi-msg-per-chunk resync), Basic-mode batch split; real E2E — custom-framed TCP source↔dest; web tests.
- **Risk:** M (protocol-critical; keep the DoS `maxFrameBytes` guard). **Unblocks:** non-standard-LLP / delimited TCP partners.

### P4 — Binary data type across TCP/HTTP (rank 10) · M
- A **Buffer-based content path** so TCP + HTTP can carry binary (imaging/PDF/non-text); `BINARY` in the data-type layer; base64 at the sandbox boundary. Coordinates with non-connector **N2** (declared pass-through datatypes) but scoped to the binary connector path here.
- **Tests:** binary round-trip E2E (TCP + HTTP); data-type-layer units.
- **Risk:** M (touches the content/serialization spine — guard the string path).

### P5 — Document Writer connector (rank 7) · M
- New **PDF/RTF destination** connector: `documentType`, template with `${…}` subst, page size, encrypt + password (`pdf-lib`/`pdfkit` for PDF). Register in `registry.ts` + the connector-type enum.
- **Tests:** connector units (PDF/RTF bytes, encryption); real E2E — message → rendered document (assert PDF magic + encryption); web form.
- **Risk:** L (self-contained new connector). **Unblocks:** rendered clinical docs/letters (no workaround today).

### P6 — Multi-vendor Database connector (rank 1) · L · own PR(s)
Driver abstraction, per D-182.
- **P6a** — `DbDriver` interface + adapters: **pg** (existing) + **mysql2** + **tedious** (SQL Server), bundled; JSON rows only; keep `FOR UPDATE SKIP LOCKED` for pg. Web: driver select.
- **P6b** — Database **JavaScript mode** (source + dest); aggregate-into-one-message, result cache, fetch size, connector-level retry, encoding.
- **P6c** — **oracledb** as an *optional* lazy peer (absence never breaks a non-Oracle install; Instant Client documented).
- **Infra:** add **mysql** + **mssql** to docker-compose `--profile test`; integration lane suites per driver.
- **Tests:** per-driver integration (real DBs) + JS-mode units + real E2E per driver.
- **Risk:** M–H (native deps, per-vendor SQL). **Unblocks:** the most channels (Oracle/SQL Server/MySQL/JS-mode DB).

### P7 — File scheme abstraction (rank 2 + fold SFTP + rank 12 WebDAV) · L · own PR(s)
Scheme abstraction (Mirth's `FileSystemConnectionFactory`), per D-182.
- **P7a** — scheme abstraction + **local** + **FTP/FTPS**; **fold SFTP** in as a scheme with the standalone connector kept as a thin **alias + migration shim** (deprecate the standalone config); File depth options (regex filter, recursion, batch, `errorReadingAction`/`errorMoveToDirectory`, size min/max, rename-on-move).
- **P7b** — **SMB** + **S3**.
- **P7c** — **WebDAV** (rank 12).
- **Infra:** add **ftp** + **smb (samba)** + **minio (S3)** to docker-compose `--profile test`.
- **Tests:** per-scheme upload↔poll cascade E2E (integration lane) + an SFTP-alias back-compat test.
- **Risk:** M–H. **Unblocks:** FTP/FTPS/SMB/S3 File channels; fixes the connector-proliferation anti-pattern.

### P8 — Web Service / SOAP connector (rank 6) · L · own PR
- New **SOAP dispatcher** (WSDL-driven; `strong-soap`/`soap`) + **SOAP listener** source. Register.
- **Tests:** dispatcher units (WSDL binding, envelope); real E2E — SOAP source↔dispatcher loopback (or a local mock SOAP server); web forms.
- **Risk:** M. **Unblocks:** brownfield SOAP (LIS/RIS/EMR, IHE, state registries).

### Deferred
- **Rank 11 (JMS)** — deferred until a customer signal (D-182).

## 3. Recommended sequence & the value-vs-risk call

Two defensible orders:

- **A. Momentum-first (recommended): P1 → P2 → P3 → P4 → P5, then P6 → P7 → P8.**
Closes five partial gaps quickly at low/medium risk, and builds the reusable `${…}` helper (P2) and generalized cert-resolver (P1) *before* the big phases consume them. The three large phases then land as isolated, heavily-reviewed PRs.
- **B. Blockers-first: P6 → P7 → P1…P5 → P8.**
Doc 12 ranks by *channels unblocked*, so DB (P6) and File (P7) are the highest-value — but also the riskiest, and doing them first means the big native-dep changes land while the shared helpers don't exist yet.

**Recommendation: A.** Same total scope, but every large phase benefits from infra built by the mediums, and early PRs stay small and fast to review.

## 4. Cross-cutting

- **Test infra:** P6/P7 need new docker `--profile test` services (mysql, mssql, ftp, smb, minio); the integration-lane gate pattern (`integration/gates.ts`) extends cleanly.
- **New deps:** mysql2, tedious, (oracledb optional), pdf-lib/pdfkit, basic-ftp, an SMB client, `@aws-sdk/client-s3`, webdav, strong-soap/soap. Each added in its phase.
- **`connectorType` enum + registry** grow with each new connector (Document Writer, SOAP); web connector-type lists + defaults + settings forms follow.
- **Out of scope here:** the non-connector gaps (doc 13, N1–N15) and beyond-Mirth (doc 14) — separate plans.

## 5. Rough effort

| Phase | Gaps | Effort | New infra/deps |
|---|---|---|---|
| P1 TLS-via-certs | 8 (+TCP TLS) | M | — |
| P2 HTTP dispatcher | 5 | M | — |
| P3 TCP framing | 3 | M | — |
| P4 Binary type | 10 | M | — |
| P5 Document Writer | 7 | M | pdf-lib |
| P6 Multi-vendor DB | 1 | L (×3) | mysql2, tedious, oracledb*, docker mysql/mssql |
| P7 File schemes | 2, 12 | L (×3) | basic-ftp, smb, s3, webdav, docker ftp/smb/minio |
| P8 SOAP | 6 | L | strong-soap |

*optional peer.
Loading