Real-message E2E testing (11/11 connectors) + DICOM dcmjs-dimse port - #41
Merged
Conversation
Adds a reusable harness that actually sends messages through channels
instead of just executing scripts in isolation:
- e2e-harness.ts: deployChannel(spec) assembles a live channel (sandbox +
8-stage pipeline + real source/dest connectors) over an in-memory store.
Source/destination filter+transformer+responseTransformer scripts are
written as TS/JS and compiled through the real esbuild compiler with
code-template injection — the same path production uses. Includes an
in-memory CaptureDestination sink and a full MessageStore implementation.
- tcp-helpers.ts: real MLLP capture server + client for TCP connector tests.
- real-messaging.e2e.test.ts:
1. TCP/MLLP in -> TS source transformer calling a FUNCTION code template
-> real TCP/MLLP out; asserts the transformed wire output + RAW/SENT rows.
2. A -> B -> C cascade over the in-memory Channel connector converting the
payload at each hop (HL7 -> JSON -> XML); asserts the final delivered XML
and per-channel message records.
Fixes discovered while wiring the harness:
- prependTemplates takes a script-key context (sourceTransformer, etc.), not
the mapped template-context value; passing the wrong string silently skipped
injection.
- onMessage must map the pipeline SENT status to the source-dispatch PROCESSED
status (mirrors the production engine's toDispatchStatus).
Engine suite: 374 passing (+2), build + lint clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F1ad6jb5mCYxzFVXHbksqi
Extends the connector matrix with two more real-I/O connectors, each
actually used end to end:
- File: drop a *.hl7 into a temp source dir → File source polls it up →
transform → File destination writes ${messageId}.out to a temp dir.
Asserts the written file content and that the source consumed the input.
- HTTP: POST to a real HttpReceiver → transform → HttpDispatcher POSTs to a
downstream http.Server. Asserts the body that landed downstream.
Proves the harness generalises past TCP/Channel to polling and request/
response sources. Connectors now covered by real-message E2E: TCP/MLLP,
Channel, File, HTTP.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F1ad6jb5mCYxzFVXHbksqi
ChannelService.create only wrote channel/scripts/destinations/metadata rows — input.filters and input.transformers were silently dropped, so a create carrying them (clone, import, or a direct API create) lost the channel's source filter and transformer. update already persisted them; create did not. - Extract the filter/transformer delete-and-reinsert logic from update into two shared helpers (syncFilters, syncTransformers) and call them from both paths. - create now captures inserted destination IDs (metaDataId -> id) so destination- scoped filters/transformers map to their connector row, same as update. - Regression test (real Postgres): create a channel with a source filter + source transformer, read it back, assert both persist with their rule/step. Not the cause of the reported save-500 (create + update + script-validation all handle that payload correctly on the dev DB); this is a separate data-loss bug found while investigating it. Server suite: 998 unit passing; integration roundtrip 3 passing; build + lint clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1ad6jb5mCYxzFVXHbksqi
- sandbox-globals.d.ts: canonical ambient type surface a channel script sees (msg/tmp/maps, logger, parseHL7/createACK, getCollection/getResource/httpFetch/ routeMessage/dbQuery, HL7 proxy). Placed at package root so it never leaks into the engine's own build. Authors type-check scripts against it. - tsconfig.sandbox.json + examples/sandbox-script-example.ts + `typecheck:scripts` script: proves the shipped types resolve for real authored scripts (tsc clean). - ts-channel-scripts.e2e.test.ts: runs real TS transformers (interfaces, generics, typed helpers, as-const unions) through a live channel via the harness — the production esbuild path transpiles them and they transform correctly. - sandbox-types-consistency.test.ts: drift guard asserting the engine .d.ts and the web Monaco string (sandbox-types.ts) declare the same global surface, since web can't import engine and the two are maintained by hand. Engine suite: 379 passing; build + lint + script-typecheck clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1ad6jb5mCYxzFVXHbksqi
Wires a real sandbox-backed ScriptRunner (as the production engine does) and drives messages through the JS connectors: - JS source: a polling script generates a message that flows to the sink. - JS destination: a script uppercases the message; its return value is the dispatch response, and the runner is observed receiving the exact content. Connectors covered by real-message E2E: TCP/MLLP, Channel, File, HTTP, JavaScript. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1ad6jb5mCYxzFVXHbksqi
Adds an engine integration lane (vitest.integration.config.ts, *.itest.ts) for connector suites that need real infrastructure. Each suite self-skips when its service env is absent, so the lane is safe to run anywhere. - gates.ts: env-driven gates for Postgres (*_test DB), SFTP, and SMTP/IMAP. - connector-db.itest.ts: TCP source -> transform -> Database dispatcher INSERT into a real table; reads the row back to prove delivery. Runs against a *_test Postgres, skips otherwise. - `pnpm --filter @mirthless/engine test:integration`. Verified green against mirthless_test; skips cleanly without a test DB. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1ad6jb5mCYxzFVXHbksqi
- docker-compose: add a `test` profile with sftp-test (atmoz/sftp) and mail-test (GreenMail SMTP+IMAP), matching the integration gates' env. Start with `docker compose --profile test up`. - connector-sftp.itest.ts: the vision's SFTP listen ↔ SFTP dest cascade — Channel A (TCP source → SFTP destination) uploads a file; Channel B (SFTP source polling the same dir) picks it up, transforms, and delivers to a sink. Verified against the real atmoz/sftp server; skips without SFTP_TEST_HOST. Connectors covered by real-message E2E: TCP/MLLP, Channel, File, HTTP, JavaScript, Database, SFTP. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1ad6jb5mCYxzFVXHbksqi
- connector-mail.itest.ts: Channel A (TCP source → SMTP destination) emails the
message; Channel B (Email/IMAP source polling the inbox) picks it up, transforms,
and delivers to a sink. Verified against GreenMail; skips without SMTP_TEST_HOST.
- gates.ts: split the mail login (GreenMail user "mirth") from the envelope address
("mirth@example.com") — IMAP authenticates with the login, SMTP addresses the email.
- docker-compose: drop greenmail.auth.disabled (it broke IMAP LOGIN); the predefined
user handles auth.
Connectors covered by real-message E2E: TCP/MLLP, Channel, File, HTTP, JavaScript,
Database, SFTP, SMTP, IMAP.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F1ad6jb5mCYxzFVXHbksqi
connector-matrix: TCP source -> transform HL7 into a FHIR Patient resource -> FhirDispatcher POSTs it to a local FHIR endpoint. Asserts the POST path (/Patient), the application/fhir+json content type, and the resource body. No external infra (local http mock), so it runs in the default unit lane. Connectors covered by real-message E2E: TCP/MLLP, Channel, File, HTTP, JavaScript, Database, SFTP, SMTP, IMAP, FHIR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1ad6jb5mCYxzFVXHbksqi
Wires the DICOM listen↔dest cascade with the real @ubercode/dcmtk binaries: Channel A (TCP source → DICOM destination / storescu C-STORE) → Channel B (DICOM source / storescp SCP) → sink. Vendors a small real DICOM object as a fixture (sample.dcm). FINDING (why it's opt-in, gated on DICOM_TEST_ENABLED and skipped by default): the SCP is reachable but rejects the association at DICOM negotiation — "Rejected Permanent, Source: Service User". The receiver wrapper (dicom-receiver.ts defaultReceiverFactory) creates the dcmtk DicomReceiver with no accepted SOP-class / presentation-context configuration, so it won't negotiate a context for the object's SOP class. This is a real DICOM-connector gap; once the SCP accepts the context, the test passes unchanged. Kept as a ready reproducer rather than grinding on dcmtk negotiation here. Real-message E2E coverage: 10/11 connector types PASSING (TCP/MLLP, Channel, File, HTTP, JavaScript, Database, SFTP, SMTP, IMAP, FHIR); DICOM wired with a documented reproducer pending the SCP negotiation fix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1ad6jb5mCYxzFVXHbksqi
…ion) → 11/11
The DICOM SCP rejected every C-STORE association ("Rejected Permanent, Source:
Service User") because the @ubercode/dcmtk wrapper created the receiver with no
accepted presentation contexts — a DICOM SCP must inspect the requested contexts
and accept the SOP-class + transfer-syntax pairs it supports before accepting the
association.
Swap to dcmjs-dimse (pure JS — the same library the production MedFusion DIMSE
service uses):
- dcmjs-dimse-adapter.ts: DimseReceiver (Scp that negotiates presentation contexts
— accept all StorageClass SOP classes + Verification + common transfer syntaxes,
per MedFusion's scp.ts/presentation-contexts.ts — writes each received instance
to a .dcm via Dataset.toFile) and DimseSender (Client + CStoreRequest(path)).
- Drop-in behind the existing DcmtkReceiver/DcmtkSender factory seams: no connector
API/config change; the 39 mock-injecting DICOM unit tests are untouched.
- Drop @ubercode/dcmtk (native binaries, process spawn, Windows path issues);
add dcmjs-dimse. Library loglevel set to warn to suppress protocol chatter.
The DICOM SCU→SCP cascade test moves from an opt-in native reproducer to the
DEFAULT lane (in-process, no binaries): connector-dicom.e2e.test.ts. That
completes real-message E2E for all 11 connector types.
connectors 491 + engine 383 passing; build + lint clean. Docs: CHANGELOG + D-181.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F1ad6jb5mCYxzFVXHbksqi
There was a problem hiding this comment.
Pull request overview
This PR replaces mock-heavy connector tests with a real-message E2E harness that pushes messages through live connectors, adds env-gated integration suites for infra-backed connectors, ports the DICOM connector implementation from @ubercode/dcmtk to pure-JS dcmjs-dimse, and fixes a server-side persistence bug where ChannelService.create dropped filters/transformers.
Changes:
- Introduces an engine E2E harness + connector matrix tests that send real messages end-to-end (default lane) and adds an infra-backed
*.itest.tsintegration lane (Postgres/SFTP/SMTP/IMAP). - Ports DICOM SCU/SCP implementation to
dcmjs-dimseand updates dependencies accordingly. - Fixes
ChannelService.createto persist filters/transformers (sharedsyncFilters/syncTransformers) and adds a real-Postgres regression test.
Reviewed changes
Copilot reviewed 28 out of 31 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| pnpm-lock.yaml | Updates lockfile for dcmjs-dimse and related transitive deps, removing @ubercode/dcmtk. |
| packages/web/src/lib/sandbox-types.ts | Clarifies canonical source and drift-guard expectations for Monaco sandbox TS defs. |
| packages/server/test/integration/channel-message-roundtrip.itest.ts | Adds regression coverage ensuring filters/transformers persist on channel CREATE. |
| packages/server/src/services/channel.service.ts | Extracts shared filter/transformer persistence logic and applies it to create + update. |
| packages/server/src/services/tests/channel.service.test.ts | Updates transaction mocks to support new create/update persistence flows. |
| packages/server/src/services/tests/channel-clone.service.test.ts | Updates clone/create mocks for new tx delete/insert behavior. |
| packages/engine/vitest.integration.config.ts | Adds separate Vitest config to run *.itest.ts suites only. |
| packages/engine/tsconfig.sandbox.json | Adds script-authoring typecheck config against shipped sandbox ambient types. |
| packages/engine/src/tests/ts-channel-scripts.e2e.test.ts | Adds E2E coverage proving TS-authored scripts compile + run through real channels. |
| packages/engine/src/tests/support/tcp-helpers.ts | Adds real TCP/MLLP helpers for E2E testing. |
| packages/engine/src/tests/support/e2e-harness.ts | Adds deployChannel(spec) harness and in-memory MessageStore for real-message E2Es. |
| packages/engine/src/tests/sandbox-types-consistency.test.ts | Adds drift-guard test to keep engine .d.ts and web Monaco string globals aligned. |
| packages/engine/src/tests/real-messaging.e2e.test.ts | Adds real-message TCP + channel cascade E2Es exercising pipeline + templates + persistence. |
| packages/engine/src/tests/integration/gates.ts | Adds env-gated helpers for infra-backed integration suites. |
| packages/engine/src/tests/integration/connector-sftp.itest.ts | Adds real SFTP cascade integration test (env-gated). |
| packages/engine/src/tests/integration/connector-mail.itest.ts | Adds real SMTP→IMAP cascade integration test (env-gated). |
| packages/engine/src/tests/integration/connector-db.itest.ts | Adds real Postgres destination integration test (env-gated, _test DB). |
| packages/engine/src/tests/connector-matrix.e2e.test.ts | Adds connector E2Es for File/HTTP/FHIR using real downstream effects. |
| packages/engine/src/tests/connector-js.e2e.test.ts | Adds JS source + destination connector E2Es with a real sandbox-backed runner. |
| packages/engine/src/tests/connector-dicom.e2e.test.ts | Adds in-process DICOM SCU→SCP cascade E2E using the new DIMSE adapter. |
| packages/engine/sandbox-globals.d.ts | Adds canonical shipped ambient sandbox types for TS channel script authoring. |
| packages/engine/package.json | Adds typecheck:scripts and test:integration scripts. |
| packages/engine/examples/sandbox-script-example.ts | Adds an example TS script file verified by the new sandbox tsconfig. |
| packages/connectors/src/dicom/dicom-receiver.ts | Switches DICOM receiver factory to the dcmjs-dimse adapter (removes dcmtk default factory). |
| packages/connectors/src/dicom/dicom-dispatcher.ts | Switches DICOM sender factory to the dcmjs-dimse adapter (removes dcmtk default factory). |
| packages/connectors/src/dicom/dcmjs-dimse-adapter.ts | Introduces DIMSE SCP/SCU adapter including presentation context negotiation and file IO. |
| packages/connectors/package.json | Replaces @ubercode/dcmtk dependency with dcmjs-dimse. |
| docs/progress/DECISIONS.md | Documents D-181 decision for the DICOM port and rationale. |
| docs/progress/CHANGELOG.md | Records the new E2E harness, connector coverage, DICOM port, and ChannelService.create fix. |
| docker/docker-compose.yml | Adds --profile test services for integration lane (SFTP + GreenMail). |
Files not reviewed (1)
- pnpm-lock.yaml: Generated file
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+57
to
+60
| const SRC_PORT_1 = 17681; | ||
| const DEST_PORT_1 = 17682; | ||
| const SRC_PORT_2 = 17683; | ||
|
|
Comment on lines
+45
to
+68
| export async function sendMllp(port: number, message: string, timeoutMs = 10_000): Promise<string> { | ||
| return new Promise<string>((resolve, reject) => { | ||
| const client = net.createConnection({ host: '127.0.0.1', port }, () => { | ||
| client.write(wrapMllp(message)); | ||
| }); | ||
|
|
||
| const parser = new MllpParser(); | ||
| client.on('data', (chunk: Buffer) => { | ||
| const messages = parser.parse(chunk); | ||
| const first = messages[0]; | ||
| if (first !== undefined) { | ||
| client.end(); | ||
| resolve(first); | ||
| } | ||
| }); | ||
|
|
||
| client.on('error', reject); | ||
| const timer = setTimeout(() => { | ||
| client.destroy(); | ||
| reject(new Error('sendMllp: timed out waiting for reply')); | ||
| }, timeoutMs); | ||
| timer.unref(); | ||
| }); | ||
| } |
Comment on lines
+216
to
+223
| export function createDimseReceiver(options: { | ||
| readonly port: number; | ||
| readonly storageDir: string; | ||
| readonly aeTitle: string; | ||
| readonly minPoolSize: number; | ||
| readonly maxPoolSize: number; | ||
| readonly connectionTimeoutMs: number; | ||
| }): Result<DcmtkReceiver> { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the mock-heavy connector tests with a harness that actually pushes messages through real connectors and asserts what lands on the other side. All 11 connector types now have a passing real-message E2E.
Harness
deployChannel(spec)assembles a live channel (sandbox + 8-stage pipeline + real source/dest connectors) over an in-memory store; filter/transformer/response-transformer scripts compile through the real esbuild + code-template path.CaptureDestinationsink + MLLP TCP helpers.pnpm test(no infra) andpnpm --filter @mirthless/engine test:integration(*.itest.ts, env-gated;docker compose --profile test upadds atmoz/sftp + GreenMail).Connector coverage (11/11)
TCP/MLLP + Channel (A→B→C HL7→JSON→XML cascade), File, HTTP, JavaScript (source+dest), Database (real
_testPostgres), SFTP (listen↔dest cascade), SMTP + IMAP (GreenMail cascade), FHIR (dest), DICOM (SCU→SCP cascade).DICOM ported to dcmjs-dimse (D-181)
The DICOM SCP rejected every C-STORE association because the
@ubercode/dcmtkwrapper created it with no accepted presentation contexts. Swapped todcmjs-dimse(pure JS, the library the production MedFusion DIMSE service uses): the new adapter negotiates presentation contexts, writes received instances viaDataset.toFile, and sends viaClient+CStoreRequest. Drop-in behind the existing factory seams — no connector API change; existing mock-injecting unit tests untouched. The cascade test now runs in the default lane, in-process, no native binaries.Also in this PR
packages/engine/sandbox-globals.d.ts(drift-guarded vs the web Monaco string).ChannelService.createsilently droppedinput.transformers/filters(onlyupdatepersisted them) — data loss on clone/import/programmatic create. Extracted sharedsyncFilters/syncTransformers; real-Postgres regression test.Verification
connectors 491 + engine 383 (unit) + integration lane (DB/SFTP/SMTP/IMAP) green; server 998 unit green; build + lint clean across touched packages. Docs: CHANGELOG + D-181.
🤖 Generated with Claude Code