Real-message E2E testing (11/11 connectors) + DICOM dcmjs-dimse port

docker/docker-compose.yml

@@ -17,5 +17,32 @@ services:
       timeout: 5s
       retries: 5
 
+  # --- Integration-test services (only start with: docker compose --profile test up) ---
+  # Back the engine's connector integration lane (packages/engine *.itest.ts).
+  # Match the env the gates expect (packages/engine/src/__tests__/integration/gates.ts).
+
+  sftp-test:
+    image: atmoz/sftp:alpine
+    container_name: mirthless-sftp-test
+    profiles: ["test"]
+    # user:password:::dir  → creates /home/mirth/upload, writable, chrooted.
+    command: mirth:mirthpw:::upload
+    ports:
+      - "2222:22"
+
+  mail-test:
+    # GreenMail: SMTP (3025) + IMAP (3143) in one container for mail connectors.
+    image: greenmail/standalone:2.1.0
+    container_name: mirthless-mail-test
+    profiles: ["test"]
+    environment:
+      GREENMAIL_OPTS: >-
+        -Dgreenmail.setup.test.all
+        -Dgreenmail.hostname=0.0.0.0
+        -Dgreenmail.users=mirth:mirthpw@example.com
+    ports:
+      - "3025:3025"
+      - "3143:3143"
+
 volumes:
   postgres_data:

docs/progress/CHANGELOG.md

@@ -2,6 +2,33 @@
 
 > Session-by-session log of what was built. Enables any future Claude instance to pick up where we left off.
 
+## 2026-07-14 — Real-message E2E testing + DICOM dcmjs-dimse port (branch `feature/real-e2e-testing`)
+
+Replaced mock-heavy connector tests with a harness that actually pushes messages through real
+connectors and asserts what lands on the other side. **All 11 connector types now covered by a
+passing real-message E2E.**
+
+- **E2E harness** (`packages/engine/src/__tests__/support/`) — `deployChannel(spec)` assembles a
+  live channel (sandbox + 8-stage pipeline + real source/dest connectors) over an in-memory store;
+  filter/transformer/response-transformer scripts compile through the real esbuild+template path.
+  `CaptureDestination` sink + MLLP TCP helpers. Two lanes: default `pnpm test` (no infra) and
+  `test:integration` (`*.itest.ts`, env-gated via `integration/gates.ts`; docker `--profile test`
+  adds atmoz/sftp + GreenMail).
+- **Coverage**: TCP/MLLP + Channel (A→B→C HL7→JSON→XML cascade), File, HTTP, JavaScript (src+dest),
+  Database (real `_test` PG), SFTP (listen↔dest cascade), SMTP+IMAP (GreenMail cascade), FHIR (dest),
+  DICOM (SCU→SCP cascade).
+- **DICOM connector ported to `dcmjs-dimse`** (pure JS) from `@ubercode/dcmtk` (native binaries).
+  Root cause of the prior association rejection: the dcmtk wrapper created the SCP with no accepted
+  presentation contexts. New `dcmjs-dimse-adapter.ts` negotiates contexts (accept all StorageClass
+  SOP classes + Verification + common transfer syntaxes) — pattern lifted from the production
+  MedFusion DIMSE service. Drop-in behind the existing `DcmtkReceiver`/`DcmtkSender` factory seams;
+  no connector API change. DICOM cascade now runs in the default lane, in-process, no binaries. (D-181)
+- **TypeScript channel scripts** compiled+run end-to-end; shipped ambient types
+  `packages/engine/sandbox-globals.d.ts` (drift-guarded against the web Monaco string).
+- **Bug fix** (`4c6af8f`): `ChannelService.create` silently dropped `input.transformers`/`filters`
+  (only `update` persisted them) — data loss on clone/import/programmatic create. Extracted shared
+  `syncFilters`/`syncTransformers`, wired into both paths; real-Postgres regression test.
+
 ## 2026-07-13 — Dashboard/Channels/Messages overhaul (branch `feature/dashboard-overhaul`)
 
 Reworked the three main views so everything happens from the Dashboard; the standalone

docs/progress/DECISIONS.md

@@ -357,3 +357,5 @@ D-178: dbQuery script bridge = named Data Sources, not script-supplied connectio
 D-179: Fixed migration `when`-timestamp ordering so drizzle-kit applies 0009/0010 (and future migrations). Root cause: the hand-written migrations 0007/0008 were registered in `meta/_journal.json` with future-dated round `when` values (1784000000000 / 1784000100000 ≈ 2026-07-14), while 0009 (collections) and 0010 (data_sources) were generated 2026-07-13 with real lower `Date.now()` values. `drizzle-kit migrate` applies journal entries in `when` order and skips any whose `when` ≤ the last-applied — so once 0008 was applied, 0009/0010 were silently skipped (migrate reported "success" as a no-op). Effect: a **fresh install** created neither `collections` nor `data_sources`, and the dev DB was missing `data_sources` (the Data Sources page 500'd on "relation does not exist"). Fix: bumped 0009→1784000200000 and 0010→1784000300000 in the journal so all `when`s are monotonic by idx (verified: a from-scratch `drizzle-kit migrate` now creates collections + data_sources). Reconciled the dev DB by updating the already-applied 0009 record's `created_at` to match, then `db:migrate` applied 0010 cleanly. LESSON: never hand-set a migration's journal `when` to a future timestamp — a later generated migration will get a lower real timestamp and be skipped. Since it is now past those dates, newly generated migrations sort correctly again; this was a one-time skew affecting only 0009/0010. — 2026-07-14 (branch fix/migration-timestamp-ordering)
 
 D-180: RBAC now resolves permissions LIVE from the user's role (single source of truth), replacing the per-user user_permissions snapshot. Root cause of the recurring bug (new permissions like collections:*/datasources:* never reaching existing users, requiring a re-seed): permissions were snapshotted into user_permissions only at user create / role-change, but the enforcement/login paths READ that stale snapshot. Meanwhile UserService.getPermissions already resolved live — two divergent truths. Fix: all four readers (auth.middleware authenticate + optionalAuthenticate, auth.service login, socket.ts) now call `permissionNamesForRole(user.role)` (the same resolver getPermissions uses); removed the dead write path `syncPermissionsForRole` from UserService (create/update no longer touch user_permissions). Effect: adding a permission to a role in code reaches every existing user of that role on their next request — no re-seed, no re-login for server enforcement. The `user_permissions` table is now vestigial (kept to avoid a migration this pass; slated for a drop-migration + scope-column cleanup in RBAC Round 2). Chosen over snapshot+reconcile-on-login (user's call) because one source of truth eliminates the whole staleness class. This is the P0 security/correctness fix; admin-managed role CRUD (roles-as-data) and sub-resource ACLs are deferred to RBAC Round 2 per D-audit plan. — 2026-07-14 (branch feature/quality-pass-1)
+
+D-181: DICOM connector runs on `dcmjs-dimse` (pure JS), not `@ubercode/dcmtk` (native binaries). Root cause of the DICOM SCP rejecting every C-STORE association ("Rejected Permanent, Source: Service User"): the dcmtk wrapper (`defaultReceiverFactory`) called `DicomReceiver.create` with NO accepted presentation contexts, so the SCP could not negotiate a context for the sender's proposed SOP class + transfer syntax. A DICOM SCP MUST inspect `association.getPresentationContexts()` and `setResult(Accept, transferSyntax)` for the ones it supports before `sendAssociationAccept()`. The native wrapper gave us no seam to do that; it also spawned processes (Windows path issues, not testable in-process). Decision: swap to `dcmjs-dimse` (the same library the production MedFusion DIMSE service uses at `vns/medfusion`), implementing a `DimseReceiver`/`DimseSender` adapter (`packages/connectors/src/dicom/dcmjs-dimse-adapter.ts`) that negotiates presentation contexts (accept all `StorageClass` SOP classes + Verification + common transfer syntaxes, mirroring MedFusion's `presentation-contexts.ts`/`scp.ts`), writes each received instance to a `.dcm` via `Dataset.toFile`, and sends via `Client`+`CStoreRequest(filePath)`. The swap is a drop-in behind the existing injectable `DcmtkReceiver`/`DcmtkSender` factory interfaces (D-086) — no connector API/config change, existing mock-injecting unit tests untouched. Result: the DICOM SCU→SCP cascade now passes in-process in the DEFAULT test lane (no native binaries, no external SCP), completing 11/11 connectors under real-message E2E. `dcmjs-dimse`'s loglevel is set to `warn` to suppress per-association protocol chatter. Chosen over patching dcmtk (we don't control its create() options; still native + spawn) or leaving DICOM as a documented reproducer (10/11). — 2026-07-14 (branch feature/real-e2e-testing)

packages/connectors/package.json

@@ -23,7 +23,7 @@
     "@mirthless/core-models": "workspace:*",
     "@mirthless/core-util": "workspace:*",
     "@mirthless/engine": "workspace:*",
-    "@ubercode/dcmtk": "^0.3.0",
+    "dcmjs-dimse": "^0.3.2",
     "generic-pool": "^3.9.0",
     "imapflow": "^1.0.171",
     "nodemailer": "^9.0.1",