Harden post-storage ultrasound compression

[royradimalai]

Summary

Harden the sample into a fail-closed pre-storage pipeline for newly received multiframe ultrasound DICOM objects.

For an eligible lossless ultrasound object, the callback archives the byte-identical original to S3 and transcodes an in-memory copy to JPEG Baseline at quality 70. Orthanc stores the compressed derivative only after both the archive and the derivative pass validation. If archival, transcoding, or validation fails, Orthanc keeps the original unchanged.

Data flow

1. Inspect the incoming DICOM bytes before Orthanc stores them.
2. Skip non-US modalities, single-frame ultrasound, and objects already marked or encoded as lossy.
3. Begin uploading the original bytes to the configured lossless archive while transcoding runs in parallel.
4. Transcode to JPEG Baseline (1.2.840.10008.1.2.4.50) at quality 70.
5. Replace the transcoder-generated SOP Instance UID with a deterministic derived UID based on the source UID and compression profile. Retransmitting the same source therefore produces the same derivative instead of duplicates.
6. Mark the stored object as derived and lossy, record ISO_10918_1, and reference the source SOP Instance UID.
7. Validate patient, study, series, SOP class, derived SOP UID, frame count, transfer syntax, and lossy metadata.
8. Wait for S3 checksum and manifest confirmation, then return the validated compressed bytes to Orthanc for storage.

Lossless archive behavior

• Stores the byte-identical source under a content-addressed SHA-256 key.
• Writes a manifest keyed by the original Study, Series, and SOP Instance UIDs.
• Uses conditional S3 writes and checksums so retries are idempotent.
• Rejects a reused source SOP UID when its bytes conflict with the existing manifest.
• Keeps the original in Orthanc if archive confirmation is unavailable or invalid.

Safety properties

• No post-storage replacement and no duplicate lossless/lossy objects in the active Orthanc study.
• No change to existing studies. The callback applies only when a new eligible object arrives.
• No change to DX, CR, CT, MRI, single-frame US, or already-lossy US.
• Compression cannot proceed without pydicom, boto3, the archive bucket, quality 70, and compatible ingest-transcoding settings.
• The callback may hold an eligible ingest while it waits for archive confirmation, up to the configured timeout. Archival and transcoding run concurrently to reduce that delay.

Radimal deployment dependency

The deployment integration is radimal-terraform PR #363. It vendors this exact script, supplies the Python runtime dependencies, provisions the archive bucket and IAM access, configures quality 70, and enables the callback only in VEG staging.

Merge this PR before #363. Any change to this script must be synchronized into #363 before that Terraform PR is merged.

Production rollout

This sample does not deploy itself. Radimal production will be enabled in a separate Terraform PR after #23 and #363 merge. That rollout will target Radimal production only, not VEG production, and must be reviewed through an orthanc-1 Atlantis plan before apply.

Testing

py -3 -m unittest discover -s python-samples -p 'test_compress_us_to_lossy.py'

All 24 focused tests pass. They cover eligibility, archive checksums and manifests, deterministic UID behavior, metadata, frame and identity validation, retransmission, and fail-closed behavior.

[amazy]

Hi @royradimalai

This looks pretty good to me !

Since we now have multiple files related to this sample, could you move it into a specific folder like python-samples/lossy-us-compression

Thanks !