refactor(dlr): defer delivery observability

Author: lykakis

docs/01-architecture.md

@@ -151,9 +151,9 @@ sequenceDiagram
 
 ## DLR Handling
 
-Outbound HTTP messages can include a Kannel-style `dlr-url`. Before accepting a submission, Sendium stores its gateway message ID. After the upstream SMSC returns `submit_sm_resp`, the client worker links that gateway ID to the exact `(provider name, provider message ID)` pair. The provider name defaults to the worker's full name; workers sharing an SMSC message-ID namespace can use the same `msg.hash.prefix`.
+Outbound HTTP messages can include a Kannel-style `dlr-url`, while downstream SMPP submissions request receipts through `registered_delivery`. Before accepting either submission, Sendium stores one `dlr_message` row containing the gateway message ID and downstream delivery target. After the upstream SMSC returns `submit_sm_resp`, the client worker links that gateway ID to the exact `(provider name, provider message ID)` pair in `provider_correlation`. The provider name defaults to the worker's full name; workers sharing an SMSC message-ID namespace can use the same `msg.hash.prefix`.
 
-Different providers can reuse the same message ID independently. Reusing the same pair within one provider moves the correlation to the newest gateway message and clears it from the previous owner. Link and resolve transactions take a composite-key advisory lock and lock affected gateway rows in canonical UUID order, preventing crossed rebind and resolve deadlocks. Resolving a receipt transactionally consumes the tracked message and all of its correlations before callback forwarding or internal DLR queueing.
+Different providers can reuse the same message ID independently. Reusing the same pair within one provider moves the correlation to the newest gateway message and clears it from the previous owner. Link and resolve transactions take a composite-key advisory lock and lock affected gateway rows in canonical UUID order, preventing crossed rebind and resolve deadlocks. Intermediate `ACCEPTD` and `ENROUTE` receipts are acknowledged without invoking the tracker or consuming correlation. The first terminal receipt records its exact state and error, consumes every correlation for the gateway message, and either deletes a `NONE` delivery row or retains an HTTP/SMPP row as `PENDING`.
 
 ```mermaid
 sequenceDiagram
@@ -164,8 +164,9 @@ sequenceDiagram
     participant Tracker as StandardMessageTracker
     participant Service as DlrService
     participant Database as PostgreSQL DLR storage
-    participant DLRHook as ForwardDlrService
-    participant App as Originating application
+    participant HTTP as HTTP DLR dispatcher
+    participant App as Originating HTTP application
+    participant SMPPApp as Originating SMPP client
 
     Ingress->>Service: saveInitialState(gateway message ID)
     Service->>Database: Insert DLR message
@@ -187,16 +188,29 @@ sequenceDiagram
         Tracker->>Service: resolveDlr(provider pair, exact state/error)
         Service->>Database: Lock, resolve, consume correlations, retain pending delivery
         Database-->>Service: Resolved message state
-        opt DLR callback URL exists
-            Service->>DLRHook: Forward DLR callback
-            DLRHook->>App: HTTP GET callback
+        alt persistence succeeds
+            Worker-->>SMSC: deliver_sm_resp (success)
+        else persistence fails
+            Worker-->>SMSC: deliver_sm_resp (SYSERR)
         end
-        Service-->>Tracker: Resolved message state
+    end
+
+    alt HTTP delivery channel
+        loop Poll durable due rows
+            HTTP->>Database: Start fenced attempt
+            HTTP->>App: HTTP GET callback
+            HTTP->>Database: Delete on success or persist retry/failure
+        end
+    else SMPP delivery channel
         Tracker->>Router: Enqueue internal MSG_DLR
-        Worker-->>SMSC: deliver_sm_resp
+        Router->>SMPPApp: deliver_sm receipt part(s)
+        SMPPApp-->>Router: deliver_sm_resp for every part
+        Router->>Database: Delete only after all responses succeed
     end
 ```
 
+HTTP and SMPP delivery are acknowledgement-driven and at-least-once. A crash after the receiver accepts a callback or response can cause the same receipt, including already acknowledged multipart SMPP parts, to be delivered again.
+
 ## MO Handling
 
 Mobile-originated messages received from upstream SMPP providers are handled by the SMPP client worker. If the worker instance has an MO forwarding URL configured, `SmppClientWorker` forwards the MO through `ForwardMoService` using the configured forwarding format.

docs/07-webhooks.md

@@ -4,7 +4,7 @@ Sendium can call external HTTP endpoints for delivery receipts and mobile-origin
 
 ## Delivery Receipt Callbacks
 
-HTTP submissions can include a `dlr-url` query parameter. Sendium stores the callback URL with the submitted message and calls it when the message state changes.
+HTTP submissions can include a `dlr-url` query parameter. Sendium stores the callback URL with the submitted message and calls it for the first terminal provider outcome. Intermediate `ACCEPTD` and `ENROUTE` receipts are acknowledged to the provider but are not forwarded.
 
 Example HTTP submission:
 
@@ -31,10 +31,12 @@ curl -G http://localhost:8080/sendsms \
 | :--- | :--- |
 | `1` | Delivered. |
 | `2` | Failed. |
-| `4` | Buffered or accepted for processing. |
-| `8` | Submitted to SMSC. |
+| `4` | Buffered or accepted for processing; retained as a compatibility mapping and not normally emitted by final-only receipt handling. |
+| `8` | Submitted to SMSC; retained as a compatibility mapping and not normally emitted by final-only receipt handling. |
 
-DLR callbacks are sent as HTTP `GET` requests. HTTP status codes from `200` to `399` are treated as successful. Sendium makes up to 10 attempts with a 120 second delay between failed attempts. The retry schedule is process-local and does not survive a Sendium restart.
+DLR callbacks are sent as HTTP `GET` requests. The durable dispatcher checks PostgreSQL on a one-second schedule in serial batches of up to 100; a running batch delays the next check rather than overlapping it. Each request has a five-second timeout, and redirects are not followed; the original response status from `200` to `399` is treated as successful. Failures on attempts 1 through 9 are scheduled 120 seconds later. A failure on attempt 10 marks the row `FAILED`, and pending or failed rows are eligible for cleanup seven days after provider resolution. A malformed callback URI fails immediately without starting an HTTP attempt.
+
+The retry schedule and attempt count survive a Sendium restart, but delivery is at-least-once rather than exactly-once. A crash or storage failure after the receiver accepts a callback can cause another request. Callback handlers must be idempotent and should use the gateway message ID supplied through `%s` as their deduplication key.
 
 ## Mobile-Originated Message Forwarding
 
@@ -85,7 +87,7 @@ outSms.instance.testRoute.forward.mo.url = https://example.com/mo?from=%p&to=%P&
 outSms.instance.testRoute.forward.mo.format = FORM
 ```
 
-MO callbacks are sent as HTTP `POST` requests. HTTP status codes from `200` to `399` are treated as successful. Sendium makes up to 10 attempts with a 120 second delay between failed attempts. The retry schedule is process-local and does not survive a Sendium restart.
+Unlike durable DLR callbacks, MO callbacks are sent as process-local HTTP `POST` requests. HTTP status codes from `200` to `399` are treated as successful. Sendium makes up to 10 attempts with a 120 second delay between failed attempts. The MO retry schedule does not survive a Sendium restart.
 
 ## Security Notes
 

docs/08-monitoring-observability.md

@@ -18,7 +18,7 @@ For a local Docker or development run, verify the endpoint with:
 curl http://localhost:8080/q/metrics
 ```
 
-The endpoint exposes Quarkus, JVM, HTTP server, and Micrometer runtime metrics. Sendium-specific business metrics require explicit instrumentation in code, such as counters, timers, or gauges registered through Micrometer.
+The endpoint exposes Quarkus, JVM, HTTP server, and Micrometer runtime metrics. Sendium's PostgreSQL DLR subsystem also registers its selected-backend gauge and storage-operation timers.
 
 ## Prometheus Configuration
 

docs/13-dlr-persistence.md

@@ -12,6 +12,12 @@ The `sendium.dlr.persistence.enabled` build-time property controls this boundary
 
 Message paths degrade rather than fail when the subsystem is absent. HTTP and downstream SMPP submissions are accepted and routed without gateway DLR state, undelivered downstream receipts fall back to the worker's in-memory retry, and provider receipts are not correlated, so Sendium emits no delivery receipts of its own. An application that embeds `sendium-core` without this subsystem is expected to supply its own `Tracker` and message store if it needs delivery receipts.
 
+## Storage Model
+
+`sendium_dlr.dlr_message` contains one row per gateway UUID. The row holds ingress metadata, the exact terminal provider outcome, the downstream delivery channel and status, the common HTTP/SMPP payload, the retry schedule, and the monotonically increasing attempt number used for fencing.
+
+`sendium_dlr.provider_correlation` maps the exact `(provider_name, provider_message_id)` pair to the gateway UUID. Multiple provider correlations, including multipart provider IDs, can point to one message. Terminal resolution stores the resolving provider pair and outcome in `dlr_message`, removes every correlation for that gateway message, and retains the message row only when HTTP or SMPP delivery is required.
+
 ## Quick Start PostgreSQL
 
 The generated Quick Start runtime creates:
@@ -98,9 +104,9 @@ PostgreSQL is fail-closed. If required persistence is unavailable, new HTTP subm
 
 Provider message IDs are correlated within the outbound provider namespace rather than globally. The worker instance name is the default namespace; workers connected to the same SMSC account can share `msg.hash.prefix` when that SMSC may deliver their receipts interchangeably. Different providers may therefore return the same message ID without overwriting each other's state. The namespace must remain stable while correlations are outstanding: changing `msg.hash.prefix` or renaming a worker using the default makes earlier receipts unresolvable.
 
-Sendium requests final delivery receipts from upstream SMPP providers. A valid unsolicited `ACCEPTD` or `ENROUTE` receipt is acknowledged successfully but is not forwarded and does not consume its provider correlation. The first terminal receipt consumes the correlation and produces the downstream DLR; later receipts for that provider message ID cannot resolve it. Multipart submissions retain this first-terminal behavior and do not aggregate delivery states across every segment.
+Sendium requests final delivery receipts from upstream SMPP providers. A valid unsolicited `ACCEPTD` or `ENROUTE` receipt, including a receipt identified through its SMPP ESM class, is acknowledged successfully but is not forwarded and does not consume its provider correlation. The first terminal receipt consumes every correlation for the gateway message and produces the downstream DLR; later receipts for those provider message IDs cannot resolve it. Multipart submissions retain this first-terminal behavior and do not aggregate delivery states across every segment. A terminal persistence failure returns `deliver_sm_resp` with `STATUS_SYSERR` so the provider can retry. A successful provider acknowledgement confirms durable resolution only; it does not wait for downstream HTTP or SMPP delivery.
 
-A terminal receipt remains in `sendium_dlr.dlr_message` while HTTP or SMPP delivery is pending. The delivery attempt number is a fencing token: a stale completion or failure cannot mutate a newer attempt, and an adapter-local active-ID guard prevents duplicate starts within one process.
+A terminal receipt remains in `sendium_dlr.dlr_message` while HTTP or SMPP delivery is pending or after HTTP delivery reaches terminal `FAILED` status. The delivery attempt number is a fencing token: a stale completion or failure cannot mutate a newer attempt, and a process-local storage guard prevents duplicate starts within one Sendium instance.
 
 ## Retention
 
@@ -114,6 +120,8 @@ The V1 retention thresholds are fixed application behavior, not environment sett
 
 Cleanup is triggered by storage activity and runs no more than once per hour. These values are therefore eligibility thresholds, not exact physical deletion deadlines: idle records can remain in the database longer, and an active deployment can retain newly eligible state until the next cleanup pass. A provider receipt cannot be matched after its correlation has been removed. Making the thresholds or cleanup schedule configurable is outside the V1 storage replacement.
 
+An HTTP row marked `FAILED` after attempt 10 remains eligible based on the original provider-resolution time, not the time of its final request. SMPP delivery has no fixed attempt cap, but a still-pending SMPP row is also eligible for cleanup seven days after resolution.
+
 Cleanup is best-effort maintenance and is isolated from message handling. One caller at a time runs a pass while every other caller proceeds immediately, and a failed pass is logged and left until the next interval rather than rejecting the submission that triggered it.
 
 ## Durability Boundaries
@@ -122,13 +130,16 @@ Cleanup is best-effort maintenance and is isolated from message handling. One ca
 | :--- | :--- | :--- |
 | Initial DLR state for HTTP and downstream SMPP submissions | Persisted before HTTP routing or a successful SMPP acknowledgement. | Router and worker queues remain in memory. A process crash can lose queued outbound work even though its DLR row remains until cleanup. |
 | Gateway-to-provider message correlation | Survives Sendium restart after the provider message ID is linked. Intermediate `ACCEPTD` and `ENROUTE` receipts leave it intact. | The first terminal receipt consumes every correlation for the gateway message. |
-| Terminal HTTP/SMPP delivery | The common payload and exact provider outcome remain in one row until fenced completion. | Delivery scheduling and SMPP response batching are separate runtime concerns. |
+| Terminal HTTP/SMPP delivery | The common payload and exact provider outcome remain in one row until fenced completion. | Delivery is at-least-once; acknowledgement can be received before the final delete commits. |
 | Active delivery attempt | The database attempt number fences stale completion, retry, and failure updates. | The active-ID guard is process-local. Adapter recreation may start a new attempt for an attempt that was active before a crash. |
 | Multipart submission | Each acknowledged segment has provisional DLR state; completed aggregates update the primary state. | Multipart assembly and its pending timers are process-local and are not reconstructed after restart. |
-| HTTP DLR callback retry | Pending state and the next-attempt timestamp are durable. | The scheduler that consumes due rows is implemented separately. |
+| HTTP DLR callback retry | Pending state, attempt count, and next-attempt timestamp are durable. Checks are scheduled every second in non-overlapping serial batches; failures retry after 120 seconds and attempt 10 failures become `FAILED`. | A request accepted before a crash or failed completion update can be repeated. A slow batch delays later due callbacks. |
+| SMPP DLR delivery | One attempt covers every generated receipt part and completes only after matching successful `deliver_sm_resp` PDUs for all parts. Pending rows are enqueued when the same `system_id` binds. | Timeout, `generic_nack`, wrong/non-OK response, enqueue/send failure, or session closure releases the attempt. Replay is bind-driven rather than periodic, and partial success is not checkpointed. |
 | Database files | The Quick Start named volume survives normal container replacement and `docker compose down`. | Volume deletion, host-disk loss, and disaster recovery require backups or external PostgreSQL replication managed by the operator. |
 
-These limits are intentional V1 boundaries. PostgreSQL provides DLR persistence and delivery fencing; it is not a distributed worker coordinator or a replacement for the router and worker queues.
+Downstream delivery uses bounded at-least-once attempt semantics, not exactly-once delivery. A crash or storage failure after an HTTP receiver accepts a callback, or after an SMPP client sends a successful `deliver_sm_resp`, can cause the receipt to be delivered again. Multipart SMPP replay can repeat already acknowledged parts. Consumers must be idempotent using the gateway or receipted message ID. HTTP retry limits, SMPP bind availability, and seven-day retention mean this is not an unlimited eventual-success guarantee.
+
+These limits are intentional V1 boundaries. PostgreSQL provides DLR persistence and delivery fencing; it is not a distributed worker coordinator or a replacement for the router and worker queues. Attempt guards are process-local, so multiple active Sendium replicas sharing one database can start duplicate deliveries.
 
 ## Related Documentation
 

sendium-core/src/main/java/gr/cytech/sendium/core/smpp/server/DlrDeliveryBatch.java

@@ -58,6 +58,10 @@ public String getGatewayMessageId() {
         return message.serial;
     }
 
+    public M getMessage() {
+        return message;
+    }
+
     public void partSucceeded(int partOrdinal) {
         boolean complete = false;
         synchronized (this) {

sendium-core/src/main/java/gr/cytech/sendium/core/smpp/server/tasks/OutTask.java

@@ -28,6 +28,7 @@ public void run() {
         boolean success;
         DlrDeliverSmReference<?> dlrReference =
                 pdu.getReferenceObject() instanceof DlrDeliverSmReference<?> reference ? reference : null;
+        StandardMessage durableDlr = dlrReference == null ? null : dlrReference.batch().getMessage();
         if (dlrReference != null && !dlrReference.batch().isActive()) {
             return;
         }
@@ -65,14 +66,18 @@ public void run() {
 
         if (!success) {
             if (dlrReference != null) {
+                if (MessageTrace.shouldLog(worker.getConfigurationProvider(), MessageTrace.EVENT_DELIVER_FAILED)) {
+                    logger.warn("message.deliver.failed worker={} {}", worker.getFullName(),
+                            MessageTrace.identifiers(durableDlr));
+                }
                 dlrReference.batch().fail("send_failed");
             } else {
                 worker.outTaskFailed(pdu, msg);
             }
-        } else if (!pdu.isResponse() && msg != null) {
+        } else if (!pdu.isResponse() && (msg != null || durableDlr != null)) {
             if (MessageTrace.shouldLog(worker.getConfigurationProvider(), MessageTrace.EVENT_DELIVER_SENT)) {
                 logger.info("message.deliver.sent worker={} deliverMsgId={} {}", worker.getFullName(),
-                        MessageTrace.value(deliverMsgId), MessageTrace.identifiers(msg));
+                        MessageTrace.value(deliverMsgId), MessageTrace.identifiers(msg != null ? msg : durableDlr));
             }
         }
     }