feat(dlr): replace MVStore with PostgreSQL persistence

.github/CONTRIBUTING.md

@@ -26,18 +26,51 @@ git clone https://github.com/cytechmobile/sendium.git
 cd sendium
 ```
 **2. Start the application in development mode:**
+
+Start a local PostgreSQL instance:
+
 ```bash
-./mvnw -pl sendium-app -am quarkus:dev
+docker run --rm -d --name sendium-postgres-dev \
+  -e POSTGRES_DB=sendium \
+  -e POSTGRES_USER=sendium \
+  -e POSTGRES_PASSWORD=sendium-dev \
+  -p 5432:5432 \
+  postgres:17-alpine
+
+SENDIUM_DLR_POSTGRESQL_JDBC_URL=jdbc:postgresql://localhost:5432/sendium \
+SENDIUM_DLR_POSTGRESQL_USERNAME=sendium \
+SENDIUM_DLR_POSTGRESQL_PASSWORD=sendium-dev \
+  ./mvnw -pl sendium-app -am quarkus:dev
 ```
 Note: This will start the server with live reload enabled. Any changes you make to the Java code will automatically trigger a compilation and reload.
 
-On Windows PowerShell, replace `./mvnw` with `.\mvnw.cmd`.
+On Windows PowerShell:
+
+```powershell
+docker run --rm -d --name sendium-postgres-dev `
+  -e POSTGRES_DB=sendium `
+  -e POSTGRES_USER=sendium `
+  -e POSTGRES_PASSWORD=sendium-dev `
+  -p 5432:5432 `
+  postgres:17-alpine
+
+$env:SENDIUM_DLR_POSTGRESQL_JDBC_URL = "jdbc:postgresql://localhost:5432/sendium"
+$env:SENDIUM_DLR_POSTGRESQL_USERNAME = "sendium"
+$env:SENDIUM_DLR_POSTGRESQL_PASSWORD = "sendium-dev"
+.\mvnw.cmd -pl sendium-app -am quarkus:dev
+```
 
 **3. 🧪 Testing**
 
 We value reliability. Before submitting any changes, please ensure all tests pass.
 
-You do not need Docker running locally to execute the test suite. Simply run the following command to execute all unit and integration tests:
+Unit tests do not require Docker:
+```bash
+./mvnw test
+```
+
+The complete verification suite includes PostgreSQL migration, adapter, Quarkus, outage, and protocol tests. Start Docker and run:
+
 ```bash
 ./mvnw verify
 ```

.github/workflows/native_e2e.yml

@@ -9,6 +9,7 @@ on:
       - ".mvn/**"
       - "mvnw"
       - "mvnw.cmd"
+      - ".github/workflows/run_tests.yml"
       - ".github/workflows/native_e2e.yml"
   workflow_dispatch:
 
@@ -24,6 +25,20 @@ jobs:
     if: github.event_name != 'pull_request' || !startsWith(github.head_ref, 'release-please--')
     runs-on: ubuntu-latest
     timeout-minutes: 60
+    services:
+      postgres:
+        image: postgres:17-alpine
+        env:
+          POSTGRES_DB: sendium
+          POSTGRES_USER: sendium
+          POSTGRES_PASSWORD: sendium-test
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd "pg_isready -U sendium -d sendium"
+          --health-interval 5s
+          --health-timeout 5s
+          --health-retries 10
     steps:
       - name: Checkout code
         uses: actions/checkout@v7
@@ -46,7 +61,10 @@ jobs:
 
       - name: Run native E2E smoke tests
         env:
-          SENDIUM_NATIVE_IMAGE: sendium:native-e2e
+          SENDIUM_E2E_IMAGE: sendium:native-e2e
+          SENDIUM_DLR_POSTGRESQL_JDBC_URL: jdbc:postgresql://host.docker.internal:5432/sendium
+          SENDIUM_DLR_POSTGRESQL_USERNAME: sendium
+          SENDIUM_DLR_POSTGRESQL_PASSWORD: sendium-test
         run: |
           ./mvnw -B -pl sendium-core -DskipTests test-compile org.codehaus.mojo:exec-maven-plugin:3.5.1:java \
             -Dexec.classpathScope=test \

.github/workflows/run_tests.yml

@@ -26,6 +26,20 @@ jobs:
     permissions:
       contents: read
     runs-on: ubuntu-latest
+    services:
+      postgres:
+        image: postgres:17-alpine
+        env:
+          POSTGRES_DB: sendium
+          POSTGRES_USER: sendium
+          POSTGRES_PASSWORD: sendium-test
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd "pg_isready -U sendium -d sendium"
+          --health-interval 5s
+          --health-timeout 5s
+          --health-retries 10
     steps:
       - name: Checkout code
         uses: actions/checkout@v7
@@ -47,12 +61,22 @@ jobs:
         env:
           SENDIUM_TEST_IMAGE: sendium:quick-start-test
         run: sh tests/quick-start-test.sh
+      - name: Test PostgreSQL restart behavior on JVM
+        env:
+          SENDIUM_E2E_IMAGE: sendium:quick-start-test
+          SENDIUM_DLR_POSTGRESQL_JDBC_URL: jdbc:postgresql://host.docker.internal:5432/sendium
+          SENDIUM_DLR_POSTGRESQL_USERNAME: sendium
+          SENDIUM_DLR_POSTGRESQL_PASSWORD: sendium-test
+        run: |
+          ./mvnw -B -pl sendium-core -DskipTests test-compile org.codehaus.mojo:exec-maven-plugin:3.5.1:java \
+            -Dexec.classpathScope=test \
+            -Dexec.mainClass=utils.NativeE2eSmoke
       - id: result
         name: Sendium Test Result
         if: always()
         run: |
-          echo "result=${{job.status}}" >> $GITHUB_ENV
-          echo "result=${{job.status}}" >> $GITHUB_OUTPUT
+          echo "result=${{job.status}}" >> "$GITHUB_ENV"
+          echo "result=${{job.status}}" >> "$GITHUB_OUTPUT"
       - name: Cancel current workflow run
         if: failure()
         uses: actions/github-script@v9

README.md

@@ -64,15 +64,15 @@ Download and run the setup script:
 curl -fsSLo quick-start.sh https://raw.githubusercontent.com/cytechmobile/sendium/main/quick-start.sh && sh quick-start.sh
 ```
 
-The script creates a `sendium/` runtime directory, generates random HTTP and SMPP credentials, writes Docker Compose and all required configuration files, starts Sendium, and waits for the HTTP API.
+The script creates a `sendium/` runtime directory, generates random HTTP, SMPP, and database credentials, writes Docker Compose and all required configuration files, starts PostgreSQL and Sendium, and waits for PostgreSQL-backed readiness.
 
 It asks you to choose one upstream option:
 
 1. **ProSMS:** Uses `smpp.prosms.gr:2775` with a transceiver connection. [Create a ProSMS account](https://prosms.gr/sms-tool/?v=2&m=8) if needed; SMPP credentials require manual approval from ProSMS. Until credentials are approved, the script creates a local-only setup without a failing placeholder connection.
 2. **Existing SMPP provider:** Enter your provider host, port, credentials, and TLS choice. Quick Start uses a transceiver connection.
 3. **Local setup only:** Starts Sendium's local HTTP and SMPP interfaces without an outbound provider. You can explore the API, but messages cannot be delivered until an upstream route is configured.
 
-HTTP and SMPP ports are bound to `127.0.0.1` by default. Use the [Docker deployment guide](docs/02-docker-deployment.md) for generated-file details, manual setup, native images, and non-local deployments.
+HTTP and SMPP ports are bound to `127.0.0.1` by default. Use the [Docker deployment guide](docs/02-docker-deployment.md) for generated-file details, manual setup, native images, and non-local deployments. Review [DLR persistence](docs/13-dlr-persistence.md) before changing database backends or volume handling.
 
 When startup completes, the script prints the Swagger URL and an exact command for following live logs
 

docs/01-architecture.md

@@ -23,7 +23,7 @@ flowchart LR
     workerQueues[Worker queues]
     smppClients["SMPP client workers<br/>smppclient instances"]
     carriers["Upstream SMSCs<br/>carriers or SMPP providers"]
-    dlrStore["DLR correlation store<br/>InMemoryDlrService"]
+    dlrStore["DLR storage<br/>PostgreSQL"]
     webhooks["HTTP webhooks<br/>DLR and MO callbacks"]
     config["Runtime config files<br/>credentials.yml<br/>smsg.properties<br/>routingTable.conf"]
 
@@ -87,7 +87,7 @@ sequenceDiagram
     participant Client as HTTP client
     participant API as KannelResource
     participant Creds as CredentialFileWatcher
-    participant DLR as InMemoryDlrService
+    participant DLR as DlrStorage
     participant Queue as Router queue
     participant Router as StandardRoutingManager
     participant Worker as SmppClientWorker
@@ -103,7 +103,7 @@ sequenceDiagram
     Router->>Worker: Enqueue to selected worker
     Worker->>SMSC: submit_sm
     SMSC-->>Worker: submit_sm_resp
-    Worker->>DLR: Link gateway UUID to operator message ID
+    Worker->>DLR: Link gateway UUID to provider message ID
 ```
 
 ## SMPP Server Flow
@@ -116,6 +116,8 @@ sequenceDiagram
     participant Server as SmppServerWorker
     participant Auth as BasicSmppAuthenticationProvider
     participant Submit as BasicSubmitSmProcessor
+    participant Store as SMPP message store
+    participant DLR as DlrStorage
     participant Queue as Router queue
     participant Router as StandardRoutingManager
 
@@ -124,7 +126,12 @@ sequenceDiagram
     Auth-->>Server: Bind accepted or rejected
     Client->>Server: submit_sm
     Server->>Submit: Validate and convert PDU
-    Submit->>Queue: Enqueue StandardMessage
+    Submit-->>Server: Valid submission event
+    Server->>Store: Add event to persistence batch
+    Store->>DLR: Persist initial DLR state
+    DLR-->>Store: Commit successful
+    Store-->>Server: Handle persisted event
+    Server->>Queue: Enqueue StandardMessage
     Server-->>Client: submit_sm_resp
     Router->>Queue: Dequeue and route message
 ```
@@ -144,26 +151,66 @@ sequenceDiagram
 
 ## DLR Handling
 
-Outbound HTTP messages can include a Kannel-style `dlr-url`. Sendium stores the gateway message ID and later links it to the operator/SMSC message ID returned by the SMPP provider. When a DLR arrives, the DLR service resolves the correlation and forwards the callback.
+Outbound HTTP messages can include a Kannel-style `dlr-url`, while downstream SMPP submissions request receipts through `registered_delivery`. Before accepting either submission, Sendium stores one `dlr_message` row containing the gateway message ID and downstream delivery target. After the upstream SMSC returns `submit_sm_resp`, the client worker links that gateway ID to the exact `(provider name, provider message ID)` pair in `provider_correlation`. The provider name defaults to the worker's full name; workers sharing an SMSC message-ID namespace can use the same `msg.hash.prefix`.
+
+Different providers can reuse the same message ID independently. Reusing the same pair within one provider moves the correlation to the newest gateway message and clears it from the previous owner. Link and resolve transactions take a composite-key advisory lock and lock affected gateway rows in canonical UUID order, preventing crossed rebind and resolve deadlocks. Intermediate `ACCEPTD` and `ENROUTE` receipts are acknowledged without invoking the tracker or consuming correlation. The first terminal receipt records its exact state and error, consumes every correlation for the gateway message, and either deletes a `NONE` delivery row or retains an HTTP/SMPP row as `PENDING`.
 
 ```mermaid
 sequenceDiagram
-    participant SMSC as Upstream SMSC
-    participant Worker as SmppClientWorker
-    participant Tracker as InMemoryMessageTracker
-    participant Store as InMemoryDlrService
+    participant Ingress as HTTP/SMPP ingress
     participant Router as Router queue
-    participant DLRHook as ForwardDlrService
-    participant App as Originating application
+    participant Worker as SmppClientWorker
+    participant SMSC as Upstream SMSC
+    participant Tracker as StandardMessageTracker
+    participant Service as DlrService
+    participant Database as PostgreSQL DLR storage
+    participant HTTP as HTTP DLR dispatcher
+    participant App as Originating HTTP application
+    participant SMPPApp as Originating SMPP client
+
+    Ingress->>Service: saveInitialState(gateway message ID)
+    Service->>Database: Insert DLR message
+    Database-->>Service: Commit
+    Service-->>Ingress: State persisted
+    Ingress->>Router: Enqueue accepted message
+    Router->>Worker: Route outbound message
+    Worker->>SMSC: submit_sm
+    SMSC-->>Worker: submit_sm_resp(provider message ID)
+    Worker->>Tracker: linkProviderMessageId
+    Tracker->>Service: Link gateway ID and provider pair
+    Service->>Database: Lock and upsert provider correlation
 
     SMSC->>Worker: deliver_sm delivery receipt
-    Worker->>Tracker: createAndEnqueueDLR
-    Tracker->>Store: Resolve operator message ID
-    Store->>DLRHook: Forward DLR callback if URL exists
-    DLRHook->>App: HTTP GET callback
-    Tracker->>Router: Enqueue internal MSG_DLR
+    alt ACCEPTD or ENROUTE receipt
+        Worker-->>SMSC: deliver_sm_resp (success)
+    else Terminal receipt
+        Worker->>Tracker: createAndEnqueueDLR
+        Tracker->>Service: resolveDlr(provider pair, exact state/error)
+        Service->>Database: Lock, resolve, consume correlations, retain pending delivery
+        Database-->>Service: Resolved message state
+        alt persistence succeeds
+            Worker-->>SMSC: deliver_sm_resp (success)
+        else persistence fails
+            Worker-->>SMSC: deliver_sm_resp (SYSERR)
+        end
+    end
+
+    alt HTTP delivery channel
+        loop Poll durable due rows
+            HTTP->>Database: Start fenced attempt
+            HTTP->>App: HTTP GET callback
+            HTTP->>Database: Delete on success or persist retry/failure
+        end
+    else SMPP delivery channel
+        Tracker->>Router: Enqueue internal MSG_DLR
+        Router->>SMPPApp: deliver_sm receipt part(s)
+        SMPPApp-->>Router: deliver_sm_resp for every part
+        Router->>Database: Delete only after all responses succeed
+    end
 ```
 
+HTTP and SMPP delivery are acknowledgement-driven and at-least-once. A crash after the receiver accepts a callback or response can cause the same receipt, including already acknowledged multipart SMPP parts, to be delivered again.
+
 ## MO Handling
 
 Mobile-originated messages received from upstream SMPP providers are handled by the SMPP client worker. If the worker instance has an MO forwarding URL configured, `SmppClientWorker` forwards the MO through `ForwardMoService` using the configured forwarding format.
@@ -209,9 +256,9 @@ Sendium expects runtime files in the configured `conf` directory.
 
 ## Persistence Boundaries
 
-Most runtime queues are in-memory. The DLR correlation service uses H2 MVStore at `data/dlr-mvstore.db` by default and falls back to in-memory maps if the store cannot be opened.
+Most runtime queues are in memory. DLR messages, provider correlations, and terminal HTTP/SMPP delivery state use PostgreSQL. Sendium completes the required storage operation before HTTP routing or successful downstream SMPP acknowledgement. Queued and in-flight messages remain process-local.
 
-This means operators should treat queued, in-flight messages as process-local state, while DLR correlation has lightweight local persistence.
+PostgreSQL does not make multipart assembly or router and worker queues durable. See [DLR Persistence](13-dlr-persistence.md) for retention, restart guarantees, and the remaining crash windows.
 
 ## Related Documentation
 
@@ -221,3 +268,4 @@ This means operators should treat queued, in-flight messages as process-local st
 * [Routing Engine](05-routing-engine.md)
 * [Webhooks](07-webhooks.md)
 * [Docker Deployment](02-docker-deployment.md)
+* [DLR Persistence](13-dlr-persistence.md)