In plain terms (for product)
What breaks: citizens and staff cannot file or update complaints. Everything else about the system looks normal.
What it looks like to a user: the page does not show an error. It hangs. The spinner keeps going until the browser gives up, typically after a minute or two, and no complaint is recorded. Trying again does the same thing. From the citizen's side the service is simply broken with no explanation.
Why it is easy to miss: the software does not crash. Most failures announce themselves — the service dies, something restarts it, an alert fires. This one keeps running and stays silent. The site is still up, people can still log in, dashboards still load, existing complaints are still searchable. Only filing and updating are dead. A status page or an uptime check would show green throughout.
How long it lasts: indefinitely. It does not recover on its own and nothing restarts it. In the case we observed it stayed broken for six hours and only came back when someone restarted it by hand. Left alone it would have stayed down.
What sets it off: a burst of traffic. We hit it during load testing, but the same thing could follow a public announcement, a media mention, a service campaign, or a flood or outage that makes many people complain at once — precisely the moments when a complaints system matters most.
Who is affected: any deployment, not just the one we tested. Two other services share the same flaw.
What we are asking for: a small change so that when the internal message queue stalls, requests fail quickly with a visible error instead of hanging forever, plus an automatic restart so the service recovers without someone noticing first. The second part matters as much as the first — today, recovery depends on a human spotting it.
How to tell it is happening: requests to file a complaint hang rather than erroring, while the rest of the site behaves normally. Monitoring should watch for "accepting requests but not answering them", not only for crashes.
Summary (technical)
A single Kafka producer failure puts pgr-services into a permanent, silent, unrecoverable hang. The service keeps accepting TCP connections and answers none of them. It does not crash, does not exit, and is not restarted by Docker, so it stays in that state indefinitely.
Observed on the Bomet deployment on 2026-08-31 during load testing. The service was wedged for ~6 hours and only recovered when restarted by hand.
Mechanism
org.egov.tracer.kafka.CustomKafkaTemplate.send waits on CompletableFuture.get() with no timeout.
It is called synchronously on the request thread:
RequestsApiController.requestsCreatePost
-> PGRService.create (backend/pgr-services/.../service/PGRService.java:141)
-> Producer.push (backend/pgr-services/.../producer/Producer.java:25)
-> CustomKafkaTemplate.send (tracer, CustomKafkaTemplate.java:27)
-> CompletableFuture.get() <-- unbounded wait
If the producer's sender thread is gone, no future can ever complete, so every request thread parks forever. There is no timeout, no circuit breaker, and no bound on how many threads can accumulate in this state.
Evidence
Thread dump taken ~6 hours after the triggering event:
|
|
Tomcat http-nio workers |
130 |
In WAITING on CompletableFuture.get() |
130 (all of them) |
— inside PGRService.create |
113 |
— inside PGRService.update |
17 |
kafka-producer-network-thread alive |
0 |
| Container CPU |
0.15% |
| Consecutive healthcheck failures |
512 |
"http-nio-8080-exec-1" #35 daemon prio=5 os_prio=0 tid=0x... nid=0x45 waiting on condition
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@17.0.20.1/Native Method)
- parking to wait for <0x00000000f2f00000> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.CompletableFuture.waitingGet(java.base@17.0.20.1/CompletableFuture.java:1898)
at java.util.concurrent.CompletableFuture.get(java.base@17.0.20.1/CompletableFuture.java:2072)
at org.egov.tracer.kafka.CustomKafkaTemplate.send(CustomKafkaTemplate.java:27)
at org.egov.pgr.producer.Producer.push(Producer.java:25)
at org.egov.pgr.service.PGRService.create(PGRService.java:141)
at org.egov.pgr.web.controllers.RequestsApiController.requestsCreatePost(RequestsApiController.java:71)
CPU near zero with a load average in the double digits is the signature: the threads are blocked, not busy.
Why it never recovers
- The trigger was
java.lang.OutOfMemoryError: Java heap space, which stopped Spring's KafkaMessageListenerContainer (Stopping container due to an Error) and took the producer's sender thread with it.
- The
OutOfMemoryError hit a listener thread, not main, so the JVM never exited and Docker never restarted the container.
- The healthcheck (
wget -qO- http://localhost:8080/pgr-services/health) times out rather than returning non-200 quickly, and nothing acts on the failure. It had failed 512 consecutive times with no effect.
- Redpanda was healthy throughout —
cluster health reported no leaderless or under-replicated partitions. The broker was never the problem.
The OOM is the trigger, but it is not the bug being reported here. Any cause that kills the producer's sender thread produces the same permanent outage. The unbounded wait is what turns a transient fault into a terminal one.
Impact
Complaint filing is completely unavailable while this persists. Read paths served by other services keep working, which makes the outage easy to miss from the outside.
Every service using CustomKafkaTemplate shares the failure mode:
backend/pgr-servicesbackend/novu-bridgeutilities/default-data-handler
Where the fix belongs
CustomKafkaTemplate is not in this repository. It comes from the eGov tracer library, declared in backend/pgr-services/pom.xml and backend/novu-bridge/pom.xml as:
<artifactId>tracer</artifactId>
<version>2.9.0-SNAPSHOT</version>
The root fix therefore belongs upstream in tracer. This issue tracks it here because the impact lands on CCRS services and some mitigations are available in this repo.
Suggested remedies
- Upstream (root fix): bound the wait —
get(timeout, unit) — and translate expiry into a failure the caller can surface as a 5xx, rather than parking the thread forever.
- Upstream or here: don't block the request thread on the send at all where the downstream write is already asynchronous via the persister. The 200 already means "queued", not "committed", so the blocking wait buys little.
- Here, cheap mitigation: make the health endpoint fail fast under thread starvation and act on the healthcheck — as configured, 512 consecutive failures produced no restart, so the wedge persisted until manual intervention.
- Related but separate:
pgr-services runs with JAVA_OPTS=-Xmx384m on a 30.6 GiB host with no container memory limit. Raising it makes this trigger rarer but does not fix this issue — any producer failure reproduces it. Worth filing separately.
Reproduction
Drive sustained load until the JVM exhausts its heap (at -Xmx384m this happened within minutes), or otherwise kill the producer's sender thread. Then any POST /pgr-services/v2/request/_create hangs until the client gives up; observed a 120s client timeout with no response and no server-side error.
Context
Found while running the load-test campaign in #1854 / #1848. The finding also invalidated the top two rungs of that campaign's concurrency ladder, since those levels were measured against an already-wedged service — the docs on #1848 have been corrected accordingly.
In plain terms (for product)
What breaks: citizens and staff cannot file or update complaints. Everything else about the system looks normal.
What it looks like to a user: the page does not show an error. It hangs. The spinner keeps going until the browser gives up, typically after a minute or two, and no complaint is recorded. Trying again does the same thing. From the citizen's side the service is simply broken with no explanation.
Why it is easy to miss: the software does not crash. Most failures announce themselves — the service dies, something restarts it, an alert fires. This one keeps running and stays silent. The site is still up, people can still log in, dashboards still load, existing complaints are still searchable. Only filing and updating are dead. A status page or an uptime check would show green throughout.
How long it lasts: indefinitely. It does not recover on its own and nothing restarts it. In the case we observed it stayed broken for six hours and only came back when someone restarted it by hand. Left alone it would have stayed down.
What sets it off: a burst of traffic. We hit it during load testing, but the same thing could follow a public announcement, a media mention, a service campaign, or a flood or outage that makes many people complain at once — precisely the moments when a complaints system matters most.
Who is affected: any deployment, not just the one we tested. Two other services share the same flaw.
What we are asking for: a small change so that when the internal message queue stalls, requests fail quickly with a visible error instead of hanging forever, plus an automatic restart so the service recovers without someone noticing first. The second part matters as much as the first — today, recovery depends on a human spotting it.
How to tell it is happening: requests to file a complaint hang rather than erroring, while the rest of the site behaves normally. Monitoring should watch for "accepting requests but not answering them", not only for crashes.
Summary (technical)
A single Kafka producer failure puts
pgr-servicesinto a permanent, silent, unrecoverable hang. The service keeps accepting TCP connections and answers none of them. It does not crash, does not exit, and is not restarted by Docker, so it stays in that state indefinitely.Observed on the Bomet deployment on 2026-08-31 during load testing. The service was wedged for ~6 hours and only recovered when restarted by hand.
Mechanism
org.egov.tracer.kafka.CustomKafkaTemplate.sendwaits onCompletableFuture.get()with no timeout.It is called synchronously on the request thread:
If the producer's sender thread is gone, no future can ever complete, so every request thread parks forever. There is no timeout, no circuit breaker, and no bound on how many threads can accumulate in this state.
Evidence
Thread dump taken ~6 hours after the triggering event:
http-nioworkersWAITINGonCompletableFuture.get()PGRService.createPGRService.updatekafka-producer-network-threadaliveCPU near zero with a load average in the double digits is the signature: the threads are blocked, not busy.
Why it never recovers
java.lang.OutOfMemoryError: Java heap space, which stopped Spring'sKafkaMessageListenerContainer(Stopping container due to an Error) and took the producer's sender thread with it.OutOfMemoryErrorhit a listener thread, notmain, so the JVM never exited and Docker never restarted the container.wget -qO- http://localhost:8080/pgr-services/health) times out rather than returning non-200 quickly, and nothing acts on the failure. It had failed 512 consecutive times with no effect.cluster healthreported no leaderless or under-replicated partitions. The broker was never the problem.The OOM is the trigger, but it is not the bug being reported here. Any cause that kills the producer's sender thread produces the same permanent outage. The unbounded wait is what turns a transient fault into a terminal one.
Impact
Complaint filing is completely unavailable while this persists. Read paths served by other services keep working, which makes the outage easy to miss from the outside.
Every service using
CustomKafkaTemplateshares the failure mode:backend/pgr-servicesbackend/novu-bridgeutilities/default-data-handlerWhere the fix belongs
CustomKafkaTemplateis not in this repository. It comes from the eGovtracerlibrary, declared inbackend/pgr-services/pom.xmlandbackend/novu-bridge/pom.xmlas:The root fix therefore belongs upstream in
tracer. This issue tracks it here because the impact lands on CCRS services and some mitigations are available in this repo.Suggested remedies
get(timeout, unit)— and translate expiry into a failure the caller can surface as a 5xx, rather than parking the thread forever.pgr-servicesruns withJAVA_OPTS=-Xmx384mon a 30.6 GiB host with no container memory limit. Raising it makes this trigger rarer but does not fix this issue — any producer failure reproduces it. Worth filing separately.Reproduction
Drive sustained load until the JVM exhausts its heap (at
-Xmx384mthis happened within minutes), or otherwise kill the producer's sender thread. Then anyPOST /pgr-services/v2/request/_createhangs until the client gives up; observed a 120s client timeout with no response and no server-side error.Context
Found while running the load-test campaign in #1854 / #1848. The finding also invalidated the top two rungs of that campaign's concurrency ladder, since those levels were measured against an already-wedged service — the docs on #1848 have been corrected accordingly.