engine is not responding

[mqtt-fan]

Hi all,

I always get this error after the call with opencode:

`startup [ 46.4 s] opening the socket
lash_serve: a prompt admitted beside active streams prefills in pieces of 32768 tokens with a decode tick between them (HALOGEN_ADMIT_CHUNK; 0 = the engine's own chunk, which keeps the admitted prompt bitwise its alone self; -1 = the whole prompt at once)
lash_serve: listening on 127.0.0.1:8730, 4 slots over ONE 262144-position KV pool, each request up to 262144 (it reserves prompt + max_tokens), max_tok 32768, BATCHED greedy or sampled; MTP depth-1 selectable (drafter 1, speculates while alone)
alogen: engine listening after 50s
alogen: engine watchdog on, 180s of silence takes the container down
halogen/tools/serve_api.py:2731: DeprecationWarning:
on_event is deprecated, use lifespan event handlers instead.

   Read more about it in the                                                                                                                                                                                                                                       
   [FastAPI docs for Lifespan Events](https://fastapi.tiangolo.com/advanced/events/).                                                                                                                                                                              

@app.on_event("startup")
NFO: Started server process [73]
NFO: Waiting for application startup.
erve_api: engine at 127.0.0.1:8730, listening on 0.0.0.0:8731
NFO: Application startup complete.
NFO: Uvicorn running on http://0.0.0.0:8731 (Press CTRL+C to quit)
NFO: 192.168.2.45:38726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
alogen: the engine has not answered PING for 45s
alogen: the engine has not answered PING for 90s
alogen: the engine has not answered PING for 135s
alogen: the engine has not answered PING for 180s
alogen: the engine process is alive and has answered nothing for 180s.
PING is answered between decode rounds and between prefill chunks, so this is
a wedged engine rather than a slow one. Shutting the container down so a
restart policy can recover it; raise or disable HALOGEN_ENGINE_WATCHDOG_S
(0 = off) if you would rather it stayed up for diagnosis.
`

I am just using the default command:

podman run --rm -p 8731:8731
--device /dev/kfd --device /dev/dri --group-add keep-groups
--security-opt seccomp=unconfined --ipc=host --ulimit memlock=-1:-1
-e HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next
-v ~/halogen-models:/models
ghcr.io/peonist-ai/halogen-flash-server:0.5.0

[wenis]

Thanks for the report, and for the exact launch command. That helps.

What that message means. The container has a watchdog that pings the engine on a side connection every 15 seconds. If the engine does not answer for 180 seconds the container is shut down, on the reasoning that a live process answering nothing is a wedge that a restart can recover and nothing else can.

Why it fired here, and why it is probably a false alarm. The engine answers that ping between decode rounds, but on 0.5.0 it does not answer it while a prompt is being read in. So a prompt that takes longer than 180 seconds to prefill looks exactly like a wedged engine to the watchdog, and gets the same treatment. The 200 OK line just above the warnings is logged when the streamed response opens, before any work happens, so all 180 seconds of silence are that one request.

That is our bug, not yours. 0.5.1 makes the engine answer between prefill chunks, which bounds the silence for long prompts, but it does not help a prompt shorter than one chunk (32,768 tokens by default), which is likely your case. So please do not expect an upgrade alone to fix this.

Try this now:

-e HALOGEN_ENGINE_WATCHDOG_S=0

added to your podman run. That disables the watchdog. If the request then completes, even slowly, we know it was a slow prefill and the watchdog was wrong to fire. If it instead sits there indefinitely with nothing happening, that is a real wedge and a different bug.

What would help us most:

1. The full container log from the very first line, not just the tail. The startup lines report how much memory the machine has and how the KV pool was sized, and yours was reduced from its default, which suggests the machine is tighter on RAM than the one we develop on.
2. MemTotal from /proc/meminfo, and whether anything else large was running.
3. Roughly how many tokens opencode is sending on that first call.
4. With the watchdog off, what the request does: how long it takes to produce a first token, and whether the GPU is busy while you wait.

Your startup took about 50 seconds to reach listening, where we typically see 6 to 18. That points at slower storage or a cold page cache, and it is consistent with a first prefill being much slower than ours. Related but not the same: issue #2 turned out to be host memory in the wrong shape, and it is worth a look.

Leaving this open. Either branch is something we need to fix.