grok error

[Damian-o2]

I get

Turn failed in 0.1s: Internal error: "serialization error: missing field completion_tokens at line 1 column 194"

[giannisanni]

Real bug, fixed on main in 78b62cf.

The first SSE chunk was sending a partial usage object:

"usage": {"prompt_tokens": 17}

That is there so a UI can show context use and prefill tok/s before the first token lands. Lenient clients ignore the missing members, which is why the web UI and curl never tripped on it, but a client that deserializes usage into a struct with non-optional fields rejects the whole turn on the opening event. Hence the failure at line 1 column 194 rather than at the end of the stream.

It now sends all three from the start:

"usage": {"prompt_tokens": 17, "completion_tokens": 0, "total_tokens": 17}

The zeros are accurate at that point, and the final chunk still reports the real counts. Pull and rebuild, and grok-cli should get through the stream. If it still errors, paste the failing payload and I will look at what else it wants.

[Damian-o2]

I update code.
It compiled (but not show any version number )
It starting
( --host '0.0.0.0' --port 8081 --ctx 56700 )

pulsar-serve: loading /home/user/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf
pulsar: using CUDA device 0
pulsar: KV auto: f32 KV at ctx 56700 would be 2.2GiB of 4.1GiB free -> defaulting to fp8 (0.6GiB); set PULSAR_KV=f32 to force exact f32 KV
pulsar: fp8 KV cache on (8.65 GiB -> 2.20 GiB over 40 layers)
pulsar: auto budget: 3.3GiB VRAM free -> expert cache 2.1GiB, staging 0.5GiB, prefill chunk 256
pulsar: no census yet - warm set seeded from built-in hotlist (12288 slabs)
pulsar: warm start: 11077 slabs in 7.4s
pulsar-serve: listening on http://0.0.0.0:8081  (web UI at /, API at /v1)

I run grok and get

 Turn failed in 1h0m: Internal error: "inference idle timeout after 3600s with no chunks"

I use GLM-4.7-Flash-REAP-23B-A3B-UD-Q5_K_XL.gguf
and this same efect: Turn failed in 1h0m: Internal error: "inference idle timeout after 3600s with no chunks"

simple not working
Qwen3.5-9B-Q4_K_M.gguf
pulsar: using CUDA device 0
pulsar: cudaMalloc(0.23 GB) failed on device 0
pulsar-serve: cuda kernel op failed: cudaMalloc

[giannisanni]

Good news first: the serialization errors are gone, the server loads and serves. The two things left look like separate problems, and neither is the schema.

1. "no chunks for 3600s" is almost certainly a non-streaming request, not a hang.

pulsar writes the SSE role chunk immediately after the response headers, before prefill starts, precisely so a long prefill does not look like a dead upstream. So on a streaming request you get bytes in milliseconds no matter how slow the model is. Getting literally nothing for an hour points at stream: false, where by definition nothing is sent until the entire completion is finished.

Check which it is:

curl -N -s http://127.0.0.1:8081/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Say OK."}],"max_tokens":5,"stream":true}'

If data: {...”role”:”assistant”...} comes back instantly, the server is fine and grok is sending stream:false.

2. Your real problem is that this box is far too small for that context.

From your own log:

auto budget: 3.3GiB VRAM free -> expert cache 2.1GiB
fp8 KV cache on (8.65 GiB -> 2.20 GiB over 40 layers)

4.1GiB of VRAM total, of which --ctx 56700 eats 2.2GiB as KV, leaving ~2.1GiB of expert cache for a 35B-A3B model. Every token then streams most of its experts from disk. That is not broken, it is just going to be extremely slow, and a 56700-token context makes prefill enormous before the first token even appears.

Start much smaller and grow only if it is comfortable:

pulsar-serve -m model.gguf --host 0.0.0.0 --port 8081 --ctx 4096

--ctx 4096 frees roughly 2GiB back to the expert cache, which is the number that actually sets your tok/s here.

3. cudaMalloc(0.23 GB) failed on the 9B is a different thing: the GPU is already full.

A quarter-gig allocation failing means something else holds the memory, very often a previous pulsar-serve that did not exit. Check with nvidia-smi and make sure no stale process is left:

nvidia-smi --query-compute-apps=pid,used_memory --format=csv

If a previous run is still there, kill it by PID and retry.

Worth saying plainly: pulsar is built for streaming very large MoE models off SSD, and it will run things far bigger than your VRAM, but "runs" and "runs at a usable speed" are different. On 4GiB, a dense small model at a modest context will feel much better than a 35B at 56700.

[Damian-o2]

In my opinion t/s is not problem. Problem is context, agents and good json

Meybe puts sha256 for working model file?

pulsar-serve: loading ./unsloth/DeepSeek-V4-Flash-0731-GGUF/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf
pulsar: split gguf: 5 shards as one virtual file
pulsar: using CUDA device 0
pulsar-serve: blk.2.attn_compressor_kv.weight: expected f16/q8_0, got BF16

Typy tensorów:

Parametry architektury:
  Liczba warstw                    = 43
  Kontekst treningowy              = 1048576
  Wymiar reprezentacji             = 4096
  Liczba głów attention          = 64
  Liczba głów KV                 = 1
  Typ skalowania RoPE              = yarn
  Współczynnik skalowania RoPE   = 16
  Oryginalny kontekst RoPE         = 65536
  rope.scaling.yarn_beta_fast      = 32
  rope.scaling.yarn_beta_slow      = 1
  Bazowa częstotliwość RoPE     = 10000
  Epsilon RMSNorm                  = 1e-06
  expert_gating_func               = 4
  Wymiar głowy Q/K                = 512
  Wymiar głowy V                  = 512
  Wymiar RoPE                      = 64
  attention.q_lora_rank            = 1024
  attention.sliding_window         = 128
  Wymiar FFN eksperta              = 2048
  expert_shared_count              = 1
  expert_weights_scale             = 1.5
  expert_weights_norm              = tak
  swiglu_clamp_exp                 = [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10]
  swiglu_clamp_shexp               = [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10]
  attention.indexer.head_count     = 64
  attention.indexer.key_length     = 128
  attention.indexer.top_k          = 512
  attention.output_group_count     = 8
  attention.output_lora_rank       = 1024
  attention.compress_ratios        = [0, 0, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 0, 0, 0]
  attention.compress_rope_freq_base = 160000
  hyper_connection.count           = 4
  hyper_connection.sinkhorn_iterations = 20
  hyper_connection.epsilon         = 1e-06
  hash_layer_count                 = 3

---

./pulsar-serve -m ~/Qwen3.5-9B-Q4_K_M.gguf --host '0.0.0.0' --port 8081 --ctx 4096
pulsar-serve: loading Qwen3.5-9B-Q4_K_M.gguf
pulsar: using CUDA device 0
pulsar: cudaMalloc(0.04 GB) failed on device 0
pulsar-serve: cuda kernel op failed: cudaMalloc

llama-server working

Architektura: qwen35
Tensorów: 427
Parametry: 9,0 mld (8 953 803 264)
Parametry aktywne: 9,0 mld (8 953 803 264)

Typy tensorów:
f32 tensorów: 177 parametry: 1,1 mln (1 061 376)
q4_K tensorów: 204 parametry: 6,3 mld (6 304 038 912)
q5_K tensorów: 24 parametry: 805,3 mln (805306368)
q6_K tensorów: 22 parametry: 1,8 mld (1 843 396 608)

Parametry architektury:
Liczba warstw = 32
Kontekst treningowy = 262144
Wymiar reprezentacji = 4096
Wymiar FFN = 12288
Liczba głów attention = 16
Liczba głów KV = 4
rope.dimension_sections = [11, 11, 10, 0]
Bazowa częstotliwość RoPE = 10000000
Epsilon RMSNorm = 1e-06
Wymiar głowy Q/K = 256
Wymiar głowy V = 256
ssm.conv_kernel = 4
ssm.state_size = 128
ssm.group_count = 16
ssm.time_step_rank = 32
ssm.inner_size = 4096
full_attention_interval = 4
Wymiar RoPE = 64

=========

I can run chat www only one model : 8ca8224c5519b50f286b9739f9ab1c2bc0759e02cfb6cbf48a09f0759eb2d319 Qwen3.6-35B-A3B-UD-Q5_K_M.gguf
RTX 3050 about 0.4 t/s from www, this is aceptable for agent but not for humans.

But only chat www, not agent!
I'cant run any agent with pulsar.

[Damian-o2]

Please install grok in yours machine and chceck himself. or claude cli.
I can run context 33K this is small but aceptable. (minimal is 64K)

[giannisanni]

Fixed on main in 03f13d0 — thanks for pasting the exact tensor name, that pinned it immediately.

pulsar-serve: blk.2.attn_compressor_kv.weight: expected f16/q8_0, got BF16

The unsloth UD-Q8_K_XL shards ship the compressor projections as BF16. BF16 and F16 are both two bytes wide but use a different exponent bias, so a loader that assumes f16 has to either refuse the file (what you hit) or silently decode noise. This is the same trap in a different function: read_tensor_bytes learned BF16 in c8d5190, but read_f16_as_q8 on the deepseek4 attention path had not. It now decodes BF16 through quant::bf16_to_f32 and quantizes to q8_0 exactly like the f16 arm, so nothing downstream changes.

Pull main and rebuild; that load error should be gone. I do not have that checkpoint on disk, so please tell me whether it gets past loading and actually generates.

On your other points, taken separately:

Context. 33K on a 3050 is the KV cache competing with the expert cache for 4.1GiB. PULSAR_KV=fp8 is already on by default in your log; if you want more context, the next lever is --ctx with a smaller expert cache, and long-context decode got much better recently (split-K flash decode). It will not get you to 64K on 4GiB of VRAM though; that is an honest hardware limit.

Agents and JSON. This is the useful thread. "Cannot run any agent" is not something I can act on directly, so could you open a separate issue with one concrete failing exchange: the request body your agent sends, and the raw response bytes pulsar returns? If the agent needs strict tool-call JSON, that is a specific parser gap I can fix. There is open work in that area right now (#29 adds DSML and Hy3 tool-call parsers), so a real failing case would help make sure it covers yours.

sha256 of known-good model files. Reasonable idea, but I would rather fix the loader for each format than publish a whitelist — a hash list goes stale the moment a repacker uploads a new quant, and "your file is not on the list" is a worse error message than the one you got. What would genuinely help is exactly what you did here: paste the failing tensor name and its type.

[Damian-o2]

I can create a agent trouble thread, but how show what pulsar send and what agent send?
I must wrote a proxy?
I have a request for you, too. If you want feedback, you need to add the --help and --version options.
or run Qwen3.8 27B on agent grok/claude etc.

---

(this same DeepSeek gguf file --port 8081 --ctx 4096 and commit 03f13d0)

pulsar-serve: loading ./unsloth/DeepSeek-V4-Flash-0731-GGUF/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf
pulsar: split gguf: 5 shards as one virtual file
pulsar: using CUDA device 0
pulsar-serve: blk.2.indexer.proj.weight: expected f16/f32, got BF16

[giannisanni]

Both of your points are fixed on main in c9e26b5.

The indexer BF16 load error. Same trap as the compressor one, in a different reader:

blk.2.indexer.proj.weight: expected f16/f32, got BF16

read_f16_as_q8 learned BF16 in 03f13d0, but read_f16_as_f32 (which loads the router, hyper-connection and indexer weights) still only knew F16 and F32. It now decodes BF16 through quant::bf16_to_f32 and produces identical f32 output, so nothing downstream changes.

I swept the rest of the tensor readers while I was in there rather than wait for a seventh report: upload_as_f32 and quant::row_to_f32 already handled BF16, so that was the last gap on the deepseek4 path.

--help and --version, as you asked. A build.rs now stamps the git sha into the binary, since a version number that never moves would not have helped you:

$ pulsar-cli --version
pulsar-cli 0.1.0 (b94e42c0525f)

--help on both binaries lists the flags and the PULSAR_* environment variables, which previously you could only find by reading the source.

On whether V4 Flash works. I bought disk and pulled the 0731 release, so I can now test this myself instead of asking you to be my hardware. On UD-Q2_K_XL it loads and generates:

pulsar: split gguf: 3 shards as one virtual file
pulsar: loaded in 6.8s (43 layers, 256 experts x top-6)
pulsar: 24 tokens in 3.39s (7.08 tok/s), vram cache 28% hits, tier 5114 resident slots

Two honest caveats. That is Q2_K_XL, not your Q8_K_XL, and Q2_K_XL does not happen to ship indexer.proj as BF16, so the specific fix above is not exercised by my copy. The decode function it routes to is the same one already in production elsewhere, but I would rather you confirm than take my word for it. Please pull main, rebuild, and paste what happens.

Also worth knowing for your context problem: this GGUF has no nextn block, so PULSAR_MTP=1 does nothing on it. And if 33K context is tight for you, PULSAR_KV=turbo4 is now the one to try. I measured the codecs properly this time rather than guessing from bit width, and on V4 Flash turbo4 holds the same top-1 agreement as q8_0 at half the bytes: docs/kv-codecs.md.

On the agent issue, a proxy is more work than you should have to do. If you can paste the raw request your agent sends and the raw response pulsar returns, that is enough. --webui-mcp-proxy logs tool calls, or curl -N against /v1/chat/completions with the same body your agent uses will show it. That belongs in its own issue though, since it has nothing to do with the serialization bug this one started as.

[Damian-o2]

pulsar-serve: loading ./unsloth/DeepSeek-V4-Flash-0731-GGUF/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf
pulsar: split gguf: 5 shards as one virtual file
pulsar: using CUDA device 0
pulsar: cudaMalloc(0.04 GB) failed on device 0
pulsar-serve: cuda kernel op failed: cudaMalloc
(context 4096)

ok, meybe too big. I start small qwen3.8 27B
I can run it in llama-server but it is verrry slowly. I can run gemma-4-26B-A4B-it-UD-Q5_K_M.gguf it is better.
But I cant run this 27B models on pulsar.

 ./pulsar-serve -m ./Qwen3.8-27B-Q4_K_M.gguf --ctx 4096 
pulsar-serve: loading ./Qwen3.8-27B-Q4_K_M.gguf
pulsar: using CUDA device 0
pulsar: cudaMalloc(0.03 GB) failed on device 0
pulsar-serve: cuda kernel op failed: cudaMalloc
 ./pulsar-serve --version
pulsar-serve 0.1.0 (f09551501668)

I download Q4K_M and Q5 from https://huggingface.co/bartowski/Qwen3.8-27B-GGUF/tree/main
Any idea how run it? Meybe add more information for debugging?

Please show me short list or working models on 8GiB VRAM.

P.S. please make similar option '-c' for context (in llama) no remowe old but add similar '--ctx' and '-c'.