/api/generate and /api/chat return duration fields in inconsistent units (breaks Ollama-compatible tok/s calculations)

[Hi5808]

Summary

The real Ollama API returns every *_duration field in nanoseconds — that's the documented contract client tooling relies on (e.g. tokens_per_second = eval_count / (eval_duration / 1e9)). rkllama's /api/generate and /api/chat responses don't follow that consistently:

• total_duration — nanoseconds (matches Ollama's convention, correct)
• eval_duration, prompt_eval_duration — milliseconds, not nanoseconds
• load_duration — a hardcoded constant (100000000) returned on every single request regardless of how long loading actually took — not a real measurement at all

Any client written against the documented Ollama API contract (e.g. anything computing eval_count / (eval_duration / 1e9)) will compute a tokens/sec figure that's off by 6 orders of magnitude against rkllama.

Environment

• rkllama version: 0.0.75
• Board: Radxa ROCK 5B+ (RK3588), Armbian 26.8.1, kernel 6.1.115-vendor-rk35xx

Steps to reproduce

curl -s -X POST http://localhost:8080/api/chat -d '{
  "model": "<any loaded model>",
  "messages": [{"role":"user","content":"Explain what a neural network is in two sentences."}],
  "stream": false
}' | python3 -m json.tool

Actual response (example, TinyLlama-1.1B)

{
  "eval_count": 101,
  "eval_duration": 12158,
  "load_duration": 100000000,
  "prompt_eval_count": 27,
  "prompt_eval_duration": 352,
  "total_duration": 12536301851
}

total_duration (12536301851 ns ≈ 12.54s) is consistent with wall-clock time. But eval_duration (12158) is only consistent with wall-clock time if interpreted as milliseconds (12.158s), not nanoseconds (which would be 0.000012s). Same for prompt_eval_duration (352 → 0.352s as milliseconds, not nanoseconds). load_duration is identical (100000000) across every request we tested regardless of actual load time, which was frequently several seconds in practice (tokenizer download + model init) — it appears to be a stub/placeholder value rather than a measurement.

Expected behavior

All *_duration fields should be in nanoseconds, matching Ollama's documented API and what total_duration already correctly does. load_duration should reflect actual measured load time, or be omitted/clearly marked as unavailable rather than returning a fixed fake value.

Suggested fix

In whatever code path builds the /api/generate / /api/chat response (looks like it's using the raw millisecond values from the RKLLM runtime's internal timers directly), multiply eval_duration and prompt_eval_duration by 1_000_000 before returning them, to match total_duration's existing nanosecond scale. For load_duration, instrument the actual model-load call rather than returning a constant.

Additional context

Full writeup with the underlying benchmark data (this bug is called out as Finding 2): https://claude.ai/code/artifact/09b27c30-c837-4204-8941-1d107ee75c9d

[danielferr85]

we will fix it