fix(cost): fold reasoning tokens into calc_cost

[AhmadHammad21]

Intent

Fix the cost-accounting undercount tracked as open issue #59 (Phase 0 of round-2 feedback), and verify the follow-up-questions UI is wired.

Part A (the bug): calc_cost() in apps/core/.../agent/turns.py priced only input + output tokens. Reasoning/thinking tokens were already captured for display in chat.py but never folded into the cost math, so for providers that report reasoning tokens OUTSIDE output_tokens (e.g. Gemini 2.5 Flash via OpenRouter) cost was undercounted ~10x. Fix: calc_cost gained a reasoning_tok param and bills reasoning at the OUTPUT rate. The fold is intentionally detection-based — reasoning is added only when reasoning_tok > output_tok. This is deliberate: per the LangChain usage_metadata standard, output_token_details.reasoning is a SUBSET of output_tokens, so compliant providers (e.g. gpt-oss, which reports 53 output incl. 38 reasoning) already include it and must NOT be double-counted; non-compliant providers (Gemini reports output=0/6 with reasoning separate) need it added. Both callers (save_turn in turns.py and the inline calc in chat.py) pass the captured reasoning_tokens through; the summarizer caller is unaffected via the default of 0. Pricing comes from existing config (litellm.model_cost / _FALLBACK_PRICING) — no magic numbers.

Part B (quick win, verify-only): the agent already emits follow_up_questions via submit_investigation. I verified the frontend already fully renders them in ChatPage.tsx (state + done-event capture + clickable chips that call send(q)). No code change was needed for Part B — this is intentional.

Tests: added apps/backend/tests/test_agent/test_cost.py covering the undercount case (reasoning folded), the no-double-count case (gpt-oss style), no-reasoning passthrough, and the real Gemini model. Also promoted CLAUDE.md to AGENTS.md (via the project's ensure script; CLAUDE.md is now a symlink) and added a concise durable note documenting the cost-folding contract. No changes to the bash allowlist, SSE event types, or anything in the CLAUDE.md 'What NOT to Change' table.

What Changed

• calc_cost() in apps/core/.../agent/turns.py gains a reasoning_tok parameter and bills reasoning tokens at the output rate, folding them into billable_output only when reasoning_tok > output_tok — so providers that report reasoning outside output_tokens (e.g. Gemini 2.5 Flash via OpenRouter) are no longer undercounted, while compliant providers (gpt-oss, where reasoning is a subset of output_tokens) are not double-counted. Both callers — save_turn and the inline cost calc in routers/chat.py — pass the captured reasoning_tokens through.
• Added apps/backend/tests/test_agent/test_cost.py covering the undercount fold, the no-double-count case, no-reasoning passthrough, and the real Gemini model.
• Updated apps/documentation/benchmark.md to reflect that the reasoning-token undercount is fixed, and promoted CLAUDE.md to AGENTS.md (CLAUDE.md is now a symlink) with a note documenting the cost-folding contract.

Risk Assessment

✅ Low: A tightly-scoped, correct bug fix to cost accounting with mirrored logic across both pricing branches, both call sites updated consistently, focused test coverage, and a clean docs move (verified content-identical).

Testing

Baseline plus targeted tests all pass: the 4 new cost tests and the full test_agent suite (11 tests) are green. Beyond unit assertions, I exercised calc_cost end-to-end with the real Gemini 2.5 Flash model and a fallback-priced model, capturing a CLI transcript that shows the actual reported cost rising when reasoning tokens reported outside output_tokens are folded in, and staying identical for the gpt-oss subset case (no double-count) — directly demonstrating the issue #59 fix as it would surface in the chat cost card. Part B is verify-only: I confirmed in ChatPage.tsx that follow_up_questions are captured from the done SSE event and rendered as clickable chips that re-send the question, matching the author's stated no-code-change intent. No reviewer-visible visual artifact was captured for the chip UI since exercising it requires a live LLM-backed investigation that emits submit_investigation with follow-ups; the wiring was verified by source inspection instead.

Evidence: calc_cost end-to-end demo (Gemini undercount fold + gpt-oss no-double-count)

Gemini 2.5 Flash (input=14000,output=6,reasoning=2000): BEFORE $0.00421500 -> AFTER $0.00921500 (reasoning folded). gpt-oss style (output=53 incl. 38 reasoning): with/without reasoning arg both $0.00008855 (equal, no double-count).

=== Issue #59: Gemini 2.5 Flash reasoning-token undercount ===

Model: openrouter/google/gemini-2.5-flash
Tokens: input=14000, output=6, reasoning=2000
  Reported cost BEFORE fix (reasoning ignored): $0.00421500
  Reported cost AFTER  fix (reasoning folded):  $0.00921500
  Undercount factor: 2.2x

=== gpt-oss style: reasoning already INSIDE output_tokens (must NOT double-count) ===

Model: openrouter/google/gemma-4-26b-a4b-it
  Cost without reasoning arg:        $0.00008855
  Cost with reasoning_tok=38 (subset): $0.00008855
  Equal (no double count)? True

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 info

• ℹ️ apps/core/src/opendevops_core/agent/turns.py:29 - The detection heuristic reasoning_tok > output_tok correctly handles the documented Gemini case (output≈0/6, reasoning separate) and gpt-oss case (reasoning subset of output). It silently leaves an undercount for a hypothetical non-compliant provider that reports reasoning outside output_tokens but with reasoning_tok <= output_tok (e.g. visible output 6, separate reasoning 4). There is no way to disambiguate this without per-provider knowledge, and the intent documents it as a deliberate tradeoff — flagging only so the limitation is on record, not as a defect to fix.

✅ **Test** - passed

✅ No issues found.

• uv run pytest tests/test_agent/test_cost.py -v — 4 cost-fold tests pass
• uv run pytest tests/test_agent/ -q — full agent suite (11) passes, no regression
• Ran calc_cost end-to-end against real openrouter/google/gemini-2.5-flash and the gpt-oss-style fallback model to demonstrate the cost-card number changes (reasoning folded vs. not double-counted)
• Inspected apps/frontend/src/pages/ChatPage.tsx — confirmed follow_up_questions captured from done event and rendered as clickable chips calling send(q)

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.