Upgrade to RubyLLM 1.16 and adopt new capabilities#34
Merged
Conversation
Bump the dependency to ruby_llm >= 1.16.0 and adopt four features that landed in 1.15/1.16, closing real gaps in cost analytics, tooling, and instrumentation: - Accurate cost: price cache-read/cache-write/reasoning components via RubyLLM::Cost on top of authoritative (attempt-aggregated) text tokens, folding them into total_cost and recording the breakdown in execution metadata. No schema change; the async job no longer recomputes a text-only total over an accurate one. - Concurrent tools: new inheritable tool_concurrency DSL on BaseAgent, passed as concurrency: to with_tools when set (else RubyLLM's global). - HTTP instrumentation: subscribe to ruby_llm's request.ruby_llm events during the LLM call and record real provider latency/count into execution metadata (llm_request_ms, llm_request_count). - Config passthroughs: forward bedrock/mistral/perplexity/vertexai/xai api_base, faraday_adapter, deprecation_behavior, and tool_concurrency. Also delete an orphan WIP spec (attachments_spec) that referenced a nonexistent DSL::Attachments module and crashed the suite at load time. Full suite: 4689 examples, 0 failures. standardrb clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- CHANGELOG: add Unreleased section for the four new capabilities - README: bump RubyLLM requirement to >= 1.16.0 - wiki/Configuration: new forwarded knobs (provider api_base, faraday_adapter, deprecation_behavior, tool_concurrency) - wiki/Agent-DSL + wiki/Tools: tool_concurrency DSL and concurrent execution - wiki/Pricing: cache- and reasoning-aware token costs + cost_breakdown - wiki/ActiveSupport-Notifications: RubyLLM library events and the captured llm_request_ms / llm_request_count latency metadata - wiki/Execution-Tracking: document the new metadata fields Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
adham90
added a commit
that referenced
this pull request
Jun 21, 2026
Five issues found reviewing the RubyLLM 1.16 upgrade (PR #34): 1. Scope llm_request_count/ms to the originating execution. The request.ruby_llm subscription was process-global, so the metric counted HTTP requests from concurrent threads and nested sub-agents. Now uses a fiber-local stack crediting only the innermost active capture on the emitting thread. 2. Stop double-charging separately-priced reasoning tokens. Providers fold reasoning tokens into output_tokens; when a model prices reasoning apart from output, those tokens are now excluded from the output charge since extra_token_costs already bills them at the reasoning rate. 3. Persist input_cost/output_cost and keep the cache-aware total. They were dropped on the async-logging path; now persisted, and total_cost is derived as input + output + metadata["cost_breakdown"] so the cache/reasoning extra is not clobbered by the before_save callback. 4. Reconcile cost_breakdown with total_cost by summing the rounded components instead of the unrounded sum. 5. Tighten bare rescues in extra_token_costs/response_cost to capture and debug-log the swallowed error. extra_component_cost ignores non-numeric breakdown values so user-supplied agent metadata can never break execution persistence. Full suite green (4696 examples), standardrb clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bumps the dependency to
ruby_llm >= 1.16.0and adopts four capabilities that landed in RubyLLM 1.15/1.16, closing real gaps in cost analytics, tooling, and instrumentation. Fully backwards compatible — no schema migration required.total_costnow prices prompt-cache reads, cache writes, and reasoning/thinking tokens viaRubyLLM::Coston top of authoritative (attempt-aggregated) text tokens. Per-component breakdown stored inexecutions.metadata["cost_breakdown"]. The asyncExecutionLoggerJobno longer recomputes a text-only total over an accurate one.tool_concurrencyDSL onBaseAgent(false/true/:threads/:fibers); passed asconcurrency:towith_toolswhen set, else RubyLLM's global default.request.ruby_llmevents during the LLM call and records real provider latency + count intoexecutions.metadata(llm_request_ms,llm_request_count), distinct from total pipeline duration.bedrock/mistral/perplexity/vertexai/xai_api_base,faraday_adapter,deprecation_behavior, andtool_concurrencytoRubyLLM.config.Already-supported 1.15/1.16 features (image editing, transcription word timing, additive callbacks) needed no changes.
Design notes
context.total_costdirectly, so making it accurate flows straight through the existingtotal_costcolumn. The cache/reasoning breakdown lives inmetadatarather than new columns, avoiding a host-app migration. Promoting the breakdown to dedicated cost columns + dashboard display is noted as a follow-up.attachments_spec) that referenced a nonexistentDSL::Attachmentsmodule and crashed the suite at load time.Testing
standardrbclean.Docs updated: CHANGELOG, README requirement, and the Configuration / Agent-DSL / Tools / Pricing / ActiveSupport-Notifications / Execution-Tracking wiki pages. An internal ADR is in
changelog/.🤖 Generated with Claude Code