Add typed output schemas for CrewAI tools - #6236
Conversation
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds first-class Pydantic ChangesTyped Tool Outputs
TavilyResearchTool Schema Field Alignment
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
lib/crewai/tests/tools/test_tool_usage.py (1)
241-244: ⚡ Quick winAvoid asserting exact JSON string formatting in hook-result expectations.
This currently couples the test to compact serializer formatting (
{"query":"crew","score":0.7}) rather than the JSON payload itself. Parsecontext.tool_resultbefore comparison to keep the test stable across harmless formatter changes.Suggested test assertion adjustment
- assert seen_results == [ - ('{"query":"crew","score":0.7}', SearchOutput(query="crew", score=0.7)), - ('{"query":"crew","score":0.7}', SearchOutput(query="crew", score=0.7)), - ] + assert len(seen_results) == 2 + for tool_result, raw_tool_result in seen_results: + assert json.loads(tool_result or "") == {"query": "crew", "score": 0.7} + assert raw_tool_result == SearchOutput(query="crew", score=0.7)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai/tests/tools/test_tool_usage.py` around lines 241 - 244, The test assertion at line 241-244 is coupling the test to exact JSON string formatting by comparing the compact JSON string directly. Instead of asserting on the raw JSON string format, parse the JSON strings in the seen_results list before comparison. For each tuple in seen_results, convert the first element (the JSON string) to a parsed dictionary using json.loads() and compare the parsed objects rather than the exact string formatting. This keeps the test stable across formatter changes while still validating the actual JSON payload content.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/crewai/src/crewai/tools/structured_tool.py`:
- Around line 72-80: In the exception handler where the validation warning is
raised, the warning message currently includes the full exception object using
{exc} which can expose sensitive data. Replace {exc} with
{exc.__class__.__name__} in the f-string on the line that constructs the warning
message (within the warnings.warn call) so that only the exception type name is
logged instead of the full exception details that may contain raw tool output,
secrets, or PII.
---
Nitpick comments:
In `@lib/crewai/tests/tools/test_tool_usage.py`:
- Around line 241-244: The test assertion at line 241-244 is coupling the test
to exact JSON string formatting by comparing the compact JSON string directly.
Instead of asserting on the raw JSON string format, parse the JSON strings in
the seen_results list before comparison. For each tuple in seen_results, convert
the first element (the JSON string) to a parsed dictionary using json.loads()
and compare the parsed objects rather than the exact string formatting. This
keeps the test stable across formatter changes while still validating the actual
JSON payload content.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 2fa4a316-135f-4588-aaa2-a75977ee5565
📒 Files selected for processing (20)
docs/edge/en/concepts/tools.mdxdocs/edge/en/guides/tools/publish-custom-tools.mdxdocs/edge/en/learn/create-custom-tools.mdxdocs/edge/en/learn/execution-hooks.mdxdocs/edge/en/learn/tool-hooks.mdxlib/crewai/src/crewai/agents/crew_agent_executor.pylib/crewai/src/crewai/agents/tools_handler.pylib/crewai/src/crewai/experimental/agent_executor.pylib/crewai/src/crewai/hooks/tool_hooks.pylib/crewai/src/crewai/tools/base_tool.pylib/crewai/src/crewai/tools/structured_tool.pylib/crewai/src/crewai/tools/tool_usage.pylib/crewai/src/crewai/utilities/agent_utils.pylib/crewai/src/crewai/utilities/tool_utils.pylib/crewai/tests/agents/test_native_tool_calling.pylib/crewai/tests/hooks/test_tool_hooks.pylib/crewai/tests/tools/test_base_tool.pylib/crewai/tests/tools/test_structured_tool.pylib/crewai/tests/tools/test_tool_usage.pylib/crewai/tests/utilities/test_agent_utils.py
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@lib/crewai-tools/src/crewai_tools/tools/tavily_research_tool/tavily_research_tool.py`:
- Around line 90-92: The current logic in the output_schema handling block only
renames output_schema to tavily_output_schema when tavily_output_schema is not
already present, but fails to handle the case when both are provided. When both
output_schema and tavily_output_schema exist in kwargs, the legacy output_schema
dict remains in kwargs and causes BaseTool.__init__ to fail due to type
incompatibility. Modify the conditional logic to check if tavily_output_schema
exists in kwargs and remove the legacy output_schema in that case, then only
perform the remapping of output_schema to tavily_output_schema when
tavily_output_schema is absent from kwargs.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 2ac50565-192f-422e-9712-f4ddd7d2dd4b
📒 Files selected for processing (3)
docs/edge/en/tools/search-research/tavilyresearchtool.mdxlib/crewai-tools/src/crewai_tools/tools/tavily_research_tool/tavily_research_tool.pylib/crewai-tools/tool.specs.json
b0394d1 to
23b2df7
Compare
Tools can now declare an `output_schema`, set explicitly or inferred from a Pydantic return annotation. `format_output_for_agent` validates the raw result against it and serializes to JSON for the agent, while `run` keeps returning the raw value. Falls back to `str(raw_result)` with a warning when validation or serialization fails.
Tools with an `output_schema` returned a Python repr to the agent instead of clean JSON. Send every tool result through `format_output_for_agent` so the agent reads valid JSON, across all executors and `ToolUsage`. The cache still stores the raw result, so cache callbacks keep getting the original typed object.
Typed tools format their output to JSON before handing it to the agent, so `tool_result` reaches after hooks as a string and the original Python object is lost. Thread the unformatted result through as `raw_tool_result` so hooks can inspect the typed value. This covers all execution paths: native tool calls, the ReAct `ToolUsage` path, cached results, and error/blocked branches.
When a `BaseTool` subclass overrides `format_output_for_agent`, route the agent-facing text through it instead of the default JSON/`str()` serialization. The structured tool wrapper now delegates to the original tool via `_original_tool`, so a tool can present Markdown or any custom representation to the agent while `tool.run(...)` still returns the raw Python value.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit f6c971d. Configure here.
|
Hi Vinicius, I maintain agent-eval (agent_regress), an OSS stats-based regression harness for agent behavior. I tried to build a real before-and-after test of this PR using the tool-level runner in agent-eval (crewai_tool_runner() + schema_conformance_scorer()), and I found something worth flagging. As currently implemented, it does not actually see this change. The crewai_tool_runner() calls tool.run() or tool._run() directly and scores the raw result. However, your result_schema validate-or-fallback logic lives one layer downstream in ToolUsage, which calls the new tool.format_output_for_agent(). The .run() and ._run() methods never touch this. I installed crewai==1.14.7 (pre-merge) and 1.15.0 (post-merge) and confirmed this empirically. Both versions produced identical raw tool output and identical schema-conformance scores, yielding zero detectable signal. The PR itself works exactly as documented once I called format_output_for_agent() directly. A conforming dict round-trips to compact JSON, and a non-conforming one fires a real RuntimeWarning naming the tool and schema before falling back to str(). I confirmed this against both the happy and failure paths. This is not a knock on the PR; it is a gap on my end. I am going to update crewai_tool_runner() to optionally route through format_output_for_agent() so that tool-level regression tests actually exercise this code path. I am flagging it here since #6236 is what surfaced this issue. You can find the repository here: https://github.com/RudrenduPaul/agent-eval Best, |
3 participants
Currently, tools have a strong input contract through
args_schema, but no output contract. This means that anything a tool outputs is converted to string.Not only the contract is weak, but the "invisible" conversion to string can have unexpected effects when the tool returns complex objects like dicts and arrays.
With this PR, a tool can optionally define an output contract with
output_schema. CrewAI validates the raw result and sends the agent JSON.If the result does not match the schema, CrewAI warns and falls back to
str(raw_result)instead of failing the run:This is additive and non-breaking. Existing tools do not need to change. Tools without
output_schemakeep the old string behavior. Invalid typed outputs warn and fall back to the old formatting path.Note
Medium Risk
Changes span shared tool execution, caching, and hook context across multiple executors; behavior is intended to be backward compatible but alters how non-string tool results reach agents.
Overview
Optional typed tool outputs mirror input
args_schema: tools can declareresult_schema(or infer it from a Pydantic_runreturn type), validate results, and send JSON strings to agents whiletool.run()still returns the raw value.Execution paths (ReAct
ToolUsage, native tool calling increw_agent_executor/agent_utils, experimental executor) now useformat_output_for_agentinstead of blindstr()conversion.@toolacceptsresult_schema;BaseToolexposes overridable formatting for Markdown/summaries. Invalid or unserializable typed output warns and falls back tostr(raw_result).Tool hooks gain
raw_tool_resultalongside agent-facingtool_result; caching andToolsHandler.on_tool_usestore raw outputs. Docs cover typed outputs, publishing, and hook semantics.Reviewed by Cursor Bugbot for commit 1c327a2. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by CodeRabbit
Release Notes
New Features
Documentation