[feat]: langsmith tracing integration#71
Conversation
|
Caution Review failedThe pull request is closed. WalkthroughThis update introduces a modular, agent-based backend architecture with a focus on developer relations (DevRel) automation. It adds a new Discord bot integrated with a queue-based orchestration system, agent state management, and specialized DevRel agent workflows. The FastAPI server is replaced by an asynchronous application coordinating agent execution and Discord interactions. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant DiscordBot
participant QueueManager
participant AgentCoordinator
participant DevRelAgent
User->>DiscordBot: Sends message in Discord
DiscordBot->>QueueManager: Enqueue message (with classification)
QueueManager->>AgentCoordinator: Dispatch devrel_request
AgentCoordinator->>DevRelAgent: Run agent with initial state
DevRelAgent->>DevRelAgent: Process workflow (intent, context, handlers)
DevRelAgent->>AgentCoordinator: Return response
AgentCoordinator->>QueueManager: Enqueue response for Discord
QueueManager->>DiscordBot: Deliver agent response
DiscordBot->>User: Post response in Discord thread
Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (25)
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hello @smokeyScraper, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
Summary of Changes
As Gemini, I've reviewed this pull request titled "[feat]: langsmith tracing integration". The primary intent of this PR is to integrate Langsmith tracing into the application's backend, providing better observability into the execution of the AI agents and their components. This integration is part of a larger effort to refactor the backend towards a more robust, agent-based, and queue-driven architecture.
The changes introduce a new structure for defining agents using LangGraph, implement a core DevRel agent with various nodes and tools (like FAQ and web search via Tavily), and establish an asynchronous queue manager and agent coordinator for handling incoming messages and orchestrating agent execution. The Discord bot has been updated to classify messages and enqueue them for processing by this new system. Configuration for Langsmith, Gemini, and Tavily APIs has been added, and dependencies have been updated accordingly.
Highlights
- Langsmith Tracing Integration: Adds Langsmith dependency, configuration options in
.env.exampleandapp/core/config.py, and applies@traceabledecorators across various agent nodes, tools, and orchestration components to enable detailed tracing of agent runs. - New Agent Architecture (LangGraph): Introduces a new
agentsdirectory structure withsharedanddevrelsubdirectories. Defines aBaseAgentclass and anAgentStatePydantic model. Implements theDevRelAgentusing LangGraph, defining a workflow with distinct nodes for intent classification, context gathering, handling specific request types (FAQ, web search, onboarding, technical support), and generating responses. - Asynchronous Queue and Orchestration: Adds
AsyncQueueManagerandAgentCoordinatorclasses inapp/core/orchestration. This establishes a queue-based system for handling incoming messages asynchronously and coordinating their processing by the appropriate agents based on classification and priority. - Discord Bot Integration: Updates the
DiscordBotto use the newClassificationRouterto classify incoming messages and enqueue them via theAsyncQueueManagerfor processing by the agent coordinator. It also registers a handler to receive and send responses back to Discord threads. - New Tools and Prompts: Adds
TavilySearchToolfor web search and a basicFAQTool. Includes new prompt templates (GENERAL_LLM_RESPONSE_PROMPT,EXTRACT_SEARCH_QUERY_PROMPT) used by the DevRel agent nodes. - Dependency and Configuration Updates: Adds necessary libraries like
langsmith,langgraph,langchain-tavily,tavily-python,langchain[google-genai], andpydantic-settingstopyproject.toml. Removeslangchain-groqandgroq. Updates.env.exampleandapp/core/config.pyto include API keys and settings for the new services. - Application Entry Point Change: Refactors
main.pyto replace the direct FastAPI/uvicorn run with a newDevRAIApplicationclass that manages the lifecycle of the queue manager, agent coordinator, and Discord bot usingasyncio.run.
Changelog
Click here to see the changelog
- backend/.env.example
- Added example environment variables for Langsmith tracing (LANGSMITH_TRACING, LANGSMITH_ENDPOINT, LANGSMITH_API_KEY, LANGSMITH_PROJECT).
- Added example variables for GEMINI_API_KEY and TAVILY_API_KEY.
- Commented out some previous variables.
- backend/app/agents/init.py
- Added imports for new base agent components and the DevRel agent.
- Updated
__all__list.
- backend/app/agents/devrel/agent.py
- New file: Defines the
DevRelAgentclass using LangGraph. - Implements the agent's workflow graph with various nodes.
- Integrates
ChatGoogleGenerativeAI,TavilySearchTool, andFAQTool. - Adds
@traceabledecorator to therunmethod and_route_to_handler.
- New file: Defines the
- backend/app/agents/devrel/nodes/classify_intent_node.py
- New file: Defines the
classify_intent_nodefunction. - Uses
ClassificationRouterto determine message intent. - Adds
@traceabledecorator.
- New file: Defines the
- backend/app/agents/devrel/nodes/gather_context_node.py
- New file: Defines the
gather_context_nodefunction. - Adds
@traceabledecorator.
- New file: Defines the
- backend/app/agents/devrel/nodes/generate_response_node.py
- New file: Defines the
generate_response_nodefunction and helpers. - Generates final responses based on task results (FAQ, web search, etc.).
- Adds
@traceabledecorators to all functions.
- New file: Defines the
- backend/app/agents/devrel/nodes/handle_faq_node.py
- New file: Defines the
handle_faq_nodefunction. - Uses the
faq_toolto get responses. - Adds
@traceabledecorator.
- New file: Defines the
- backend/app/agents/devrel/nodes/handle_onboarding_node.py
- New file: Defines the
handle_onboarding_nodefunction. - Adds
@traceabledecorator.
- New file: Defines the
- backend/app/agents/devrel/nodes/handle_technical_support_node.py
- New file: Defines the
handle_technical_support_nodefunction. - Adds
@traceabledecorator.
- New file: Defines the
- backend/app/agents/devrel/nodes/handle_web_search_node.py
- New file: Defines the
handle_web_search_nodefunction and helper. - Uses the
search_tool(Tavily) after extracting a query. - Adds
@traceabledecorators to both functions.
- New file: Defines the
- backend/app/agents/devrel/prompts/base_prompt.py
- New file: Defines the
GENERAL_LLM_RESPONSE_PROMPTstring.
- New file: Defines the
- backend/app/agents/devrel/prompts/search_prompt.py
- New file: Defines the
EXTRACT_SEARCH_QUERY_PROMPTstring.
- New file: Defines the
- backend/app/agents/devrel/state.py
- New file: Placeholder comment.
- backend/app/agents/devrel/tools/faq_tool.py
- New file: Defines the
FAQToolclass with hardcoded responses. - Includes basic fuzzy matching.
- Adds
@traceabledecorators to methods.
- New file: Defines the
- backend/app/agents/devrel/tools/search_tool.py
- New file: Defines the
TavilySearchToolclass. - Integrates with the Tavily API for web search.
- Adds
@traceabledecorator to thesearchmethod.
- New file: Defines the
- backend/app/agents/shared/base_agent.py
- New file: Defines the abstract
BaseAgentclass.
- New file: Defines the abstract
- backend/app/agents/shared/classification_router.py
- New file: Defines
MessageCategory,DevRelNeed, andClassificationRouter. - Implements message classification using LLM and pattern matching.
- Adds
@traceabledecorators to classification methods.
- New file: Defines
- backend/app/agents/shared/state.py
- New file: Defines the
AgentStatePydantic model for agent state management.
- New file: Defines the
- backend/app/core/config.py
- New file: Defines the
Settingsclass usingpydantic-settings. - Adds configuration fields for Gemini, Tavily, and Langsmith.
- Removes Groq-related configuration.
- New file: Defines the
- backend/app/core/orchestration/agent_coordinator.py
- New file: Defines the
AgentCoordinatorclass. - Manages agent instances and routes messages from the queue.
- Handles sending responses back to platforms.
- Adds
@traceabledecorators to handling methods.
- New file: Defines the
- backend/app/core/orchestration/queue_manager.py
- New file: Defines the
AsyncQueueManagerclass. - Implements an asynchronous priority queue for messages.
- Manages message handlers and worker tasks.
- New file: Defines the
- backend/bots/discord/discord_bot.py
- New file: Defines the
DiscordBotclass. - Uses the
ClassificationRouterandAsyncQueueManager. - Handles incoming messages, creates threads, and enqueues messages for agents.
- Handles responses from agents and sends them to Discord.
- Adds
!resetand!help_devrelcommands.
- New file: Defines the
- backend/main.py
- Replaced the previous FastAPI/uvicorn setup with a new
DevRAIApplicationclass. - Initializes and starts the
AsyncQueueManager,AgentCoordinator, andDiscordBot. - Configures logging and signal handling for graceful shutdown.
- Adds checks for required environment variables.
- The main entry point now uses
asyncio.runto start the application.
- Replaced the previous FastAPI/uvicorn setup with a new
- pyproject.toml
- Removed
langchain-groqandgroqdependencies. - Added
langgraph,langchain-tavily,tavily-python,langchain[google-genai],pydantic-settings, andlangsmithdependencies.
- Removed
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
|
Closing this PR, added way too much granular tracing leading to duplicated results. Screenshots
|
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive Langsmith tracing integration and a new DevRel agent powered by LangGraph, complete with a Discord bot interface. The architecture is well-thought-out, featuring an asynchronous queue manager, Pydantic-based configuration, and modular agent components. The code quality is high, with good use of async programming, type hints, and error handling. The Langsmith integration via @traceable decorators is correctly implemented.
Summary of Findings
- Agent Routing Logic: A critical issue was found in
DevRelAgentwhere the keys for conditional routing inadd_conditional_edgesdo not match the string values returned by the_route_to_handlerfunction. This needs to be corrected for the agent to route intents properly. - LLM JSON Parsing: The JSON parsing in
ClassificationRouter._parse_llm_responserelies on string searching for '{' and '}'. Using Langchain's dedicated output parsers could make this more robust. - Discord Thread Creation Fallback: In
DiscordBot._get_or_create_thread, if thread creation fails, it falls back to using the original channel ID as the thread ID. This could lead to responses appearing in the main channel unexpectedly. - f-string in Fallback Classification: In
backend/app/agents/shared/classification_router.pyline 190, thereasoningstring in_fallback_classificationappears to intend to use an f-string fororiginal_message[:50], but it's currently a literal string. It should bef"Fallback classification for message: {original_message[:50]}...". I did not comment on this inline due to the review settings.
Merge Readiness
This pull request introduces valuable Langsmith tracing and a powerful DevRel agent. However, there is a critical issue in the agent's routing logic that must be addressed before merging. Additionally, a couple of medium-severity issues related to JSON parsing and Discord thread handling should be considered for improvement.
I recommend that the critical issue be fixed, and the medium-severity issues be reviewed by the author. As a reviewer, I am not authorized to approve pull requests; please ensure further review and approval from authorized team members after addressing the feedback.
| workflow.add_conditional_edges( | ||
| "gather_context", | ||
| self._route_to_handler, | ||
| { | ||
| MessageCategory.FAQ: "handle_faq", | ||
| MessageCategory.WEB_SEARCH: "handle_web_search", | ||
| MessageCategory.ONBOARDING: "handle_onboarding", | ||
| MessageCategory.TECHNICAL_SUPPORT: "handle_technical_support", | ||
| MessageCategory.COMMUNITY_ENGAGEMENT: "handle_technical_support", | ||
| MessageCategory.DOCUMENTATION: "handle_technical_support", | ||
| MessageCategory.BUG_REPORT: "handle_technical_support", | ||
| MessageCategory.FEATURE_REQUEST: "handle_technical_support", | ||
| "default": "handle_technical_support" | ||
| } |
There was a problem hiding this comment.
There appears to be a mismatch in how conditional edges are defined. The _route_to_handler method (lines 84-101) returns string keys (e.g., "faq", "web_search") based on the intent. However, the dictionary provided to add_conditional_edges (lines 55-65) uses MessageCategory enum objects (e.g., MessageCategory.FAQ) as keys for most routes, instead of the strings returned by _route_to_handler.
According to LangGraph documentation, the keys in the mapping dictionary for add_conditional_edges must be the string outputs from the conditional function (_route_to_handler in this case).
This mismatch will likely cause the routing to not work as expected for enum-keyed routes, potentially always falling back to a default or failing if the returned string key isn't found.
Could you update the keys in the add_conditional_edges dictionary to be the string values that _route_to_handler returns (e.g., use "faq" instead of MessageCategory.FAQ)? The "default" key should also match the default string returned by _route_to_handler (which is "technical_support").
workflow.add_conditional_edges(
"gather_context",
self._route_to_handler,
{
"faq": "handle_faq",
"web_search": "handle_web_search",
"onboarding": "handle_onboarding",
"technical_support": "handle_technical_support", # This will catch explicit technical_support and defaults
# Ensure _route_to_handler returns these exact strings for the respective categories
# For categories that map to the same handler, ensure _route_to_handler returns "technical_support"
# e.g., COMMUNITY_ENGAGEMENT, DOCUMENTATION, BUG_REPORT, FEATURE_REQUEST should result in "technical_support" from _route_to_handler
}
)Style Guide References
| # Extract JSON from response | ||
| json_start = response.find('{') | ||
| json_end = response.rfind('}') + 1 | ||
|
|
||
| if json_start != -1 and json_end != -1: | ||
| json_str = response[json_start:json_end] | ||
| parsed = json.loads(json_str) | ||
| return parsed | ||
| raise ValueError("No JSON found in response") |
There was a problem hiding this comment.
The current method for parsing JSON from the LLM response by finding { and } characters can be a bit fragile if the LLM output isn't perfectly clean (e.g., includes preamble/postamble text outside the JSON block despite instructions).
Have you considered using Langchain's built-in output parsers, like JsonOutputParser or PydanticOutputParser? They are generally more robust at extracting and validating JSON from LLM responses and can simplify this parsing logic significantly. This would also align well with the Pydantic models used elsewhere.
Style Guide References
| except Exception as e: | ||
| logger.error(f"Failed to create thread: {str(e)}") | ||
|
|
||
| return str(message.channel.id) # Fallback to original channel |
There was a problem hiding this comment.
The fallback return str(message.channel.id) in _get_or_create_thread if thread creation fails (line 157) might lead to agent responses being sent to the main channel instead of a dedicated thread. This could be confusing for users if they expect interactions to be contained within threads.
Would it be more appropriate to perhaps notify the user in the original channel that thread creation failed and they should try again, or handle this error more explicitly rather than silently falling back to the channel ID as the thread_id? This would ensure that the agent's context and response delivery mechanism remain consistent with the threaded conversation model.
Style Guide References
1 participant
Summary by CodeRabbit
New Features
Bug Fixes
Chores