ROS Package bob_llm

The bob_llm package provides a ROS 2 node (llm node) that acts as an autonomous AI agent. Rather than just wrapping an external Large Language Model (LLM) API, the node implements a complete runtime environment for agentic workflows. It manages conversational state, integrates standard tool calls, and hosts a dynamic skill execution engine capable of discovering, loading, and executing self-contained skills at runtime.

Features

• OpenAI-Compatible: Connects to any LLM backend that exposes an OpenAI-compatible API endpoint (e.g., Ollama, vLLM, llama-cpp-python, commercial APIs).
• Stateful Conversation: Maintains chat history to provide conversational context to the LLM.
• Dynamic Tool System: Dynamically loads Python functions from user-provided modules and exposes them to the LLM. The model can selectively trigger these tools to execute shell scripts, interface with Qdrant vector databases, or control local hardware.
• Autonomous Skill Execution: Implements the Anthropic Agent Skills standard, allowing the agent to dynamically discover available skills, inspect their implementation (SKILL.md), write new scripts (.sh or .py), and execute them in a multi-directory configuration (separating protected core skills from writable user sandboxes).
• High Performance Streaming: Optimized byte-stream parsing ensures zero-latency delivery of reasoning tokens and response chunks directly from the socket (no internal buffering).
• Reasoning/Thinking Support: Real-time extraction and publishing of model reasoning (e.g., from Gemma 2 or DeepSeek) to a dedicated topic.
• Interactive Chat CLI: Includes a premium terminal interface with Markdown rendering and multi-line support.
• Multi-modality: Supports multimodal input (e.g., images) via JSON prompts.
• Lightweight: The node core requires only standard Python libraries (requests, rich, prompt_toolkit).
• Multi-arch Docker Support: Ready-to-use Docker images for amd64 and arm64, fully configurable via environment variables for easy deployment.

Docker Usage

The bob_llm node is available as a multi-arch Docker image. All ROS parameters can be configured via environment variables (prefixed with LLM_).

Running with Docker

docker run -it --rm \
  --name bob-llm \
  -e LLM_API_URL="http://192.168.1.100:8000/v1" \
  -e LLM_API_KEY="your_secret_token" \
  -e LLM_API_MODEL="llama3" \
  -e LLM_TEMPERATURE="0.5" \
  ghcr.io/bob-ros2/bob-llm:latest

Running with Docker Compose

services:
  llm:
    image: ghcr.io/bob-ros2/bob-llm:latest
    container_name: bob-llm
    environment:
      - LLM_API_URL=http://llm-backend:8000/v1
      - LLM_API_KEY=sk-12345
      - LLM_API_MODEL=gpt-4
      - LLM_SYSTEM_PROMPT="You are a helpful robot assistant named Bob."
      - LLM_TEMPERATURE=0.8
    restart: always

Installation

1. Clone the Repository Navigate to your ROS 2 workspace's src directory and clone the repository:

   cd ~/ros2_ws/src
   git clone https://github.com/bob-ros2/bob_llm.git

2. Install Dependencies The node requires a few Python packages. It is recommended to install these within a virtual environment.

   pip install requests PyYAML rich prompt_toolkit

3. Build and Source

   cd ~/ros2_ws
   colcon build --packages-select bob_llm
   source install/setup.bash

Usage

1. Start the Brain (LLM Node)

Ensure your LLM server is active and the api_url in your params file is correct.

ros2 run bob_llm llm --ros-args --params-file /path/to/your/ros2_ws/src/bob_llm/config/node_params.yaml

2. Enter Interactive Chat

Interact with Bob through a dedicated, interactive terminal client.

# Start standard chat
ros2 run bob_llm chat

# Start with boxed UI (visual panels)
ros2 run bob_llm chat --panels

CLI Arguments for chat

usage: ros2 run bob_llm chat [-h] [--topic_in TOPIC_IN] [--topic_out TOPIC_OUT]
                             [--topic_response TOPIC_RESPONSE]
                             [--topic_tools TOPIC_TOOLS]
                             [--topic_reasoning TOPIC_REASONING] [--panels]

options:
  -h, --help            show this help message and exit
  --topic_in TOPIC_IN   ROS Topic to send prompts to (default: llm_prompt)
  --topic_out TOPIC_OUT
                        ROS Topic to receive streamed responses (default: llm_stream)
  --topic_response TOPIC_RESPONSE
                        ROS Topic to receive final complete responses (default: llm_response)
  --topic_tools TOPIC_TOOLS
                        ROS Topic to receive tool call notifications (default: llm_tool_calls)
  --topic_reasoning TOPIC_REASONING
                        ROS Topic to receive model reasoning content (default: llm_reasoning)
  --panels              Enable boxed UI (default: off)

Chat Configuration

The chat client supports the following ROS parameters and environment variables:

• queue_size (Integer): ROS parameter to control the subscription queue depth.
• CHAT_QUEUE_SIZE (Environment Variable): Default value for the queue_size parameter (default: 1000).

Chat Example

Chat for https://github.com/bob-ros2/bob_llm
Usage: Press Enter to send, or Alt+Enter for a new line.

YOU: Was kannst du über dieses System sagen?

[*] SKILL: list_nodes({})

LLM: Ich sehe folgende aktive Komponenten im System:
- /llm (Das Gehirn)
- /bob_chat_client (Dieser Chat)
- /eva/logic (Zustandssteuerung)

3. Advanced Input & Multi-modality

The node supports advanced input formats beyond simple text. If the input message on /llm_prompt is valid JSON, it is parsed as a message object.

Generic JSON Input: You can pass any valid JSON dictionary. If it contains a role field (e.g., user), it is treated as a standard message object and appended to the history.

Image Helper: If process_image_urls is enabled, the node automatically base64-encodes images from file:// or http:// URLs.

ros2 topic pub /llm_prompt std_msgs/msg/String "data: '{\"role\": \"user\", \"content\": \"Describe this\", \"image_url\": \"file:///tmp/cam.jpg\"}'" -1

ROS 2 API

Topic	Type	Description
/llm_prompt	std_msgs/msg/String	(Subscribed) Receives user prompts.
/llm_response	std_msgs/msg/String	(Published) Final, complete response from the LLM.
/llm_stream	std_msgs/msg/String	(Published) token-by-token chunks of the response.
/llm_reasoning	std_msgs/msg/String	(Published) Live reasoning/thinking content from the model.
/llm_tool_calls	std_msgs/msg/String	(Published) JSON info about tool execution for clients.
/llm_latest_turn	std_msgs/msg/String	(Published) Latest turn as JSON array of messages.
/llm_stats	std_msgs/msg/String	(Published) Execution stats like token count and generation speed.

Configuration

The node is configured through a ROS parameters YAML file. Most parameters support dynamic reconfiguration at runtime.

Parameter	Type	Default	Description
api_type	string	openai_compatible	The type of the LLM backend API (e.g., "openai_compatible"). [ENV: LLM_API_TYPE]
api_url	string	http://localhost:8000/v1	The base URL of the LLM backend API. [ENV: LLM_API_URL]
api_key	string	no_key	The API key for authentication with the LLM backend. [ENV: LLM_API_KEY]
api_model	string	""	The specific model name to use (e.g., 'gpt-4', 'llama3'). [ENV: LLM_API_MODEL]
system_prompt	string	""	The system prompt to set the LLM context. [ENV: LLM_SYSTEM_PROMPT]
system_prompt_file	string	""	Path to a file containing the system prompt. [ENV: LLM_SYSTEM_PROMPT_FILE]
initial_messages_json	string	[]	A JSON string of initial messages for few-shot prompting. [ENV: LLM_INITIAL_MESSAGES_JSON]
max_history_length	integer	10	Max turns to keep in history. [ENV: LLM_MAX_HISTORY_LENGTH] Range: [0, 1000]
max_tool_calls	integer	5	Max consecutive tool calls allowed. [ENV: LLM_MAX_TOOL_CALLS] Range: [0, 50]
stream	bool	true	Enable/disable streaming for the final LLM response. [ENV: LLM_STREAM]
temperature	double	0.7	Controls the randomness of the output. [ENV: LLM_TEMPERATURE] Range: [0.0, 2.0]
top_p	double	1.0	Nucleus sampling diversity control. [ENV: LLM_TOP_P] Range: [0.0, 1.0]
max_tokens	integer	0	Max tokens to generate. 0 means no limit. [ENV: LLM_MAX_TOKENS]
stop	string array	['stop_llm']	A list of sequences to stop generation at. [ENV: LLM_STOP]
presence_penalty	double	0.0	Penalizes new tokens based on presence. [ENV: LLM_PRESENCE_PENALTY] Range: [-2.0, 2.0]
frequency_penalty	double	0.0	Penalizes new tokens based on frequency. [ENV: LLM_FREQUENCY_PENALTY] Range: [-2.0, 2.0]
api_timeout	double	120.0	Timeout in seconds for API requests. [ENV: LLM_API_TIMEOUT]
tool_interfaces	string array	[]	A list of Python modules or file paths to load as tools. [ENV: LLM_TOOL_INTERFACES]
skill_dir	string	./config/skills	Directory where skills are stored. Supports a comma-separated list of paths for dual-loading. [ENV: LLM_SKILL_DIR]
message_log	string	""	If set, appends conversation turns to this JSON file. [ENV: LLM_MESSAGE_LOG]
process_image_urls	bool	false	If true, processes image_url in JSON prompts. [ENV: LLM_PROCESS_IMAGE_URLS]
response_format	string	""	JSON string defining the output format. [ENV: LLM_RESPONSE_FORMAT]
tool_choice	string	auto	Tool calling behavior ('auto', 'none', 'required'). [ENV: LLM_TOOL_CHOICE]
tool_timeout	double	60.0	Maximum time in seconds to wait for a tool to execute. [ENV: LLM_TOOL_TIMEOUT]
eof	string	""	Optional string to publish on llm_stream when generation is finished. [ENV: LLM_EOF]
model_context_limit	integer	0	The total context window size (limit) of the model in tokens. [ENV: LLM_MODEL_CONTEXT_LIMIT]
stream_options_include_usage	bool	true	Request usage details from the API in the stream options. [ENV: LLM_STREAM_OPTIONS_INCLUDE_USAGE]
stats_mode	integer	0	Statistics publishing mode: 0 for completed only, 1 for all. [ENV: LLM_STATS_MODE]

Security Note: API Key Cloaking

For security reasons, the api_key parameter is "cloaked" immediately after the node initializes. Once the key has been read into the node's internal memory, the public ROS parameter is cleared. This prevents the key from being accidentally exposed or read via ros2 param get /llm api_key.

Tip

For production environments, it is best practice to use an API Gateway or Reverse Proxy to inject authentication tokens. This avoids passing sensitive keys through the ROS parameter system entirely.

Structured JSON Output

The response_format parameter enables structured output, forcing the LLM to respond with valid JSON.

JSON Object Mode

Force the LLM to output valid JSON by setting response_format to {"type": "json_object"}. Note: Your prompt must mention the word "JSON".

JSON Schema Mode (Strict)

Define an exact schema the LLM must follow:

response_format: |
  {
    "type": "json_schema",
    "json_schema": {
      "name": "robot_command",
      "strict": true,
      "schema": {
        "type": "object",
        "properties": {
          "action": {"type": "string", "enum": ["move", "stop"]},
          "speed": {"type": "number"}
        },
        "required": ["action"]
      }
    }
  }

Conversation Logging

Set the message_log parameter to an absolute file path (e.g., /home/user/chat.json) to save the entire conversation history.

---

Tool System

Creating a Tool File

A tool file is a standard Python script containing functions with docstrings. The system automatically mirrors these functions as tools for the LLM.

Skill System (Agentskills)

The bob_llm node implements the Anthropic Agent Skills specification.

1. Skill Discovery

Add the path of config/skill_tools.py to your tool_interfaces to enable the skill discovery API (load_skill_info, execute_skill_script, etc.).

Configuration: Ensure the skill_dir parameter points to a valid directory where your Agentskills are stored. A sample collection of skills is provided in the ./config/skills directory of the package.

Dual Skill Loading Architecture: The skill_dir parameter supports a comma-separated list of paths (e.g., /ros2_ws/src/core_skills,/home/user/learned_skills). When multiple paths are provided:

• The node will seamlessly merge and load skills from all configured directories.
• The agent is strictly prevented from overwriting or deleting skills that reside in any "core" directory (all paths except the last one).
• New skills created by the agent will always be written exclusively to the last directory in the list, allowing for a protected core and a writable memory sandbox.

2. Ready-to-use Tools

• ROS CLI Tools (config/ros_cli_tools.py): Inspect and control the ROS system.
• Qdrant Memory Tools (config/qdrant_tools.py): Semantic long-term memory.
  • Configured via LLM_QDRANT_LOCATION, LLM_QDRANT_COLLECTION, etc.

---

Development & Integration Testing

The package includes a comprehensive integration test to verify the full flow (Reasoning, Streaming, Tool Calls) against a real LLM provider. This test is not run automatically by colcon test as it requires an external API key and internet access.

1. Setup Environment

Navigate to the tests directory and create a .env file from the template:

cd src/bob_llm/tests
cp .env.template .env

Edit the .env file and configure the following:

• LLM_API_KEY: Your real API key (e.g., from DeepSeek or OpenAI).
• LLM_TOOL_INTERFACES: This must be an absolute path to your tool configuration (e.g., /home/user/ros2_ws/src/bob_llm/config/ros_cli_tools.py).

2. Run the Test

Use pytest directly to run the integration test. The -s flag is recommended to see the beautiful rendered output.

# From the workspace root
source install/setup.bash
pytest src/bob_llm/tests/test_llm_integration.py -s -v

The test will:

1. Launch the llm node in an isolated background process (using a random ROS_DOMAIN_ID to avoid collisions).
2. Send a complex prompt requiring tool use.
3. Verify that reasoning, streaming, and tool execution work as expected.
4. Print a clean, colored log of the interaction.