VISION-ENABLED DIALOGUE

A dialogue system that combines speech and images in a conversation history with summarisation strategies to improve contextual understanding. It uses a configurable (for example, text-only or multimodal) language model to generate responses. It can be used as a Python library or as a ROS 2 node.

This repository contains the implementation described in the paper "I Was Blind but Now I See: Implementing Vision-Enabled Dialogue in Social Robots" by Giulio Antonio Abbo and Tony Belpaeme (March 2025) available here. Link to the HRI'25 LBR paper. For the version of the code used in the paper, see the v0.2.0.

Contents

• Features
• Installation
• Usage - Get started
• Usage - Generating responses
• Usage - Rendering outputs and handling interruptions
• Usage - Manage prompt length
• ROS 2 compatibility
• Furhat robot compatibility
• Contributing
• Reference

Features

• Separates conversation management from response generation, so the dialogue flow, model choice, etc., can be changed independently.
• Supports both text-only generation and vision-language generation through a shared response-generator interface.
• Can choose between the text model and the vision model with a lightweight chooser model, or force the next turn to use the VLM.
• In addition to the usual prompt instructions, the system allows for steering instructions to be sent per turn or persistently; these are put at the end of the prompt to further guide the model's output.
• Applies background history-compaction strategies so older frames (images) and mixed content can be summarised instead of keeping every raw item in the prompt.
• Streams generated text in sentence-sized chunks for faster interactions.
• Stores camera frames, spoken turns, thoughts, and generated turns in a typed conversation history.
• Preserves generation metadata in the history for later inspection.

Installation

For the standalone Python entrypoint, install the main dependencies:

pip install vision-enabled-dialogue

# Or, if you want to install from source, clone the repository and run:
pip install -r requirements.txt

# To run the examples, you will also need OpenCV:
pip install opencv-python

The default GPT client in this repository is configured in code and points to http://localhost:11434/v1 with the API key ollama. If you want to use a different backend, set the environment variables OPENAI_API_BASE_URL and OPENAI_API_KEY before running the code, or pass a custom ResponseGenerator to the DialogueManager constructor. To use OpenAI models, set OPENAI_API_BASE_URL to an empty string.

Usage - Get started

The script in main.py shows a simple example of using the DialogueManager with a webcam and a vision-enabled LLM. Through the default SummariseHeavy strategy with a `HidingSummariser, it always keeps four images in memory and hides the rest. It also contains a more in-depth example of how to customise the dialogue manager.

Ensure you have Ollama running with the gemma3:27b model on port 11434, then run the script:

python main.py

Usage - Generating responses

Responses are generated by the DialogueManager class, which takes a ResponseGenerator, which represent a set of instructions for the LLM. It uses a LLM implementation to produce the actual responses. Currently only the GPT implementation is available, which uses the OpenAI conventions, also compatible with Ollama.

The response is streamed in sentence-sized chunks, so the first part of the response is available before the entire generation is complete. The metadata about the generation is stored in the conversation history, so you can inspect it later.

Usage - Rendering outputs and handling interruptions

Provide a function to handle the generated text chunks to the DialogueManager constructor, for example to print them to the console or send them to a TTS engine. If the speech is interrupted, this function should return the text that was actually spoken, so the dialogue manager can store it in the conversation history. The full generated text is also stored in the conversation history.

The ConversationHistory class uses two callback functions to notify when a new part is added or when a part is summarised. You can use these callbacks to update the UI or trigger other actions.

Usage - Manage prompt length

Conversation length is controlled by the history layer in vision_enabled_dialogue/conversation_history. The main building blocks are:

• ConversationHistory, which stores the ordered ConversationPart objects and applies growth strategies in the background.
• GrowthStrategy, which defines how to summarise parts of the conversation using a Summariser.
  • SummariseHeavy, summarises older heavy parts (e.g., frames) once their count exceeds a limit.
  • KeepLastNStrategy, which keeps the last N important parts (e.g., text) and summarises the rest.
• Summariser, which defines how to summarise conversation parts, when a growth strategy requires it.
  • FrameSummariser, which produces a summary of images.
  • MixedSummariser, which produces a text summary for mixed conversation parts.
  • HidingSummariser, which hides parts from the transcript (but does not remove them from memory).

For instance, to summarise the initial part of the conversation when it becomes too long, you can use a KeepLastNStrategy with a MixedSummariser.

If you want to reduce the prompt size, substituting consecutive frames with a text summary, you can use SummariseHeavy strategy with a FrameSummariser, defining the minimum and maximum number of frames to keep in the prompt. If instead you want to hide older parts from the prompt, while keeping them in memory, you can use HidingSummariser instead.

ROS 2 compatibility

This package is compatible with ROS 2 (tested on Humble). An example node is provided in vision_enabled_dialogue/main_ros.py. It declares the persistent_steering parameter and uses these interfaces:

• Subscribed topics: /humans/voices/anonymous_speaker/speech, dialogue/steering, /camera/image_raw, /camera/image_raw/compressed
• Published topics: /conversation, /transcript
• Action client: /tts_engine/tts

To test the node, you can install this repo as a ROS 2 Python package (in your-workspace/src/vision_enabled_dialogue), make sure the dependencies are installed, then build the workspace, source and run it with:

colcon build --symlink-install
source ./install/setup.bash
ros2 run vision_enabled_dialogue main

Furhat robot compatibility

Following the deprecation of Furhat's RemoteAPIs, this repository no longer provides a Furhat-specific entrypoint. You can use this library in your existing Furhat project.

Contributing

Contributions are welcome! Please open an issue or submit a pull request. For major changes, please open an issue first to discuss what you would like to change.

Reference

If you use this code in your work, please cite the paper:

@inproceedings{10.5555/3721488.3721641,
author = {Abbo, Giulio Antonio and Belpaeme, Tony},
title = {I Was Blind but Now I See: Implementing Vision-Enabled Dialogue in Social Robots},
year = {2025},
publisher = {IEEE Press},
booktitle = {Proceedings of the 2025 ACM/IEEE International Conference on Human-Robot Interaction},
pages = {1176–1180},
numpages = {5},
keywords = {conversation, dialogue, hri, large language model, prompt engineering, ros, vision language model},
location = {Melbourne, Australia},
series = {HRI '25}
}