Ontology-Driven LLM-TAMP

Abstract

Performing complex manipulation tasks in dynamic environments requires efficient Task and Motion Planning (TAMP) approaches that combine high-level symbolic plans with low-level motion control. Advances in Large Language Models (LLMs), such as GPT-4, are transforming task planning by offering natural language as an intuitive and flexible way to describe tasks, generate symbolic plans, and reason. However, the effectiveness of LLM-based TAMP approaches is limited due to static and template-based prompting, which limits adaptability to dynamic environments and complex task contexts. To address these limitations, this work proposes a novel Onto-LLM-TAMP framework that employs knowledge-based reasoning to refine and expand user prompts with task- contextual reasoning and knowledge-based environment state descriptions. Integrating domain- specific knowledge into the prompt ensures semantically accurate and context-aware task plans. The proposed framework demonstrates its effectiveness by resolving semantic errors in symbolic plan generation, such as maintaining logical temporal goal ordering in scenarios involving hierarchical object placement. The proposed framework is validated through both simulation and real-world scenarios, demonstrating significant improvements over the baseline approach in terms of adaptability to dynamic environments and the generation of semantically correct task plans.

Citation

@article{DIN2026105404,
title = {Onto-LLM-TAMP: Knowledge-oriented Task and Motion Planning using Large Language Models},
author = {Muhayy Ud Din and Jan Rosell and Waseem Akram and Isiah Zaplana and Maximo A. Roa and Irfan Hussain},
journal = {Robotics and Autonomous Systems},
pages = {105404},
year = {2026},
issn = {0921-8890},
doi = {https://doi.org/10.1016/j.robot.2026.105404},
url = {https://www.sciencedirect.com/science/article/pii/S0921889026000771}
}

Install dependencies

To run the code create a conda environment with python 3.9 using the below command

conda create --name tamp-env python=3.9
conda activate tamp-env

To download the Spacy language model, run the following command

python -m spacy download en_core_web_sm

git clone https://github.com/Muhayyuddin/llm-tamp.git
cd llm-tamp
pip install -r requirements.txt

Before Running

Please, create a folder openai_keys under the project directory; and create a file openai_key.json under the folder openai_keys; fill in this json file with your openAI API key:

{
    "key": "",
    "org": "",
    "proxy" : ""
}

Run Ontology-driven LLM-TAMP planning

To run the ontology-driven LLM-TAMP with the following scene, run the command below in the terminal

python main.py --config-name=llm_tamp env=easy_ycb_objects_scene planner=llm_sample_params max_llm_calls=10 play_traj=true use_gui=true

To run the ontology-driven LLM-TAMP with the following scene, run the command below in the terminal

python main.py --config-name=llm_tamp env=easy_ycb_tabe_obj planner=llm_sample_params max_llm_calls=10 play_traj=true use_gui=true

Ontological Knowledge

We provided the minimal classes and rules of our developed ontological knowledge to reproduce the results. The full Ontology rdf file and clean version of the code will be updated on acceptance.

Acknowledgment

Thank you for the nice work done by LLM3, FoundationPose, and FoundationPose-ROS2 , we use some code from these repositories to implement our Ontology-deriven-LLM-TAMP framework.

https://github.com/AssassinWS/LLM-TAMP

https://github.com/NVlabs/FoundationPose

https://github.com/ammar-n-abbas/FoundationPoseROS2