ROS2 Maze Robot Q-Learning

A ROS2 package for training a robot to navigate through a maze using Q-Learning reinforcement learning.

Overview

This project implements a Q-Learning agent that learns to navigate a robot through a maze environment. The robot uses reinforcement learning to discover optimal paths while avoiding obstacles.

Features

• Q-Learning Agent: Tabular Q-learning implementation with epsilon-greedy exploration
• Maze Environment: Grid-based maze environment with customizable dimensions
• Reward System: Distance-based rewards with goal, collision, and step penalties
• Training Manager: Complete training loop with episode management
• Visualization Tools: Plot training progress, Q-table heatmaps, and learning curves
• Data Logging: CSV and JSON logging of training metrics
• Model Management: Save and load trained Q-tables
• Collision Detection: Laser scan-based obstacle detection
• Path Planning: A* algorithm for optimal path finding
• Performance Metrics: Comprehensive evaluation statistics
• Experience Replay: Buffer for storing and replaying experiences
• Hyperparameter Tuning: Grid search and random search utilities

Installation

Prerequisites

• ROS2 (Humble or later)
• Python 3.8+
• Required Python packages:
  • numpy
  • matplotlib
  • rclpy
  • geometry_msgs
  • nav_msgs
  • sensor_msgs

Build

cd ~/ros2_ws/src
git clone https://github.com/Lucii-666/ROS2-Maze_Robot.git
cd ~/ros2_ws
colcon build --packages-select maze_robot_qlearning
source install/setup.bash

Usage

Launch the Simulation

ros2 launch maze_robot_qlearning maze_launch.py

Train the Agent

from maze_robot_qlearning.training_manager import TrainingManager

trainer = TrainingManager()
trainer.train()

Evaluate Trained Agent

from maze_robot_qlearning.evaluator import AgentEvaluator

evaluator = AgentEvaluator(model_path='q_table.pkl')
evaluator.run_evaluation(environment)

Visualize Results

from maze_robot_qlearning.visualizer import TrainingVisualizer

viz = TrainingVisualizer()
viz.load_data('training_data.pkl')
viz.plot_training_progress()
viz.plot_q_table_heatmap(q_table)

Configuration

Edit config/config.yaml to customize:

• Learning rate
• Discount factor
• Epsilon decay
• Number of episodes
• Reward values
• Grid size
• Robot speeds

Project Structure

maze_robot_qlearning/
├── config/
│   └── config.yaml
├── launch/
│   └── maze_launch.py
├── maze_robot_qlearning/
│   ├── q_learning_agent.py
│   ├── maze_environment.py
│   ├── reward_calculator.py
│   ├── training_manager.py
│   ├── visualizer.py
│   ├── data_logger.py
│   ├── model_manager.py
│   ├── collision_detector.py
│   ├── state_representation.py
│   ├── action_space.py
│   ├── episode_manager.py
│   ├── path_planner.py
│   ├── performance_metrics.py
│   ├── experience_replay.py
│   ├── hyperparameter_tuner.py
│   └── evaluator.py
├── urdf/
│   └── simple_robot.urdf
├── worlds/
│   └── maze_world.sdf
└── README.md

License

MIT License

Author

Lucii-666

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.