Welcome to the LunarLander repository. This project is part of my undergraduate these and explores the evolutionary optimization of neural network hyperparameters applied to the Deep Q-Network (DQN) algorithm. The focus of the research is to demonstrate how optimization techniques can improve the performance of reinforcement learning agents in complex tasks.
Our study compared the performance of DQN agents trained with and without hyperparameter optimization. By applying a genetic algorithm, we were able to achieve significant improvements in performance metrics. The table below summarizes the results of the optimized models compared to the baseline:
| Metric | Training 3 (No Optimization) | Training 6 (With Optimization) |
|---|---|---|
| Max Reward | 317.15 | 325.25 |
| Average Reward | 30.91 | 41.94 |
| Average Time Per Episode (s) | 1.15 | 1.68 |
| Average Actions | 275 | 254 |
| Times Correct | 305 | 350 |
- Maximum Reward: The optimized model achieved a maximum reward of 325.25, representing an increase of 2.56% compared to the baseline model without optimization.
- Average Reward: An increase of 35.64% in the average reward was observed with the optimized model, indicating a more consistent performance during the training episodes. - Action Efficiency: The optimized model required fewer actions to achieve the same reward level, resulting in a 7.64% reduction in the average number of actions.
- Number of Times Correct: The optimization led to a 14.75% increase in the number of correct answers, suggesting greater accuracy and efficiency in the decisions made by the agent.
The results highlight the positive impact of hyperparameter optimization on the performance of DQN, demonstrating that careful adjustments can lead to more effective and efficient solutions in reinforcement learning environments.
The initial configuration of the DQN agent was kept constant to establish a reliable baseline. The parameters used were the following:
- Batch Size: 64
- Gamma (Discount Factor): 0.99
- Start Epsilon: 1.0
- Final Epsilon: 0.01
- Epsilon Decay: 0.996
- Target Update Frequency: 10
- Number of Episodes: 350
- Learning Rate: 0.001
We implemented a genetic algorithm to explore different hyperparameter settings of the neural network. The parameter variations included:
- Number of Layers: between 1 and 8
- Number of Neurons per Layer: between 64 and 512
- Dropout Value: 0%, 30%, 50%
- Activation Function: ReLU, Sigmoid, Softmax
The optimized parameters demonstrated substantial improvements in the agent's performance, as evidenced by the results presented.