P-DQN: Parametrized Deep Q-Networks

A PyTorch implementation of Parametrized Deep Q-Networks (P-DQN), a reinforcement learning algorithm designed to handle environments with discrete-continuous hybrid action spaces.

🌟 Features

• Hybrid Action Space Support: Seamlessly handles both discrete and continuous action components
• Modular Architecture: Clean separation of agent, networks, and replay buffer
• Flexible Training: Configurable hyperparameters via command-line arguments
• Tensorboard Integration: Real-time monitoring of training metrics
• Checkpoint System: Automatic saving and loading of model checkpoints
• Multiple Environments: Support for various gym-hybrid environments (Moving, Sliding, etc.)

🚀 Quick Start

1. Clone the Repository

git clone https://github.com/ZhBF/ParaDQN.git
cd ParaDQN

2. Install Dependencies

# Install required packages
pip install -r requirements.txt

# Install PyTorch (visit https://pytorch.org for your specific configuration)
pip install torch torchvision torchaudio

3. Install Gym-Hybrid Environment

cd ..
git clone https://github.com/thomashirtz/gym-hybrid.git
cd gym-hybrid
pip install -e .
cd ../ParaDQN

4. Run Training

# Train with default parameters
python main.py

# Train with custom parameters
python main.py --env_name moving --episodes 2000 --lr 0.001

🎮 Usage

Training Arguments

python main.py [OPTIONS]

Options:
  --env_name          Environment name (default: 'moving')
  --episodes          Number of training episodes (default: 1600)
  --batch_size        Batch size for training (default: 128)
  --lr                Learning rate (default: 0.0001)
  --gamma             Discount factor (default: 0.99)
  --buffer_size       Replay buffer size (default: 100000)
  --tau               Target network update rate (default: 0.001)

Monitoring Training

# View training progress with Tensorboard
tensorboard --logdir runs/

Then open your browser to http://localhost:6006

Loading Checkpoints

Checkpoints are automatically saved in runs/run_<env>_<id>_<timestamp>/checkpoints/. To resume training or evaluate a trained model, load the checkpoint in your script.

📈 Results

The agent learns to solve hybrid action space tasks effectively. Training progress can be monitored through Tensorboard, showing metrics such as:

• Episode rewards
• Q-value estimates
• Loss curves
• Success rates

📝 Citation

If you use this code in your research, please cite the original paper:

@article{xiong2018parametrized,
  title={Parametrized deep q-networks learning: Reinforcement learning with discrete-continuous hybrid action space},
  author={Xiong, Jiechao and Wang, Qing and Yang, Zhuoran and Sun, Peng and Han, Lei and Zheng, Yang and Fu, Haobo and Zhang, Tong and Liu, Ji and Liu, Han},
  journal={arXiv preprint arXiv:1810.06394},
  year={2018}
}

📚 References

• Parametrized Deep Q-Networks Learning: Reinforcement Learning with Discrete-Continuous Hybrid Action Space
• Gym-Hybrid Environment

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

✉️ Contact

For questions or discussions, please open an issue on GitHub.