Advanced Deep Reinforcement Learning for Heat Pump Control in Residential Buildings

⚠️ Note: Last update on 05.01.2026

This repository contains the official code of our paper "Advanced Deep Reinforcement Learning for Heat Pump Control in Residential Buildings". It features a custom Gymnasium environment for smart heat pump control in residential buildings, inspired by the Heat Pump House at the Living Lab Energy Campus (LLEC), KIT.

1. Introduction LLECBuildingGym

Click to expand/collapse

1.1 Description

The base_building_gym.py simulates thermal building dynamics with heat pump control in 5-minute intervals. This framework leverages the Gymnasium and Pyomo libraries, making it suitable for both reinforcement learning agents and advanced control strategies.

To simulate real-world uncertainty, the environment includes:

• Wiener Process Noise introduces random fluctuations into the outdoor temperature
• Sensor Noise simulates inaccurate indoor and outdoor temperature measurements

These features support the evaluation under uncertainty and help assess the robustness of control strategies.

Key Features

• Single-zone indoor thermal model with electric heat pump control and heat loss dynamics
• Dynamic energy pricing and weather inputs
• Configurable heat pump control every 5 minutes
• Exogenous variables outdoor temperature and dynamic energy prices
• Modular design supporting custom reward modes and controllers (RL, PI, PID, Fuzzy, MPC)

1.2 Project Structure

LLECBuildingGym/                              # Root directory of the project
├── data/                                     # Input data (e.g., weather, pricing)
├── llec_building_gym/                        # Main Python package: Gym environment and controllers
│   ├── controllers/                          # Other controllers; Fuzzy, MPC, PI, PID
│   │   ├── __init__.py                       # Exports controller classes
│   │   ├── fuzzy_controller.py               # Fuzzy controller
│   │   ├── mpc_controller.py                 # MPC controller
│   │   ├── pi_controller.py                  # PI controller
│   │   ├── pid_controller.py                 # PID controller
│   │   └── README_MPC.md                     # MPC documentation and usage instructions
│   ├── envs/                                 # Submodule with environment definitions
│   │   ├── __init__.py                       # Exports environments for external use
│   │   └── base_building_gym.py              # Main environment logic and control integration
│   └── __init__.py                           # Registers environments
├── models/                                   # Saved trained models (PPO, SAC, DDPG,TD3, A2C)
├── plot-paper/                               # Notebooks to generate figures and tables
│   ├── check_envs_registration.ipynb         # Verifies registered Gymnasium environments
│   ├── generate_table03_summary_stats.ipynb  # Generate Table 03
│   ├── plot_fig03_temperature_data.ipynb     # Plots indoor/outdoor temperature data for Figure 03
│   ├── plot_fig04_price_data.ipynb           # Plots dynamic energy prices for Figure 04
│   ├── plot_fig05_indoor_temp_setpoint.ipynb # Plots dynamic indoor temp setpoints for Figure 05
│   └── preprocess_outdoor_temperature.ipynb  # Prepares outdoor temperature time series
├── slurm_logs_eval/                          # SLURM logs from evaluation jobs
├── slurm_logs_train/                         # SLURM logs from training jobs
├── slurm_script/                             # SLURM job submission scripts
├── results/                                  # Evaluation logs and result CSVs
├── .gitignore                                # Ignore in version control
├── LICENSE                                   # Licensing
├── README.md                                 # Repo documentation and usage instructions
├── pyproject.toml                            # Build system configuration
├── requirements.txt                          # Python dependencies
├── run_evaluation.py                         # Evaluate trained models
└── run_train_rl.py                           # Train RL models (PPO, SAC, DDPG,TD3, A2C)

2. Installation and Environment Setup

Click to expand/collapse

2.1a Haicore (Linux):

Clone the repository:

git clone https://github.com/KIT-IAI/LLECBuildingGym
python3.9 -m venv llec_env
source llec_env/bin/activate
cd LLECBuildingGym

pip install --upgrade pip
pip install -r requirements.txt
pip install -e .

The virtual environment and project directory should be organized as shown below:

llec_env/        # Python virtual environment
LLECBuildingGym/ # Root directory of the project

2.1b Local (Windows):

Install Python 3.9.18 from https://www.python.org/downloads/release/python-3918 (newer Python versions may work but are not tested).

git clone https://github.com/KIT-IAI/LLECBuildingGym
py -3.9 -m venv llec_env
.\llec_env\Scripts\activate
cd LLECBuildingGym

python -m pip install --upgrade --force-reinstall pip
pip install -r requirements_windows.txt
pip install -e .

2.2 Reinstallation (after code changes):

pip uninstall llec_building_gym -y
pip install -e .

2.3 Environment Check (verify that the environment is registered correctly):

python check_envs_registration.ipynb

2.4 For using Jupyter notebooks:

source llec_env/bin/activate
pip install ipykernel
python -m ipykernel install --user --name=llec_env --display-name "Python (llec_env)"
jupyter kernelspec list

Always activate the virtual environment (source llec_env/bin/activate) before starting Jupyter to ensure correct dependencies. After registering the kernel, restart Jupyter so the Python (llec_env) kernel becomes available.

3.Training and Evaluation

Click to expand/collapse

This repository supports both RL agent training and controller evaluation via script-based workflows. RL training is handled using stable-baselines3 algorithms, while evaluation supports classical control strategies such as PI, PID, Fuzzy Logic, and MPC Controllers.

3.1 RL Training:

Train RL agents using the script run_train_rl.py.

Two reward modes and multiple observation variants are supported for flexible evaluations.

• temperature: Temperature-based reward (single-objective)
• combined: Temperature and energy cost combined (multi-objective)

Command-line Arguments:

Argument	Type	Default Value	Choices	Description
--algorithm	str	"ppo"	ppo, sac, ddpg,td3, a2c	RL algorithm to use (from Stable-Baselines3).
--timesteps	float	1e6	Any positive float	Total number of environment steps.
--num-envs	int	4	>= 1	Number of parallel environments (for vectorized training).
--seed	int	42	Any integer	Random seed for reproducibility.
--eval-freq	int	5000	>= 1	Evaluation frequency (in timesteps).
--reward_mode	str	"temperature"	temperature, combined	Reward mode: temperature (single-reward) or combined (multi-reward).
--energy-price-path	str	"data/price_data_2025.csv"	Valid CSV path	Path to normalized energy price CSV file.
--training	flag	False	False, True	Use training data for energy prices (default: TOU Prices).
--obs_variant	str	T01	T01,T02.T03,T04,C01,C02.C03,C04	Select observation variant (see detailed list below).

Observation Variants:

Variant	Features Included	Description
T01	noisy_temp_deviation	Temperature deviation only
T02	noisy_temp_deviation, time_of_day	Add normalized time of day
T03	noisy_temp_deviation, prev_action	Add previous normalized action
T04	noisy_temp_deviation, time_of_day, prev_action	Full thermal state
C01	noisy_temp_deviation, energy_price, future_prices	Thermal + current and future energy prices
C02	C01 + prev_action	C01 + previous action
C03	C01 + time_of_day	C01 + time of day
C04	C01 + time_of_day, prev_action	Full combined state

---

Example Usage

python run_train_rl.py --algorithm ppo --reward_mode temperature --training

3.2 Evaluation:

The evaluation supports both RL agents and advanced control strategies from control theory.
These include:

• PI/PID Control – widely used feedback controllers based on proportional, integral, and derivative action
• Fuzzy Control – heuristic rule-based controller using fuzzy logic for handling uncertainty
• MPC Control – model predictive control with configurable prediction horizon

Command-line Arguments:

Argument	Type	Default Value	Choices	Description
--algorithms	list	["ppo", "sac", "ddpg", "td3", "a2c", "PI Control", "PID Control", "Fuzzy Control", "MPC Control"]	Any combination of supported controllers and RL models	List of algorithms or controllers to evaluate.
--episodes	int	10	>= 1	Number of evaluation episodes per algorithm.
--seed	int	58	Any integer	Random seed for reproducibility.
--model_seed	int	42	Any integer	Seed number used during training for selecting the correct model file.
--mpc_horizon	int	72	>= 1 (typically multiples of 12)	Prediction horizon for MPC (in 5-minute steps, e.g., 12 = 1 hour).
--reward_mode	str	"temperature"	temperature, combined	Reward mode: temperature or combined (multi-objective).
--energy_price_path	str	"data/price_data_2025.csv"	data/price_data_2025.csv	Path to normalized energy price CSV.
--outdoor_temperature_path	str	"data/LLEC_outdoor_temperature_5min_data.csv"	data/LLEC_outdoor_temperature_5min_data.csv	If not provided, a synthetic temperature profile is used.
--obs_variant	str	T01	T01,T02.T03,T04,C01,C02.C03,C04	Select observation variant (see detailed list below).
--prefer_best	flag	False	False,True	If set, prefers loading best_model.zip instead of <algorithm>_model_seed<seed>.zip (e.g., ppo_model_seed42.zip) during evaluation. Supported algorithms: ppo, sac, ddpg,td3, a2c.

---

Example Usage

# Evaluate PPO agent for temperature based rewards
python run_evaluation.py --algorithms ppo --reward_mode temperature --obs_variant T01

# Evaluate all available agents and controllers
chmod +x slurm_script/slurm_train_01_rl_batch.sh
./slurm_script/slurm_train_01_rl_batch.sh

The modular design allows users to plug in their own controllers or extend the environment with new features, e.g., building dynamics or pricing schemes.

4. Citation 📝

If you use this framework in your research, please consider citing our paper 📝 and giving the repository a star ⭐:

@inproceedings{demirel2025_LLECBuildingGym,
  title     = {{Advanced Deep Reinforcement Learning for Heat Pump Control in Residential Buildings}},
  author    = {Demirel, Gökhan and Ekin, Ömer and Liu, Jianlei and Spatafora, Luigi and Förderer, Kevin and Hagenmeyer, Veit},
  year      = {2025},
  booktitle = {2025 IEEE PES Innovative Smart Grid Technologies Conference Europe (ISGT Europe)},
  pages     = {1--5},
  doi       = {10.1109/ISGTEurope64741.2025.11305332},
}

License

This code is licensed under the MIT License. For any issues or any intention of cooperation, please feel free to contact me at goekhan.demirel@kit.edu.