Replication package

For the paper A Model-Driven Approach for Developing Families of Reinforcement Learning Environments.

Accepted for MODELS 2026.

About

Virtual training environments are software-intensive systems in which reinforcement learning (RL) agents learn, adapt, and demonstrate meaningful behavior. Virtual training environments offer a safe and cost-efficient alternative to training agents in real-world settings. However, to converge, most realistic RL problems require training in multiple, mostly similar but slightly different environments---i.e., families of environment variants. The typical development process of environment families is a labor-intensive and error-prone manual endeavor that does not scale well. To alleviate these issues, in this paper, we propose a model-driven approach for developing families of RL training environments. To obtain the family of environments, we develop an approach and prototype tool. In our approach, a hybrid genetic algorithm---a combination of population-based global search and heuristic local search---generates environment families. Mutations and constraints are expressed as model transformations and are operationalized into a search process by a state-of-the-art model transformation engine. We demonstrate the soundness of our approach in a wildfire mitigation scenario and curriculum learning---a particular learning paradigm that relies on environment families.

Table of contents

• Content description
• Reproduction of analysis
• Reproduction of experimental data
• Experiment settings
• Results

Content description

01-configurations: contains the environment and learning configuration files

• environment-config.json — the generated curriculum
• learning-config.json — RL hyperparameters for the normal training budget (50,000 steps/env)
• learning-config-shorter.json — RL hyperparameters for the shorter training budget (10,000 steps/env)
• learning-config-longer.json — RL hyperparameters for the longer training budget (100,000 steps/env)
• random-configs/ — 30 generated random environment sequences used in the additional evaluation

02-train-data: contains trained agents' data produced by running the scripts in 03-scripts

• <experiment>/start_stage_1/ … start_stage_5/ — final checkpoints (final_env_N.npy) for each curriculum run
• <experiment>/baseline/<label>/ — checkpoints for each baseline agent
• comparison_<budget>/ — checkpoints for the random-baseline comparison experiment

03-scripts: contains Python scripts to reproduce the training results and plots in 04-results

04-results: contains the plots and visualizations used in the publication

• map_visualization/ — visualization of each environment in the generated curriculum
• plots/ — cumulative reward figures during training
• statistics/: evaluation results of the success rate and the mean episodic return.

Here's the revised version:

---

Reproduction of analysis

Requirements

Python 3.11 or higher is required. Set up a virtual environment from the 03-scripts directory:

cd 03-scripts
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Step 0 — Configuration files

The experiments are controlled by two JSON files in 01-configurations/:

• environment-config.json: the family of environments. To apply the pipeline to a different environment family, replace this file with one following the same schema (a top-level "environments" list where each entry has env_id, desc, reward_schedule, and complexity).
• learning-config.json: RL hyperparameters and training budget. Three variants are provided:

File	Steps per environment
learning-config.json	50,000 (default)
learning-config-shorter.json	10,000
learning-config-longer.json	100,000

Step 1 — Generate map visualizations

python map_visualizer.py

Saves the visualization of each environment to 04-results/map_visualization/.

To visualize a different environment config:

python map_visualizer.py ../01-configurations/your-environment-config.json

Step 2 — Training

python runner.py --all-stages --baseline all

This will:

1. Build the full curriculum (E1 → E6) ordered by environment complexity
2. Run 5 partial curricula starting from E1, E2, E3, E4, and E5 through the target environment (E6)
3. Run a direct-training baseline for each environment
4. Save agent checkpoints to 02-train-data/
5. Save the plots to 04-results/plots/.

To run a single partial curriculum, use --start-stage N (where N is 1–5) instead of --all-stages. For example, to start from environment 3 (E3 → E6):

python runner.py --start-stage 3 --baseline all

To change the training budget, swap --learn-config:

python runner.py --all-stages --baseline all \
  --learn-config ../01-configurations/learning-config-shorter.json

To skip baselines, omit --baseline all. To train only the target environment baseline (E6), use --baseline 5.

Step 3 — Evaluation

python evaluator.py

Reads checkpoints from 02-train-data/ and writes evaluation_results.csv to 04-results/statistics/.

--n-eval-episodes (default: 3000): controls how many episodes are used for the evaluation.

--learn-config must match the config used in Step 2.

Step 4 — Additional evaluation (random baseline comparison)

# Normal budget
python run_comparison.py \
  --learn-config ../01-configurations/learning-config.json \
  --output ../02-train-data/comparison_normal

# Shorter budget
python run_comparison.py \
  --learn-config ../01-configurations/learning-config-shorter.json \
  --output ../02-train-data/comparison_shorter

# Longer budget
python run_comparison.py \
  --learn-config ../01-configurations/learning-config-longer.json \
  --output ../02-train-data/comparison_longer

Each run saves summary_cumulative_reward.png, evaluation_per_seed.csv, and evaluation_summary.csv to its output directory.

--plot-only: regenerates plots and CSVs from already completed training without rerunning.

--seed (default: 42): global random seed for the curriculum and baseline runs.

Reproduction of experimental data

Setting up Eclipse

For the following steps, refer to the tool's official GitHub code repository.

Obtaining the generated curriculum

The curriculum can be generated by running experiments encoded in unit tests in the Example.

• To locate the unit tests, navigate to https://github.com/ssm-lab/ga4cl/blob/main/tests/ca.mcmaster.ssm.mde4rl.tests/src/ca/mcmaster/ssm/mde4rl/tests/Example.xtend.
• Right-click the file name.
• Go to Run as and select JUnit Plug-in Test to run.
• The generated curriculum is environment-config.json under the folder.

Experiment settings

Problem

The target environment used in the experiments:

Settings and hyperparameters

Parameter	Value
Mutation rate	0.85
Complexity measure	The number of feasible paths between the start and the goal
Diversity measure	Shannon entropy
Population size	6
Training budget per environment	50,000 steps
RL method	Q-learning
Learning rate (α)	0.1
Discount factor (γ)	0.99

Results

Generated curricula

(1) (2) (3)

(4) (5) (6)

Cumulative reward during training

Cumulative reward across all curricula

Cumulative reward with different prefixes

Additional evaluation results

Refer to official GitHub data repository.