Repository for the paper While My RecSys Gently Weeps: Energy-Efficent Early Stopping with Exponential Moving Average for Green Recommender Systems.

Abstract

The environmental sustainability of Recommender Systems has become a topic of increasing interest in the research community. With this paper, we propose E3S (Energy-Efficient Early Stopping), a novel strategy to early stop the training of the recommendation model when the marginal improvements in the validation metric are not justified by the increase in emissions generated by the training.

To this end, we consider the accuracy/emission ratio epoch by epoch. We analyze this ratio's trend using the Exponentially Weighted Moving Average, which provides a smoothed, continuously updated estimate of the average efficiency ratio, assigning greater weight to recent values while gradually discounting older observations. Finally, we stop the training when the model consistently fails to maintain an acceptable level of energy efficiency relative to its recent historical trend, suggesting that further computation would not yield meaningful improvements given the increase in emissions.

In the experimental session, we compared two variants of our E3S and evaluated them against several state-of-the-art recommendation models trained on two well-known datasets. Results show that our methodology effectively decreases the emissions of the recommendation models without excessively compromising the quality of the suggestions. The comparison with the conventional early stopping method highlights the advantages of the proposed approach in terms of sustainability and performance preservation.

---

How does E3S work?

The system keeps track of the emissions of a given recommendation model on a specific dataset. During the execution of each model, the metrics, the emissions, and the configuration of the parameters used are saved. In particular, for this paper we exploit the EWMA to analyze the historical trend of the the AER (Accuracy/Emissions Ratio) and stop the training when it is not sustainable anymore.

Models and metrics refer to the implementation by @Recbole. Datasets are provided in a compatible format.

Emission tracking was made possible through the @CodeCarbon library.

---

📁 Repository Structure

├── data/                   # Contains the datasets used
├── deps/                   # Directory containing specific dependenciestechniques work
│   ├── codecarbon/         # Files and configurations related to the CodeCarbon library
│   ├── ema_files/          # Files related to the EMA early stopping strategy (with upper limit)
│   ├── ema_no_cap_files/   # Files related to the EMA early stopping strategy (WITHOUT upper limit)
│   └── utility_files/      # Files related to the Utility early stopping strategy
├── experiment_results/     # Directory for detailed results of each experiment (for each dataset, variant and configuration of the E3S)
├── src/                    # Contains the project’s source code
├── LICENSE.md
├── README.md
└── requirements.txt        # List of Python dependencies

---

🛠️ Requirements

For our experiments, we used :
python/3.10.8--gcc--8.5.0 and cuda/12.1

The main requirements are related to @Recbole, in which we implemented the E3S criterion, and @CodeCarbon, that we used to track the emissions. We suggest to refer to their requirements to install the most recente and compatible versions.

We also share our requirements, which can be installed by running:

pip install -r requirements.txt

We suggest to create a virtual environment with these requirements.

---

⚙️ Installation

1. Once the environment has been create, locate the dependencies directory (e.g., .env/lib/python3.10/site-packages/).

2. Copy the deps/codecarbon/ into this directory.

3. Select the E3S variant and copy the files from deps/<variant>/recbole to .env/lib/python3.10/site-packages/.

   variant can be ema_cap_files for the bounded variant and ema_no_cap_files for unbounded variant.

4. Copy deps/<variant>/src to the root of the project

Done! The E3S is now installed, and you can run the experiments!

---

⚙️ Run the experiment

To run the experiments, just run:

python src/run.py 

This will run the experiments, for the two datasets and for all the recommendation models with some default parameters, for which we choose (see the paper for more details about these two parameters):

• tolerance step: 10
• smoothing factor: 2/(tolerance_step+1)

Alternatively, it is possible to set any model, dataset and parameter value directly at the this step by running:

python src/tracker.py --dataset={dataset} --model={model} --tolerance_step={tolerance_step} --smoothing_factor={smoothing_factor}

For our experiments, following the literature, we have always considered as smoothing factor the value 2/(tolerance_step+1).

---

📊🤖 Datasets and Recommendation Models

We conducted the experiments against the following datasets and recommendation models:

• Datasets: AmazonBooks e MovieLens-1M
• RecSys: BPR, CFKG, CKE, DMF, KGCN, KGNNLS, LINE, MultiDAE, LightGCN, NFCF e DGCF