This repository contains code and supplementary materials for the paper A Unifying Information-theoretic Perspective on Evaluating Generative Models. It includes implementations for our proposed metric Precision Cross-Entropy (PCE), Recall Cross-Entropy (RCE), and Recall Entropy (RE), along with scripts to produce experimental results.
- Alexis Fox (alexis.fox@duke.edu)
Duke University - Samarth Swarup (swarup@virginia.edu)
University of Virginia - Abhijin Adiga (abhijin@virginia.edu)
University of Virginia
Conference: AAAI25
ArXiv Link: https://arxiv.org/abs/2412.14340
Considering the difficulty of interpreting generative model output, there is significant current research focused on determining meaningful evaluation metrics. Several recent approaches utilize “precision” and “recall,” borrowed from the classification domain, to individually quantify the output fidelity (realism) and output diversity (representation of the real data variation), respectively. With the increase in metric proposals, there is a need for a unifying perspective, allowing for easier comparison and clearer explanation of their benefits and drawbacks. To this end, we unify a class of kth-nearest neighbors (kNN)-based metrics under an information-theoretic lens using approaches from kNN density estimation. Additionally, we propose a tri-dimensional metric composed of Precision Cross-Entropy (PCE), Recall Cross-Entropy (RCE), and Recall Entropy (RE), which separately measure fidelity and two distinct aspects of diversity, inter- and intra-class. Our domain-agnostic metric, derived from the information-theoretic concepts of entropy and cross-entropy, can be dissected for both sample- and mode-level analysis. Our detailed experimental results demonstrate the sensitivity of our metric components to their respective qualities and reveal undesirable behaviors of other metrics.
- metrics/: Implementation of our metric and others discussed.
- notebooks/: Jupyter notebook for running representative experiments in paper.
- docs/: Paper PDF.
To set up the repo and test out the metrics:
- Clone the repository
git clone https://github.com/unified-generative-metrics.git
cd unified-generative-metrics- Install dependencies
pip install -r requirements.txt- Install the package
pip install -e .Example Usage:
import numpy as np
from metrics import compute_metrics
num_real_samples = num_fake_samples = 10000
feature_dim = 1000
nearest_k = 5
real_features = np.random.normal(loc=0.0, scale=1.0, size=[num_real_samples, feature_dim])
fake_features = np.random.normal(loc=0.0, scale=1.0, size=[num_fake_samples, feature_dim])
metrics = compute_metrics(real_features, fake_features, nearest_k)
print(metrics)To cite our work, use:
title={A Unifying Information-theoretic Perspective on Evaluating Generative Models},
author={Fox, Alexis and Swarup, Samarth and Adiga, Abhijin},
booktitle={Proceedings of AAAI25},
year={2025},
url={https://arxiv.org/abs/2412.14340}
}