Currently, LLMeter provides point estimates of quantile stats (p90, p99, etc) and general warnings in the docs about the unreliability of extreme quantiles for small sample sizes.
Maybe we could consider providing more robust estimates of this uncertainty, to help guide when the quantiles aren't appropriate to use?
For example there's a proposed implementation here of confidence interval estimation for a quantile, and Claude suggested a similar approach as below:
from scipy.stats import binom
def quantile_ci(data, p, confidence=0.95):
sorted_data = sorted(data)
n = len(sorted_data)
alpha = 1 - confidence
# Find lower and upper order statistic indices
lower = binom.ppf(alpha / 2, n, p) # e.g. index of CI lower bound
upper = binom.ppf(1 - alpha / 2, n, p)
l, u = int(lower), int(min(upper, n - 1))
if l == u:
return None # Can't form a CI — sample too small for this quantile
return (sorted_data[l], sorted_data[u])I'd need to spend longer staring at the stats to fully validate, but the general gist seems to be that there should be possibilities - with the main dependency being some kind of representation of the PMF/CDF of Binomial distribution... Which might be possible without taking on the heavy scipy library?
If we don't like the complexity of actually exposing the confidence intervals to users, then maybe it could just form the basis of an internal validity check that omits quantile stats we can't reliably estimate given the collected sample set?
Currently, LLMeter provides point estimates of quantile stats (p90, p99, etc) and general warnings in the docs about the unreliability of extreme quantiles for small sample sizes.
Maybe we could consider providing more robust estimates of this uncertainty, to help guide when the quantiles aren't appropriate to use?
For example there's a proposed implementation here of confidence interval estimation for a quantile, and Claude suggested a similar approach as below:
I'd need to spend longer staring at the stats to fully validate, but the general gist seems to be that there should be possibilities - with the main dependency being some kind of representation of the PMF/CDF of Binomial distribution... Which might be possible without taking on the heavy scipy library?
If we don't like the complexity of actually exposing the confidence intervals to users, then maybe it could just form the basis of an internal validity check that omits quantile stats we can't reliably estimate given the collected sample set?