Skip to content

Commit 6cc9632

Browse files
feat: add consistency metrics (#198)
* add consistency metrics * Update doc/estimators/ConsistencyMetrics.md Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
1 parent dcbd55c commit 6cc9632

9 files changed

Lines changed: 462 additions & 3 deletions

File tree

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ Refer to the documentation to quickly integrate and utilize the library's signal
2020
| [Control Analysis](doc/control_analysis/README.md) | Frequency Response, Root Locus, Controllability/Observability Matrices & Gramians |
2121
| [Controllers](doc/controllers/README.md) | Bang-Bang/Hysteresis, PID, LQR, LQI (Integral/Servo State Feedback), MPC, Saturation, Rate Limiter, Slew-Limited Saturation, Feedforward/2-DOF, Gain-Scheduled Controller, Lead-Lag Compensator, Luenberger Observer |
2222
| [Dynamics](doc/dynamics/README.md) | Euler-Lagrange, Newton-Euler, Recursive Newton-Euler, ABA |
23-
| [Estimators](doc/estimators/README.md) | Linear Regression, Polynomial Fitting, Yule-Walker (offline), Recursive Least Squares, LMS / NLMS Adaptive Filter (online) |
23+
| [Estimators](doc/estimators/README.md) | Linear Regression, Polynomial Fitting, Yule-Walker (offline), Recursive Least Squares, LMS / NLMS Adaptive Filter (online), Consistency Metrics / NEES / NIS |
2424
| [Filters](doc/filters/README.md) | Kalman, Extended Kalman, Unscented Kalman, Alpha-Beta/Alpha-Beta-Gamma, FIR, IIR, Exponential Moving Average, Moving Average, Complementary, Median Filter, CIC (Cascaded Integrator-Comb), Notch/Comb Filter, Savitzky-Golay Filter, Biquad/Second-Order-Section Cascade |
2525
| [Kinematics](doc/kinematics/README.md) | Forward Kinematics |
2626
| [Neural Network](doc/neural_network/README.md) | Layers, activations, losses, model |

ROADMAP.md

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,6 @@ Difficulty legend:
4848
| 45 | IIR filter design (Butterworth/Chebyshev + bilinear) | `filters/passive` | ★★★★★ |
4949
| 46 | H∞ state-feedback control | `robust_control` (new) | ★★★★★ |
5050
| 47 | Model Reference Adaptive Control (MRAC) | `nonlinear_control` (new) | ★★★★★ |
51-
| 52 | Estimator consistency metrics (NEES / NIS) | `estimators` | ★★★☆☆ |
5251

5352
Items 48–52 are the **evaluation & metrics primitives** — reusable quantities the per-family
5453
unit-test reference [`TESTING.md`](TESTING.md) depends on but which
@@ -566,7 +565,7 @@ on bounded `math::Vector`/`math::Matrix` inputs; tests are `TEST_F` on `float`.
566565
the existing `solvers::DurandKerner` for the full spectrum.
567566
- **Reuses:** `math::Matrix`, `solvers::DurandKerner`.
568567

569-
### 52. Estimator consistency metrics (NEES / NIS) ★★★☆☆ — `estimators`
568+
### ~~52. Estimator consistency metrics (NEES / NIS) ★★★☆☆ — `estimators`~~ ✓ Done
570569
- **What:** Normalised Estimation Error Squared and Normalised Innovation Squared, with χ²
571570
confidence-gate helpers.
572571
- **Metric value:** M8 (statistical consistency) — the only rigorous correctness test for the Kalman
Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
# Estimator Consistency Metrics (NEES / NIS)
2+
3+
## Overview & Motivation
4+
5+
State estimators such as the Kalman filter, Extended Kalman filter, and Unscented Kalman filter produce both a state estimate and a covariance matrix that quantifies estimation uncertainty. A filter is said to be consistent when the true errors are statistically compatible with the reported covariance — that is, the filter neither overestimates nor underestimates its own uncertainty.
6+
7+
Raw error metrics such as RMSE cannot distinguish a consistent filter (correct covariance) from an inconsistent one (wrong covariance that happens to produce low errors in a specific trial). Consistency metrics address this gap by normalising errors with respect to the predicted covariance and comparing the result against a chi-squared distribution.
8+
9+
## Mathematical Theory
10+
11+
### Normalised Estimation Error Squared (NEES)
12+
13+
Given the state error $\varepsilon_k = x_k - \hat{x}_{k|k}$ and the posterior state covariance $P_{k|k}$, the NEES at time step $k$ is
14+
15+
$$\varepsilon_k^{\mathsf{T}} P_{k|k}^{-1} \varepsilon_k$$
16+
17+
Under the hypothesis that the filter is consistent and the errors are Gaussian, this quantity is chi-squared distributed with $n_x$ degrees of freedom, where $n_x$ is the state dimension.
18+
19+
### Normalised Innovation Squared (NIS)
20+
21+
Given the innovation $\nu_k = z_k - \hat{z}_{k|k-1}$ and the innovation covariance $S_k$, the NIS at time step $k$ is
22+
23+
$$\nu_k^{\mathsf{T}} S_k^{-1} \nu_k$$
24+
25+
Under consistency, NIS follows a chi-squared distribution with $n_z$ degrees of freedom, where $n_z$ is the measurement dimension. Unlike NEES, NIS is computable without knowledge of the true state and is therefore usable in real deployments.
26+
27+
### Chi-Squared Confidence Gate
28+
29+
A two-sided 95% confidence region for a single NEES or NIS sample is
30+
31+
$$\chi^2_{n,\,0.025} \;\le\; \varepsilon_k^{\mathsf{T}} P_{k|k}^{-1} \varepsilon_k \;\le\; \chi^2_{n,\,0.975}$$
32+
33+
where $\chi^2_{n,p}$ is the $p$-th quantile of the chi-squared distribution with $n$ degrees of freedom.
34+
35+
### Time-Averaged Gate
36+
37+
Averaging NEES or NIS over $N$ independent samples gives a statistic that is chi-squared with $N \cdot n$ degrees of freedom, scaled by $1/N$. The 95% bounds for the time-averaged value are therefore
38+
39+
$$\frac{\chi^2_{Nn,\,0.025}}{N} \;\le\; \bar{\varepsilon} \;\le\; \frac{\chi^2_{Nn,\,0.975}}{N}$$
40+
41+
Averaging reduces the variance of the consistency estimate and is the standard approach in Monte-Carlo filter evaluation.
42+
43+
### Embedded Computation
44+
45+
Rather than forming $P^{-1}$ explicitly, the quadratic form $\varepsilon^{\mathsf{T}} P^{-1} \varepsilon$ is computed by solving $P z = \varepsilon$ for $z$ (via Gaussian elimination with partial pivoting) and then computing $\varepsilon^{\mathsf{T}} z$. This avoids matrix inversion, reduces floating-point operations, and is numerically more stable.
46+
47+
Chi-squared quantiles for degrees of freedom 1–10 at the 95% confidence level are stored in a compile-time array. Dimensions outside this range are rejected via `static_assert`.
48+
49+
## Complexity Analysis
50+
51+
| Case | Time | Space | Notes |
52+
|---------|----------|----------|--------------------------------------------|
53+
| Best | $O(n^2)$ | $O(n^2)$ | Dominated by back-substitution in GE |
54+
| Average | $O(n^3)$ | $O(n^2)$ | Gaussian elimination with partial pivoting |
55+
| Worst | $O(n^3)$ | $O(n^2)$ | All pivots require row swaps |
56+
57+
All storage is stack-allocated; $n$ is bounded at compile time by `Dim`.
58+
59+
## Step-by-Step Walkthrough
60+
61+
Consider a 2-D state ($n = 2$) with error $\varepsilon = [1, 2]^{\mathsf{T}}$ and covariance $P = \mathrm{diag}(1, 4)$.
62+
63+
1. Solve $P z = \varepsilon$: $z_1 = 1/1 = 1$, $z_2 = 2/4 = 0.5$.
64+
2. Compute dot product: $\varepsilon^{\mathsf{T}} z = 1 \cdot 1 + 2 \cdot 0.5 = 2.0$.
65+
3. Compare against $\chi^2_{2, 0.025} \approx 0.051$ and $\chi^2_{2, 0.975} \approx 7.38$.
66+
4. Since $2.0 \in [0.051, 7.38]$, the filter is declared consistent at the 95% level.
67+
68+
## Pitfalls & Edge Cases
69+
70+
A singular covariance matrix causes the linear solve to fail; the implementation detects near-zero diagonal pivots and returns `std::nullopt`. Callers must check the optional before using the value.
71+
72+
A single sample NEES or NIS value has high variance under chi-squared; a filter may appear inconsistent simply due to random variation. Time-averaging over many Monte-Carlo runs is the statistically correct procedure.
73+
74+
NEES requires ground-truth state access and is therefore only applicable in simulation. NIS is the consistency monitor of choice for deployed systems.
75+
76+
## Variants & Generalizations
77+
78+
The average NEES over a Monte-Carlo ensemble of $M$ runs and $K$ time steps yields $M \cdot K$ samples; consistency must hold jointly across the ensemble, not just per run. Extensions to non-Gaussian cases (e.g., particle filters) use empirical quantiles rather than chi-squared bounds.
79+
80+
## Applications
81+
82+
- Monte-Carlo evaluation of Kalman-family filters during design and tuning.
83+
- Online innovation gating in Kalman filters (rejecting outlier measurements whose NIS exceeds the threshold).
84+
- Diagnosing process-noise or measurement-noise mis-specification.
85+
- Formal consistency testing as part of filter validation before deployment.
86+
87+
## Connections to Other Algorithms
88+
89+
NEES and NIS are statistical companions to the standard Kalman filter update step. They depend on the covariance propagation produced by the `filters/active` family (KF, EKF, UKF). The linear solve reuses `solvers::GaussianElimination`, and the state-error representation aligns with `estimators::EstimationMetrics`.
90+
91+
## References & Further Reading
92+
93+
- Y. Bar-Shalom, X. R. Li, T. Kirubarajan, *Estimation with Applications to Tracking and Navigation*, Wiley, 2001. Chapter 5 (Estimation Consistency).
94+
- T. D. Barfoot, *State Estimation for Robotics*, Cambridge University Press, 2017. Chapter 9.
95+
- S. Julier, J. Uhlmann, "A New Extension of the Kalman Filter to Nonlinear Systems," *Proc. SPIE*, 1997.

doc/estimators/README.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,3 +16,9 @@ Statistical estimation algorithms for fitting models to observed data and making
1616
| Algorithm | Description |
1717
|-----------------------------------------------------|--------------------------------------------------------------------------|
1818
| [Recursive Least Squares](RecursiveLeastSquares.md) | Sample-by-sample parameter estimation with exponential forgetting factor |
19+
20+
## Consistency Metrics
21+
22+
| Algorithm | Description |
23+
|---------------------------------------------------------|-------------------------------------------------------------------------------------|
24+
| [Consistency Metrics (NEES/NIS)](ConsistencyMetrics.md) | Normalised Estimation Error Squared and Normalised Innovation Squared with χ² gates |
Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,25 @@
1+
numerical_add_header_library(numerical.estimators)
2+
3+
target_include_directories(numerical.estimators ${NUMERICAL_VISIBILITY}
4+
"$<BUILD_INTERFACE:${CMAKE_CURRENT_LIST_DIR}/../../>"
5+
"$<INSTALL_INTERFACE:${CMAKE_INSTALL_INCLUDEDIR}>"
6+
)
7+
8+
target_link_libraries(numerical.estimators ${NUMERICAL_VISIBILITY}
9+
infra.util
10+
numerical.math
11+
numerical.solver
12+
)
13+
14+
target_sources(numerical.estimators PRIVATE
15+
ConsistencyMetrics.hpp
16+
Estimator.hpp
17+
)
18+
19+
numerical_add_coverage_sources(numerical.estimators
20+
ConsistencyMetrics.cpp
21+
)
22+
123
add_subdirectory(offline)
224
add_subdirectory(online)
25+
add_subdirectory(test)
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
// Copyright 2025 Numerical Toolbox Contributors
2+
// SPDX-License-Identifier: MIT
3+
#include "numerical/estimators/ConsistencyMetrics.hpp"
4+
5+
namespace estimators
6+
{
7+
template class ConsistencyMetrics<float, 1>;
8+
template class ConsistencyMetrics<float, 2>;
9+
template class ConsistencyMetrics<float, 3>;
10+
}
Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
#pragma once
2+
3+
#if defined(__GNUC__) || defined(__clang__)
4+
#pragma GCC optimize("O3", "fast-math")
5+
#endif
6+
7+
#include "numerical/math/CompilerOptimizations.hpp"
8+
#include "numerical/math/Matrix.hpp"
9+
#include "numerical/solvers/GaussianElimination.hpp"
10+
#include <array>
11+
#include <cmath>
12+
#include <cstddef>
13+
#include <optional>
14+
15+
namespace estimators
16+
{
17+
namespace detail
18+
{
19+
static constexpr std::size_t kMaxChiSquareDim = 10;
20+
static constexpr std::size_t kNumAlpha = 1;
21+
22+
static constexpr std::array<float, kMaxChiSquareDim> kChi2Lo95 = {
23+
0.000982f, 0.050636f, 0.215795f, 0.484419f, 0.831212f,
24+
1.237344f, 1.689869f, 2.179731f, 2.700389f, 3.246973f
25+
};
26+
27+
static constexpr std::array<float, kMaxChiSquareDim> kChi2Hi95 = {
28+
5.023886f, 7.377759f, 9.348404f, 11.143480f, 12.832502f,
29+
14.449376f, 16.012764f, 17.534546f, 19.022768f, 20.483177f
30+
};
31+
}
32+
33+
template<typename T, std::size_t Dim>
34+
class ConsistencyMetrics
35+
{
36+
static_assert(std::is_floating_point_v<T>, "ConsistencyMetrics supports floating-point types");
37+
static_assert(Dim >= 1 && Dim <= detail::kMaxChiSquareDim,
38+
"ConsistencyMetrics Dim must be in [1, kMaxChiSquareDim]");
39+
40+
public:
41+
using StateVector = math::Vector<T, Dim>;
42+
using CovarianceMatrix = math::Matrix<T, Dim, Dim>;
43+
44+
[[nodiscard]] static OPTIMIZE_FOR_SPEED std::optional<T> Nees(const StateVector& error, const CovarianceMatrix& covariance);
45+
[[nodiscard]] static OPTIMIZE_FOR_SPEED std::optional<T> Nis(const StateVector& innovation, const CovarianceMatrix& innovationCovariance);
46+
[[nodiscard]] static bool IsConsistent(T value);
47+
[[nodiscard]] static bool IsTimeAveragedConsistent(T averagedValue, std::size_t numSamples);
48+
49+
private:
50+
[[nodiscard]] static OPTIMIZE_FOR_SPEED std::optional<StateVector> Solve(const CovarianceMatrix& matrix, const StateVector& rhs);
51+
[[nodiscard]] static OPTIMIZE_FOR_SPEED T DotProduct(const StateVector& a, const StateVector& b);
52+
};
53+
54+
template<typename T, std::size_t Dim>
55+
OPTIMIZE_FOR_SPEED std::optional<T> ConsistencyMetrics<T, Dim>::Nees(const StateVector& error, const CovarianceMatrix& covariance)
56+
{
57+
auto z = Solve(covariance, error);
58+
if (!z.has_value())
59+
return std::nullopt;
60+
return DotProduct(error, z.value());
61+
}
62+
63+
template<typename T, std::size_t Dim>
64+
OPTIMIZE_FOR_SPEED std::optional<T> ConsistencyMetrics<T, Dim>::Nis(const StateVector& innovation, const CovarianceMatrix& innovationCovariance)
65+
{
66+
auto z = Solve(innovationCovariance, innovation);
67+
if (!z.has_value())
68+
return std::nullopt;
69+
return DotProduct(innovation, z.value());
70+
}
71+
72+
template<typename T, std::size_t Dim>
73+
bool ConsistencyMetrics<T, Dim>::IsConsistent(T value)
74+
{
75+
return value >= static_cast<T>(detail::kChi2Lo95[Dim - 1]) &&
76+
value <= static_cast<T>(detail::kChi2Hi95[Dim - 1]);
77+
}
78+
79+
template<typename T, std::size_t Dim>
80+
bool ConsistencyMetrics<T, Dim>::IsTimeAveragedConsistent(T averagedValue, std::size_t numSamples)
81+
{
82+
if (numSamples == 0)
83+
return false;
84+
const std::size_t dof = numSamples * Dim;
85+
const float lo = (dof <= detail::kMaxChiSquareDim)
86+
? detail::kChi2Lo95[dof - 1] / static_cast<float>(numSamples)
87+
: detail::kChi2Lo95[detail::kMaxChiSquareDim - 1] / static_cast<float>(numSamples);
88+
const float hi = (dof <= detail::kMaxChiSquareDim)
89+
? detail::kChi2Hi95[dof - 1] / static_cast<float>(numSamples)
90+
: detail::kChi2Hi95[detail::kMaxChiSquareDim - 1] / static_cast<float>(numSamples);
91+
return static_cast<float>(averagedValue) >= lo &&
92+
static_cast<float>(averagedValue) <= hi;
93+
}
94+
95+
template<typename T, std::size_t Dim>
96+
OPTIMIZE_FOR_SPEED std::optional<typename ConsistencyMetrics<T, Dim>::StateVector> ConsistencyMetrics<T, Dim>::Solve(const CovarianceMatrix& matrix, const StateVector& rhs)
97+
{
98+
for (std::size_t i = 0; i < Dim; ++i)
99+
{
100+
float pivot = std::abs(static_cast<float>(matrix.at(i, i)));
101+
if (pivot < 1e-10f)
102+
return std::nullopt;
103+
}
104+
105+
solvers::GaussianElimination<T, Dim> solver;
106+
return solver.Solve(matrix, rhs);
107+
}
108+
109+
template<typename T, std::size_t Dim>
110+
OPTIMIZE_FOR_SPEED T ConsistencyMetrics<T, Dim>::DotProduct(const StateVector& a, const StateVector& b)
111+
{
112+
T result{};
113+
for (std::size_t i = 0; i < Dim; ++i)
114+
result += a.at(i, 0) * b.at(i, 0);
115+
return result;
116+
}
117+
118+
#ifdef NUMERICAL_TOOLBOX_COVERAGE_BUILD
119+
extern template class ConsistencyMetrics<float, 1>;
120+
extern template class ConsistencyMetrics<float, 2>;
121+
extern template class ConsistencyMetrics<float, 3>;
122+
#endif
123+
}
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
add_executable(numerical.estimators_test)
2+
emil_build_for(numerical.estimators_test BOOL NUMERICAL_TOOLBOX_BUILD_TESTS)
3+
emil_add_test(numerical.estimators_test)
4+
5+
target_link_libraries(numerical.estimators_test PUBLIC
6+
gmock_main
7+
numerical.estimators
8+
)
9+
10+
target_sources(numerical.estimators_test PRIVATE
11+
TestConsistencyMetrics.cpp
12+
)

0 commit comments

Comments
 (0)