estimation_evaluation_toolbox

A Python toolkit for evaluating robotics state estimators. It provides:

• Trajectory I/O — load/save trajectories in TUM format (SE(3) and SE(2,3))
• Trajectory alignment — align estimated trajectories to ground truth via first-pose or Umeyama SVD methods, with support for known sensor–body extrinsics
• APE metrics — Root-Mean-Square Absolute Pose Error (rotation and translation) via the evo library
• Orchestration framework — abstract Orchestrator base class and Analysis driver for systematic, reproducible multi-dataset / multi-parameter evaluation runs
• Hand-eye calibration — quaternion-based SO(3) calibration (Daniilidis) and OpenCV SE(3) calibration
• Plotting utilities — trajectory, velocity, Euler-angle, and timestamp plots
• Simulation — synthetic trajectory generation and ROS bag writer

Installation

pip install -e .

Two small optional extras for less-used features:

pip install -e ".[geo]"   # adds pyproj, needed for ecef_to_enu.py
pip install -e ".[cv]"    # adds opencv, needed for hand_eye_cv2()

ROS features (BagDataset.gt_states(), BagDataset.timestamps(), simulation/create_sim_trajectories.py) require a ROS environment with rosbag available — these cannot be installed via pip.

Quick start

1. Define your dataset

from estimation_evaluation_toolbox import BagDataset, TopicConfig, SensorType

dataset = BagDataset(
    dataset_id="my_run",
    bag_path="/data/my_run.bag",
    gt_topic="/groundtruth",
    pipeline_specific_data={"config_template": "/cfg/default.yaml"},
    topic_configs=[
        TopicConfig("/lidar0", SensorType.LIDAR),
        TopicConfig("/imu",    SensorType.IMU),
    ],
)

2. Implement an Orchestrator

import functools
from estimation_evaluation_toolbox import Orchestrator
from estimation_evaluation_toolbox.io.text_files import load_from_tum_format
from estimation_evaluation_toolbox.alignment import align_sensor_traj_to_gt
from estimation_evaluation_toolbox.metrics import compute_ape
from estimation_evaluation_toolbox.utils.misc import run_terminal_command

class MyOrchestrator(Orchestrator):
    def __init__(self, *args, binary_path: str, **kwargs):
        super().__init__(*args, **kwargs)
        self.binary_path = binary_path

    def setup_config(self):
        # write self.config_path from self.dataset and self.overrider
        ...

    def execute(self, rerun=None):
        self.setup_config()
        run_terminal_command(
            cwd=self.run_folder,
            cmd=[self.binary_path, "--config", str(self.config_path)],
            wait=True,
            command_fname="run.log",
        )

    def align(self, rerun=None):
        estimated = load_from_tum_format(
            self.run_folder / "output.txt", C_ba=True
        )
        aligned = align_sensor_traj_to_gt(self.dataset.gt_states(), estimated)
        ape_rot, ape_pos = compute_ape(self.dataset.gt_states(), aligned)
        self._results = {"ape_rot_deg": ape_rot, "ape_pos_m": ape_pos}

    def plot(self, rerun=None):
        pass  # optional

    def to_rows(self, rerun=None):
        return [{
            **self.metadata,
            "dataset": self.dataset.dataset_id(),
            **self._results,
        }]

3. Run an Analysis

from pathlib import Path
from estimation_evaluation_toolbox import Analysis, OverrideConfig

factories = {
    "my_run": functools.partial(
        MyOrchestrator, binary_path="/usr/local/bin/my_estimator"
    ),
}

overriders = [
    OverrideConfig([(["Params", "MaxRange"], "max_range", 50.0)]),
    OverrideConfig([(["Params", "MaxRange"], "max_range", 100.0)]),
]

analysis = Analysis(
    analysis_id="range_sweep",
    orchestrator_factories=factories,
    dataset_list=[dataset],
    overriders=overriders,
    output_dir=Path("/results"),
)

df = analysis.execute()
print(df[["dataset", "max_range", "ape_rot_deg", "ape_pos_m"]])

Dependencies

Package	Purpose
numpy, scipy, matplotlib, seaborn	Core numerics and plotting
navlie	State representations, B-spline
pymlg	Lie group operations
evo	APE / RPE trajectory metrics
pyproj (optional, [geo])	ECEF ↔ ENU coordinate conversion
opencv-python (optional, [cv])	hand_eye_cv2 SE(3) calibration
rosbag (ROS environment)	Bag file loading and simulation output