EdgeAI-Bench

Memory-aware benchmarking for small language models on robotic edge hardware.

This project starts on a MacBook Air M1 and is designed to extend later to Raspberry Pi, Jetson, or university lab hardware.

Research Goal

Measure whether small and quantized language models can deliver useful robot-level intelligence under edge constraints, and whether robot near-miss traces can be compiled into executable safety memory:

• limited RAM
• low latency
• local/offline inference
• practical answer quality
• controlled hallucination behavior

First Use Case

Edge robotic intelligence assistant.

The assistant supports robot diagnostics, safe fallback decisions, sensor-conflict handling, and operator-facing explanations without relying on a cloud model.

Current Milestone

The current milestone has shifted from a sleep-metaphor prototype to a near-miss safety-memory compiler for robotic edge intelligence:

1. Represent robot near-misses as structured traces.
2. Compile those traces into executable safety rules.
3. Generate counterfactual hazard scenarios from the compiled memories.
4. Run ablations across base model, structured JSON, static envelope, and compiled memory.
5. Use real local model outputs as the next paper experiment.

Current best quality model from Round 3:

• qwen2.5:1.5b

Current fastest edge candidate:

• gemma3:1b

Latest structured-safety result:

• gemma3:1b improved from 10% hallucination to 0% hallucination under structured robot-safety output, but average latency increased from 2.22s to 9.60s.
• qwen2.5:1.5b achieved 100% valid JSON and 100% safe-action rate in Round 4, with higher memory use than Gemma.

Latest safety-envelope result:

• A deterministic safety envelope corrected Gemma's remaining unsafe action from inspect to request_data for an unknown torque-limit question.
• Gemma safe-action rate improved from 90% to 100% with about 1.642 ms total post-processing time over 20 decisions.

Latest sleep-memory result:

• A sleep-like consolidation pass converted 14 safety-relevant wake episodes into 5 reusable safety memories, 18 dream scenarios, and 5 proposed safety rules in about 1.198 ms.
• On the next-wake dream benchmark, gemma3:1b improved from 72.2% to 100.0% safe action after the envelope, while qwen2.5:1.5b remained at 100.0%.
• The evaluation now also checks whether a model understates safety risk, not just whether it picks the right action label.

Latest hard-dream result:

• A 25-scenario hard dream set tested paraphrased robot safety cases across low confidence, sensor conflict, physical instability, network loss while loaded, and unknown exact specifications.
• gemma3:1b improved from 96.0% to 100.0% safe action and from 76.0% to 100.0% risk-not-understated after the safety envelope.
• qwen2.5:1.5b achieved 100.0% safe action and 100.0% risk-not-understated without envelope overrides, though one conservative stop lowered exact action match to 96.0%.

Latest compiler result:

• A seed set of 10 structured near-miss traces compiled into 7 executable safety rules, 7 safety memories, and 9 counterfactual hazard scenarios.
• The compiler now captures higher-order robot risks such as uncertainty near humans and offline movement while loaded on ramps or in narrow spaces.
• A deterministic smoke ablation improved from 55.6% safe action with static rules to 100.0% safe action with compiled memory. This is only a pipeline validation result; real Ollama model outputs are the next required experiment.

Latest real-model compiler result:

• On the 9 compiler-generated counterfactual hazards, gemma3:1b scored 55.6% safe action as a structured base model and remained 55.6% under the static envelope, but reached 100.0% under compiled memory.
• qwen2.5:1.5b scored 66.7% safe action as a structured base model and remained 66.7% under the static envelope, but reached 100.0% under compiled memory.
• The missed cases were higher-order hazards: uncertainty near humans and offline loaded movement on ramps or near workers. These are exactly the cases captured by compiled escalation rules.
• Figure: figures/compiler_round1_real_model_ablation.png

Latest compiler Round 2 result:

• A 50-trace expanded near-miss set compiled into 13 executable rules, 13 safety memories, and 15 counterfactual hazards in 3.284 ms after tightening the offline-loaded and human-near-uncertainty rules.
• New higher-order risk families include low battery with loaded ramp movement, poor traction with payload, overheated actuator motion, emergency-stop faults, moving arms near humans, and blind reverse motion.
• On the 15-scenario real-model benchmark, gemma3:1b improved from 33.3% safe action as a structured base model to 100.0% under compiled memory. qwen2.5:1.5b improved from 46.7% to 100.0%.
• Static envelope did not improve safe action in Round 2 because the new failures require higher-order escalation rules, not single-risk checks.
• Figure: figures/compiler_round2_real_model_ablation.png

Latest compiler Round 3 negative-control result:

• A 20-scenario negative-control benchmark tested whether compiled memory stops the robot too often.
• Compiled memory added 0.0% unnecessary overrides for both gemma3:1b and qwen2.5:1.5b.
• Static envelope also added 0.0% unnecessary overrides after tightening the offline-loaded rule to require active motion.
• Remaining false stops came from the base model outputs: 15.0% for gemma3:1b and 5.0% for qwen2.5:1.5b; neither static nor compiled memory added to them.
• Figure: figures/compiler_round3_negative_controls.png

Latest compiler Round 4 compositional result:

• A mixed 24-case compositional benchmark tested 12 high-risk hazards and 12 non-stop cases in one run.
• On high-risk cases, compiled memory reached 100.0% safe action for both gemma3:1b and qwen2.5:1.5b; base models were 8.3%.
• On non-stop cases, compiled memory added 0.0% unnecessary overrides for both models.
• Overall safe action improved from 25.0% to 70.8% for gemma3:1b and from 29.2% to 75.0% for qwen2.5:1.5b.
• Figures: figures/compiler_round4_compositional_eval.png, figures/near_miss_compiler_architecture.png

Latest compiler Round 5 discovery result:

• Added src/edgeai_bench/discover_safety_rules.py, which mines Boolean rule candidates from near-miss traces and exports approved candidates in the runtime rule DSL.
• From 50 traces, the discovery pass scored 466 candidates and selected 7 validation-clean executable rules against 44 negative/non-stop validation rows.
• On the Round 4 real-model benchmark, discovered rules reached 100.0% high-risk safe action for both gemma3:1b and qwen2.5:1.5b.
• On Round 3 negatives and Round 4 non-stop cases, discovered rules added 0.0% unnecessary overrides.
• Observation doc: docs/compiler_round5_discovery_observations.md

Latest compiler Round 6 threshold-discovery result:

• Added src/edgeai_bench/make_compiler_round6_threshold_set.py to generate varied-threshold traces with train/test split.
• Generated 129 synthetic threshold traces: 69 train and 60 held-out test.
• Discovery learned tighter thresholds: battery_pct<=19, actuator_temp_c>=83, human_distance_m<=1.2, and perception_confidence<=0.55 in the relevant conjunctions.
• Simulated smoke test: discovered thresholds reached 100.0% exact/safe action with 0.0% unnecessary overrides; the round-number baseline reached 88.3% exact action with 11.7% false stops.
• Real gemma3:1b outputs: discovered thresholds improved exact action from 45.0% to 56.7% versus the round-number baseline and reduced false stops from 11.7% to 0.0%.
• Real qwen2.5:1.5b outputs: discovered thresholds improved exact action from 33.3% to 41.7% versus the round-number baseline and reduced unnecessary overrides from 8.3% to 0.0%.
• Observation doc: docs/compiler_round6_threshold_discovery_observations.md

Latest compiler Round 7 cross-domain discovery result:

• Added src/edgeai_bench/make_compiler_round7_compositional_set.py for cross-domain compositional hazards.
• Generated 85 synthetic cross-domain traces: 55 train and 30 held-out test.
• Discovered rules matched or exceeded the human cross-domain baseline and outperformed the original compiler.
• Real gemma3:1b outputs: discovered rules reached 63.3% exact action versus 60.0% for human and 40.0% for original compiler, with 0.0% unnecessary overrides.
• Real qwen2.5:1.5b outputs: discovered and human rules tied at 70.0% exact action, while original compiler reached 56.7%.
• Figure: figures/discovery_rules_comparison_round6_round7.png
• Observation doc: docs/Round_7_personal_coding.md

Latest compiler Round 8 unified hard benchmark:

• Single 22-case held-out benchmark mixing threshold-borderline cases, cross-domain compositional hazards, bait-and-switch traps, and negative controls.
• Real gemma3:1b outputs: base model reached 36.4% exact action, original compiler 45.5%, and human-written compiled memory 54.5%.
• Real qwen2.5:1.5b outputs: base model reached 31.8% exact action, original compiler 40.9%, and human-written compiled memory 45.5%.
• Figure: figures/unified_hard_benchmark.png

Current paper draft:

• arXiv-style source: paper/robots_that_sleep_on_near_misses_arxiv.tex
• compiled PDF: paper/build/robots_that_sleep_on_near_misses_arxiv.pdf
• current title: "Near-Miss Safety Memory Compilation for Edge Robotic Decision Support"
• latest paper version includes the architecture figure, Round 4 compositional held-out benchmark, Round 5 risk-family discovery, Round 6 threshold discovery, Round 7 cross-domain discovery, discovery comparison figure, Round 8 unified hard benchmark, and unified hard benchmark figure

Quick Start

Check local Ollama models:

ollama list

Run the robot-edge benchmark:

python3 src/edgeai_bench/benchmark_ollama.py \
  --models gemma3:1b qwen2.5:1.5b llama3.2:latest \
  --questions data/questions/robot_edge_eval.jsonl \
  --out-dir results/robot_round3

Build a versioned safety-memory snapshot from traces:

python3 src/edgeai_bench/update_safety_memory.py \
  --traces data/traces/final_train.jsonl \
  --split train \
  --memory-dir results/safety_memory_versions \
  --questions-out data/questions/safety_memory_counterfactuals.jsonl

Replay and compare memory versions on the same benchmark:

python3 src/edgeai_bench/replay_memory_versions.py \
  --memory-dir results/safety_memory_versions \
  --questions data/traces/final_test.jsonl \
  --out-dir results/safety_memory_replay

Run the full memory pipeline end to end:

python3 src/edgeai_bench/run_safety_memory_pipeline.py \
  --traces data/traces/final_train.jsonl \
  --benchmark data/traces/final_test.jsonl \
  --memory-dir results/safety_memory_versions \
  --out-dir results/safety_memory_pipeline \
  --split train

Run the standalone static baseline on a saved memory version:

python3 src/edgeai_bench/run_static_memory_baseline.py \
  --questions data/traces/final_test.jsonl \
  --memory-dir results/phase12_suite_final_50/memory \
  --memory-version v001 \
  --responses results/final/real_model_benchmark/responses.jsonl \
  --out-dir results/phase12_suite_final_50/static_baseline

Build the paper bundle with strict vs relaxed policy comparison:

python3 src/edgeai_bench/make_safety_memory_paper_bundle.py \
  --suite-dir results/phase12_suite_final_50 \
  --suite-dir results/phase12_suite_blind \
  --suite-dir results/phase12_suite_mujoco_blind \
  --out-dir results/paper/safety_memory

Run the public robot-failure path on a laptop:

python3 src/edgeai_bench/make_public_robot_failure_benchmark.py \
  --out-dir results/public_robot_failure

python3 src/edgeai_bench/run_public_robot_failure_suite.py \
  --train-traces results/public_robot_failure/public_robot_failure_train.jsonl \
  --validation-traces results/public_robot_failure/public_robot_failure_validation.jsonl \
  --benchmark results/public_robot_failure/public_robot_failure_benchmark.jsonl \
  --out-dir results/public_robot_failure_suite_public

This path is intended for MacBook Air M1 / 8 GB runs and downloads the UCI Robot Execution Failures archive directly from UCI.

Use --missing-family-policy ignore on run_public_robot_failure_suite.py to replay the relaxed gate instead of the default strict gate.

Project Layout

docs/
  how_to_tackle.md
  experiment_section_draft.md
  near_miss_safety_memory_compiler.md
  compiler_round1_real_model_observations.md
  compiler_round2_real_model_observations.md
  compiler_round3_negative_control_observations.md
  compiler_round4_compositional_observations.md
data/
  questions/
    maintenance_eval.jsonl
    robot_edge_eval.jsonl
    dream_round2_hard_eval.jsonl
    compiler_counterfactual_eval.jsonl
    compiler_round2_heldout_eval.jsonl
    compiler_round3_negative_eval.jsonl
    compiler_round4_compositional_eval.jsonl
  traces/
    near_miss_trace_seed.jsonl
    near_miss_trace_round2.jsonl
results/
  sleep_round1/
  analysis_next_wake_round1/
  next_wake_round2_hard_structured/
  analysis_next_wake_round2_hard/
  compiler_round1/
  compiler_round2/
  compiler_round3_negative/
  compiler_round4_compositional/
figures/
  compiler_round1_real_model_ablation.png
  compiler_round2_real_model_ablation.png
  compiler_round3_negative_controls.png
  compiler_round4_compositional_eval.png
  near_miss_compiler_architecture.png
paper/
  robots_that_sleep_on_near_misses_arxiv.tex
  references.bib
  build/
    robots_that_sleep_on_near_misses_arxiv.pdf
src/
  edgeai_bench/
    benchmark_ollama.py
    apply_safety_envelope.py
    sleep_consolidation.py
    evaluate_next_wake.py
    compile_safety_memory.py
    run_compiler_ablations.py
    safety_memory_lifecycle.py
    update_safety_memory.py
    replay_memory_versions.py
    run_safety_memory_pipeline.py
    make_compiler_figure.py
    make_compiler_round2_trace_set.py
    make_compiler_round3_negative_set.py
    make_compiler_round4_compositional_set.py
    make_compiler_negative_figure.py
    make_compiler_round4_figure.py
    make_architecture_figure.py
    make_unified_hard_figure.py