Skip to content

Benchmark ModernTCN and DLinear on real REDD data - #9

Merged
nipunbatra merged 2 commits into
mainfrom
agent/t0-modern-models
Jul 17, 2026
Merged

Benchmark ModernTCN and DLinear on real REDD data#9
nipunbatra merged 2 commits into
mainfrom
agent/t0-modern-models

Conversation

@nipunbatra

@nipunbatra nipunbatra commented Jul 17, 2026

Copy link
Copy Markdown
Member

Summary

  • register ModernTCN and DLinear as NILMbench models
  • advance the single immutable nilmtk-contrib pin to merge commit 8d745493ed9f84dd00fb502ffe85943eaeedc4c8
  • build one CUDA campaign image for both models, pinned to runner adaf03e42f5b8dd6b9ab95942ee191999f2d3b25
  • publish six real REDD/A100 result bundles (two models × seeds 10, 20, 42)
  • regenerate the living leaderboard automatically from all 57 immutable runs
  • document the batched image policy: one campaign image for an eligible model batch, not one image per algorithm

Real T0 results

Protocol: corrected-t1-redd / fridge / 60s / seq 299 / 3 epochs / max 3960

Rank Model MAE (W) F1 Parameters Time Peak CUDA memory
7 ModernTCN 56.62 ± 1.77 0.659 ± 0.007 41,921 11.55 ± 1.33 s 91.3 MiB
14 DLinear 75.92 ± 2.73 0.570 ± 0.021 600 10.53 ± 0.57 s 64.6 MiB

The generated leaderboard now has one evaluation cohort and contiguous ranks 1–19.

Provenance

  • image: nilmbench:t0-adaf03e-cuda
  • image content digest: sha256:d7253754a6a9133235076fa1c1555104aa8be8128443da96bc16ae3d46809aa8
  • runner: adaf03e42f5b8dd6b9ab95942ee191999f2d3b25
  • nilmtk-contrib: 8d745493ed9f84dd00fb502ffe85943eaeedc4c8
  • hardware: NVIDIA A100-SXM4-80GB
  • REDD SHA-256: 757e694a420157aa035c8fd345915080c2396cb7a37bfeb5f925bcb84f9532b8

The fail-closed preflight and final read-only candidate verifier both passed.

Verification

  • uv lock --check
  • uv run ruff check src tests
  • uv run pytest — 129 passed, 1 skipped
  • existing 17-model leaderboard regenerated byte-for-byte before the campaign
  • final 19-model leaderboard regenerated byte-for-byte after publication
  • JSON commits to the exact generated CSV SHA-256
  • six unique result IDs; exact seeds 10/20/42 per model; 3,960 train and 2,448 test samples per run
  • all 19 aggregates are smoke-verified

@nipunbatra
nipunbatra merged commit 68fcbc4 into main Jul 17, 2026
1 check passed
@nipunbatra
nipunbatra deleted the agent/t0-modern-models branch July 17, 2026 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant