docs(research): record TODO(nowcast) decoding-scheme plan in lstm deep dive - #1
Open
jajupmochi wants to merge 21 commits into
Open
docs(research): record TODO(nowcast) decoding-scheme plan in lstm deep dive#1jajupmochi wants to merge 21 commits into
jajupmochi wants to merge 21 commits into
Conversation
…p dive Per-step vs DMS vs produce-then-truncate comparison across the swiss benchmark, TSL, and this repo; action items before any nowcast experiment (per-step head for lstm, audit other models, optional per-step aux loss). Mirrors the inline TODO(nowcast) in liulian/models/torch/lstm.py. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Repo-visible record of work items (the agent session task list was not visible in the repo). Includes the nowcast decoding TODO, multi_channel coordinates injection, Case-2 outer sweep, sin/random backfill, Ray local_mode cleanup, and the completed 2026-05/06 milestones. Convention added to CLAUDE.md: update the ledger in the same commit when work finishes or a new item is decided. mkdocs nav: Task Ledger entry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
swiss_river.py CASE-1/2 note, lstm.py benchmark URL line, trainer.py batch todo — comments only, no behaviour change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ledger now lives at liulian-docs/docs/tasks/liulian-python/<YYYY-MM>.md (per-repo sub-dirs, monthly files, topic tags). docs/tasks.md stays as a pointer so in-repo links and the mkdocs nav keep working; CLAUDE.md convention updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sinusoidal_dim / random_identifier_dim were gated out of per_entity HPO
because the loaders are frozen per run (a sampled dim would have been a
dead knob). Per user requirement they are now REAL inner-HPO knobs in
both split modes:
- make_trainable gains loaders_factory: when a trial samples a
DATA_LAYER_DIM_KEYS param, the trial rebuilds dataset+loaders with the
sampled dim inside the Ray worker and re-syncs enc_in from a real batch
- experiment.py passes a zero-capture factory (build_dataset+build_loaders)
and, post-HPO, rebuilds the loaders with the WINNING dim before the
final checkpoint-load + evaluation (enc_in synced on best_args too)
- resolve_search_space: per_entity gating removed; DATA_LAYER_DIM_KEYS
promoted to the public contract
Verified by a real 4-trial HPO on swiss-river-zurich+sinusoidal: trials
sampled dims {3, 14, 16, 17}, best_config carries sinusoidal_dim=3, and
the final evaluation ran on rebuilt dim-3 loaders with finite metrics.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rapper
Per-entity transparent identifiers (onehot/coordinates/sinusoidal/random/
numeric_id) now follow the same mechanism as embedding and multi_channel:
MODEL-layer injection via a wrapper that is rebuilt per HPO trial.
- EntityTransparentWrapper: per-sample fixed-feature concat (NO projection,
no learnable params); table built via the data-layer make_entity_features
for constructive bitwise equivalence with the legacy bake-in
- id_injection config knob: 'model' (canonical) | 'data' (legacy fallback);
dataset skips baking under 'model' while still emitting entity ids
- pipeline: enc_in widened by the identifier dim + wrapper mounted
- trainer: pass_entity_ids covers model-injected transparent modes
- experiment: per-trial factory rebuilds inner model at enc_in = base + dim;
loaders_factory + post-HPO loader rebuild now gated to id_injection='data'
- FIX latent overwrite bug: SwissRiverDataset.super().__init__ silently
reset sinusoidal_dim / random_identifier_dim / random_identifier_seed /
id_injection to parent defaults — custom dims never reached the data
path (this also invalidated the loaders_factory dim-tuning verification;
v2 verification now checks trial checkpoint weight shapes)
- tests: 15-case equivalence suite (table / layout / swiss end-to-end)
Verified: 4-trial HPO on zurich+sinusoidal — every trial checkpoint's
LSTM input width == 1 + sampled dim ({3,14,16,17}), best=3, finite final
metrics; 358 guard tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR adds a model-layer “transparent” entity-identifier injection path (id_injection='model') intended to be bitwise-equivalent to the legacy data-layer bake-in, while also updating HPO plumbing so identifier-dimension knobs can be tuned without becoming dead parameters. It also includes research/docs updates (task ledger pointer and LSTM nowcast decoding plan).
Changes:
- Introduce
EntityTransparentWrapperand wireid_injectionthrough dataset/model/trainer/experiment so per-entity transparent modes can be injected in the model layer. - Update Ray Tune integration to rebuild loaders per trial when tuning data-layer identifier dimensions, and resync
enc_infor best-config retraining. - Add equivalence tests plus docs/config updates (MkDocs nav + ledger stub + LSTM deep-dive TODO(nowcast) note).
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
tests/models/torch/test_transparent_injection_equivalence.py |
Adds tests asserting bitwise equivalence between data-bake-in and model-layer transparent injection. |
mkdocs.yml |
Adds “Task Ledger” to MkDocs navigation. |
liulian/runtime/trainer.py |
Passes entity_ids to models for id_injection='model' transparent modes (not just embedding). |
liulian/runtime/experiment.py |
Extends HPO rebuild logic for per-entity transparent injection and adds per-trial loader rebuild support for tuned data-layer dims. |
liulian/pipeline.py |
Threads id_injection to SwissRiverDataset and adds model-layer transparent injection wrapping + enc_in widening. |
liulian/optim/search_spaces.yaml |
Updates documentation/comments to reflect that identifier dims are now tunable with proper per-trial rebuild behavior. |
liulian/optim/search_spaces.py |
Renames/exports data-layer dim keys and removes prior gating that made some dims untunable in per-entity splits. |
liulian/optim/ray_optimizer.py |
Adds loaders_factory support to rebuild loaders per trial when sampling data-layer identifier dims and resync enc_in. |
liulian/models/torch/lstm.py |
Clarifies reference notes around decoding scheme; keeps TODO(nowcast) pointer. |
liulian/models/torch/entity_mixin.py |
Adds EntityTransparentWrapper for per-sample fixed feature injection in per-entity mode. |
liulian/data/ts/timeseriesdataset.py |
Adds id_injection option to skip data-layer bake-in when model-layer injection is selected. |
liulian/data/swiss_river.py |
Threads id_injection through SwissRiverDataset and down into per-station TimeSeriesDataset construction. |
liulian/config.py |
Adds default id_injection: model (documented as canonical). |
experiments/swiss_river/default_config.yaml |
Adds id_injection: model to the default Swiss River experiment config. |
docs/tasks.md |
Adds an in-repo stub page pointing to the external ledger hub repo. |
docs/research/entity-id-deep/lstm.md |
Records TODO(nowcast) decoding scheme plan and action items. |
CLAUDE.md |
Documents the task ledger location/conventions. |
Comment on lines
+336
to
+340
| mode = str(config.get('identifier_mode', 'none')).strip().lower() | ||
| if str(config.get('id_injection', 'data')).strip().lower() != 'model': | ||
| return False | ||
| if config.get('split_mode') == 'multi_channel': | ||
| return False |
Comment on lines
+427
to
+433
| self.inner = inner_model | ||
| self.mode = str(mode) | ||
| self.station_ids = [str(s) for s in station_ids] | ||
| self._coordinates = coordinates | ||
| self.sinusoidal_dim = int(sinusoidal_dim) | ||
| self.random_dim = int(random_dim) | ||
| self.random_seed = int(random_seed) |
Comment on lines
375
to
380
| from liulian.models.torch.entity_mixin import ( | ||
| ChannelEntityWrapper, | ||
| ChannelTransparentWrapper, | ||
| EntityTransparentWrapper, | ||
| EntityWrapper, | ||
| ) |
- tools/plot_swiss3dt_results.py: reads denorm RMSE from real results.json (no hardcoded numbers), emits grouped-bar + coordinates old/new figures and a CSV - docs/research/2026-06-13-swiss3dt-results.md: 18-cell table, research- critic caveats (single seed, enc_in capacity confound, same domain — no "transparent > embedding" claim), and the coordinates flip explanation (old = zero-vector bug, retracted; new = real + min-max normalized) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… baseline The 2026-05-15 advisor slide reported coordinates as regressing the LSTM (+4.1%). That was a zero-vector bug (topology never loaded under graph_mode='none' -> make_entity_features zero-filled), not "unscaled lat/lon". Annotate the slide in place (comment-don't-delete): top correction banner + inline [R] retractions on the table cell, heatmap column, §2(4), §5.1, and the next-steps line. Original snapshot text preserved as the as-presented record. Also fix the coordinate comparison figure/writeup to use the run the slide actually came from (swissriver-lstm-REAL-20260512, whose onehot 1.1717 matches the slide's 1.171) instead of the under-trained fullmatrix-0511: old coord 1.630 vs none 1.561 (+4.4%) -> new 1.155 vs 1.723 (-33%), same denorm-RMSE units, comparable none levels. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…r/patchtst
Four multi_channel bugs surfaced by a smoke matrix (swiss × {dlinear,
patchtst} × all modes), all fixed:
- P1 transparent identifiers were a DEAD KNOB in mc: ChannelTransparentWrapper
is only mounted by the adapter's _init_entity_support, but pipeline.build_model
instantiates the model class directly (not via adapter) and only mounted the
embedding wrapper. So onehot/sinusoidal/random/coordinates RMSE == none
(this also invalidates the 2026-05 dlinear/patchtst transparent results).
Fix: pipeline.build_model now mounts ChannelTransparentWrapper for mc +
transparent (via _transparent_injection_kind), unifying with the per_entity
EntityTransparentWrapper path; this also activates the previously-dead
per-trial rebuild in experiment.py.
- P2 coordinates injection was never wired (config['coordinates'] unset);
now pulled from dataset.topology at mount time.
- P3 mc swiss-river-2010/zurich were all-NaN (target _wt carries NaNs;
feature _at is clean). Fix: nan_mask_loss:true in the swiss mc configs +
_compute_metrics now masks NaN targets unconditionally (no-op on clean data).
- P4 was a consequence of P1 (rebuild branch unreachable); now live.
Hyperparameters (user decision 2026-06-14): new dlinear_swiss / patchtst_swiss
spaces with batch_size FIXED at 32 (not tuned, matches swiss lstm) and
patch_len/stride fixed 16/8; traffic/electricity keep the batch-tuned global
spaces. matrix: coordinates restored for all swiss pairs.
Verified: 36-cell smoke (2 models × 3 datasets × 6 modes) all PASS, 0 NaN,
transparent ≠ none; mc sinusoidal_dim genuinely tuned per trial (4-trial HPO,
channel_features width == sampled dim {8,20,23,25}); 493 guard tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mc HPO rebuild read coordinates/station_ids from config, but those are never written there (pipeline pulls them from dataset.topology). So every trial rebuilt with integer station ids: harmless for onehot/sinusoidal (index-based) but the coordinates mode raised "missing coordinate for ['0','1',...]" and ALL 6 swiss mc coordinate cells failed in the full run (smoke missed it — smoke runs hpo=False, skipping the rebuild path). Fix mirrors the per_entity EntityTransparentWrapper: ChannelTransparentWrapper now stores mode/station_ids/coordinates/dims/seed, and experiment.py captures them from the model instance for the per-trial factory. Verified: coordinates survives a real 3-trial HPO rebuild (rmse 1.3327, no raise); 74 guard tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…udit 36-cell multi_channel results (read from real results.json). Audit-corrected framing: DLinear shows no measurable identifier effect (flat, val mirrors test); on PatchTST the INJECTION POINT drives it — add_after_patch embedding helps (~-5%), pre-patch concat_to_x transparent fusion hurts (+30-60%). NOT claimed: "transparent harmful" (embedding is also an identifier and helps) or "DLinear immune" (can't separate model from projection). Key caveat: mc transparent results reflect ChannelTransparentWrapper's fusion projection, not intrinsic identifier value. Contrast table vs per_entity LSTM notes the split-mode confound. Follow-up ablations listed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jajupmochi
force-pushed
the
exp/entity-identifier-2026-06
branch
from
June 14, 2026 13:35
450911c to
59f44bb
Compare
…OOM)
The fusion projection was applied to a materialized (B,T,N,1+D) tensor,
which is O(B·T·N²) for onehot (D=N) and OOMed on big multi_channel datasets
(traffic N=862 tried to allocate ~9 GiB; smoke FAILed). Since the projection
is linear, proj(cat([x, feats])) == w_val*x + (feats@w_feat + bias) per
channel — O(B·T·N + N·D), no 4D materialization. Numerically identical
(verified, max diff ~1e-7).
Adds ChannelTransparentWrapper tests: algebraic == materializing reference
for all transparent modes, and onehot at N=800 runs without the N² blowup.
Verified: traffic+electricity × {lstm,dlinear,patchtst} × 5 modes smoke all
PASS (onehot no longer OOMs).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tools/build_entity_id_figures.py scans all run-tag results.json (no
hardcoded numbers, resume-safe — latest per cell) and emits:
- heatmap-vs-none.png: % test-RMSE change vs each (dataset,model)'s own
none baseline (green=better, red=worse)
- results-table.{tex,pdf}: actual RMSE, best cell per dataset bolded,
compiled with pdflatex
Current snapshot (55 cells): swiss lstm per_entity all green (-11..-35%);
swiss patchtst mc transparent all red (+32..+85%, pre-patch fusion);
swiss dlinear flat; traffic/electricity lstm in progress. Re-run with
--pull after new cells finish.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Table now has two emphases: bold = best over all models & modes for a dataset; underline = best id-mode within each (model, dataset) row. Legend row added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…identifier Root-cause mechanism (verified numerically, max diff 8e-6): PatchTST's per-channel instance normalization subtracts each channel's time-mean, and all transparent identifiers are per-channel CONSTANTS injected pre-norm via concat_to_x, so the norm erases them exactly — leaving only the fusion projection's distortion (the +30-85% regression). embedding survives via add_after_patch (post-norm, d_model token space). Transparent can't use add_after_patch today: gated to embedding + dimension mismatch (needs a Linear(D,d_model) projection) — the clean ablation to run next. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…on (task #40) Refactors collect() -> collect_tags(tags) and adds build_ablation_table(): a separate concat_to_x-vs-add_after_patch table for patchtst transparent modes (per-dataset, bolds the better injection, shows none ref). Skips cleanly (no file) until ABLATION_TAGS is populated, so it's safe on every watcher refresh. Adds MAIN_PATCHTST_SOURCE knob (concat|add_after_patch| better) for which injection the MAIN table uses per cell — defaults to concat; final pick is the user's once the ablation runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…m subagent) Transparent identifiers (onehot/sin/random/coordinates) can now inject via add_after_patch: PatchTST builds the fixed (N,D) table internally, projects it to d_model with a learned Linear, and adds it to patch tokens downstream of instance-norm — bypassing the norm that erases the concat_to_x (pre-norm) constant injection. The clean ablation for "injection point vs identifier type". Opt-in via env LIULIAN_PATCHTST_TRANSPARENT_ADD_AFTER_PATCH; default concat_to_x unchanged. patchtst.py + test_patchtst.py taken from the subagent (its base was clean there). pipeline.py + matrix.py were RE-DONE by hand on the current structure: the subagent's worktree branched from an old commit (a3dc2f9, pre the id_injection / unified-wrapper / mc-mount work), so its pipeline/ matrix edits didn't apply — its "concat_to_x doesn't inject" flag was a misread of that old tree (verified: concat_to_x DOES mount ChannelTransparentWrapper on current main). _transparent_injection_kind now returns None for patchtst add_after_patch (handled internally, no wrapper); build_model injects coords/station_ids/dims onto config for that path. Verified on current tree: concat_to_x onehot still mounts the wrapper; add_after_patch onehot/coordinates build the internal id_proj+id_table (no wrapper, no raise); HPO sanity (2-trial Ray) PASS; 88 model/matrix tests green. Smoke hint: add_after_patch onehot 1.42 vs concat_to_x 2.19 — the instance-norm hypothesis looks right, pending the full HPO ablation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sume) build_hpo_experiment_name appended a timestamp unconditionally, so every resubmit after a walltime TIMEOUT started a FRESH Ray experiment and re-ran all trials from scratch. Harmless when a cell finishes within one walltime (swiss/electricity), but the big traffic cell (>12h, 50 trials) never finished: 3 fresh experiments, ~129 trials, 0 cells done, ~36 GPU-h wasted. Now the timestamp is omitted when hpo_resume is set, so a resumed run targets the SAME experiment dir and Ray continues the remaining trials. Each matrix cell has its own hpo_storage_path, so the stable name stays unique per cell. Non-resume runs keep the timestamp. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…env passthrough) sbatch is a fresh shell, so the matrix opt-in env var must be exported in the job script. ExperimentParams + CLI flag now export LIULIAN_PATCHTST_TRANSPARENT_ADD_AFTER_PATCH=1 when set, enabling the task #40 ablation to run on the cluster (paygo). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ABLATION_TAGS Monthly cap reset (user, 2026-06-16): the 10 CHF paygo cap is per-month, so June starts fresh (May's 4.42 doesn't count). Ablation = swiss patchtst transparent add_after_patch on paygo, 3 jobs ~1.2 CHF each projected. Filled ABLATION_TAGS so build_entity_id_figures emits the concat-vs-add_after_patch table once results land. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Per-step vs DMS vs produce-then-truncate comparison across the swiss
benchmark, TSL, and this repo; action items before any nowcast experiment
(per-step head for lstm, audit other models, optional per-step aux loss).
Mirrors the inline TODO(nowcast) in liulian/models/torch/lstm.py.
Co-Authored-By: Claude Fable 5 noreply@anthropic.com