Skip to content

Commit c3898a4

Browse files
committed
feat: update benchmark/docs pipeline and add forecasting guide demo
1 parent 106a00b commit c3898a4

156 files changed

Lines changed: 13025 additions & 2912 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/copilot-instructions.md

Lines changed: 421 additions & 0 deletions
Large diffs are not rendered by default.

.gitignore

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,4 +61,3 @@ wandb/
6161

6262
# Scratch files
6363
_test_batch.py
64-
extra_info_work.md

INTEGRATION_PLAN.md

Lines changed: 205 additions & 193 deletions
Large diffs are not rendered by default.

_record_baselines.py

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ def _base_config(
1515
model='lstm',
1616
split_mode='per_entity',
1717
id_integration=None,
18+
**extra,
1819
):
1920
from liulian.config import load_config
2021

@@ -49,7 +50,12 @@ def _base_config(
4950
model=model,
5051
d_model=16,
5152
e_layers=1,
53+
d_layers=1,
54+
label_len=5,
5255
enc_in=None,
56+
individual=False,
57+
moving_avg=25,
58+
embed='none', # Skip temporal embedding (swiss-river has single time feature)
5359
batch_size=16,
5460
max_train_iters=5,
5561
max_eval_iters=5,
@@ -77,6 +83,7 @@ def _base_config(
7783
seed=2026,
7884
quick_test=False,
7985
)
86+
cfg.update(extra)
8087
return cfg
8188

8289

@@ -121,6 +128,11 @@ def run_scenario(name, cfg):
121128
'patchtst_single_emb': lambda: _base_config('embedding', False, model='patchtst', split_mode='multi_channel'),
122129
'patchtst_tune_no_emb': lambda: _base_config('none', True, model='patchtst', split_mode='multi_channel'),
123130
'patchtst_tune_emb': lambda: _base_config('embedding', True, model='patchtst', split_mode='multi_channel'),
131+
# Informer scenarios (multi_channel mode)
132+
'informer_single_no_emb': lambda: _base_config('none', False, model='informer', split_mode='multi_channel', factor=3),
133+
'informer_single_emb': lambda: _base_config('embedding', False, model='informer', split_mode='multi_channel', factor=3),
134+
'informer_tune_no_emb': lambda: _base_config('none', True, model='informer', split_mode='multi_channel', factor=3),
135+
'informer_tune_emb': lambda: _base_config('embedding', True, model='informer', split_mode='multi_channel', factor=3),
124136
}
125137

126138

docs/dataset_config_audit.md

Lines changed: 251 additions & 0 deletions
Large diffs are not rendered by default.

docs/datasets.md

Lines changed: 15 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -90,13 +90,19 @@ dec_inp = cat([batch_y[:, :label_len, :], zeros(pred_len)])
9090
# Ground-truth warm-up + zero placeholders for prediction
9191
```
9292

93-
### Encoder-only models (DLinear, PatchTST)
93+
### Encoder-only and decoder-light models
9494

95-
These models accept `(x_enc, x_mark_enc, x_dec, x_mark_dec)` for API
96-
compatibility but **completely ignore** `x_dec` and `x_mark_dec`.
97-
**Set `label_len=0`** for these models to avoid constructing unused
98-
decoder inputs. There is no data-leakage risk either way — the overlap
99-
region `[seq_len-label_len : seq_len]` is already visible to the encoder.
95+
Several models accept `(x_enc, x_mark_enc, x_dec, x_mark_dec)` for API
96+
compatibility but ignore part or all of the decoder path during the
97+
forecast forward pass. That means `label_len` often has no behavioral
98+
effect even when the benchmark config still carries a non-zero value.
99+
100+
For strict TSL benchmark parity, many published `PatchTST` and `DLinear`
101+
scripts still keep the dataset default decoder overlap (`48` for the
102+
96-step benchmarks, `18` for ILI) even though the model does not use the
103+
decoder warm-up. By contrast, the TSL `TimeMixer` scripts intentionally
104+
set `label_len=0` across datasets, and we preserve that as a model-
105+
specific convention.
100106

101107
### Encoder-decoder models (Informer, Autoformer, etc.)
102108

@@ -159,7 +165,9 @@ Produces **continuous features normalised to `[-0.5, 0.5]`**:
159165
| Monthly (`m`) | MonthOfYear | 1 |
160166

161167
This is the encoding used by **all standard TSL benchmark scripts**
162-
(via `--embed timeF`).
168+
(via `--embed timeF`). Detailed minute offsets such as `10min` and
169+
`15min` are supported through pandas offset parsing and are preferable
170+
to ambiguous shorthand when the dataset cadence is known precisely.
163171

164172
### `timeenc=0` (categorical — manual)
165173

docs/demo_gallery.md

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Demo Gallery
2+
3+
This page collects runnable forecasting demos built on the unified LIULIAN pipeline.
4+
5+
## Forecasting Demos
6+
7+
- [PatchTST + Swiss River 1990](demo_patchtst_swiss1990.md)
8+
9+
## How to Use
10+
11+
1. Pick a demo page.
12+
2. Run the command exactly as shown.
13+
3. Inspect generated artifacts and metrics.
14+
4. Copy the pattern to your own dataset/model YAML.
15+
16+
## Next Demos to Add
17+
18+
- DLinear on ETTh1 (multi-channel)
19+
- Informer on ETTm1
20+
- TimeXer with identifier embeddings

docs/demo_patchtst_swiss1990.md

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# Demo: PatchTST on Swiss River 1990
2+
3+
This demo is a minimal, runnable baseline for LIULIAN forecasting.
4+
5+
## Goal
6+
7+
Run PatchTST on `swiss-river-1990` using the unified pipeline and verify that metrics + artifacts are generated.
8+
9+
## Option A: Run the Python Demo Script
10+
11+
```bash
12+
python examples/forecasting_patchtst_swiss1990_pipeline.py --quick-test
13+
```
14+
15+
Full run:
16+
17+
```bash
18+
python examples/forecasting_patchtst_swiss1990_pipeline.py
19+
```
20+
21+
## Option B: Run Directly from CLI
22+
23+
```bash
24+
liulian run experiments/swiss_river/patchtst_config.yaml --quick_test
25+
```
26+
27+
## What This Uses
28+
29+
- Config preset: `experiments/swiss_river/patchtst_config.yaml`
30+
- Runtime entrypoint: `liulian.pipeline.run_experiment`
31+
- Config merger: `liulian.config.load_config`
32+
33+
## Expected Outputs
34+
35+
At run completion, check the generated `artifacts` directory (path printed in terminal). You should find:
36+
37+
- `results.json`
38+
- `predictions.npz`
39+
- `figures/` plots (if visualization is enabled)
40+
41+
## Validation Checklist
42+
43+
- Process exits with code `0`
44+
- Metrics are printed in terminal
45+
- `results.json` exists and includes `metrics` + `timing`
46+
47+
## Adapt This Demo
48+
49+
Try these incremental changes:
50+
51+
1. Change `pred_len` from `7` to `14`
52+
2. Change `batch_size` from `32` to `16`
53+
3. Toggle `quick_test` off for full training
54+
4. Swap model config to another YAML under `experiments/swiss_river/`

docs/forecasting_pipeline.md

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
# Forecasting Pipeline Guide
2+
3+
This guide shows how to run forecasting experiments using the unified pipeline in `liulian/pipeline.py`.
4+
5+
## Why This Pipeline
6+
7+
The forecasting pipeline standardizes the full workflow:
8+
9+
1. Seed and reproducibility setup
10+
2. Dataset construction
11+
3. Model construction
12+
4. Data loader creation
13+
5. Experiment run (train/eval)
14+
6. Artifacts and `results.json` export
15+
16+
Use it when you want consistent behavior across datasets and models.
17+
18+
## Pipeline API Map
19+
20+
Core APIs:
21+
22+
- `liulian.config.load_config`: merge `DEFAULT_CONFIG < YAML < CLI`
23+
- `liulian.pipeline.build_dataset`: create dataset from config
24+
- `liulian.pipeline.build_model`: create model from config
25+
- `liulian.pipeline.build_loaders`: create train/val/test loaders
26+
- `liulian.pipeline.build_experiment`: wire Task + Data + Model + Runtime
27+
- `liulian.pipeline.run_experiment`: run full end-to-end pipeline
28+
29+
Most users should call only `run_experiment(config)`.
30+
31+
## Quick Start (CLI)
32+
33+
Run PatchTST on Swiss River 1990:
34+
35+
```bash
36+
liulian run experiments/swiss_river/patchtst_config.yaml
37+
```
38+
39+
Quick test mode (small, fast smoke run):
40+
41+
```bash
42+
liulian run experiments/swiss_river/patchtst_config.yaml --quick_test
43+
```
44+
45+
Override key hyperparameters from CLI:
46+
47+
```bash
48+
liulian run experiments/swiss_river/patchtst_config.yaml \
49+
--pred_len 14 \
50+
--train_epochs 20 \
51+
--batch_size 16
52+
```
53+
54+
## Quick Start (Python)
55+
56+
```python
57+
from liulian.config import load_config
58+
from liulian.pipeline import run_experiment
59+
60+
cfg = load_config(
61+
'experiments/swiss_river/patchtst_config.yaml',
62+
cli_overrides={
63+
'quick_test': True,
64+
'hpo': False,
65+
},
66+
)
67+
summary = run_experiment(cfg)
68+
print(summary.get('artifacts_dir'))
69+
```
70+
71+
## Configuration Flow
72+
73+
The config merge order is:
74+
75+
1. `liulian.config.DEFAULT_CONFIG`
76+
2. YAML config file (for example `experiments/swiss_river/patchtst_config.yaml`)
77+
3. CLI or programmatic overrides
78+
79+
Practical advice:
80+
81+
- Keep experiment presets in YAML files under `experiments/`
82+
- Use CLI overrides for one-off comparisons
83+
- Use `quick_test: true` for sanity checks before long runs
84+
85+
## Artifacts and Results
86+
87+
After each run, the pipeline writes artifacts to an experiment directory (under `artifacts/`), typically including:
88+
89+
- `results.json`: structured summary for auditing and comparison
90+
- `predictions.npz`: raw predictions and ground truth arrays
91+
- `figures/`: generated forecast visualizations (if enabled)
92+
93+
For `results.json` fields, see [Results JSON](results_json.md).
94+
95+
## Common Troubleshooting
96+
97+
### OOM or very slow training
98+
99+
- Reduce `batch_size`
100+
- Reduce model width (`d_model`, `d_ff`, `n_heads`)
101+
- Start with `--quick_test` to validate pipeline integrity first
102+
103+
### Dataset not found
104+
105+
- Confirm dataset key in config `data:` is valid
106+
- The pipeline attempts auto-download for supported datasets
107+
108+
### Unexpected split behavior
109+
110+
- Confirm `split_mode` and `train_split`
111+
- Check whether your model expects `multi_channel` or `per_entity`
112+
113+
## Demo Gallery
114+
115+
For ready-to-run examples, see [Demo Gallery](demo_gallery.md).
116+
117+
## External Design References
118+
119+
These resources were used as style references for clear forecasting quickstarts:
120+
121+
- [PyTorch Forecasting tutorial](https://pytorch-forecasting.readthedocs.io/en/stable/tutorials/stallion.html)
122+
- [GluonTS quick start](https://ts.gluon.ai/stable/tutorials/forecasting/quick_start_tutorial.html)
123+
- [NeuralForecast quick start](https://nixtlaverse.nixtla.io/neuralforecast/docs/getting-started/quickstart.html)

docs/index.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,8 @@ python examples/quick_run.py
2525
## Documentation
2626

2727
- [Architecture](architecture.md) — Design decisions and module overview
28+
- [Forecasting Pipeline Guide](forecasting_pipeline.md) — End-to-end run flow using `liulian.pipeline`
29+
- [Demo Gallery](demo_gallery.md) — Runnable forecasting demos
2830
- [Adapter Guide](adapter_guide.md) — How to write model adapters
2931
- [Manifest Specification](manifest_spec.md) — YAML dataset manifest format
3032
- [Contributing](contributing.md) — How to contribute to liulian

0 commit comments

Comments
 (0)