temporal: close the non-DST Temporal TCK cluster (+30)

.metis/initiatives/GQLITE-I-0049/initiative.md

@@ -0,0 +1,83 @@
+---
+id: tck-deep-item-conformance-temporal
+level: initiative
+title: "TCK deep-item conformance (Temporal, Quantifier, Merge, ordering)"
+short_code: "GQLITE-I-0049"
+created_at: 2026-05-30T14:17:28.737710+00:00
+updated_at: 2026-05-30T14:17:28.737710+00:00
+parent: 
+blocked_by: []
+archived: false
+
+tags:
+  - "#initiative"
+  - "#phase/discovery"
+
+
+exit_criteria_met: false
+estimated_complexity: L
+initiative_id: tck-deep-item-conformance-temporal
+---
+
+# TCK deep-item conformance (Temporal, Quantifier, Merge, ordering)
+
+## Context
+
+After the +44-scenario TCK push (PRs #76–#87, main at **3728 pass / 148 non-pass**
+as of 2026-05-30), the remaining gaps are no longer one-line fixes — they cluster
+into a handful of deep, multi-scenario features. This initiative tracks those
+clusters, one task each, so each can be decomposed and ground out independently
+while progress survives compaction.
+
+Baseline: `angreal test tck` → pass=3728 fail=123 error=25 skipped=4.
+Verification gate for every task: rigorous full pass-set diff (zero regressions),
+unit 944/944, functional clean. Work stacks on one branch per the PR-batching
+preference; squash-merge per cluster or per batch as the user directs.
+
+## Goals & Non-Goals
+
+**Goals:**
+- Close the large TCK clusters: Temporal (~42), Quantifier (~30), Merge (~14),
+  ordering/WithOrderBy (~10), with zero regressions.
+- Decompose each cluster into a task with its own root-cause analysis + sub-steps.
+
+**Non-Goals:**
+- Boolean type preservation (architecturally blocked — see memory
+  `boolean_type_preservation_blocked`; SQLite has no bool type / can't bind subtypes).
+- Full rewrites of the value-representation layer.
+
+## Cluster inventory (task-per-cluster)
+
+| Task | Cluster | ~Count | Root cause | Difficulty |
+|------|---------|--------|------------|------------|
+| T-A | Temporal | ~42 | duration arithmetic, duration.between, date selection, parsing, accessors, named TZ | Medium, decomposable |
+| T-B | Quantifier | ~30 | all/any/none/single through the WITH list pipeline | Medium (one root cause may cascade) |
+| T-C | Merge | ~14 | bind path, bound-var reuse, direction, multi-row MATCH+MERGE cartesian iteration | Medium-hard |
+| T-D | WithOrderBy/ordering | ~10 | ORDER BY stability/tie-break across types (refines GQLITE-T-0340 comparator) | Low-medium |
+| T-E | Existential subqueries (2/3) | 5 | full/nested EXISTS { full-query / aggregation / nesting } | Hard |
+| T-F | Write-path list/UNWIND (List12, Pattern2) | 7 | entity-list through SET / pattern-comprehension write path | Hard |
+| T-G | Misc contained singletons | ~6 | smallest-int literal (Literals2/3/4 [8]), varlen path-length (Path3 [1], Path1), runtime TypeErrors (TypeConversion3) | Mixed |
+
+## Recommended sequence
+
+1. **Temporal (T-A)** — biggest, cleanly decomposable, steady high-yield. **STARTING HERE.**
+2. **Quantifier (T-B)** — best count-to-effort; existing analysis in memory `quantifier_equality_dropped`.
+3. **Merge (T-C)** — multi-row MATCH+MERGE iteration unlocks several at once.
+4. **WithOrderBy (T-D)** — small comparator refinement.
+
+## Detailed Design
+
+Per-cluster design lives in each child task. The shared method: (a) enumerate the
+cluster's failing scenarios + diagnostics, (b) find the common root cause(s),
+(c) fix smallest-first, (d) rigorous pass-set diff after each, (e) record findings
+in the task as working memory.
+
+## Implementation Plan
+
+Phase per cluster, in the sequence above. Each cluster task is decomposed into
+sub-steps once started (human check-in before decomposing, per Metis HITL).
+
+## Status
+
+- 2026-05-30: Initiative created. Structure approved (initiative + task-per-cluster).
+  Starting with Temporal (T-A).

.metis/initiatives/GQLITE-I-0049/tasks/GQLITE-T-0341.md

@@ -0,0 +1,191 @@
+---
+id: temporal-cluster
+level: task
+title: "Temporal cluster: duration arithmetic, parsing, selection, DST"
+short_code: "GQLITE-T-0341"
+created_at: 2026-05-30T14:17:28.737710+00:00
+updated_at: 2026-05-30T14:17:28.737710+00:00
+parent: GQLITE-I-0049
+blocked_by: []
+archived: false
+
+tags:
+  - "#task"
+  - "#phase/todo"
+
+exit_criteria_met: false
+initiative_id: tck-deep-item-conformance-temporal
+---
+
+# Temporal cluster: duration arithmetic, parsing, selection, DST
+
+## Parent Initiative
+[[GQLITE-I-0049]]
+
+## Objective
+
+Close the Temporal TCK cluster (~42 failing examples across Temporal1/2/3/5/7/8/10)
+with zero regressions. Temporal code lives in `src/backend/transform/transform_func_temporal.c`
+and the temporal UDFs/helpers in `src/backend/runtime/udf_helpers.c` (duration JSON
+builder, ISO parsing, epoch-ns conversions). Durations are stored as the JSON object
+`{"_iso8601": "...", "months": M, "days": D, "seconds": S, "nanosecondsOfSecond": N}`.
+
+## Sub-steps (decomposition, smallest-yield-first ordering TBD after analysis)
+
+1. **Duration arithmetic — Temporal8 (~12)**: add/subtract durations; add/subtract a
+   duration to/from date/time/localtime/datetime; multiply/divide a duration by a number.
+   Likely the biggest single win; concentrated in duration +/-/*//.
+2. **Duration between + DST + large — Temporal10 (~10)**: `duration.between` variants,
+   durations across daylight-saving boundaries, very large durations (seconds overflow).
+3. **Date/time selection — Temporal3 (~9)**: `date({...})` / `time({...})` /
+   `datetime({...})` selecting/combining component fields (truncation/selection forms).
+4. **Parsing edge cases — Temporal2 (~6)**: parse date/time/duration from string,
+   named time zones (`[Europe/Stockholm]`), fractional-second edge cases.
+5. **Construction / accessors / comparison — Temporal1/5/7 (~5)**: construct duration
+   (example 3), time-offset construction, datetime accessors, time & duration equality.
+
+## Method
+
+Per sub-step: enumerate failing examples + read each scenario's query and expected
+output; reproduce via `sqlite3 :memory:` (NOT ad-hoc chains — single queries through
+the harness for verification); find the common root cause; fix; rigorous full pass-set
+diff after each (zero regressions); unit 944/944; functional clean. Record findings here.
+
+## Acceptance Criteria
+- [ ] Temporal8 duration arithmetic examples pass
+- [ ] Temporal10 duration-between / DST / large examples pass
+- [ ] Temporal3 selection examples pass
+- [ ] Temporal2 parsing examples pass
+- [ ] Temporal1/5/7 construction/accessor/comparison examples pass
+- [ ] Zero TCK regressions across the whole cluster; unit 944/944; functional clean
+
+## Status Updates
+
+- 2026-05-30: Task created with sub-step breakdown from the failure inventory.
+  Baseline main = 3728 pass. Starting analysis with sub-step 1 (Temporal8 duration
+  arithmetic) — biggest concentrated win.
+
+### 2026-05-30: Sub-step 1 (Temporal8) deep analysis
+
+Durations are stored as JSON `{"_iso8601","months","days","seconds","nanosecondsOfSecond"}`.
+Reusable runtime helpers in `udf_helpers.c`: `is_duration_value()`, `dur_field_ll()`,
+`emit_duration_json(ctx, months, days, total_ns)` (keeps months/days independent, only
+splits total_ns→seconds+nanos), `apply_duration_to_temporal()`, `format_iso_duration()`.
+ADD/SUB route through `gql_dyn_addsub_func` (`_gql_dyn_add`/`_gql_dyn_sub`), which already
+handles dur+dur and dur+temporal.
+
+Concrete Temporal8 bugs found:
+- **[7] multiply/divide by number → returns `0`.** `BINARY_OP_MUL`/`DIV` in
+  transform_expr_ops.c emit a bare ` * ` / ` / `; SQLite coerces the duration JSON text
+  to 0. FIX: route MUL/DIV through new `_gql_dyn_mul`/`_gql_dyn_div` UDFs (mirror the
+  ADD/SUB dispatch) that detect a duration operand and scale months/days/total_ns by the
+  number (Cypher rounding for non-integer factors TBD — check expected values per example).
+- **[2]-[5] duration ± temporal: wrong values** (e.g. local time off by hours:
+  exp `22:29:27.5` vs act `17:14:54.5`). `apply_duration_to_temporal` mis-applies some
+  components — investigate which (days/seconds vs hours). NOT just formatting.
+- **[6] duration ± duration: wrong values AND wraps as object.** exp `P25Y4M43DT50H...`
+  vs act `{_iso8601:'P25Y4M44DT20H...'}`. Two issues: (a) value differs by ~6h (computation
+  or the test's input durations carry hours we mis-split between days/seconds), (b) the
+  result renders as the full duration object, but the expected is the ISO string — the
+  renderer already extracts `_iso8601` for duration values (udf_helpers.c:905), so check
+  why the arithmetic result loses that path (likely the column type/rendering path).
+
+NEXT (next work session): implement `_gql_dyn_mul`/`_gql_dyn_div` for durations ([7],
+clearest win), then chase the value-computation discrepancy in apply_duration_to_temporal
+([2]-[5]) and dur+dur ([6]). Verify each with rigorous pass-set diff.
+
+### 2026-05-30: Sub-step 1 (Temporal8) COMPLETE — Temporal8 27/27
+
+Two commits (branch i0049-temporal): duration mul/div (+3), fractional construction
++ date arithmetic (+11). Branch 3728 -> 3742. Key learnings (durations are NOT
+normalized across components; 1 month = 30.436875 days, 1 day = 86400 s; only
+date+duration rolls whole-day time into the date). Remaining Temporal:
+Temporal10 (10, duration.between + DST), Temporal3 (9, date selection),
+Temporal2 (6, parsing), Temporal1 [13]/Temporal5 [6]/Temporal7 [3] (1 each).
+NEXT: Temporal10 duration.between.
+
+### 2026-05-30: Sub-step 2 (Temporal10) partial — inMonths/inDays tz fix (+2)
+
+duration.inMonths/inDays now compare time-of-day in UTC when both sides have tz
+(Temporal10 [3] ex19, [4] ex17). Branch 3742 -> 3744. Remaining Temporal10 (8):
+DST-aware durations [8] (5 — needs real DST transition handling, hard) and large
+durations [9]/[10] (2 — int overflow in the calendar/seconds path). Deferring DST
++ overflow; next consider Temporal3 (date selection, 9) and Temporal2 (parsing, 6).
+
+### 2026-05-30: Session checkpoint — branch i0049-temporal at 3744 (+16 vs main 3728)
+
+Commits: duration mul/div (+3), fractional construction+date arith (+11),
+inMonths/inDays tz (+2). Temporal8 fully closed (27/27).
+
+Remaining Temporal (next sub-steps):
+- Temporal3 [1] date selection (5): `date({date: other, quarter: N})` must
+  preserve monthOfQuarter + day (we reset to quarter-start month + day 1, giving
+  1984-07-01 vs expected 1984-08-11). Plus week/ordinalDay base-selection forms.
+- Temporal3 [3] time() with named tz (2): `time(...)` must DROP the `[Region]`
+  suffix, keeping only the offset (datetime keeps it; time drops it).
+- Temporal3 [10] (2): datetime named-tz DST offset (+01 vs +02).
+- Temporal2 parsing (6), Temporal1 [13], Temporal5 [6], Temporal7 [3].
+- Temporal10 DST [8] (5) + large/overflow [9]/[10] (2) — hardest, deferred.
+
+### 2026-05-31: checkpoint — branch i0049-temporal at 3749 (+21 vs main 3728)
+
+Closed this session: Temporal8 (27/27), Temporal10 inMonths/inDays, Temporal3
+quarter selection + time() region-drop, Temporal1 [12], Temporal7 [6].
+7 commits, all rigorous-diff zero-regression, unit 944/944.
+
+Remaining Temporal (precise, with root cause for next session):
+- **Temporal2 [3]/[5]** (2): tz offset `-00:00`/`+00:00` must render as `Z`.
+  Formatting fix in the offset emitters.
+- **Temporal1 [13]** (3 examples): tz offset with seconds — `+02:05:00` -> `+02:05`
+  (drop `:00`), `+02:05:59` -> keep `:59`. Offset parse must retain seconds and
+  the formatter must emit `:SS` only when non-zero. (Currently offsets stored as
+  minutes; need second precision.)
+- **Temporal2 [7]** (2): parse duration FROM string `P22DT19H51M49.5S` — fractional
+  / multi-field ISO duration parse broken (gives P22DT12H / PT0S).
+- **Temporal3 [1] ex8/15** (2): date selection returns None — investigate which
+  selection form (likely week/weekYear combo) crashes.
+- **Temporal5 [6]** (1): datetime accessors (one accessor value off).
+- **Temporal7 [3]** (1): compare times — boolean vector inverted (tz/offset compare).
+- **DST cluster (hard, deferred): Temporal3 [10] (2), Temporal2 [6] (2),
+  Temporal10 [8] (6)** — need real DST-transition handling at named-zone dates
+  (named_tz_offset is a month approximation; these test exact transition days).
+- **Overflow (deferred): Temporal10 [9]/[10] (2)** — int64 overflow in calendar/
+  seconds path for billion-year / huge-second durations.
+
+### 2026-05-31: Temporal cluster nearly closed — branch 3758 (+30 vs main 3728)
+
+All non-DST Temporal now passes. Remaining = DST only (12):
+- Temporal10 [8] (8): duration.inSeconds across the 2017-10-29 fall-back —
+  needs across-transition elapsed-time (24 wall hrs = 25 real hrs).
+- Temporal3 [10] (2) / Temporal2 [6] (2): named-zone offset on a DST-transition
+  date (e.g. 1984-03-28 -> +02).
+GOTCHA: named_tz_offset's month approximation (Apr–Sep summer) is LOAD-BEARING.
+A clean last-Sunday-rule rewrite REGRESSED -29 (many passing tests depend on the
+coarse rule). DST must be done empirically/per-date against the TCK's exact
+expectations, NOT a simple rule swap. Recommend PR the +30 branch; tackle DST
+as a separate focused task.
+
+### 2026-06-01: DST cluster — ROOT CAUSE = needs an IANA tz database (NOT closeable by rules)
+
+Dug into the 12 remaining DST scenarios empirically. Findings:
+- **Why the offset approximation is load-bearing / risky:** `named_tz_offset` is
+  called from many paths (construction, parsing, duration.between, accessors,
+  rendering). The TCK encodes HISTORICALLY-ACCURATE IANA data, so any rule must
+  match history exactly. A modern "last Sunday of March..October" EU rule
+  REGRESSED -52 examples because pre-1996 EU DST ended the last Sunday of
+  SEPTEMBER (e.g. 1984-10-11 Stockholm = +01:00, not +02:00). The OLD coarse
+  Apr–Sep approximation happens to match those pre-1996 Sept endings.
+- A regression-free improvement IS possible: historical end-month
+  (`(y>=1996)?Oct:Sep`) + threading the real base date into the datetime
+  named-tz offset lookup (`_gql_tz_offset_for(tz, _gql_date_compose(... $.date,
+  $.datetime))`). Verified 0 regressions, and it fixes individual examples
+  (e.g. Temporal3 [10] ex14) — but flips NO full scenario, so net +0. Reverted
+  to keep the +30 branch coherent; re-derive from this note when doing IANA work.
+- **Hard stop:** full DST is impossible without an embedded IANA tz database.
+  Temporal2 [6] expects `1818-07-21 Stockholm = +00:53:28` — Local Mean Time
+  before standardization. No rule produces that. Temporal10 [8] additionally
+  needs across-transition elapsed time (24 wall-clock hrs = 25 real hrs on the
+  fall-back day). Both require the real tzdata.
+- RECOMMENDATION: close T-0341 as "Temporal non-DST done (+30)"; open a separate
+  task "embed IANA tzdata for full temporal DST conformance" (large, data-heavy)
+  for the 12 DST scenarios.

docs/testing/semantic-coverage-matrix.md

@@ -548,3 +548,79 @@ pass-set diff: zero regressions, +3; unit 944/944; functional clean):
   error. keys() on a parameter/expression emits a single-eval subquery over
   json_each, using the value's `properties` object when present (node/rel) else
   its own keys (map). Fixes Graph3 [6], Graph4 [5], Map3 [2].
+
+## Coverage update (2026-05-30) — duration multiply/divide by a number (Temporal8 [7])
+
+`transform_expr_ops.c`, `udf_helpers.c`, `udf_register.c`. Verified via the TCK
+harness (3728 -> 3731, rigorous full pass-set diff: zero regressions, +3; unit
+944/944; functional clean):
+
+- **`duration * n` / `duration / n` now scale the duration component-wise**
+  instead of coercing the JSON to 0. New `_gql_dyn_mul` / `_gql_dyn_div` UDFs
+  (mirroring the ADD/SUB `_gql_dyn_*` dispatch) detect a Duration operand and
+  scale months/days/seconds/nanos by the factor, cascading each unit's
+  fractional remainder down via 1 month = 30.436875 days and 1 day = 86400 s
+  (components otherwise NOT normalized across each other); plain numerics keep
+  native int/float semantics. BINARY_OP_MUL/DIV route through the helpers unless
+  both operands are numeric literals. Fixes Temporal8 [7]. Part of GQLITE-T-0341.
+
+## Coverage update (2026-05-30) — fractional duration construction + date arithmetic (Temporal8 [6]/[1])
+
+`udf_helpers.c`. Verified via the TCK harness (3731 -> 3742, rigorous full
+pass-set diff: zero regressions, +11; unit 944/944; functional clean):
+
+- **`duration({...})` with fractional components cascades correctly.** The
+  fractional-month→day carry used 30.0 days/month; corrected to 30.436875 (avg
+  Gregorian month), matching Cypher (Temporal8 [6] examples 3/6/7/8/9, Temporal1
+  [12], Temporal7 [6]).
+- **Durations no longer normalize sub-day time into days.** The composer's
+  day-overflow roll (seconds → days) was removed so `duration({hours:25})` stays
+  `PT25H` and a fractional duration keeps e.g. `PT67H` — consistent with
+  `emit_duration_json` used by duration addition.
+- **date + duration rolls the duration's whole-day time into the date.** Since
+  the duration value is no longer pre-normalized, `apply_duration_to_temporal`
+  now adds `time_ns / DAY_NS` whole days (trunc toward zero) to a pure-date input
+  and drops the sub-day remainder (Temporal8 [1] example 3). Part of GQLITE-T-0341.
+
+## Coverage update (2026-05-30) — duration.inMonths/inDays tz normalization (Temporal10 [3]/[4])
+
+`udf_helpers.c`. Verified via the TCK harness (3742 -> 3744, rigorous full
+pass-set diff: zero regressions, +2; unit 944/944; functional clean):
+
+- **`duration.inMonths` / `duration.inDays` compare the time-of-day in UTC**
+  when both operands carry a tz offset, instead of the local clock face. A
+  tz-offset difference (e.g. `+0200` vs `+0100`) no longer spuriously drops a
+  whole month/day (Temporal10 [3] ex19 `P1Y`, [4] ex17 `P337D`). inDays was also
+  rewritten to count whole days from the calendar day difference minus a partial
+  trailing day (time-of-day comparison), using `days_from_civil` (unbounded)
+  instead of `timegm`. Part of GQLITE-T-0341. Remaining Temporal10: DST-aware
+  durations [8] and large-duration overflow [9]/[10] (deferred).
+
+## Coverage update (2026-05-30) — time() drops named-zone region; date() quarter selection (Temporal3)
+
+`udf_helpers.c`, `transform_func_temporal.c`, `udf_register.c`. Verified via the
+TCK harness (rigorous full pass-set diffs: zero regressions; unit 944/944;
+functional clean):
+
+- **`date({date: other, quarter: N})` preserves month-of-quarter + day** (+3,
+  3744->3747). Was resetting to the quarter's first month/day 1.
+- **`time()` / `localtime()` drop a named-zone `[Region]` suffix**, keeping only
+  the numeric offset, while `datetime()` retains it (+2, 3747->3749). The shared
+  `_gql_time_compose` UDF gained a `drop_region` arg (1 for time, 0 for datetime).
+  Part of GQLITE-T-0341.
+
+## Coverage update (2026-05-31) — Temporal cluster grind (T-0341), 3728 -> 3758 (+30)
+
+`udf_helpers.c`, `transform_func_temporal.c`, `udf_register.c`. Closed nearly the
+whole Temporal cluster across many rigorous-diff-verified commits: duration
+multiply/divide; fractional duration construction + date arithmetic;
+duration.inMonths/inDays tz normalization; date() quarter selection; time()
+region-drop; zero-offset -> Z; offset zero-seconds drop; date(datetime); ISO
+fractional-month cascade; UTC-instant temporal comparison; .timezone accessor;
+alternate ISO duration form. Unit 944/944, functional clean throughout.
+
+Remaining Temporal = the DST cluster only (12): Temporal10 [8] (across-DST-
+transition elapsed time), Temporal3 [10] / Temporal2 [6] (offset resolution on a
+DST-transition date). `named_tz_offset`'s month approximation is load-bearing
+(an accurate last-Sunday-rule swap regressed -29), so DST needs a careful,
+empirical, per-zone effort + across-transition interval math. Deferred.