GFQL searchAny: WYSIWYG search coverage — match user-visible formatting (float, datetime) + format options

[lmeyerov]

Follow-up from the searchAny work (#1680). searchAny currently searches string and integer columns (numeric-term-gated), matching the streamgl-viz inspector's shouldSearch for those dtypes. It does not search float or date columns, which the inspector does (numeric-term-gated). This issue tracks closing that gap.

Reference (ground truth)

~/Work/graphistry/apps/core/viz/src/worker/services/dataframe/sortAndFilterRowsByQuery.js + src/formatters/defaultFormat.js. The inspector searches number/integer/date columns (iff the term is numeric) via defaultFormat: fractional floats → sprintf('%.4f'); whole numbers → raw String(); dates → moment MMM D YYYY, h:mm:ss a z. Full analysis: plans/viz-filter-pipeline/research/searchany-inspector-parity.md.

Float — proven native cross-engine path (dgx-probed, NO host-bridge, NO broad-NIE)

The naive astype(str) diverges in the exponent regime (1e16 → '1e16' vs '1e+16') — the original reason floats were excluded. The fix:

• round(4) then fixed-point decimal render: cudf .astype(Decimal128(scale=4)).astype(str) ≡ polars .cast(Decimal(scale=4)).cast(str) exactly (vectorized, no exponent, GPU-native).
• Apply the same round(4)+decimal convention on pandas too (not its f\"{v:.4f}\", which uses true float bits and diverges at half-boundaries like 0.12345→0.1235 vs 0.1234) so all four engines agree.

Implementation sketch (on graphistry/compute/gfql/search_any.py + lazy/engine/polars/search.py)

1. _is_float_dtype + include float in the auto gate for numeric terms.
2. _canonical_float_str(series) = round(4) → fixed-4-decimal string, engine-aware (pandas/cudf/polars Decimal).
3. Wire into search_any_mask (kernel) + search_any_polars.
4. Remove the now-unneeded exclusions: the cuDF explicit-float NIE (search_any.py) and the polars _stringify_ok float-exclusion (search.py).
5. Verify with a dgx rounding fuzzer — cross-engine round(4) half-rounding consistency is the one residual risk (stringification-parity class); pandas is the oracle, so either canonicalize to agreement or pin the boundary behavior.

Date (lower priority, likely a separate slice)

Inspector formats dates via moment MMM D YYYY, h:mm:ss a z; exact cross-engine replication (tz/locale) is hard → candidate honest-NIE + doc.

Test hooks

test_engine_polars_conformance_matrix.py::test_search_any_* + test_viz_pipeline_conformance.py trick matrix — add float-search pins (fractional, whole, negative, large-magnitude, null) 4-engine parity-or-NIE.

[lmeyerov]

Datetime search — scope clarification (raised in review)

The inspector's date search is narrow, so parity here is subtle:

• Date columns are searched only when the term matches /^[0-9.-]+$/ (same numeric gate as numbers). Text terms (weds, jun) and slash-dates (6/1/2025) are excluded from date columns entirely.
• When searched, the value is rendered via moment 'MMM D YYYY, h:mm:ss a z' (e.g. "jun 1 2025, 12:00:00 am utc") and substring-matched — no weekday field, no slash format. So only numeric substrings (2025, 1, 12) can match.

Consequences:

• weds / 6/1/2025 match no date column in the inspector — and we already return no-match (dates excluded), so we're at parity for these inputs today. No work needed for them.
• The only date parity gap is numeric-term vs the moment-rendered string — hard to replicate cross-engine (tz/locale). Candidate: honest-NIE + doc rather than reproduce moment formatting on cuDF/polars.

Optional larger scope (NOT parity — a deliberate improvement)

If we want searchAny to beat the inspector on dates — parse the term as a date (6/1/2025, weekday, relative) and match date columns semantically (equality/range) instead of substring-on-rendered-string — that's a separate, larger feature (term-as-date parsing + typed date comparison). Flag for product decision before building.

[lmeyerov]

Inspector datetime rendering — exact spec (for the date half)

From src/formatters/{formatDate,castToMoment}.js:

Output (search uses the long form), always UTC:

• long: moment 'MMM D YYYY, h:mm:ss a z' → "Jun 1 2025, 12:00:00 am UTC" (lowercased for search)
• short (display only): 'MMM D YY, h:mm:ss a' → "Jun 1 25, 12:00:00 am"
• month abbreviation; no leading zero on day or hour; 12h lowercase am/pm; tz label; empty/0/null → "".

Input (castToMoment):

• number → moment.unix (seconds) first; if invalid or year>5000 → moment.utc (millis). Auto seconds-vs-millis.
• string/Date → moment.utc(value) (ISO-8601 etc, UTC).
• numeric string → parseInt retry via the number path.

Cross-engine hazards (why NIE is the likely call for date): reproducing moment's English-locale render on pandas/cuDF/polars strftime diverges — %I zero-pads the hour (01 vs moment 1), %-d is platform-specific, plus the UTC label and the seconds/millis heuristic. Exact parity is fragile; recommend honest-NIE + doc for date columns unless a canonical formatter is proven byte-identical across all four engines with a fuzzer (same bar as float).

[lmeyerov]

Reframe (per review): the goal is WYSIWYG search, not just inspector-parity

The unifying principle: searchAny should match what the user SEES — the display-rendered value — for 100% search coverage. This is a superset of the inspector's current behavior and even closes an inspector gap: the inspector displays dates as "Jun 1 2025, 12:00:00 am UTC" but numeric-gates their search, so today you can't find that row by typing jun or jun 1 2025. WYSIWYG search fixes that.

This generalizes float + datetime (and any formatted column) under one rule: search the display-formatted string, not the raw value. The format used for search must equal the format used for display.

The architectural crux: search-format must track display-format

Display formatting lives in the frontend (streamgl-viz JS: defaultFormat, per-column overrides). searchAny runs in the GFQL backend (Python). For TRUE WYSIWYG they must render identically. So the design should let the caller pass its display-format spec into searchAny, with sensible defaults when it doesn't — the format string/options become the shared display↔search contract.

Proposed API — format options with nice defaults (searchAny(entity, term, {...}))

Additive to the existing {caseSensitive, regex, columns}; all optional:

• temporal: 'wysiwyg' (default — render datetime cols to a human string and text-search them) | 'iso' (search ISO-8601) | 'skip' (exclude, today's behavior) | 'raw'.
• temporalFormat: strftime-style spec; default = a canonical human format matching the common inspector display (e.g. %b %-d %Y, %-I:%M:%S %p %Z → "Jun 1 2025, 12:00:00 am UTC"), overridable so a caller can pass the exact per-column display format for pixel-parity.
• tz: default UTC (matches the inspector's moment.utc); overridable.
• floatPrecision: default 4 (matches the inspector's %.4f); render floats fixed-point at this precision.
• numeric/float/bool per-dtype toggles if we want to diverge from the inspector's dtype gate.

Defaults chosen so the zero-arg call behaves like the inspector's display (WYSIWYG), not its narrower search — i.e. searchAny(n, 'jun') finds "Jun 1 2025…" rows. Power users / the frontend pass explicit formats for exact parity with a customized display.

Coverage this unlocks (datetime)

With temporal='wysiwyg' + the default format, ALL of these match a 2025-06-01 cell (case-insensitive substring on the rendered string):
jun · jun 1 2025 · 2025 · 12:00 · am · utc. (6/1/2025 still won't unless the display format uses slashes — WYSIWYG means it matches the rendered form; a slash-date term needs either a slash display format or the separate 'smart date parsing' feature below.)

Cross-engine + parity-or-NIE (still inviolable)

• float: fixed-point via round(precision) + decimal render — dgx-proven cuDF≡polars; canonicalize pandas to the same convention; fuzz-verify byte-identical (see earlier comments).
• datetime: per-engine strftime of the chosen format + normalization for the known divergences (%I hour zero-pad vs moment h, %-d platform-specificity, tz label, seconds-vs-millis heuristic). Must be fuzz-verified byte-identical across pandas/cuDF/polars against the pandas oracle. Where a requested format canNOT be reproduced identically on an engine → honest NIE, never silent divergence (parity-or-NIE holds — WYSIWYG does not get to relax it).
• Whichever format is defaulted, it becomes part of the conformance corpus (pin float + datetime WYSIWYG cases, 4-engine).

Optional stretch (separate, NOT WYSIWYG): smart term-as-date parsing

To also match 6/1/2025, weekdays, or relative dates regardless of display format, parse the term as a date and compare semantically (equality/range) against datetime columns. Bigger feature (term parsing + typed comparison); flag for product before building. Keep distinct from WYSIWYG substring search.

Scope note for #1680

None of this blocks #1680 — datetime/float remain excluded there (parity-safe). This issue now owns the WYSIWYG design (float = near-term + clean; datetime WYSIWYG + format-options API = the larger piece).

[lmeyerov]

Being implemented by #1700 ("searchAny WYSIWYG float + datetime search") — floatPrecision/temporalFormat/tz format call-params, decimal-cast float rendering, and datetime WYSIWYG per the specs above. #1700 references this issue but doesn't use a closing keyword, so close this when #1700 merges (or add Closes #1695 to that PR).