Document user-side t-digest via declarative numpy expression + define the user-side-code policy

[espg]

🤖 from Claude

Purpose

Two things in one place:

1. Document, with worked examples, how to declare a t-digest (and similar sketches) user-side as a declarative numpy expression — running on unmodified Lambda workers with no layer redeploy.
2. Use that as the anchor for the larger design conversation: how do we define and allow user-side code, where the line sits between "declarative numpy" and "arbitrary code," and what the security model is.

Context: #279 ships t-digest as a vectorized built-in (build_tdigest / build_tdigest_pairwise), and #213 established "no user code on Lambda — built-ins + declarative numpy config only." But zagg does have a user-side extension path today, and it's underdocumented. This issue makes it concrete.

The mechanism (what exists today)

A reducer can be declared two ways:

• function: — a dotted path resolved by importlib.import_module from the installed zagg. Built-in only; the body must be packaged → needs a layer redeploy. (This is where t-digest agg hot path is unvectorized: ~15× build + ~6× fold available in pure numpy #279's t-digest lives.)
• expression: — a numpy string eval'd by _eval_expression_raw (src/zagg/config.py:2323) in a restricted namespace, and it already supports kind: ragged output (issue Tier-2 vectors: CSR ragged (values/offsets/cell_ids) + t-digest as List<FixedSizeList<2>> #48):

ns = {"__builtins__": {}, "np": np, "numpy": np, "len": len,
      "float": float, "int": int, "abs": abs, "sum": sum, **columns}
return eval(expression, ns)

The expression travels as data in the user's config and runs on unmodified workers — no code shipped, no redeploy. This is the "extend aggregations as numpy operations" intent from #213, and it's the answer to "can a user run a custom aggregation without permission to update the Lambda layer?" — yes, if it's expressible as a control-flow-free numpy expression.

Examples to write (docs template / notebook)

A short template showing the same sketch at increasing optimization, so users can see the pattern and the ceiling:

1. Exact baseline — np.quantile / fixed-value-bin histogram per cell as a kind: ragged expression. Simplest, exact, but payload grows with resolution (no δ budget).

2. Loop-free t-digest as a ragged expression — the ideal-threshold form (straight-line numpy: k1_inv(arange(δ)) → searchsorted → reduceat, no control flow), inlinable into a single expression. This is the real worked example: a full δ-budgeted t-digest a user can drop into a config with zero deploy. (Prototype lives in the t-digest agg hot path is unvectorized: ~15× build + ~6× fold available in pure numpy #279 scratch bench as build_fullvec.)

3. Why the shipped built-in differs — the built-in uses the greedy form because loop-free shrinks the loss-free (exact) regime from ~1.27δ to ~0.64δ, turning the typical ~500-obs cell at δ=512 from exact into approximate:

   n obs	greedy (built-in)	loop-free (expression)
   300	300 exact	300 exact
   500	500 exact	401 approx
   650	650 exact	429 approx

   So: the built-in is the accurate default; the loop-free expression is the deploy-free, user-injectable option, with a documented accuracy caveat. Good teaching example of the tradeoff.

All examples must be binder-runnable with public/in-tree data (per CLAUDE.md §4 docs rule).

The larger conversation — user-side code policy

eval(expr, {"__builtins__": {}, "np": np, ...}) is a soft sandbox, not a security boundary. numpy alone can escape it (np.load → pickle → arbitrary code; other vectors exist). So today's expression path is safe when the job author is trusted (the operator submitting configs) but is not isolation for untrusted/multi-tenant users. Questions to settle here:

• Where is the intended line between "declarative numpy" (allowed as data) and "user code" (forbidden per Event interchange + catalog for the temporal pipeline (antarctic_AR_dataset integration) #213)? The expression eval blurs it — it's data that is also code.
• If we ever want untrusted user reducers, what's the boundary? Options: an AST-allowlist evaluator (numpy calls + arithmetic only, no attribute access to load/__*__), a small restricted DSL, or keeping user reducers strictly local + an upstreaming funnel (status quo per Event interchange + catalog for the temporal pipeline (antarctic_AR_dataset integration) #213).
• Do we want to bless the expression path as a documented, supported extension surface (with its trust assumptions stated), or keep it de-facto-internal?

Acceptance

• A docs page or notebook template with examples (1)–(3), binder-runnable.
• A written decision on the user-side-code trust model (even if the decision is "trusted authors only, documented as such").

Labeled discuss — no code until the trust model and doc shape are agreed.

[espg]

🤖 from Claude

The doc half is straightforward; the policy half is the real fork, so leading there.

The eval namespace is the crux. _eval_expression_raw (config.py) passes {"__builtins__": {}, "np": np, ...}. Stripping __builtins__ stops the obvious escapes but not numpy-native ones — np.load reaches pickle, and attribute access (np.__loader__, dunder-walking off any exposed object) reopens the import graph. So today's guarantee is precisely "safe iff the config author is trusted," which is fine under the operator-submits-configs model (#213) but must be stated in the docs, not implied.

Three trust-model options, in increasing cost:

• (1) Bless as-is, "trusted authors only." Document the expression path as supported, state the trust assumption explicitly, change nothing in code. Cheapest, honest, and matches the actual single-operator deployment. Risk: the moment anyone wires a multi-tenant config intake, this is a remote-code-execution surface, and a documented feature is harder to walk back than a de-facto-internal one.
• (2) AST-allowlist evaluator. Replace eval with a walk that permits arithmetic, subscripting, and calls to a whitelist of np.* names, and rejects attribute access to dunders / load / frombuffer-family. Turns "trusted-only" into "safe for semi-trusted." Cost: a real parser to maintain, and it will reject legitimate expressions (any arr.reshape(...) method call), so it needs a curated method allowlist too. Medium effort, meaningfully narrows the hole.
• (3) Keep strictly-local + the upstreaming funnel (status quo per Event interchange + catalog for the temporal pipeline (antarctic_AR_dataset integration) #213). Users prototype expression locally; anything for shared/untrusted use must become a built-in via PR. Safest, but it's the friction this issue's own worked example (loop-free t-digest, zero-deploy) is trying to relieve — so choosing this means the docs teach the pattern as a local dev tool, not a production extension surface.

My lean: (1) now, with the trust caveat written prominently, and file (2) as a separate tracked issue — don't block the docs on building an AST evaluator, but don't let the docs imply isolation the eval doesn't provide.

Clarifying questions:

1. Is there any current or near-term path where a config comes from someone other than the operator (a hosted intake, a shared-catalog submission)? If no, (1) is clearly right; if yes, the docs shouldn't ship until (2) exists.
2. For the worked example — do you want the loop-free t-digest shown as a single inlinable expression string, or as a documented multi-field decomposition? The single-string form is the impressive drop-in demo but is nearly unreadable; the decomposition teaches the mechanism better. Could show both.
3. Binder data (CLAUDE.md §4): should the examples key off an in-tree test fixture, or a public ATL06/ATL03 granule by URL? In-tree keeps binder hermetic; a real granule is more convincing but adds a network dependency to the notebook.

Risk to flag: whichever option, the docs must not show np.load/np.fromfile in any example, even innocently — a copy-pasteable expression that touches the filesystem is exactly the pattern we don't want normalized. Worth an explicit "expressions are pure array math; no I/O" line in the docs and, ideally, a lint on shipped expression: fields.

---

Generated by Claude Code