Secrets framework: policy-based credential handling and message scrubbing (#211)

[mattprintz]

Summary

Introduces a secrets subsystem for Beaker: a typed registry of the credentials the server knows about, a per-boundary policy engine describing what may cross each trust boundary, and enforcement at two of those boundaries (subkernel process environment, and Jupyter messages leaving the BeakerKernel).

The core idea is that a secret is not just a value — it's a value plus a set of policies, one per boundary it might cross. BaseSecret declares five policy slots (subkernel_message_policy, ui_message_policy, agent_message_policy, beaker_kernel_environment_policy, subkernel_environment_policy), and each secret subclass sets defaults appropriate to its tier. A system env secret is Allow into the Beaker kernel environment but Remove from the subkernel's; a user env secret is the reverse. This makes the two-tier split from #211 a property of the secret type rather than of scattered call sites.

What's here

Policy engine (src/beaker_notebook/lib/secrets/policies.py)

Allow, Redact (equal-length # masking), Remove, Last4 (fully masks secrets under 6 chars rather than revealing a large fraction of them), and MappingKey for the dict-key case. Policies recurse over str/bytes/list/dict and operate on a deep copy. Dict keys containing a secret are rewritten to a placeholder (SYSTEM-ENV-SECRET:0) and remapped after iteration completes, with next_key() disambiguating collisions. Each sanitization runs a validation chain — currently not_in, which raises SecretValidationError if the secret survives, so a policy bug fails loudly instead of leaking.

Secret types (src/beaker_notebook/lib/secrets/secret_types.py)

EnvironmentSecret and its System/User variants, BeakerConfigProviderSecret (LLM provider keys and the service-token override), and a SkillSecret stub.
to_dict/from_dict round-trip class identity, per-boundary policies, and optionally the resolved value — this is what lets a server-resolved secret be reified inside the kernel process, where get_value() often can't run.

Secrets manager (src/beaker_notebook/services/secrets/manager.py)

A LoggingConfigurable hung off the app as the secrets_manager trait, populated during initialize(). It discovers secrets from three sources: environment-variable name heuristics (configurable substring and suffix lists, each with an extra_* companion so operators extend rather than replace the defaults), app traits by dotted config string (Application.cookie_secret, IdentityProvider.token, GatewayClient.auth_token), and Beaker config providers.
It exposes sanitize_kernel_environment_vars and sanitize_subkernel_envionment_vars, both wired into BeakerKernelManager._async_pre_start_kernel — which addresses acceptance criterion (b): shared keys are stripped from the subkernel env before launch.

AppTraitSecret (services/secrets/app_secrets.py) resolves lazily by walking the traitlets Configurable graph from the Application singleton, so a trait's value is read at scrub time rather than captured at startup.

Message scrubbing (src/beaker_notebook/kernel.py, lib/jupyter_kernel_proxy.py)

intercept_message now treats None for stream_type/msg_type as a wildcard, and ProxyKernelServer.get_destination() distinguishes client-bound from subkernel-bound streams. BeakerKernel.redact_secrets registers as a catch-all filter, picks the policy slot by destination, and sanitizes metadata and content only — routing headers and binary buffers are deliberately left intact so a false-positive substring match can't corrupt correlation ids or blob bytes. The kernel fetches its secret set (values included) once at context establishment from a new authenticated GET /beaker/secrets/ endpoint (services/secrets/handlers.py), filtering out secrets with no usable value.