Skip to content

feat(provider): add dictionaryFromJson sync type - #44

Open
adrien-barret wants to merge 1 commit into
controlplane-com:mainfrom
AdeptMind:feat/dictionary-from-json
Open

feat(provider): add dictionaryFromJson sync type#44
adrien-barret wants to merge 1 commit into
controlplane-com:mainfrom
AdeptMind:feat/dictionary-from-json

Conversation

@adrien-barret

Copy link
Copy Markdown

Summary

Adds a new sync type dictionaryFromJson: <secret-path> that creates a Control Plane dictionary secret from a single source secret whose value is a JSON object. Leaf values are exposed under dot-notation keys, so consumers can reference nested fields via standard CPLN dot-notation (cpln://secret/<name>.<dot.path>).

Motivation

We have a setup where:

  • Devs across multiple teams own GCP Secret Manager secrets containing JSON config blobs (sometimes nested).
  • They consume those secrets in CPLN workloads via dot-notation, which requires the CPLN secret to be of type dictionary with flat keys.
  • The set of JSON keys evolves frequently and the infra team doesn't get notified before each change.

The existing options didn't cover this:

Option Why it doesn't fit
opaque Doesn't support dot-notation referencing
dictionary with explicit {KEY: {path, parse}} Requires enumerating every key in sync.yaml; the schema is dynamic, so any new key would need a Terraform / sync.yaml change
dictionaryFromProject: true (GCP) Bulks every secret in the GCP project into a single dict secret — wrong granularity (we want one dict per JSON secret, not one giant dict per project)

dictionaryFromJson fills the gap: ESS fetches the source secret, parses its value as JSON, recursively walks the object, and exposes each leaf as a flat dot-path key in the resulting dictionary.

Example

GCP secret catalog-async-prod contains:

{ "db": { "user": "admin", "pass": "s3cret" }, "api_key": "abc123" }

sync.yaml:

- name: catalog-async-prod
  provider: my-gcp
  dictionaryFromJson: catalog-async-prod

Result — CPLN dictionary secret catalog-async-prod:

{ "db.user": "admin", "db.pass": "s3cret", "api_key": "abc123" }

Consumers can now use cpln://secret/catalog-async-prod.db.user.

Fallback behavior

If the source value cannot be parsed as a JSON object (raw string, JSON array, scalar, or malformed JSON), ESS falls back to a single-key dictionary { __raw: <raw-value> } instead of throwing. This avoids type-flip delete/recreate cycles when a dev pushes a transient bad value, and surfaces the issue clearly in the CPLN UI under the __raw key.

Implementation

  • src/config/syncConfig.ts — add dictionaryFromJson: z.string().nonempty() to SecretSchema, include in the xor refinement and in isDictionarySecret.
  • src/provider/util/flatten.ts — new utility with flattenJson() (recursive dot-path walk) and tryParseJsonObject() (safe parse with object-root guard).
  • src/provider/provider.ts — new branch in getSecret() and checkSecret(); arrays are JSON-stringified to keep dict values as strings.
  • src/sync/sync.ts — no changes needed; existing isDictionarySecret() path handles it.
  • 17 new unit tests in flatten.spec.ts and syncConfig.spec.ts.

Test plan

  • npm run build passes (TypeScript compiles)
  • npx jest src/provider/util/flatten.spec.ts — 14 tests pass
  • npx jest -t "dictionaryFromJson" — schema validation passes
  • npx jest -t "rejects multiple sync types" — xor refinement passes
  • e2e against real GCP Secret Manager (we'll validate this on our fork before any upstream merge)

Happy to iterate on naming (dictionaryFromJson vs dictionaryFromJsonSecret vs something else), behavior (throw vs fallback), and depth limits if you have preferences.

ESS can now create a CPLN dictionary secret from a single source secret
containing a JSON object — leaf values become dot-notation keys in the
dictionary. Removes the need to enumerate keys in sync.yaml when the
schema is unknown or evolving.

When the source value is not a parseable JSON object (raw string,
array, scalar, or malformed JSON), ESS falls back to a single-key
dictionary {__raw: <value>} instead of throwing. This avoids type-flip
delete/recreate cycles on transient bad pushes.

- Schema: dictionaryFromJson (string) added to SecretSchema
- New util: flattenJson(obj) → flat dict with dot paths
- Provider: branch in getSecret() handles fetch + parse + fallback
- 17 unit tests for flatten + tryParseJsonObject + schema validation

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant