Skip to content

feat(datanode): delegate OpenSearch roles across node groups - #114

Open
monrax wants to merge 3 commits into
mainfrom
feat/dn-roles
Open

feat(datanode): delegate OpenSearch roles across node groups#114
monrax wants to merge 3 commits into
mainfrom
feat/dn-roles

Conversation

@monrax

@monrax monrax commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

This is a first pass at the DataNode roles issue. While it has been validated against a live EKS cluster, it has not been tested thoroughly.

Summary

Allow Data Nodes to be dedicated to specific OpenSearch roles so larger clusters can delegate responsibilities (e.g. a dedicated search/warm tier or dedicated cluster-manager nodes) instead of every Data Node carrying all roles.

Details

  • The top-level datanode block is now the primary node group; datanode.roles (default [] = Data Node default set of roles) sets its OpenSearch roles and is inherited as the default by extra groups.
  • New datanode.extraNodeGroups map (keyed by group name) — each entry renders its own StatefulSet, ConfigMap and PDB, inheriting every datanode.* value and overriding only what it declares. The headless Service and Secret stay shared; discovery seed hosts span all groups so they form a single OpenSearch cluster.
  • graylog.datanode.groups helper normalizes + enriches each group (fullname, configmap/pdb names, storage classes, group label) so templates iterate with ./$.
  • Group labels (graylog-datanode-group) are emitted only when extra groups exist, so a plain install keeps today's unlabeled selector and upgrades in place.
  • Guardrails: hard-fail when no group is cluster_manager-eligible; NOTES.txt warnings when no group is data-eligible or when a search group has no snapshot repository configured.
  • Dropped the | default N fallbacks from graylog.replicas / graylog.datanode.replicas values.yaml is authoritative; explicit replicas: 0 is no longer coerced); graylog.datanode.replicas now sums across groups.
  • Docs: new docs/datanode-node-roles.md guide (roles, node groups, guardrails, search-tier repository requirement, flat→groups migration), linked from the README and values reference.

Linked issues

This fixes #107

PR Checklist

Please check the items that apply to your change.

  • Tests added/updated
  • Documentation updated
  • This PR includes a new feature
  • This PR includes a bugfix
  • This PR includes a refactor

Testing Checklist

Static Validation

  • Linter check passes: helm lint ./charts/graylog
  • Helm renders local template sucessfully: helm template graylog ./charts/graylog --validate

Installation

  • Fresh installation completes successfully (verified on EKS with node groups)
  • All pods reach Running state
  • Helm unit tests pass: helm unittest charts/graylog (73 tests)

Functional (if applicable)

  • DataNodes visible / registered (all groups CONNECTED/AVAILABLE, indexer cluster green)
  • Web UI accessible and login works
  • Inputs can be created and receive data

Upgrade (if applicable)

  • Upgrade from a flat install to node groups succeeds (see below)
  • Scaling up/down works correctly

Specific to this PR

Verified live on EKS (fresh EKS cluster, provider=aws):

  • Role delegation: primary cluster_manager-only + a data,ingest,remote_cluster_client group. Each pod's generated opensearch.yml carried exactly the delegated node.roles; the two StatefulSets had distinct selectors (no cross-adoption); all datanodes registered in one cluster; indexer health green with 0 unassigned shards.
  • Guardrails: confirmed the hard-fail blocks a real helm upgrade --dry-run when no group is cluster_manager-eligible; NOTES warnings render for the no-data / search-without-repo cases.
  • Migration: enabling extraNodeGroups on a running flat install fails an in-place upgrade on the immutable StatefulSet selector; the documented kubectl delete statefulset graylog-datanode && helm upgrade recreates it (PVCs retained) and the cluster returns green as one cluster. The role-narrowing / dedicated-manager rebuild caveats are documented in the guide.

Notes for reviewers

  • Verify all applicable tests above pass
  • Validate that the linked issues are no longer reproducible, if applicable
  • Sync up with the author before merging
  • The commit history should be preserved - use rebase-merge or standard merge options when applicable

🤖 Generated with Claude Code

monrax added 3 commits July 13, 2026 00:59
@monrax
feat(datanode): delegate OpenSearch roles across node groups
2c3cb29 
Add opt-in role delegation so larger clusters can dedicate StatefulSets to
specific OpenSearch roles (cluster_manager, data, ingest, remote_cluster_client,
search) instead of every Data Node carrying all roles.

- The top-level datanode block is the primary node group (legacy unsuffixed
  names). datanode.roles (default []) sets its OpenSearch roles and is inherited
  as the default by every extra group; empty keeps the Data Node default and its
  automatic search-role-on-snapshot behavior.
- datanode.extraNodeGroups is a map keyed by group name; each value overrides the
  inherited datanode.* values. Keys may be hyphenated. Cross-values-file deep
  merge and unique names come for free.
- graylog.datanode.groups enriches every group with derived fields (fullname,
  configmapName, pdbName, groupLabel, data/nativeLibs storage class), so the
  StatefulSet, ConfigMap and PDB templates iterate with `.`/`$` and never pass
  (root, group) dicts. The StatefulSet, ConfigMap and PDB become per-group; the
  headless Service and Secret stay shared, and discovery seed hosts span all
  groups.
- Group labels (graylog-datanode-group) are emitted only when extra groups exist,
  so a plain install keeps today's unlabeled selector and upgrades in place. When
  groups are added, the primary is labeled `default` and every StatefulSet gets a
  distinct selector (no controller overlap). NOTE: this relabels the primary
  selector, which is immutable — first-time adoption requires recreating the
  graylog-datanode StatefulSet (documented in values.yaml).
- graylog.datanode.replicas sums replicas across groups; drop the `| default N`
  fallbacks from graylog.replicas / graylog.datanode.replicas so values.yaml is
  authoritative and an explicit replicas: 0 is no longer coerced.

Verified: back-compat renders identically to main (modulo random secrets and
config checksum); map mode renders primary + extras with distinct selectors,
per-group roles/heap/storage, group-spanning seed hosts, and falsy overrides
(persistence.data.enabled: false). 66 unit tests pass; helm lint clean.

Guardrails (fail without a cluster_manager group, warn on missing data role or a
search group without a repo) follow in a separate commit.

Co-Authored-By: Claude Opus 4.8 (1M context)[EMAIL_ADDRESS_REDACTED]om>
@monrax
feat(datanode): guardrails for node-group role coverage
8ece57a 
Add validation on top of the extraNodeGroups role delegation:

- Hard fail (graylog.datanode.validate, included from the datanode StatefulSet)
  when no node group is eligible to be a cluster_manager, i.e. every group sets
  explicit roles and none includes cluster_manager. Groups with empty roles use
  the Data Node default (which includes cluster_manager), so the all-defaults
  install never trips it.
- NOTES.txt warning when no group is eligible to hold data (every group sets
  explicit roles and none includes data).
- NOTES.txt warning when a group declares the search role but no snapshot
  repository is configured (datanode.config.s3ClientDefault*), listing the
  affected groups; suppressed once a repo is set.

Warnings follow the chart's existing NOTES.txt convention (fsGroup, MongoDB
worker count). 7 unit tests cover the fail path, both warnings, and their
negative cases via failedTemplate / matchRegexRaw; 73 tests pass, lint clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <[EMAIL_ADDRESS_REDACTED]>
@monrax
docs(datanode): guide for node roles / node groups
2423e05 
Add docs/datanode-node-roles.md covering the role list, the primary group +
extraNodeGroups map, a hot+search example, the guardrails, the search-tier
snapshot-repository requirement, and the flat->groups migration. The migration
recreates the primary StatefulSet by deleting it and upgrading in one chained
step, so helm creates (not patches) the immutable selector and never errors;
PVCs are retained and the primary pods restart once. Documents the
role-narrowing / data-path and dedicated-manager rebuild caveats.

Link the guide from the README Usage section and the DataNode values reference,
and trim the values.yaml comments to a short pointer.

Migration verified live on EKS: adding a data-capable group to a running flat
install migrates cleanly (PVCs retained, cluster stays green as one cluster).

Co-Authored-By: Claude Opus 4.8 (1M context) <[EMAIL_ADDRESS_REDACTED]>
@monrax
monrax requested a review from alix-graylog July 13, 2026 08:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alix-graylog alix-graylog Awaiting requested review from alix-graylog

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Allow Datanode Roles

1 participant

@monrax