[J-02] Scaling down Graylog strands unprocessed messages in abandoned PVC without drain procedure

[alix-graylog]

Summary

When scaling down Graylog replicas without draining the message journal first, unprocessed messages are left stranded in abandoned PersistentVolumeClaims. The current README
documentation shows bare scale-down commands (--set graylog.replicas=1) with no guidance on how to safely drain the journal, risking data loss.

Details

Graylog maintains an on-disk message journal for durability. When a StatefulSet is scaled down:

1. Pods are terminated in reverse ordinal order (pod-N down to pod-0)
2. The associated PVCs remain attached due to StatefulSet retention policies
3. Any unprocessed messages in the journal are left in the orphaned PVC
4. Without explicit drain procedures, these messages are effectively lost

The issue is compounded by lack of clear documentation on:

• How to check journal fill level before scaling
• How to gracefully drain messages before shutdown
• Which Graylog API endpoints to use for monitoring
• How to determine when it's safe to scale down

Reference: J-02 (Production Readiness Review)

Impact

Production deployments using the bare --set graylog.replicas=1 scale-down procedure risk data loss. Operations teams lack the procedures needed to safely scale down while
preserving unprocessed messages. This is a critical gap in production runbooks.

How to Reproduce?

Environment

• Helm chart version: v4.2.0 or later
• Kubernetes version: any (with StatefulSet support)
• Graylog with messageJournal.enabled: true and persistent storage

Pre-flight checks

• helm lint ./graylog passes
• Checked existing issues for duplicates

Steps to reproduce the issue:

1. Deploy Graylog with journal enabled:

   helm install graylog ./charts/graylog \
     --set graylog.config.messageJournal.enabled=true \
     --set graylog.persistence.enabled=true
   

2. Send messages to Graylog (e.g., via GELF input on port 12201)
3. Trigger an arbitrary scale-down (simulating the README example):
   kubectl patch statefulset graylog --type='json'
   -p='[{"op": "replace", "path": "/spec/replicas", "value":1}]'
4. Observe behavior:

Watch pods terminate

kubectl get pods -w

Check remaining PVCs after scale-down

kubectl get pvc

Orphaned PVCs remain even after pods are gone

5. Try to recover messages:

There is no clear procedure documented

Messages in the abandoned PVC are inaccessible

Expected behavior

• The README should provide a drain runbook before scale-down
• Users should be able to monitor journal fill level using documented API endpoints
• Unprocessed messages should be flushed or explicitly drained before scaling down
• The drain procedure should be automated or clearly documented

Actual behavior

• The README shows --set graylog.replicas=1 with no drain guidance
• There is no procedure to safely drain messages before shutdown
• Unprocessed messages in scaled-down pods are lost
• Operations teams must reverse-engineer drain procedures on their own

Notes for maintainers

What needs to be documented (J-02):
Create a "Scaling Down Safely" runbook in the README that covers:

1. Check journal fill level before initiating scale-down:

Get current journal fill percentage

kubectl exec graylog-0 -- curl -s http://localhost:9000/api/system/journal | jq '.uncommitted_entries'
2. Monitor load balancer status to verify pod drains:

Watch lbstatus for all nodes

for i in 0 1 2; do
kubectl exec graylog-$i -- curl -s http://localhost:9000/api/system/lbstatus
done
3. Flush buffers (if needed):

• The server automatically flips lbstatus to DEAD on shutdown
• With the readiness probe fixed (see G-02), the pod is removed from Service endpoints automatically
• Buffers flush during graceful shutdown (respecting terminationGracePeriodSeconds)

4. Scale down with partition-canary to avoid wholesale crashes:

Scale one replica at a time, verify journal is draining

kubectl patch statefulset graylog --type='json'
-p='[{"op": "replace", "path": "/spec/updateStrategy/rollingUpdate/partition", "value":2}]'

Wait for pod-2 to terminate gracefully

Verify messages are being consumed by remaining pods

Repeat for pod-1, then pod-0

Related issues (tracked separately):

• J-03: Template-time guard: fail when messageJournal.enabled=true + persistence.enabled=false
• J-04: Replace journal lock sentinel (.lock) with dedicated .initialized marker file
• G-02: HTTP readiness probe on /api/system/lbstatus (enables automatic pod removal from Service)
• U-02: Expose and document terminationGracePeriodSeconds for graceful shutdown

API endpoints to document:

• GET /api/system/lbstatus — unauthenticated, returns 200/429/503, used for load balancer integration
• GET /api/system/journal — journal statistics including uncommitted entry count
• PUT /api/system/lbstatus/override/dead — manually flip node to DEAD (requires admin auth)

Helm-specific guidance:

• Explain StatefulSet ordinal-based termination (pod-N down to pod-0)
• Document persistentVolumeClaimRetentionPolicy and why whenScaled: Retain is the safe default
• Show how to audit orphaned PVCs after scale-down