[C-07] MongoDB default topology PSA discouraged for production; should default to 3 data replicas

[alix-graylog]

Summary

The chart's default MongoDB topology uses a Primary-Secondary-Arbiter (PSA) configuration with 2
data-bearing replicas and 1 arbiter. MongoDB explicitly discourages this topology for
production. With one data member down, w:majority writes stall. The default should be changed
to 3 data-bearing replicas with no arbiter, reserving PSA as a documented cost-optimization
option.

Details

Current default (lines 364-365 of values.yaml):

mongodb:
  replicas: 2
  arbiters: 1

This configuration creates a Primary-Secondary-Arbiter replica set. According to MongoDB
documentation:

• PSA topology is not recommended for production use
• With one data member down, replica sets using w:majority cannot complete writes
• If the secondary fails and the primary is isolated, the arbiter cannot be elected as primary
  (it has no data)

Recommended production topology:

• 3 data-bearing replicas (no arbiters)
• Supports w:majority writes with one replica down
• Better supports rolling upgrades and maintenance windows

Secondary issue: The NOTES.txt template (line 16) has a replica count comparison that may not
render topology descriptions correctly. Needs verification after topology change.

Reference: C-07 (Production Readiness Review)

Impact

Production deployments using the default MongoDB configuration cannot handle replica set member
failures gracefully. This is a significant reliability issue. Users must manually adjust the
configuration, but the defaults don't guide them toward the production-recommended topology.

How to Reproduce?

Environment

• Helm chart version: v4.2.0 or later
• MongoDB chart dependency enabled (default)

Pre-flight checks

• helm lint ./graylog passes
• Checked existing issues for duplicates

Steps to reproduce the issue:

1. Deploy Graylog with default MongoDB settings:
   helm install graylog ./charts/graylog
2. Check the deployed MongoDB replica set configuration:
   kubectl exec mongodb-0 -- mongosh --eval "rs.conf()" | grep -E "members|arbiterOnly"
3. Verify the topology:
   kubectl logs graylog-0 | grep "MongoDB"

Output shows: "Primary-Secondary-Arbiter mode"

4. Simulate data member failure and attempt writes:

Kill the secondary

kubectl delete pod mongodb-1

Try to perform a write with w:majority

kubectl exec mongodb-0 -- mongosh --eval "
db.test.insert({x: 1}, {writeConcern: {w: 'majority'}})"

Expected behavior

• Default MongoDB topology should be production-grade (3 data replicas)
• Writes with w:majority should succeed even with one replica down
• PSA should be documented as a cost-optimized alternative
• NOTES.txt should accurately describe the topology

Actual behavior

• Default topology is PSA with 2 replicas + 1 arbiter
• w:majority writes fail with one data replica down
• No guidance on why PSA is being used as the default
• Users must manually reconfigure for production

Notes for maintainers

Changes needed:

1. Update values.yaml (lines 364-365):

Change from:

replicas: 2
arbiters: 1

To:

replicas: 3
arbiters: 0
2. Add documentation in values.yaml:
Add a comment explaining the topology choice and cost optimization:
mongodb:

For production deployments, use 3 data-bearing replicas with no arbiters.

This configuration supports w:majority writes even with one replica down.

For cost-optimized test/staging environments, use the PSA topology:

replicas: 2, arbiters: 1

replicas: 3
arbiters: 0
3. Verify NOTES.txt logic (line 16):
Test that the replica description renders correctly with the new defaults:

• With replicas: 3, arbiters: 0 → Should show "Primary-Secondary-Secondary"
• With replicas: 2, arbiters: 1 → Should show "Primary-Secondary-Arbiter"

Current condition if gt 1 $mongoReplicaCount may need adjustment for clarity.
4. Add README section documenting topology options:

MongoDB Topology

The default production topology uses 3 data-bearing replicas with no arbiters.
This configuration ensures w:majority writes succeed even with one replica down.

For cost-optimized deployments, you can use the Primary-Secondary-Arbiter (PSA)
topology instead:

mongodb:
  replicas: 2
  arbiters: 1

4. Note: PSA topology is not recommended by MongoDB for production use.

5. Test the changes:

• Deploy with new defaults and verify 3 replicas are created
• Simulate replica failures and verify w:majority writes continue
• Verify the NOTES.txt topology description is accurate
• Verify upgrading from old configs respects user-set values

Related improvements (tracked separately):

• Issue HA Improvements #14 (HA Improvements) covers anti-affinity and PDB defaults which complement this change