fix(plausible): stop ClickHouse OOM crashloop and fix DB hostnames - #27
Merged
Merged
Conversation
ClickHouse crashlooped on stage (Code 241 MEMORY_LIMIT_EXCEEDED), blocking the analytics pod's wait-for-clickhouse init container. Bitnami's default resourcesPreset "small" caps ClickHouse at a 768Mi limit; ClickHouse sets its memory tracker to ~90% of the cgroup limit (~691Mi), too little to run background merges and system-log flushes. Override the preset with explicit resources (1Gi request, 2Gi limit); the node has ~6.5Gi allocatable. Also fix the Postgres and ClickHouse connection URLs. The chart's defaults target `plausible-analytics-*` hostnames, but our Helm release is named `plausible`, so the bundled subchart Services are `plausible-postgresql` and `plausible-clickhouse`. Point the URLs at the real Service names (credentials still match the subchart auth defaults). Refs #18. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Helm template diffdiff -u --recursive --label base --label head base head
--- base
+++ head
@@ -218,10 +218,10 @@
app.kubernetes.io/managed-by: Helm
type: Opaque
data:
- SECRET_KEY_BASE: ZXB0UUZRZGNZR29jTDRmenFVVjJtdUxZNHdhSlM2SmI5V3h1RnBRYVVHaFZaZXV6Nk4xenZXYVgzZDNybGt0RmxraWtQc1ZrUWRhR1hlaGJTRk5kWXRlcXRU
+ SECRET_KEY_BASE: N3c2d0lzSE1QOWQ3aTR3dDdmb2dRV0lNalplOVBXMnUwbnFvUFFETEx0VTR5bmxNTzRnWDJxWVN5Mzlhb1d5eFQzc2E1dDE1dkNNS3NBemdQNFFxUVZnSzlr
TOTP_VAULT_KEY: ZHN4dmJuM2p4RGQxNmF6MlFwc1g1QjhPK2xseGpRMlNKRTJpNUJ6eDM4ST0=
- DATABASE_URL: cG9zdGdyZXM6Ly9wb3N0Z3Jlczpwb3N0Z3Jlc0BwbGF1c2libGUtYW5hbHl0aWNzLXBvc3RncmVzcWw6NTQzMi9wbGF1c2libGVfZGI=
- CLICKHOUSE_DATABASE_URL: aHR0cDovL2NsaWNraG91c2U6cGFzc3dvcmRAcGxhdXNpYmxlLWFuYWx5dGljcy1jbGlja2hvdXNlOjgxMjMvcGxhdXNpYmxlX2V2ZW50c19kYg==
+ DATABASE_URL: cG9zdGdyZXM6Ly9wb3N0Z3Jlczpwb3N0Z3Jlc0BwbGF1c2libGUtcG9zdGdyZXNxbDo1NDMyL3BsYXVzaWJsZV9kYg==
+ CLICKHOUSE_DATABASE_URL: aHR0cDovL2NsaWNraG91c2U6cGFzc3dvcmRAcGxhdXNpYmxlLWNsaWNraG91c2U6ODEyMy9wbGF1c2libGVfZXZlbnRzX2Ri
---
# Source: plausible/charts/plausible-analytics/charts/clickhouse/templates/configmap.yaml
@@ -748,13 +748,11 @@
envFrom:
resources:
limits:
- cpu: 750m
- ephemeral-storage: 2Gi
- memory: 768Mi
+ cpu: 1
+ memory: 2Gi
requests:
cpu: 500m
- ephemeral-storage: 50Mi
- memory: 512Mi
+ memory: 1Gi
ports:
- name: http
containerPort: 8123 |
Contributor
There was a problem hiding this comment.
Pull request overview
Restores Plausible availability on the stage cluster by fixing two deployment-blocking configuration issues in the Plausible Helm wrapper chart: incorrect in-cluster DB service hostnames and insufficient ClickHouse resources causing OOM crashloops.
Changes:
- Override
databaseURLandclickhouseDatabaseURLto use the actual Service names created by this Helm release (plausible-postgresql,plausible-clickhouse). - Disable the ClickHouse
resourcesPresetand set explicit CPU/memory requests/limits to prevent OOM crashlooping.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+14
to
+17
| # Only the host portion is wrong; the credentials match the subchart auth | ||
| # (postgres/postgres, clickhouse/password) set in the dependency defaults. | ||
| databaseURL: "postgres://postgres:postgres@plausible-postgresql:5432/plausible_db" | ||
| clickhouseDatabaseURL: "http://clickhouse:password@plausible-clickhouse:8123/plausible_events_db" |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes Plausible on stage, which was fully down. Two independent bugs, both confirmed on the live cluster.
Bugs
resourcesPreset: smallcaps ClickHouse at a 768Mi limit. ClickHouse sets its memory tracker to ~90% of the cgroup limit (~691Mi), too little to run background merges and system-log flushes, so it crashlooped (Code: 241 MEMORY_LIMIT_EXCEEDED, 592 restarts). This blocked the analytics pod'swait-for-clickhouseinit container. The node has ~6.5Gi allocatable.databaseURL/clickhouseDatabaseURLtargetplausible-analytics-postgresqlandplausible-analytics-clickhouse, but our Helm release is namedplausible, so the bundled subchart Services areplausible-postgresqlandplausible-clickhouse. The analytics pod'swait-for-postgresinit container had been looping onplausible-analytics-postgresql:5432 - no responsesince 2026-07-08.Fix
Verification
Application: Ready for connectionsand noCode 241. (That live patch was temporary and reverts on ArgoCD selfHeal; this PR is the durable fix.)helm templateemitsplausible-postgresql/plausible-clickhouse, matching the actual Services. The livewait-for-postgresloop confirmed the wrong host is the current blocker.Out of scope (follow-ups on #18)
postgres:postgres,clickhouse:password) should move to ESO/1Password before prod.persistence.enabled: false, so analytics data is lost on pod restart.Refs #18.