[Oztechan/CCC#4845] Run the backend as supervised systemd instances (prod + test) - #4846
Open
mustafaozhan wants to merge 2 commits into
Open
[Oztechan/CCC#4845] Run the backend as supervised systemd instances (prod + test)#4846mustafaozhan wants to merge 2 commits into
mustafaozhan wants to merge 2 commits into
Conversation
The jar is copied to the droplet by CI but nothing starts it, so the process is launched by hand over SSH and manually restarted every 10-20 days once it exhausts the droplet's RAM. Nothing in the application leaks. On a 1G/1vCPU droplet the JVM's heap defaults to a quarter of total RAM and max direct memory defaults to the max heap, so both ceilings derive from total RAM and together oversubscribe the box. The heap expands toward its maximum over time and is not returned to the OS, which is why the failure takes a week or two to surface. When the kernel OOM killer is what acts, nothing restarts. Add a systemd unit that caps heap, direct memory and metaspace explicitly, sets MemoryMax below total RAM so systemd restarts the service rather than the kernel killing it silently, and recovers on its own via Restart=always plus ExitOnOutOfMemoryError. Deliberately no MemoryHigh: it forces reclaim, but a JVM's footprint is almost entirely anonymous pages, which cannot be reclaimed without swap, so the service would stall rather than fail cleanly and restart. Claude-Session: https://claude.ai/code/session_013Km3Q6sfLatVr6YM8cffrF
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Coverage variation | ✅ +0.00% coverage variation |
| Diff coverage | ✅ ∅ diff coverage |
Coverage variation details
Coverable lines Covered lines Coverage Common ancestor commit (4109f94) 3340 1746 52.28% Head commit (0034985) 3340 (+0) 1746 (+0) 52.28% (+0.00%) Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch:
<coverage of head commit> - <coverage of common ancestor commit>
Diff coverage details
Coverable lines Covered lines Diff coverage Pull request (#4846) 0 0 ∅ (not applicable) Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified:
<covered lines added or modified>/<coverable lines added or modified> * 100%
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a deploy/ package to supervise the CCC backend jar with systemd on the droplet, aiming to eliminate manual starts/restarts and prevent kernel OOM kills from leaving the API down.
Changes:
- Introduces a systemd unit template with JVM memory ceilings + restart-on-failure/OOM behavior.
- Adds an idempotent
setup-server.shbootstrap script to install/enable the unit and perform a basic health check. - Documents the rationale and memory sizing analysis in
deploy/README.md.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
deploy/setup-server.sh |
Bootstraps the droplet: selects jar, writes unit from template, (re)starts service, performs health check. |
deploy/README.md |
Documents why systemd is needed, memory sizing, and how to run setup + operate day-to-day. |
deploy/ccc-backend.service.template |
Defines the supervised systemd service with JVM sizing flags, memory cgroup ceiling, restart policy, and hardening. |
Suppressed comments (1)
deploy/setup-server.sh:44
- This log line invokes
javavia PATH, which can differ from what the unit runs (/usr/bin/java). Using the same absolute path makes the output accurate and avoids failures if PATH is unusual under sudo/root.
log "Java : $(java -version 2>&1 | head -1)"
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| [[ -d $APP_DIR ]] || die "APP_DIR does not exist: $APP_DIR" | ||
| id "$APP_USER" &>/dev/null || die "user does not exist: $APP_USER" | ||
| APP_GROUP="$(id -gn "$APP_USER")" | ||
| command -v java &>/dev/null || die "java not found on PATH; install a JRE 21+ first" |
Comment on lines
+53
to
+59
| if [[ ! -f "$APP_DIR/$JAR_NAME" ]]; then | ||
| newest_jar="$(find "$APP_DIR" -maxdepth 1 -name 'app-*.jar' -printf '%T@ %p\n' 2>/dev/null \ | ||
| | sort -rn | head -1 | cut -d' ' -f2-)" | ||
| [[ -n $newest_jar ]] || die "no $JAR_NAME and no app-*.jar found in $APP_DIR" | ||
| log "Promoting $(basename "$newest_jar") -> $JAR_NAME" | ||
| cp "$newest_jar" "$APP_DIR/$JAR_NAME" | ||
| fi |
Comment on lines
+111
to
+118
| log "Writing $UNIT_PATH" | ||
| sed -e "s|__USER__|$APP_USER|g" \ | ||
| -e "s|__GROUP__|$APP_GROUP|g" \ | ||
| -e "s|__APP_DIR__|$APP_DIR|g" \ | ||
| -e "s|__JAR_NAME__|$JAR_NAME|g" \ | ||
| -e "s|__MEMORY_MAX__|$memory_max|g" \ | ||
| -e "s|__PROTECT_HOME__|$protect_home|g" \ | ||
| "$TEMPLATE" > "$UNIT_PATH" |
| supported way to apply new settings. It will: | ||
|
|
||
| 1. promote the newest `app-<version>.jar` to the stable name `ccc-backend.jar`; | ||
| 2. derive `MemoryHigh`/`MemoryMax` from the droplet's actual RAM; |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #4846 +/- ##
========================================
Coverage 55.29% 55.29%
========================================
Files 169 169
Lines 3033 3033
Branches 452 452
========================================
Hits 1677 1677
Misses 1331 1331
Partials 25 25 🚀 New features to boost your workflow:
|
Convert the unit to a systemd template, ccc-backend@.service, so each
environment runs as its own instance with its own working directory and
therefore its own SQLite file. Two instances sharing a directory would
corrupt a single database, since the file is created relative to CWD.
Fitting a second JVM on a 1G droplet needs three things. A swapfile,
which DO droplets ship without and which costs disk rather than RAM. An
asymmetric MemorySwapMax, pinning prod out of swap so its GC pauses stay
predictable while the mostly-idle test instance pages out — this is what
plain "add swap" would not give. And a test instance that does not sync,
keeping it off the premium API quota and leaving its pages idle enough
to be good swap candidates.
Ceilings total 800M against ~981M of MemTotal, leaving ~180M for the OS.
Per-instance cgroup limits are drop-ins rather than EnvironmentFile
values, because systemd expands ${VAR} only in ExecStart and never in
directives like MemoryMax.
The test instance additionally needs a build that reads SERVER_PORT from
the environment; the script warns when it is requested. Setup also gains
MIGRATE_FROM, which copies an existing database and jar into the prod
instance directory without touching the originals.
Claude-Session: https://claude.ai/code/session_013Km3Q6sfLatVr6YM8cffrF
mustafaozhan
changed the title
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4845
What
Adds a
deploy/directory with a templated systemd unit that supervises the backend, one instance per environment, plus a one-time setup script and the memory analysis behind the tuning.Nothing outside
deploy/changes — this PR does not touch the build, the backend code, orrelease.yml.Why
CI copies the jar to the droplet but nothing starts it, so the process is launched by hand over SSH and manually restarted every 10-20 days once it exhausts RAM.
The application is not leaking.
HttpClientis a Koinsingle, the sync loops are bounded, andConversionis capped at ~160 rows by itsbaseprimary key. The cause is an unconstrained JVM: heap defaults to ¼ of total RAM and max direct memory defaults to the max heap, so both ceilings derive from total RAM and together oversubscribe a 1 GB box. The heap expands toward its maximum over time and is not returned to the OS, which is why it takes a week or two to surface.What made it a manual chore is that when the kernel OOM killer acts, nothing restarts the service.
Two environments
ccc-backend@.serviceis a template, so%igives each environment its own working directory — and therefore its own SQLite file, sinceapplication_database.sqlite.dbis created relative to CWD.MemoryMaxMemorySwapMaxHow both fit in 1 GB
A Ktor/Netty/Koin/SQLDelight JVM has a non-heap floor of ~170-220 MB before any heap, paid per instance. Three things make the second one affordable:
MemorySwapMax— prod pinned out of swap so its GC pauses stay predictable, test allowed to page out. This is what plain "add swap" would not give.Ceilings total 800 MB against ~981 MB
MemTotal, leaving ~180 MB for the OS.vm.swappiness=10keeps swap a safety valve rather than routine.Notes on specific choices
MemoryHigh. It forces reclaim, but a JVM's footprint is almost entirely anonymous pages — the service would stall under pressure instead of failing cleanly and restarting.EnvironmentFilevalues. systemd expands${VAR}only inExecStart, never in directives likeMemoryMax;EnvironmentFilereaches the process, not the unit config.adaptiveallocator, soio.netty.allocator.*pooled-arena properties are no-ops.MaxDirectMemorySizeis the control that applies.UseSerialGCis for determinism, not effect. Ergonomics already selects it below 2 CPUs; the flag only guarantees the choice survives a droplet resize.ProtectHomeis computed by the setup script, since hardcodingyeswould hide the jar when the deploy directory is under/homeor/root.Rollout
Not automatic. Prod first:
MIGRATE_FROMcopies the existing database and newestapp-<version>.jarinto/opt/ccc/prod; originals are left as a rollback path.The test instance binds 8081 and therefore needs a build that reads
SERVER_PORTfrom the environment — that is the next PR. The script warns iftestis requested before then.Not verified
The JVM flags, script syntax and template/placeholder parity are validated locally, but this has not been run on a real droplet. The health-check loop reports per-instance failures with the
journalctlcommand to diagnose.Follow-ups
SERVER_HOST/SERVER_PORT/SYNC_ENABLEDas independent env vars, replacing the overloadedIS_PRODUCTION— required before the test instance can startdevelop→ test,master→ prod, with a restart and health check