fix(release): update the deployment script#138
Conversation
📝 WalkthroughWalkthroughThe deployment workflows add Docker BuildKit cleanup after image pruning. The PII analysis notebook reformats markdown, imports, validator construction, and DataFrame displays without changing validation or metric behavior. ChangesDeployment Docker cleanup
PII notebook formatting
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Ayush8923
changed the title
There was a problem hiding this comment.
this PR only includes formatting-related CI fixes.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/cd-production.yml:
- Line 34: Update the SSM deployment commands in
.github/workflows/cd-production.yml line 34 and .github/workflows/cd-staging.yml
line 33 to prevent docker builder prune -af from exceeding the workflow’s
polling window: either extend the associated SSM wait/poll duration beyond five
minutes or cap the prune step while preserving successful deployment completion.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 98b462b6-e45a-4071-b70c-9cb16ce2a0b7
📒 Files selected for processing (3)
.github/workflows/cd-production.yml.github/workflows/cd-staging.ymlbackend/notebooks/analysis-pii.py
| BUILD_DIRECTORY: ${{ secrets.BUILD_DIRECTORY }} | ||
| run: | | ||
| DEPLOY_CMD="cd ${BUILD_DIRECTORY} && git fetch --tags origin && git checkout --force ${{ github.sha }} && docker compose build && docker compose run --rm --entrypoint '' backend uv run --frozen alembic upgrade head && docker compose up -d --remove-orphans && docker image prune -f" | ||
| DEPLOY_CMD="cd ${BUILD_DIRECTORY} && git fetch --tags origin && git checkout --force ${{ github.sha }} && docker compose build && docker compose run --rm --entrypoint '' backend uv run --frozen alembic upgrade head && docker compose up -d --remove-orphans && docker image prune -f && docker builder prune -af" |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
for f in .github/workflows/cd-production.yml .github/workflows/cd-staging.yml; do
echo "===== $f ====="
nl -ba "$f" | sed -n '1,220p'
echo
done
echo "===== search for SSM polling / timeout logic ====="
rg -n "wait|poll|timeout|ssm|builder prune|docker builder prune|docker image prune|DEPLOY_CMD" .github/workflows . -g '!**/node_modules/**'Repository: ProjectTech4DevAI/kaapi-guardrails
Length of output: 260
🏁 Script executed:
#!/bin/bash
set -euo pipefail
for f in .github/workflows/cd-production.yml .github/workflows/cd-staging.yml; do
echo "===== $f ====="
cat -n "$f" | sed -n '1,220p'
echo
done
echo "===== search for polling / timeout logic ====="
rg -n "wait|poll|timeout|ssm|builder prune|docker builder prune|docker image prune|DEPLOY_CMD|send-command|get-command-invocation" .github/workflows . -g '!**/node_modules/**'Repository: ProjectTech4DevAI/kaapi-guardrails
Length of output: 15127
🌐 Web query:
AWS SSM send-command get-command-invocation status poll timeout default command continues after client stops polling
💡 Result:
When you use AWS Systems Manager (SSM) Run Command, the process is asynchronous [1]. Stopping your client-side polling of get-command-invocation has no effect on the execution of the command on the managed node [1]. Key points regarding this behavior: 1. Asynchronous Execution: The send-command API initiates the command and returns a command ID, but it does not wait for the command to finish [1]. The command continues to run independently on the target instance regardless of what happens to the client that issued it [1][2]. 2. Polling vs. Execution: get-command-invocation is simply a mechanism to query the current status and output of a command that is already running or has already completed on the managed node [3][4]. Your client-side polling loop (or the CLI/SDK waiters) is only responsible for observing the state; it does not control the command's lifecycle [1][5][6]. 3. Command Timeouts: The lifecycle of the command is governed by parameters defined during the send-command call, specifically TimeoutSeconds (for delivery) and, if defined in the SSM document, Execution Timeout [2][7]. If these limits are reached, the SSM Agent on the managed node will terminate the execution and report the status as DeliveryTimedOut or ExecutionTimedOut, even if your client has stopped polling entirely [2][3]. In summary, the command on the target node is completely decoupled from your polling client. It will continue to execute until it finishes, reaches a terminal state (such as Success, Failed, or TimedOut), or is explicitly cancelled by another call to the cancel-command API [2][3].
Citations:
- 1: https://stackoverflow.com/questions/77480826/how-can-i-use-aws-ssm-send-command-in-a-way-that-it-returns-the-exit-code-of-t
- 2: https://docs.aws.amazon.com/systems-manager/latest/userguide/monitor-commands.html
- 3: https://docs.aws.amazon.com/systems-manager/latest/APIReference/API_GetCommandInvocation.html
- 4: https://docs.aws.amazon.com/boto3/latest/reference/services/ssm/client/get_command_invocation.html
- 5: https://awscli.amazonaws.com/v2/documentation/api/2.34.7/reference/ssm/wait/command-executed.html
- 6: https://docs.aws.amazon.com/boto3/latest/reference/services/ssm/waiter/CommandExecuted.html
- 7: https://docs.aws.amazon.com/systems-manager/latest/APIReference/API_SendCommand.html
Extend the SSM wait window or cap the prune step.
docker builder prune -af runs inside the SSM command, but both workflows only poll for about 5 minutes (20 × 15s). If cleanup runs longer, the GitHub job fails while the remote command may still finish successfully, which creates false deployment failures and encourages unsafe retries.
.github/workflows/cd-production.yml#L34.github/workflows/cd-staging.yml#L33
📍 Affects 2 files
.github/workflows/cd-production.yml#L34-L34(this comment).github/workflows/cd-staging.yml#L33-L33
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/cd-production.yml at line 34, Update the SSM deployment
commands in .github/workflows/cd-production.yml line 34 and
.github/workflows/cd-staging.yml line 33 to prevent docker builder prune -af
from exceeding the workflow’s polling window: either extend the associated SSM
wait/poll duration beyond five minutes or cap the prune step while preserving
successful deployment completion.
1 participant
Summary
The EC2 hosts running staging and production were repeatedly filling up their EBS volumes. The culprit was BuildKit build cache: every deploy runs docker compose build, which writes a fresh set of cache layers, and nothing ever reclaimed them. The existing docker image prune -f only removes dangling images — it does not touch build cache, so the cache grew unbounded until deploys started failing on "no space left on device".
This PR appends
docker builder prune -afto the end of the deploy command chain in both workflows:Checklist
Before submitting a pull request, please ensure that you mark these task.
fastapi run --reload app/main.pyordocker compose upin the repository root and test.