Skip to content

CON-14275: Remove doks-debug Deployment Exists toleration; add debug-node helper - #37

Merged
pyadagiri-do merged 2 commits into
masterfrom
pyadagiri/slurp
Aug 3, 2026
Merged

CON-14275: Remove doks-debug Deployment Exists toleration; add debug-node helper#37
pyadagiri-do merged 2 commits into
masterfrom
pyadagiri/slurp

Conversation

@pyadagiri-do

@pyadagiri-do pyadagiri-do commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Remove the default catch-all tolerations: [{operator: Exists}] from k8s/deployment.yaml so leftover Deployments cannot freely reschedule onto draining/cordoned nodes (see CON-14275 / ESC-24421).
  • Add script/debug-node for ephemeral single-node access (pin → exec/chroot → delete on exit), with optional --tolerate-all for tainted nodes only when needed. inspired from here
  • Document DaemonSet vs debug-node usage and the upgrade-blocking risk in the README.

Testing

KEnv => cc: 366336f8-88c9-4bbb-a4e5-d062c76a2dcd / kubectx: cc
> script/debug-node t-x-s-f-p-ops-pool-d33xfq5hh-37fr7x
deployment.apps/doks-debug created
root@t-x-s-f-p-ops-pool-d33xfq5hh-37fr7x:/# ls
bin  boot  dev	etc  home  lib	lib64  lost+found  media  mnt  opt  proc  root	run  sbin  srv	sys  tmp  usr  var  worker_observability_refresh_token
root@t-x-s-f-p-ops-pool-d33xfq5hh-37fr7x:/#


> k get pods | grep doks-debug
doks-debug-5bbd475876-qll87                 1/1     Running             0                  49s     10.133.253.220   t-x-s-f-p-ops-pool-d33xfq5hh-37fr7x               <none>           <none>

k describe deployment doks-debug | grep -E -i 'Name:|Replicas:|Node-Selectors:|Tolerations:'
Name:                   doks-debug
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
  Node-Selectors:  kubernetes.io/hostname=t-x-s-f-p-ops-pool-d33xfq5hh-37fr7x
  Tolerations:     <none>
  
  
> root@t-x-s-f-p-ops-pool-d33xfq5hh-37fr7x:/# exit
exit
deployment.apps "doks-debug" deleted from kube-system namespace

> k describe deployment doks-debug | grep -E -i 'Name:|Replicas:|Node-Selectors:|Tolerations:'
Error from server (NotFound): deployments.apps "doks-debug" not found

Long-lived doks-debug Deployments with operator: Exists can reschedule onto draining nodes and block upgrades; keep Exists only on the DaemonSet and document short-lived node access via script/slurp.
@pyadagiri-do pyadagiri-do changed the title Remove doks-debug Deployment Exists toleration; add slurp helper Remove doks-debug Deployment Exists toleration; add debug-node helper Jul 28, 2026
@pyadagiri-do pyadagiri-do changed the title Remove doks-debug Deployment Exists toleration; add debug-node helper CON-14275: Remove doks-debug Deployment Exists toleration; add debug-node helper Jul 28, 2026
@pyadagiri-do
pyadagiri-do merged commit c47d258 into master Aug 3, 2026
3 checks passed
@pyadagiri-do
pyadagiri-do deleted the pyadagiri/slurp branch August 3, 2026 07:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants