[0.2.15] Intermittent "operation not permitted" in Python sandbox — default seccomp whitelist missing modern syscalls; `ALLOWED_SYSCALLS` REPLACES (not appends) the default

[6mvp6]

Description

This adds root-cause analysis and a concrete fix to #99, which tracks the same symptom ("Occasionally 'Operation not permitted'") but has had no maintainer response since it was opened on 2024-10. The symptom is deterministically reproducible, and the cause is entirely within dify-sandbox.

Happy to post this as a comment on #99 instead if the maintainers prefer to keep everything in one place.

Environment

• Used via Dify v1.14.2 (Docker Compose), which bundles langgenius/dify-sandbox:0.2.15.
• Linux x86_64, modern kernel/glibc; the sandbox image ships CPython 3.14.
• SANDBOX_ENABLE_NETWORK=true, SANDBOX_WORKER_TIMEOUT=15.
• ALLOWED_SYSCALLS is not set (stock deployment — so the default whitelist is in effect).

Symptom

~0.1% of Python Code-node executions fail non-deterministically (more under concurrency):

{"code":0,"message":"success","data":{"error":"error: operation not permitted\n","stdout":""}}

Sandbox container log at the same instant:

{"level":"ERROR","msg":"process finished with error","status":"signal: bad system call (core dumped)"}

The child process is killed by SIGSYS — i.e. seccomp blocked a syscall that is not in the whitelist. The same code succeeds 99.9% of the time; the blocked syscall is only hit on certain glibc code paths / timing.

Reproduction (deterministic)

Repro 1 — default whitelist gap (the actual production bug)

docker run --rm -d --name sbx -p 8194:8194 langgenius/dify-sandbox:0.2.15

curl -s -X POST http://localhost:8194/v1/sandbox/run \
  -H "X-Api-Key: ${SANDBOX_API_KEY:-dify-sandbox}" \
  -H "Content-Type: application/json" \
  -d '{"language":"python3","code":"import os; print(len(os.sched_getaffinity(0)))"}'
# => {"code":0,"message":"success","data":{"error":"error: operation not permitted\n","stdout":""}}

A baseline print(42) succeeds on the same container, proving the failure is specific to the blocked syscall, not the container.

Repro 2 — the ALLOWED_SYSCALLS workaround is a footgun

docker run --rm -d --name sbx2 -e ALLOWED_SYSCALLS=204 langgenius/dify-sandbox:0.2.15

curl -s -X POST http://localhost:8194/v1/sandbox/run \
  -H "X-Api-Key: ${SANDBOX_API_KEY:-dify-sandbox}" \
  -H "Content-Type: application/json" \
  -d '{"language":"python3","code":"print(42)"}'
# => {"code":0,"message":"success","data":{"error":"error: operation not permitted\n","stdout":""}}

Setting ALLOWED_SYSCALLS replaces the default whitelist, so even print(42) fails (write(1) / exit(60) are no longer allowed). In a real deployment this turns a ~0.1% intermittent failure into a 100% sandbox outage.

Probe method (how the blocked set was enumerated)

import ctypes
ctypes.CDLL(None).syscall(N)   # blocked -> process SIGSYS -> empty stdout + "operation not permitted"

Syscalls blocked on the 0.2.15 default whitelist (amd64) — each is routinely emitted by modern glibc / CPython:

nr	name	nr	name
17	pread64	204	sched_getaffinity
19	readv	213	epoll_create
20	writev	218	set_tid_address
28	madvise	237	mbind
63	uname	238	set_mempolicy
79	getcwd	239	get_mempolicy
99	sysinfo	281	epoll_pwait
187	readahead	293	pipe2
203	sched_setaffinity	302	prlimit64
332	statx	435	clone3

Root cause

Two independent issues in dify-sandbox:

1. Incomplete default whitelist. ALLOW_SYSCALLS in internal/static/python_syscall/syscalls_amd64.go omits syscalls that modern glibc (2.34+) and CPython 3.14 use (statx, clone3, sched_getaffinity, prlimit64, pipe2, madvise, uname, …).

2. REPLACE semantics. internal/core/lib/python/add_seccomp.go:

   allowed_syscall := os.Getenv("ALLOWED_SYSCALLS")
   if allowed_syscall != "" {
       // allowed_syscalls = ONLY the user-supplied numbers
       // (default ALLOW_SYSCALLS + ALLOW_NETWORK_SYSCALLS are DROPPED)
   } else {
       allowed_syscalls = append(allowed_syscalls, python_syscall.ALLOW_SYSCALLS...)
       if enable_network {
           allowed_syscalls = append(allowed_syscalls, python_syscall.ALLOW_NETWORK_SYSCALLS...)
       }
   }

   So the natural workaround (ALLOWED_SYSCALLS=...) replaces the entire whitelist. The container still reports "healthy" because the healthcheck hits /health in the parent process, not the seccomp'd child — a broken whitelist is therefore a silent outage.

Suggested fix

1. Expand the default whitelist. Add the modern syscalls to ALLOW_SYSCALLS in internal/static/python_syscall/syscalls_amd64.go (and the arm64 equivalent). Low-risk subset: statx(332), sched_getaffinity(204), madvise(28), pipe2(293), uname(63), get_mempolicy(239), set_tid_address(218), getcwd(79), pread64(17), readv(19), writev(20), readahead(187), sysinfo(99), epoll_create(213), epoll_pwait(281). clone3(435) / prlimit64(302) / sched_setaffinity(203) / mbind(237) / set_mempolicy(238) need a DoS / resource-control tradeoff assessment before adding. This is the same class of issue Docker/moby already handled when glibc adopted clone3/statx (seccomp filter breaks latest glibc (in fedora rawhide) by blocking clone3 with EPERM moby/moby#42680).

2. Change REPLACE → APPEND. In internal/core/lib/python/add_seccomp.go, always load the default lists and append the user list on top, so the whitelist is safely operator-extensible without a custom image rebuild:

   allowed_syscalls = append(allowed_syscalls, python_syscall.ALLOW_SYSCALLS...)
   if enable_network {
       allowed_syscalls = append(allowed_syscalls, python_syscall.ALLOW_NETWORK_SYSCALLS...)
   }
   if allowed_syscall != "" {
       // append user-supplied numbers instead of replacing
       ...
   }

   This matters because FAQ.md §2's prescribed remedy (edit source + rebuild) is impractical for Docker Compose deployers.

References

• Symptom tracker (same intermittent failure, no maintainer response yet): Occasionally "Operation not permitted" #99
• Precedent for expanding a default seccomp profile for modern glibc: seccomp filter breaks latest glibc (in fedora rawhide) by blocking clone3 with EPERM moby/moby#42680
• Whitelist source: internal/static/python_syscall/syscalls_amd64.go
• Seccomp installer / REPLACE logic: internal/core/lib/python/add_seccomp.go

[Harsh23Kashyap]

I'd like to work on this. I'll open a PR shortly.

[Harsh23Kashyap]

Opened PR #275 for this.

[6mvp6]

Opened PR #275 for this.

Thanks for this — verified by building this branch (fix/274-allowed-syscalls-append, amd64, via Dify v1.14.2) and re-running my repros against a running sandbox:

• Repro 1 (default whitelist gap): os.sched_getaffinity returns "operation not permitted" (SIGSYS) on 0.2.15; on this branch it returns the CPU count. ✅
• Repro 2 (REPLACE footgun): ALLOWED_SYSCALLS=204 + print(42) bricks the sandbox on 0.2.15 (the default whitelist is dropped); on this branch print(42) succeeds — confirming
  the APPEND semantics via MergeSyscalls. ✅
• Security sanity: a still-disallowed syscall (e.g. fork) is still rejected with "operation not permitted", so the seccomp boundary is intact. ✅

Both issues from #274 are resolved and the boundary holds. The clone3 → ALLOW_ERROR_SYSCALLS choice and the append logic are exactly right. Will adopt as soon as this ships in a
release. Thanks!