Skip to content

fix: prevent pipe deadlock when gh-ost row count estimate is inaccurate - #718

Open
XiRanZhang wants to merge 1 commit into
hanchuanchuan:masterfrom
XiRanZhang:fix/ghost-pipe-deadlock
Open

fix: prevent pipe deadlock when gh-ost row count estimate is inaccurate#718
XiRanZhang wants to merge 1 commit into
hanchuanchuan:masterfrom
XiRanZhang:fix/ghost-pipe-deadlock

Conversation

@XiRanZhang

Copy link
Copy Markdown

Summary

When InnoDB row count estimate is significantly lower than the actual row count, gh-ost copy progress exceeds 100% while row copy is still ongoing. The pipe reader goroutines in execCommand (session/osc.go) exit early upon detecting pct >= 100%, leaving no consumer for the stdout/stderr pipes. This causes a deadlock:

cmd.Wait() waits for gh-ost exit
  → gh-ost write(stdout) waits for pipe space
    → pipe buffer full (64KB), waits for parent to read
      → deadlock

Root cause

In execCommand, the reader goroutine calls wg.Done() + break when mysqlAnalyzeGhostOutput returns complete = true (i.e., pct >= 100). Since InnoDB's EXPLAIN-based row estimate can be significantly off (e.g., 12M estimated vs 70M actual), the percentage exceeds 100% long before row copy finishes. Both reader goroutines exit, wg.Wait() returns, and the code enters cmd.Wait() — but gh-ost is still running and writing to stdout. With no pipe consumer, the 64KB buffer fills up, gh-ost blocks on write(), and cmd.Wait() never returns.

Fix

  • Reader goroutines no longer exit on the complete signal; they continue draining pipes until EOF (process exit), preventing pipe backpressure deadlock
  • Use defer wg.Done() to ensure WaitGroup release on all exit paths
  • Fix strings.Contains argument order: strings.Contains("[info]", line)strings.Contains(line, "[info]") (was reversed, never matched)

Real-world incident

This bug caused a production gh-ost migration to deadlock for over 1 hour on a table with 70M rows (estimated 12.7M by InnoDB). Diagnosed via strace showing write(1, ...) blocked on a full pipe, with the parent process stuck in cmd.Wait().

When InnoDB row count estimate is significantly lower than the actual
row count, gh-ost copy progress exceeds 100% while row copy is still
ongoing. The pipe reader goroutines in execCommand exit early upon
detecting pct >= 100 (via mysqlAnalyzeGhostOutput returning complete),
leaving no consumer for the stdout/stderr pipes. gh-ost continues
writing status lines, eventually filling the 64KB pipe buffer and
blocking on write(). Meanwhile the parent blocks on cmd.Wait(),
creating a deadlock:

  cmd.Wait() waits for gh-ost exit
    → gh-ost write(stdout) waits for pipe space
      → pipe full, waits for parent to read
        → deadlock

Fix:
1. Reader goroutines no longer exit on complete signal; they drain
   pipes until EOF (process exit), preventing pipe backpressure.
2. Use defer wg.Done() to ensure WaitGroup release on all exit paths.
3. Fix strings.Contains argument order ("[info]", line → line, "[info]")
   which was reversed and never matched.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant