fix: prevent pipe deadlock when gh-ost row count estimate is inaccurate - #718
Open
XiRanZhang wants to merge 1 commit into
Open
fix: prevent pipe deadlock when gh-ost row count estimate is inaccurate#718XiRanZhang wants to merge 1 commit into
XiRanZhang wants to merge 1 commit into
Conversation
When InnoDB row count estimate is significantly lower than the actual
row count, gh-ost copy progress exceeds 100% while row copy is still
ongoing. The pipe reader goroutines in execCommand exit early upon
detecting pct >= 100 (via mysqlAnalyzeGhostOutput returning complete),
leaving no consumer for the stdout/stderr pipes. gh-ost continues
writing status lines, eventually filling the 64KB pipe buffer and
blocking on write(). Meanwhile the parent blocks on cmd.Wait(),
creating a deadlock:
cmd.Wait() waits for gh-ost exit
→ gh-ost write(stdout) waits for pipe space
→ pipe full, waits for parent to read
→ deadlock
Fix:
1. Reader goroutines no longer exit on complete signal; they drain
pipes until EOF (process exit), preventing pipe backpressure.
2. Use defer wg.Done() to ensure WaitGroup release on all exit paths.
3. Fix strings.Contains argument order ("[info]", line → line, "[info]")
which was reversed and never matched.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When InnoDB row count estimate is significantly lower than the actual row count, gh-ost copy progress exceeds 100% while row copy is still ongoing. The pipe reader goroutines in
execCommand(session/osc.go) exit early upon detectingpct >= 100%, leaving no consumer for the stdout/stderr pipes. This causes a deadlock:Root cause
In
execCommand, the reader goroutine callswg.Done()+breakwhenmysqlAnalyzeGhostOutputreturnscomplete = true(i.e.,pct >= 100). Since InnoDB'sEXPLAIN-based row estimate can be significantly off (e.g., 12M estimated vs 70M actual), the percentage exceeds 100% long before row copy finishes. Both reader goroutines exit,wg.Wait()returns, and the code enterscmd.Wait()— but gh-ost is still running and writing to stdout. With no pipe consumer, the 64KB buffer fills up, gh-ost blocks onwrite(), andcmd.Wait()never returns.Fix
completesignal; they continue draining pipes until EOF (process exit), preventing pipe backpressure deadlockdefer wg.Done()to ensure WaitGroup release on all exit pathsstrings.Containsargument order:strings.Contains("[info]", line)→strings.Contains(line, "[info]")(was reversed, never matched)Real-world incident
This bug caused a production gh-ost migration to deadlock for over 1 hour on a table with 70M rows (estimated 12.7M by InnoDB). Diagnosed via
straceshowingwrite(1, ...)blocked on a full pipe, with the parent process stuck incmd.Wait().