Skip to content

Pipe: avoid blocking startup on hardlink dir cleanup#17932

Open
Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:codex-pipe-startup-periodical-cleanup
Open

Pipe: avoid blocking startup on hardlink dir cleanup#17932
Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:codex-pipe-startup-periodical-cleanup

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

This PR avoids blocking DataNode startup while cleaning stale pipe hardlink/copied-file directories.

Previously, startup synchronously deleted the pipe hardlink base dir. On deployments with a huge number of pipe object files, this can keep the main startup thread in recursive filesystem deletion for a long time.

The new flow:

  • rename the existing pipe hardlink dir aside as pipe.startup-cleaning-* during startup
  • register stale dirs to the existing PipePeriodicalJobExecutor
  • delete stale dirs in throttled rounds instead of adding a new thread or blocking the startup thread
  • resume cleanup of interrupted pipe.startup-cleaning-* dirs on the next startup

Verification

  • mvn -Ddevelocity.off=true -DskipTests spotless:apply -pl iotdb-core/datanode
  • git diff --cached --check

spotless:check could not be completed in the local Windows environment because the JVM failed to start due to insufficient native/page-file memory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant