Skip to content

JBR-10347 Wayland: fix deadlock between WLClipboard dataLock and clipboard monitor - #615

Open
verglor wants to merge 1838 commits into
JetBrains:mainfrom
verglor:verglor/JBR-10347-wlclipboard-lock-inversion
Open

JBR-10347 Wayland: fix deadlock between WLClipboard dataLock and clipboard monitor#615
verglor wants to merge 1838 commits into
JetBrains:mainfrom
verglor:verglor/JBR-10347-wlclipboard-lock-inversion

Conversation

@verglor

@verglor verglor commented Jul 3, 2026

Copy link
Copy Markdown

Fixes JBR-10347 (also reported as IJPL-248197) — IDE-wide UI freezes/deadlocks on Wayland when a clipboard offer arrives while another thread reads the clipboard.

Root cause

An AB–BA lock inversion, regressed from the JBR-7896 fix by the JBR-9898 change:

  • EDT: WLDataDevice.handleSelectionWLClipboard.handleClipboardOffer takes dataLock, then SunClipboard.lostOwnershipNow() needs the clipboard monitor (SunClipboard.java:242)
  • Any thread: SunClipboard.getContents() / isDataFlavorAvailable() (synchronized, holds the monitor) → WLClipboard.getClipboardFormats() needs dataLock

The JBR-7896 fix ("Avoid performing blocking I/O while holding a lock") deliberately kept lostOwnershipNow() outside dataLock, and the JBR-8833 refactor preserved that. JBR-9898's ownership check reads dataLock-guarded state (ourDataSource), so the lostOwnershipNow() call moved inside the synchronized block, re-creating the inversion. JBR-9865 shortened the freeze window (non-blocking getMimes()) but the inversion remained.

Fix

Evaluate the ownership condition under dataLock, invoke lostOwnershipNow() after releasing it — restoring the JBR-7896 invariant of never entering clipboard-monitor code while holding dataLock.

Test

WLClipboardLockOrderTest provokes the exact interleaving deterministically: a reader thread holds the clipboard monitor (as SunClipboard.getContents() does), waits for the offer-handler thread to block, then calls getClipboardFormats(). Failure is detected via ThreadMXBean.findDeadlockedThreads() and reported with the JVM deadlock report.

Verified on JBR 25.0.3+9-b329.124 under a real GNOME Wayland session:

  • unpatched runtime → RuntimeException: Deadlock detected with stacks identical to the field thread dumps in JBR-10347
  • same runtime with the fixed class via --patch-moduleTest passed

Backport

The jbr25 tip carries the same inversion (with lostOwnershipNow(null) instead of lostOwnershipNow(), so not a clean cherry-pick). Since that line ships in 2026.1/2026.2 IDEs, I'm happy to submit the jbr25 backport PR as well once this one is reviewed.

@verglor
verglor force-pushed the verglor/JBR-10347-wlclipboard-lock-inversion branch from a76d673 to 2132857 Compare July 5, 2026 19:08
The WinAPI function `GetFileAttributesW` actually opens a file and immediately closes it. If the file is a named pipe, it triggers a connection to the listener of the pipe. The listener closes the connection also asynchronously.

If the pipe is created with `nMaxInstances=1`, this code does not work:

```java
var path = Path.of("""\\.\pipe\openssh-ssh-agent""")

// `readAttributes` calls `GetFileAttributesW`, it opens the file and then closes it.
if (Files.readAttributes(path, BasicFileAttributes.class).isRegularFile()) {
  // Very probable race condition: the listener of the pipe has already received
  // the request of opening the file triggered by `GetFileAttributesW`
  // but hasn't closed the connection on their side yet.
  Files.newByteChannel(path);  // Fails with ERROR_PIPE_BUSY.
}
```

This revision adds an explicit check if some file looks like a Windows pipe. In these cases `RandomAccessFile` and `FileInputStream` don't call `Files.readAttributes` for the file.
Support for gnome-specific dialog modality is disabled by default.
Use -Dsun.awt.wl.NativeModality=true to enable.
…ile in tmp dir only

(cherry picked from commit 4bd3f78)
(cherry picked from commit b7a3a34)
…va.util.PropertyPermission" "sun.tools.attach.tmp.only" "read")

(cherry picked from commit 3a09f6c)
(cherry picked from commit 3b2399f)
…me process

This patch makes the WLClipboard not call lostOwnershipNow() when a data
offer is available that was made previously by this clipboard. To do
this, WLClipboard's data sources announce a random cookie string as one
of the supported mime types. This string is then checked for when
receiving a data offer.
…ter press space or alt key.

Fixing the timestamps of each KeyEvent/MouseEvent/MouseWheelEvent created by WLToolkit.

(cherry picked from commit 4d1f4d9)
This patch removes holding a clipboard-global lock when receiving data
by reading from a pipe, since that has the potential to cause deadlocks
under specific circumstances.

To synchronize destroying the offer objects on the Java side, they are
now refcounted.
…ional scaling is OFF

The size is corrected only under -Dsun.awt.wl.correctLogicalSize=true.
The location is unchanged.
Relaxed the logging entry level from WARNING to INFO.

(cherry picked from commit 28c83fe)
… during Japanese typing in Editor

Make the caret within preedit text be sourced from cursor_begin of zwp_text_input_v3::preedit_string and not from cursor_end.

(cherry picked from commit 29ede7b)
…ly highlighted in search

* Completely reimplementing JavaPreeditString#fromWaylandPreeditString and JavaCommitString#fromWaylandCommitString to make sure they don't ignore UTF-8 encoding errors and that JavaPreeditString#fromWaylandPreeditString sensibly handles invalid cursor indices.
* Covering the new JavaPreeditString#fromWaylandPreeditString with tests.
* Working around https://gitlab.gnome.org/GNOME/mutter/-/issues/3547.

(cherry picked from commit e621663)
…t_manager_v3 if IM support is disabled

* Introducing a new WLToolkit method isNativeInputMethodSupportEnabled
* Deciding whether to bind to a zwp_text_input_manager_v3 based on the value returned by the new method
Removed AppContext usage
…cursive for JTree

* Use tree.getSelectionPaths() for JAVA_AX_SELECTED_CHILDREN
* For JAVA_AX_ALL_CHILDREN and JAVA_AX_VISIBLE_CHILDREN also use linear iteration to avoid expensive operations with arrays on each level
* Flip comparison order for accessible states to not call getAccessibleStateSet() when not needed because it can be expensive

(cherry picked from commit fb21f0f)
… unclickable (Wayland)

Passed origin relocation of the device to top-levels

upd: Fix popup
* Refactor WLRoundedCornersManager.RoundedCornerKind

Introduce subclasses to put additional parameters there
later for more sophisticated painting.

DEFAULT and NONE are left alone because DEFAULT
is used for instance comparison in DefaultFrameDecoration,
which we won't change here, and NONE is just a nice shortcut.

* Merge canPaintRoundedCorners and roundedCornersRequested

Rationale: they're always called in pair, and we'll need that
instanceof check in canPaintRoundedCorners now as well
to check if the border width is valid.

A side note: this also fixes roundedCornerKind without
synchronized() in canPaintRoundedCorners().

* Refactor rounded corner painting

Rename masks to paths, because we'll soon have more paths
to paint the border.

Move resetCornerPaths() to keep createCornerPaths()
overloads together.

Move the graphics.setComposite() call closer to fill()
because we'll need different composites for different parts.

* Implement macOS-like rounded corner params for Wayland

To make it easier for client code, use exactly the same format
for the parameters as for macOS.
@verglor

verglor commented Jul 6, 2026

Copy link
Copy Markdown
Author

Hi Vitaly Provodin (@vprovodin), is force pushing main branch regular workflow here ? It makes PRs based on it problematic - or should I use different branch as base for PRs ?

@tsarn

Copy link
Copy Markdown
Member

Hi,

Thank you for the pull request and your hard work. It saddens me to say it, but due to various legal reasons we cannot accept pull requests from external contributors: https://github.com/JetBrains/JetBrainsRuntime#contributing.

Consider contributing this to OpenJDK project Wakefield https://openjdk.org/projects/wakefield/ instead. Please also note that you cannot have used AI at all to help you develop this code. Otherwise it will be rejected by OpenJDK https://openjdk.org/legal/ai.

…board monitor

WLClipboard.handleClipboardOffer() called SunClipboard.lostOwnershipNow()
while holding dataLock. lostOwnershipNow() synchronizes on the clipboard's
monitor, which SunClipboard's public API (getContents(),
isDataFlavorAvailable(), ...) holds while calling back into
getClipboardFormats(), which acquires dataLock. The two opposite lock
acquisition orders deadlock when a clipboard offer arrives while another
thread is reading the clipboard.

This reinstates the invariant established by the JBR-7896 fix (never enter
clipboard-monitor code while holding dataLock), which was regressed by
JBR-9898: the new ownership check reads state guarded by dataLock, so the
lostOwnershipNow() call was moved inside the synchronized block. Evaluate
the condition under dataLock, but invoke lostOwnershipNow() after releasing
it.

Add a regression test that deterministically provokes the inverted
interleaving via reflection and fails with the JVM's deadlock detector
report when the inversion is present.
@verglor
verglor force-pushed the verglor/JBR-10347-wlclipboard-lock-inversion branch from 2132857 to e87d38a Compare July 6, 2026 14:48
@verglor

verglor commented Jul 6, 2026

Copy link
Copy Markdown
Author

Hi Nikita Tsarev (@tsarn), thank you for the information. So could you please fix it ? It seems to be trivial change and prevents serious deadlock.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.