Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions modules/ROOT/content-nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,7 @@
** xref:clustering/servers.adoc[]
** xref:clustering/databases.adoc[]
** xref:clustering/unbind-system-database.adoc[]
** xref:clustering/troubleshooting.adoc[]
** Monitoring
*** xref:clustering/monitoring/show-servers-monitoring.adoc[]
*** xref:clustering/monitoring/show-databases-monitoring.adoc[]
Expand Down
1 change: 1 addition & 0 deletions modules/ROOT/pages/clustering/index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ This chapter describes the following:
* xref:clustering/servers.adoc[Managing servers in a cluster] -- How to manage the servers in a cluster.
* xref:clustering/databases.adoc[Managing databases in a cluster] -- How to manage the databases in a cluster.
* xref:clustering/unbind-system-database.adoc[Unbind the `system` database] label:new[Introduced in 5.0] -- How to use the `neo4j-admin dbms unbind-system-db` command.
* xref:clustering/troubleshooting.adoc[Troubleshooting] -- Diagnose and mitigate common cluster issues.

* **Monitoring** -- Monitoring of a cluster.
** xref:clustering/monitoring/show-servers-monitoring.adoc[Monitor servers] -- The tools available for monitoring the servers in a cluster.
Expand Down
56 changes: 56 additions & 0 deletions modules/ROOT/pages/clustering/troubleshooting.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
:description: Troubleshooting information for common Neo4j cluster availability and replication issues.
:page-role: enterprise-edition
[[clustering-troubleshooting]]
= Troubleshooting

This page describes known cluster issues and their mitigations.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see only one issue. Do you know if we expect to add more?


[[clustering-troubleshooting-raft-sender-channel-acquisition]]
== Slow or unavailable writes when a cluster member is shut down

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we have just one issue, maybe this could be the title instead of "Troubleshooting." Something like: "Troubleshooting slow or unavailable writes"


In Neo4j 5.26 LTS, a cluster can experience delayed or unavailable writes if the leader waits for a Raft sender connection to a cluster member that has stopped but whose network endpoint still accepts connection attempts.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that we are on Neo4j 5.26, I don't think we need to say it explicitly.

Suggested change
In Neo4j 5.26 LTS, a cluster can experience delayed or unavailable writes if the leader waits for a Raft sender connection to a cluster member that has stopped but whose network endpoint still accepts connection attempts.
A cluster can experience delayed or unavailable writes if the leader waits for a Raft sender connection to a cluster member that has stopped but whose network endpoint still accepts connection attempts.

This can happen when a server, virtual machine, or container is stopped or shut down, but the network layer does not immediately refuse connections to that member.

When this happens, the leader may wait until xref:configuration/configuration-settings.adoc#config_dbms.cluster.network.connect_timeout[`dbms.cluster.network.connect_timeout`] elapses before attempting to replicate data to another cluster member.
In a three-member cluster, this can delay writes because the Raft protocol requires a quorum (two of the three members) to replicate the transaction before it can be committed.

[[clustering-troubleshooting-raft-sender-channel-acquisition-symptoms]]
=== Symptoms

Common symptoms include:

* Writes become slow, time out, or appear unavailable after one cluster member is stopped, shut down, or restarted.
* The problem lasts for approximately the configured `dbms.cluster.network.connect_timeout`, which is `30s` by default.
* The problem is more likely in environments where a stopped member's network endpoint continues to accept connection attempts instead of refusing them immediately.
This has been observed in Kubernetes environments, but it is not limited to Kubernetes.
* The Neo4j debug log can contain Raft sender connection timeout messages similar to:
+
[source, text]
----
Raft sender failed exceptionally [Address: <host>:7000]
...
io.netty.channel.ConnectTimeoutException: connection timed out after 30000 ms
----

If the network endpoint refuses connections immediately, the log can instead contain `Connection refused`.
That case is less likely to cause the write latency described here because the leader can move on to another member without waiting for the connection timeout.

[[clustering-troubleshooting-raft-sender-channel-acquisition-mitigation]]
=== Mitigation in Neo4j 5.26 LTS

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

Suggested change
=== Mitigation in Neo4j 5.26 LTS
=== Mitigation


In Neo4j 5.26 LTS, enable asynchronous acquisition of Raft sender channels by setting the following internal setting to `true` in _neo4j.conf_ on all cluster members:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and here


[source, properties]
----
internal.dbms.cluster.raft.async_acquisition_of_raft_sender_channels=true
----

Restart each member for the setting to take effect.
This allows the leader to continue replicating to other members without blocking on a slow or hanging connection attempt to one member.

[NOTE]
====
*Behavior from Neo4j 2025.02* +

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this note doesn't belong on this page. If I am on 5.26, why would I care about 2026.02? But if I'm on 2026.01, then I need to know how to fix it.
Maybe, instead of having this note here, explain a bit in the current version if we haven't already done that, and add a note that in previous version, "a cluster can experience delayed or unavailable writes if the leader waits for a Raft sender connection to a cluster member that has stopped but whose network endpoint still accepts connection attempts." and link to that page.

Starting with Neo4j 2025.02, the setting is available as link:{neo4j-docs-base-uri}/operations-manual/current/configuration/configuration-settings/#config_dbms.cluster.raft.async_channel_acquisition_enabled[`dbms.cluster.raft.async_channel_acquisition_enabled`] and defaults to `true`.
No mitigation is required for this issue in Neo4j 2025.02 or later unless this setting has been explicitly changed to `false`.
====