-
Notifications
You must be signed in to change notification settings - Fork 93
Async raft sender - document in 5.26 #3175
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: 5.x
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,56 @@ | ||||||
| :description: Troubleshooting information for common Neo4j cluster availability and replication issues. | ||||||
| :page-role: enterprise-edition | ||||||
| [[clustering-troubleshooting]] | ||||||
| = Troubleshooting | ||||||
|
|
||||||
| This page describes known cluster issues and their mitigations. | ||||||
|
|
||||||
| [[clustering-troubleshooting-raft-sender-channel-acquisition]] | ||||||
| == Slow or unavailable writes when a cluster member is shut down | ||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If we have just one issue, maybe this could be the title instead of "Troubleshooting." Something like: "Troubleshooting slow or unavailable writes" |
||||||
|
|
||||||
| In Neo4j 5.26 LTS, a cluster can experience delayed or unavailable writes if the leader waits for a Raft sender connection to a cluster member that has stopped but whose network endpoint still accepts connection attempts. | ||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Given that we are on Neo4j 5.26, I don't think we need to say it explicitly.
Suggested change
|
||||||
| This can happen when a server, virtual machine, or container is stopped or shut down, but the network layer does not immediately refuse connections to that member. | ||||||
|
|
||||||
| When this happens, the leader may wait until xref:configuration/configuration-settings.adoc#config_dbms.cluster.network.connect_timeout[`dbms.cluster.network.connect_timeout`] elapses before attempting to replicate data to another cluster member. | ||||||
| In a three-member cluster, this can delay writes because the Raft protocol requires a quorum (two of the three members) to replicate the transaction before it can be committed. | ||||||
|
|
||||||
| [[clustering-troubleshooting-raft-sender-channel-acquisition-symptoms]] | ||||||
| === Symptoms | ||||||
|
|
||||||
| Common symptoms include: | ||||||
|
|
||||||
| * Writes become slow, time out, or appear unavailable after one cluster member is stopped, shut down, or restarted. | ||||||
| * The problem lasts for approximately the configured `dbms.cluster.network.connect_timeout`, which is `30s` by default. | ||||||
| * The problem is more likely in environments where a stopped member's network endpoint continues to accept connection attempts instead of refusing them immediately. | ||||||
| This has been observed in Kubernetes environments, but it is not limited to Kubernetes. | ||||||
| * The Neo4j debug log can contain Raft sender connection timeout messages similar to: | ||||||
| + | ||||||
| [source, text] | ||||||
| ---- | ||||||
| Raft sender failed exceptionally [Address: <host>:7000] | ||||||
| ... | ||||||
| io.netty.channel.ConnectTimeoutException: connection timed out after 30000 ms | ||||||
| ---- | ||||||
|
|
||||||
| If the network endpoint refuses connections immediately, the log can instead contain `Connection refused`. | ||||||
| That case is less likely to cause the write latency described here because the leader can move on to another member without waiting for the connection timeout. | ||||||
|
|
||||||
| [[clustering-troubleshooting-raft-sender-channel-acquisition-mitigation]] | ||||||
| === Mitigation in Neo4j 5.26 LTS | ||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same here.
Suggested change
|
||||||
|
|
||||||
| In Neo4j 5.26 LTS, enable asynchronous acquisition of Raft sender channels by setting the following internal setting to `true` in _neo4j.conf_ on all cluster members: | ||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. and here |
||||||
|
|
||||||
| [source, properties] | ||||||
| ---- | ||||||
| internal.dbms.cluster.raft.async_acquisition_of_raft_sender_channels=true | ||||||
| ---- | ||||||
|
|
||||||
| Restart each member for the setting to take effect. | ||||||
| This allows the leader to continue replicating to other members without blocking on a slow or hanging connection attempt to one member. | ||||||
|
|
||||||
| [NOTE] | ||||||
| ==== | ||||||
| *Behavior from Neo4j 2025.02* + | ||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this note doesn't belong on this page. If I am on 5.26, why would I care about 2026.02? But if I'm on 2026.01, then I need to know how to fix it. |
||||||
| Starting with Neo4j 2025.02, the setting is available as link:{neo4j-docs-base-uri}/operations-manual/current/configuration/configuration-settings/#config_dbms.cluster.raft.async_channel_acquisition_enabled[`dbms.cluster.raft.async_channel_acquisition_enabled`] and defaults to `true`. | ||||||
| No mitigation is required for this issue in Neo4j 2025.02 or later unless this setting has been explicitly changed to `false`. | ||||||
| ==== | ||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see only one issue. Do you know if we expect to add more?