Error: Rpc(Transport(Error in the WebSocket handshake: i/o error: unexpected end of file

[bernardoaraujor]

while testing on kubernetes mode, I am facing the following errors:

$ ./zombienet-linux -p kubernetes test tests/stps/relay.feature
...
    ✔ alice: is up within 600 secs (73ms)
[2022-05-25T18:53:22Z INFO  utils] Checking sTPS pre-conditions (account nonces and free balances).
rust process exited with code 0
    ✔ alice: js-script ./utils.js with "check_pre_conditions" return is 0 within 600 secs (533ms)
[2022-05-25T18:53:22Z INFO  utils] Node 0: Reading funded accounts from: "tests/stps/funded-accounts.json"
[2022-05-25T18:53:22Z INFO  utils] Node 2: Reading funded accounts from: "tests/stps/funded-accounts.json"
[2022-05-25T18:53:23Z INFO  utils] Node 1: Reading funded accounts from: "tests/stps/funded-accounts.json"
[2022-05-25T18:53:23Z INFO  utils] Node 3: Reading funded accounts from: "tests/stps/funded-accounts.json"
Error: Rpc(Transport(Error in the WebSocket handshake: i/o error: unexpected end of file

Caused by:
    0: i/o error: unexpected end of file
    1: unexpected end of file))
[2022-05-25T18:53:23Z INFO  utils::sender] Node 0: signing 4096 transactions
rust process exited with code 1
child process exited
[2022-05-25T18:53:23Z INFO  utils::sender] Node 2: signing 4096 transactions
[2022-05-25T18:53:23Z INFO  utils::sender] Node 1: signing 4096 transactions
[2022-05-25T18:54:14Z INFO  utils::sender] Node 0: sending 4096 transactions in chunks of 50
[2022-05-25T18:54:15Z INFO  utils::sender] Node 1: sending 4096 transactions in chunks of 50
[2022-05-25T18:54:15Z INFO  utils::sender] Node 2: sending 4096 transactions in chunks of 50
[2022-05-25T18:54:15Z INFO  utils::sender] Node 0: 350 txs sent in 1172 ms (298.38 /s)
[2022-05-25T18:54:16Z INFO  utils::sender] Node 1: 300 txs sent in 1060 ms (283.01 /s)
[2022-05-25T18:54:16Z INFO  utils::sender] Node 2: 200 txs sent in 1409 ms (141.94 /s)
[2022-05-25T18:54:17Z INFO  utils::sender] Node 1: 600 txs sent in 1035 ms (579.69 /s)
[2022-05-25T18:54:17Z INFO  utils::sender] Node 2: 400 txs sent in 1069 ms (374.12 /s)

A few patterns that caught my attention:

• this does not happen on native mode.
• the error message happens right after utils starts reading funded accounts from: tests/stps/funded-accounts.json, although there's no RPC call in this process... it's just loading a json file locally.
• the error message happens before any utils::sender starts signing the transactions to be dispatched, which is way before any actual RPC call happens.
• there's one utils::sender per node. As I increase the number of nodes in the setup, it's always the last utils::sender that dies (e.g.: 5 nodes, no txs are dispatched for node 4)

@niklasad1 does this look like anything related to jsonrpsee to you?
all the points listed above tell me it's not, but maybe your experienced eyes can rule this out from a different perspecitve.

cc @ggwpez @pepoviola

[ggwpez]

Reproduced it with 0.9.22 in local KIND cluster.
Needs investigation.

[niklasad1]

Yes, it'a a jsonrpsee error for sure, the WebSocket handshake failed but I can't tell why without more information.
How many connections are you guys opening to the server?

You can replace env_logger with tracing-subscriber to get the logs from jsonrpsee on the client side.

Would be good get some logs from polkadot node, it looks weird with EOF error in the handshake FWIW

[ggwpez]

You can replace env_logger with tracing-subscriber to get the logs from jsonrpsee on the client side.

Do we have to modify the source for that? It uses a docker image and such...
Is there no -l rpc=trace or equivalent? @niklasad1

[bernardoaraujor]

You can replace env_logger with tracing-subscriber to get the logs from jsonrpsee on the client side.

Do we have to modify the source for that? It uses a docker image and such...
Is there no -l rpc=trace or equivalent? @niklasad1

I think niklas meant using tracing-subscriber on our utils code

there's no docker involved in that

[niklasad1]

I think niklas meant using tracing-subscriber on our utils code

Yes, it's one line in the utils code

On the polkadot node side you could do:

./polkadot -l "rpc=trace,jsonrpsee=debug"

However, I think Wireshark is the way to go as this setup seems quite complicated and probably performs huge number of calls.

[bernardoaraujor]

I haven't done anything related to tracing-subscriber on the sender side yet, but I added -l "rpc=trace,jsonrpsee=debug" to polkadot args list and started a kubernetes setup with 3 nodes.
Only node 0 received transactions, which means 1 and 2 had the error.

Here's the grafana logs for all 3 nodes:

• 0: https://grafana.parity-mgmt.parity.io/explore?orgId=1&left=%5B%22now-3h%22,%22now%22,%22loki.parity-zombienet%22,%7B%22expr%22:%22%7Bpod%3D~%5C%22zombie-2c76a920f7b73c9a916a2a96b8e748df%2Fstps-0%5C%22%7D%22,%22refId%22:%22A%22,%22range%22:true%7D%5D
• 1: https://grafana.parity-mgmt.parity.io/explore?orgId=1&left=%5B%22now-3h%22,%22now%22,%22loki.parity-zombienet%22,%7B%22expr%22:%22%7Bpod%3D~%5C%22zombie-2c76a920f7b73c9a916a2a96b8e748df%2Fstps-1%5C%22%7D%22,%22refId%22:%22A%22,%22range%22:true%7D%5D
• 2: https://grafana.parity-mgmt.parity.io/explore?orgId=1&left=%5B%22now-3h%22,%22now%22,%22loki.parity-zombienet%22,%7B%22expr%22:%22%7Bpod%3D~%5C%22zombie-2c76a920f7b73c9a916a2a96b8e748df%2Fstps-2%5C%22%7D%22,%22refId%22:%22A%22,%22range%22:true%7D%5D

They're quite dense though, I'm not sure what to look for.

[niklasad1]

@bernardoaraujor

Those nodes are not even running jsonrpsee, it's running v0.9.21 i.e, the old jsornpc library.
Can you please update them and then I can have a look at the logs?

I guess if you guys could provide some instructions how to run this locally then I could have a look

[bernardoaraujor]

@bernardoaraujor

Those nodes are not even running jsonrpsee, it's running v0.9.21 i.e, the old jsornpc library. Can you please update them and then I can have a look at the logs?

I guess if you guys could provide some instructions how to run this locally then I could have a look

damn, sorry about that!

here's another set.
this time, only node 2 failed.

• 0: https://grafana.parity-mgmt.parity.io/explore?orgId=1&left=%5B%22now-3h%22,%22now%22,%22loki.parity-zombienet%22,%7B%22expr%22:%22%7Bpod%3D~%5C%22zombie-1ed63c5b9f8f6101b8505d93186f9cf1%2Fstps-0%5C%22%7D%22,%22refId%22:%22A%22,%22range%22:true%7D%5D
• 1: https://grafana.parity-mgmt.parity.io/explore?orgId=1&left=%5B%22now-3h%22,%22now%22,%22loki.parity-zombienet%22,%7B%22expr%22:%22%7Bpod%3D~%5C%22zombie-1ed63c5b9f8f6101b8505d93186f9cf1%2Fstps-1%5C%22%7D%22,%22refId%22:%22A%22,%22range%22:true%7D%5D
• 2: https://grafana.parity-mgmt.parity.io/explore?orgId=1&left=%5B%22now-3h%22,%22now%22,%22loki.parity-zombienet%22,%7B%22expr%22:%22%7Bpod%3D~%5C%22zombie-1ed63c5b9f8f6101b8505d93186f9cf1%2Fstps-2%5C%22%7D%22,%22refId%22:%22A%22,%22range%22:true%7D%5D

there's two ways to reproduce: with zombienet's native mode or kubernetes mode.

doing ./zombienet.sh should bootstrap the project.
in case you want to try kubernetes mode, first you need to ask @pepoviola to give you permissions on the zombienet cluster and run gcloud container clusters get-credentials parity-zombienet --zone europe-west3-b --project parity-zombienet

then you do ./zombienet-linux -p kubernetes test tests/stps/relay.feature

[bernardoaraujor]

another relevant point is that if this happens both with 0.9.21 and 0.9.22 maybe it's not really jsonrpsee?

[niklasad1]

another relevant point is that if this happens both with 0.9.21 and 0.9.22 maybe it's not really jsonrpsee?

Yeah it at least makes me calmer :)

The logs are pretty much useless on the node side so you can turn them off, it could be some nginx/proxy setting or something that rejects the connection too. It never reaches the jsonrpc code servers as I can see both in v0.9.21 or v0.9.22.

I pretty much know nothing about the zombienet or kubernetes :)

On the subxt side you guys could try to open the websocket connection like this in a loop with some sleeps to try it a couple of times before killing the node but again if you find some logs from subxt/jsonrpsee on the client side that would be helpful

[ggwpez]

The RPC trace log is blank for the node that does not get the connection. Therefore it could also be the kubernetes/docker stuff which does not correctly forward the port.
To me it looks like the port of the container is not open.
both logs:
bad.log
good.txt

PS: I tried again with a python script to connect to the IPs printed in console.
For stps-0 and 1 this works, but for stps-2 not. So this is a problem with zombienet probably.

[pepoviola]

PS: I tried again with a python script to connect to the IPs printed in console. For stps-0 and 1 this works, but for stps-2 not. So this is a problem with zombienet probably.

Hi @ggwpez, I will double check this. You can't connect to the rpc endpoint?

[bernardoaraujor]

on the rust code, the error is originating on the await? from this line, which is the instantiation of the subxt ClientBuilder for utils::sender module:

	let api = ClientBuilder::new()
		.set_url(node.clone())
		.build()
		.await?
		.to_runtime_api::<runtime::RuntimeApi<DefaultConfig, PolkadotExtrinsicParams<DefaultConfig>>>(
		);

[niklasad1]

Yes, that's when connecting the RPC endpoint under the hood in subxt.

so if you could try something like this instead to try connect a couple of times if the RPC endpoint is busy or something:

let mut i = 0;

let client = loop {
       let err = match WsClientBuilder::default() {
         .connection_timeout(std::time::Duration::from_secs(60)).
         .build(&node)
         .await {
           Ok(c) => break c,
           Err(e) => e
        };
        
       i += 1;
       
      // what you think is a reasonable limit
      if i > 10 {
        return Err(err.into()); 
      }
      tokio::time::sleep(std::time::Duration::from_secs(1)).await;
}

let api = ClientBuilder::new()
		.set_client(client)
		.build()
		.await?
		.to_runtime_api::<runtime::RuntimeApi<DefaultConfig, PolkadotExtrinsicParams<DefaultConfig>>>(
		);

But as it doesn't work for Oliver in python either there is mostly something flaky with the RPC endpoint configuration.

[ggwpez]

so if you could try something like this instead to try connect a couple of times if the RPC endpoint is busy or something:

Yea I added a retry loop. First time it gets the EOF error and then just "connection refused". Nothing in the node log.