CheckConnectionState: option to consider node as not synced if el_offline is true

[nkryuchkov]

Closes #171

Depends on OffchainLabs/prysm#14226

The multi client doesn't switch clients if the current client's EL goes offline until the client sets is_syncing to true. It causes duty misses.

The PR adds a WithELConnectionCheck parameter to both http and multi, which makes the client consider a node as synced only if el_offline is false

It seems to work with Teku, Lighthouse, Nimbus correctly. It doesn't work with Prysm because of OffchainLabs/prysm#14226, its implementation of el_offline is incorrect

[nkryuchkov]

Linter and tests seem to fail due to the same reasons as in #191

[mcdee]

Which beacon node(s) exhibit this behavior? And is this an issue with the beacon node not supplying duties, or is it that the multi client fails to recognize that the beacon node is not supplying duties?

[nkryuchkov]

@mcdee Sorry for the delay. We at @ssvlabs tested this solution for a while and noticed cases when the Prysm node had been mistakenly considered unhealthy because SyncDistance grew a little. So we decided not to use this solution because the Prysm workaround is too hacky, and increasing the allowed SyncDistance gap doesn't look good either.

I removed the Prysm workaround, but unfortunately, OffchainLabs/prysm#14226 hasn't been fixed yet, so this PR wouldn't work for Prysm

Which beacon node(s) exhibit this behavior?

I tested it with Prysm, Lighthouse, Teku, Nimbus. All clients except Prysm set el_offline correctly, but Prysm didn't

And is this an issue with the beacon node not supplying duties, or is it that the multi client fails to recognize that the beacon node is not supplying duties?

The multi-client fails to recognize that the beacon node is not supplying duties (the beacon node sets el_offline to true but is still synced)