Fix docker connectivity for E2E tests on mac

[jonbarksdale]

End to end tests are failing on mac with what appear to be docker related connection issues.

For a clean build, use this:

./gradlew clean build test -Pinteg -Pe2e

Or for incremental test runs, use this

./gradlew test -Pinteg -Pe2e

The docker compose up step is failing, and it appears as though one node fails to connect to the control plane.

You can verify this manually on a mac by going into the java/endToEnd directory, and doing a docker compose up node. One of the nodes will fail with an exception like this

endtoend-node-2  | java.lang.IllegalStateException: Unable to register
endtoend-node-2  | 	at org.svarm.node.manager.RealControlPlaneManager.lambda$getStatus$3(RealControlPlaneManager.java:123)
endtoend-node-2  | 	at java.base/java.util.Optional.orElseThrow(Optional.java:403)
endtoend-node-2  | 	at org.svarm.node.manager.RealControlPlaneManager.getStatus(RealControlPlaneManager.java:123)
endtoend-node-2  | 	at org.svarm.node.manager.RealControlPlaneManager.lambda$keyForNode$0(RealControlPlaneManager.java:91)
endtoend-node-2  | 	at io.micrometer.core.instrument.AbstractTimer.record(AbstractTimer.java:163)
endtoend-node-2  | 	at com.codeheadsystems.metrics.Metrics.time(Metrics.java:204)
endtoend-node-2  | 	at com.codeheadsystems.metrics.Metrics.time(Metrics.java:166)
endtoend-node-2  | 	at com.codeheadsystems.metrics.Metrics.time(Metrics.java:136)
endtoend-node-2  | 	at org.svarm.node.manager.RealControlPlaneManager.keyForNode(RealControlPlaneManager.java:90)
endtoend-node-2  | 	at org.svarm.node.engine.impl.HsqlDatabaseEngine.getInternalConnectionUrl(HsqlDatabaseEngine.java:141)
endtoend-node-2  | 	at org.svarm.node.engine.impl.HsqlDatabaseEngine.internalDataSource(HsqlDatabaseEngine.java:122)
endtoend-node-2  | 	at org.svarm.node.module.DataSourceModule.internalDataSource(DataSourceModule.java:60)
endtoend-node-2  | 	at org.svarm.node.module.DataSourceModule_InternalDataSourceFactory.internalDataSource(DataSourceModule_InternalDataSourceFactory.java:55)
endtoend-node-2  | 	at org.svarm.node.module.DataSourceModule_InternalDataSourceFactory.get(DataSourceModule_InternalDataSourceFactory.java:44)
endtoend-node-2  | 	at org.svarm.node.module.DataSourceModule_InternalDataSourceFactory.get(DataSourceModule_InternalDataSourceFactory.java:14)
endtoend-node-2  | 	at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
endtoend-node-2  | 	at org.svarm.node.module.DataSourceModule_InternalJdbiFactory.get(DataSourceModule_InternalJdbiFactory.java:43)
endtoend-node-2  | 	at org.svarm.node.module.DataSourceModule_InternalJdbiFactory.get(DataSourceModule_InternalJdbiFactory.java:14)
endtoend-node-2  | 	at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
endtoend-node-2  | 	at org.svarm.node.module.DataSourceModule_TenantTableDaoFactory.get(DataSourceModule_TenantTableDaoFactory.java:39)
endtoend-node-2  | 	at org.svarm.node.module.DataSourceModule_TenantTableDaoFactory.get(DataSourceModule_TenantTableDaoFactory.java:13)
endtoend-node-2  | 	at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
endtoend-node-2  | 	at org.svarm.node.manager.TenantTableManager_Factory.get(TenantTableManager_Factory.java:58)
endtoend-node-2  | 	at org.svarm.node.manager.TenantTableManager_Factory.get(TenantTableManager_Factory.java:17)
endtoend-node-2  | 	at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
endtoend-node-2  | 	at org.svarm.node.manager.ControlPlaneWatcherManager_Factory.get(ControlPlaneWatcherManager_Factory.java:52)
endtoend-node-2  | 	at org.svarm.node.manager.ControlPlaneWatcherManager_Factory.get(ControlPlaneWatcherManager_Factory.java:13)
endtoend-node-2  | 	at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
endtoend-node-2  | 	at org.svarm.node.component.DaggerNodeDropWizardComponent$NodeDropWizardComponentImpl.managedObjects(DaggerNodeDropWizardComponent.java:517)
endtoend-node-2  | 	at org.svarm.server.Server.run(Server.java:73)
endtoend-node-2  | 	at org.svarm.server.Server.run(Server.java:35)
endtoend-node-2  | 	at io.dropwizard.core.cli.EnvironmentCommand.run(EnvironmentCommand.java:66)
endtoend-node-2  | 	at io.dropwizard.core.cli.ConfiguredCommand.run(ConfiguredCommand.java:98)
endtoend-node-2  | 	at io.dropwizard.core.cli.Cli.run(Cli.java:78)
endtoend-node-2  | 	at io.dropwizard.core.Application.run(Application.java:94)
endtoend-node-2  | 	at org.svarm.node.Node.main(Node.java:49)

[jonbarksdale]

On another attempt, I got two nodes failing with this error:

endtoend-node-3  | feign.FeignException$InternalServerError: [500 Server Error] during [PUT] to [http://control:8080/v1/node/2b047c9a-b1f2-44dc-8ffe-d68a4a8de079/enable] [ControlNodeService#enable(String)]: [{"code":500,"message":"There was an error processing your request. It has been logged (ID 6da10dad503cc2d2)."}]
endtoend-node-3  | 	at feign.FeignException.serverErrorStatus(FeignException.java:259)
endtoend-node-3  | 	at feign.FeignException.errorStatus(FeignException.java:206)
endtoend-node-3  | 	at feign.FeignException.errorStatus(FeignException.java:194)
endtoend-node-3  | 	at feign.FeignException.errorStatus(FeignException.java:171)
endtoend-node-3  | 	at feign.codec.ErrorDecoder$Default.decode(ErrorDecoder.java:92)
endtoend-node-3  | 	at feign.ResponseHandler.decodeError(ResponseHandler.java:136)
endtoend-node-3  | 	at feign.ResponseHandler.handleResponse(ResponseHandler.java:70)
endtoend-node-3  | 	at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:116)
endtoend-node-3  | 	at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:72)
endtoend-node-3  | 	at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:98)
endtoend-node-3  | 	at feign.micrometer.MeteredInvocationHandleFactory.lambda$create$0(MeteredInvocationHandleFactory.java:70)

[jonbarksdale]

Initial impressions suggest a race of some kind, though weird that it only happens on a mac

[wolpert]

I've added retry via resilience4j which is helping the issue in general. There is still a race condition or failure in a node registering in a few cases.

Try this
gradle endToEnd:clean endToEnd:test -Pe2e
like 3 times in a row and see if they all fail. If the do and for different reasons, a bit of deep dive is of interest.

I'm working on getting registration working better.

[wolpert]

Honestly, I love that docker network is flaky. It's a bit of chaos we need to find stability issues.

[jonbarksdale]

will do, I should have a few minutes today to take a look

[wolpert]

FYI, I just found a weird case... after the node was registered and enabled, the request to get the key for the node's tenant table failed at the node lookup. As in, when NodeManager.read(uuid) called the dao, there was no node returned.

Maybe it's an issue with the db or datastore? Need to vet it more.

[jonbarksdale]

Still fails on a non deterministic node even with the resiliency changes

[wolpert]

I updated the retry to retry on server exceptions. (It was doing client, my bad)

The issue is still there, but better hidden. I need to find a way to test this easier than with the full end to end test.

[wolpert]

Here's the latest... all on control

1. I added in NoOpCache to the jdbi usage and I could still easily get tests to fail. I ended up adding in the Caffeine plugin instead, no change either. (There is default statement caching if you do nothing)
2. I forced the JDBC connection pool to a size of 1, and the endToEnd tests consistently pass. So something is messed up, but not sure if its a rogue jdbc connection or something on the psql docker install. (Does not fail when using HSQLDB, but that test was with no pooling I think). I doubt its psql, but cannot rule it out.

  initialSize: 1
  minSize: 1
  maxSize: 1

It's sooo weird. Never seen this before. I may try to rip out the dropwizard pool and you c3p0 and see what happens.