Debian: Cannot connect to docker daemon

[vdo]

I just deployed 3 nodes on a cloud platform, using debian 10, and I followed the tutorial to connect all of them.

Everything worked fine, but when I try to run overnode ps or overnode up, it fails to connect to the docker daemons, with the output:

root@on2:~# overnode ps
[1] Couldn't connect to Docker daemon at http://10.39.240.1:2375 - is it running?
[1] 
[1] If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.
[3] Couldn't connect to Docker daemon at http://10.39.240.3:2375 - is it running?
[3] 
[3] If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.
[2] Couldn't connect to Docker daemon at http://10.39.240.2:2375 - is it running?
[2] 
[2] If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.

Maybe I missed some step, but I reviewed the docs and I cannot find any.

Some info:

root@on2:~# overnode status
[overnode] Targets status:
95.216.155.123
65.21.0.209
[overnode] Peers status:
00:00:00:00:00:03(on3)
   -> 65.21.0.209:6783      00:00:00:00:00:01(on1)                established
   -> 65.21.5.75:6783       00:00:00:00:00:02(on2)                established
00:00:00:00:00:02(on2)
   <- 65.21.0.209:41085     00:00:00:00:00:01(on1)                established
   <- 95.216.155.123:58683  00:00:00:00:00:03(on3)                established
00:00:00:00:00:01(on1)
   <- 95.216.155.123:35969  00:00:00:00:00:03(on3)                established
   -> 65.21.5.75:6783       00:00:00:00:00:02(on2)                established
[overnode] Connections status:
<- 65.21.0.209:41085     established encrypted   fastdp 00:00:00:00:00:01(on1) encrypted=truemtu=1376
<- 95.216.155.123:58683  established encrypted   fastdp 00:00:00:00:00:03(on3) encrypted=truemtu=1376
[overnode] DNS status:
overnode     10.39.240.1     82a3550003f7 00:00:00:00:00:01
overnode     10.39.240.2     a7e606b8f3e8 00:00:00:00:00:02
overnode     10.39.240.3     8f567ffbde09 00:00:00:00:00:03
[overnode] IPAM status:
00:00:00:00:00:02(on2)                  131072 IPs (25.0% of total) (0 active)
00:00:00:00:00:01(on1)                  131072 IPs (25.0% of total) 
00:00:00:00:00:03(on3)                  131072 IPs (25.0% of total) 
00:00:00:00:00:04()                     131072 IPs (25.0% of total) - unreachable!
[overnode] Endpoints status:
weave:expose 00:00:00:00:00:02
a7e606b8f3e8 f6:d3:15:44:60:ea 10.39.240.2/12

root@on2:~# docker logs weave
INFO: 2021/01/27 10:18:29.052223 Command line options: map[H:[unix:///var/run/weave/weave.sock] datapath:datapath dns-domain:weave.local. dns-listen-address:172.17.0.1:53 docker-bridge:docker0 host-root:/host http-addr:127.0.0.1:6784 ipalloc-default-subnet:10.32.0.0/12 ipalloc-init:seed=::1,::2,::3,::4 ipalloc-range:10.40.0.0/13 name:::2 nickname:on2 plugin:false port:6783 proxy:true resolv-conf:/var/run/weave/etc/resolv.conf rewrite-inspect:true status-addr:127.0.0.1:6782 weave-bridge:weave]
INFO: 2021/01/27 10:18:29.052285 weave  2.6.5
INFO: 2021/01/27 10:18:29.082665 Docker API on unix:///var/run/docker.sock: &[Platform={"Name":"Docker Engine - Community"} Components=[{"Details":{"ApiVersion":"1.40","Arch":"amd64","BuildTime":"2020-06-01T09:11:17.000000000+00:00","Experimental":"false","GitCommit":"42e35e61f3","GoVersion":"go1.13.10","KernelVersion":"4.19.0-13-amd64","MinAPIVersion":"1.12","Os":"linux"},"Name":"Engine","Version":"19.03.11"},{"Details":{"GitCommit":"269548fa27e0089a8b8278fc4fc781d7f65a939b"},"Name":"containerd","Version":"1.4.3"},{"Details":{"GitCommit":"ff819c7e9184c13b7c2607fe6c30ae19403a7aff"},"Name":"runc","Version":"1.0.0-rc92"},{"Details":{"GitCommit":"fec3683"},"Name":"docker-init","Version":"0.18.0"}] ApiVersion=1.40 MinAPIVersion=1.12 Arch=amd64 BuildTime=2020-06-01T09:11:17.000000000+00:00 Version=19.03.11 GitCommit=42e35e61f3 GoVersion=go1.13.10 Os=linux KernelVersion=4.19.0-13-amd64]
INFO: 2021/01/27 10:18:29.099127 proxy listening on unix:///var/run/weave/weave.sock
INFO: 2021/01/27 10:18:29.347561 Bridge type is bridged_fastdp
INFO: 2021/01/27 10:18:29.347585 Communication between peers via untrusted networks is encrypted.
INFO: 2021/01/27 10:18:29.408045 Our name is 00:00:00:00:00:02(on2)
INFO: 2021/01/27 10:18:29.408243 Launch detected - using supplied peer list: []
INFO: 2021/01/27 10:18:29.437516 Docker API on unix:///var/run/docker.sock: &[Platform={"Name":"Docker Engine - Community"} Version=19.03.11 ApiVersion=1.40 MinAPIVersion=1.12 GitCommit=42e35e61f3 GoVersion=go1.13.10 Os=linux BuildTime=2020-06-01T09:11:17.000000000+00:00 Components=[{"Details":{"ApiVersion":"1.40","Arch":"amd64","BuildTime":"2020-06-01T09:11:17.000000000+00:00","Experimental":"false","GitCommit":"42e35e61f3","GoVersion":"go1.13.10","KernelVersion":"4.19.0-13-amd64","MinAPIVersion":"1.12","Os":"linux"},"Name":"Engine","Version":"19.03.11"},{"Details":{"GitCommit":"269548fa27e0089a8b8278fc4fc781d7f65a939b"},"Name":"containerd","Version":"1.4.3"},{"Details":{"GitCommit":"ff819c7e9184c13b7c2607fe6c30ae19403a7aff"},"Name":"runc","Version":"1.0.0-rc92"},{"Details":{"GitCommit":"fec3683"},"Name":"docker-init","Version":"0.18.0"}] Arch=amd64 KernelVersion=4.19.0-13-amd64]
INFO: 2021/01/27 10:18:29.450313 Checking for pre-existing addresses on weave bridge
INFO: 2021/01/27 10:18:29.453841 [allocator 00:00:00:00:00:02] No valid persisted data
INFO: 2021/01/27 10:18:29.457170 [allocator 00:00:00:00:00:02] Initialising with supplied IPAM seed
INFO: 2021/01/27 10:18:29.460644 Listening for DNS queries on 172.17.0.1:53
INFO: 2021/01/27 10:18:29.464197 Sniffing traffic on datapath (via ODP)
INFO: 2021/01/27 10:18:29.466422 Listening for HTTP control messages on 127.0.0.1:6784
INFO: 2021/01/27 10:18:29.466631 Listening for status+metrics requests on 127.0.0.1:6782
INFO: 2021/01/27 10:18:36.799411 Creating container with WEAVE_CIDR "10.39.240.2/12"
INFO: 2021/01/27 10:18:37.309612 Attaching container a7e606b8f3e8ae30b74c4315644287066fe50e210d9ac026a537bd8c3d69b3b6 with WEAVE_CIDR "10.39.240.2/12" to weave network
INFO: 2021/01/27 10:18:37.310911 [allocator 00:00:00:00:00:02] Address 10.39.240.2/12 claimed by a7e606b8f3e8ae30b74c4315644287066fe50e210d9ac026a537bd8c3d69b3b6 - not in our range
INFO: 2021/01/27 10:18:37.976503 [nameserver 00:00:00:00:00:02] adding entry for a7e606b8f3e8ae30b74c4315644287066fe50e210d9ac026a537bd8c3d69b3b6: overnode.weave.local. -> 10.39.240.2
INFO: 2021/01/27 10:18:49.128250 ->[65.21.0.209:41085] connection accepted
INFO: 2021/01/27 10:18:49.130706 ->[65.21.0.209:41085|00:00:00:00:00:01(on1)]: connection ready; using protocol version 2
INFO: 2021/01/27 10:18:49.131023 overlay_switch ->[00:00:00:00:00:01(on1)] using fastdp
INFO: 2021/01/27 10:18:49.131122 ->[65.21.0.209:41085|00:00:00:00:00:01(on1)]: connection added (new peer)
INFO: 2021/01/27 10:18:49.131278 Setting up IPsec between 00:00:00:00:00:02(on2) and 00:00:00:00:00:01(on1)
INFO: 2021/01/27 10:18:49.131548 ipsec: InitSALocal: 65.21.0.209 -> 65.21.5.75 :6784 0xa19605c7
INFO: 2021/01/27 10:18:49.333495 fastdp ->[65.21.0.209:6784|00:00:00:00:00:01(on1)]: IPSec init SA remote
INFO: 2021/01/27 10:18:49.333861 ipsec: InitSARemote: 65.21.5.75 -> 65.21.0.209 :6784 0x80e2f802
INFO: 2021/01/27 10:18:49.335878 ->[65.21.0.209:41085|00:00:00:00:00:01(on1)]: connection fully established
INFO: 2021/01/27 10:18:49.699107 sleeve ->[65.21.0.209:6783|00:00:00:00:00:01(on1)]: Effective MTU verified at 1414
INFO: 2021/01/27 10:23:01.980043 Discovered remote MAC 36:27:2e:89:8e:10 at 00:00:00:00:00:01(on1)
INFO: 2021/01/27 10:25:24.836008 ->[95.216.155.123:58683] connection accepted
INFO: 2021/01/27 10:25:24.838420 ->[95.216.155.123:58683|00:00:00:00:00:03(on3)]: connection ready; using protocol version 2
INFO: 2021/01/27 10:25:24.838844 overlay_switch ->[00:00:00:00:00:03(on3)] using fastdp
INFO: 2021/01/27 10:25:24.838926 ->[95.216.155.123:58683|00:00:00:00:00:03(on3)]: connection added (new peer)
INFO: 2021/01/27 10:25:24.839214 Setting up IPsec between 00:00:00:00:00:02(on2) and 00:00:00:00:00:03(on3)
INFO: 2021/01/27 10:25:24.839813 ipsec: InitSALocal: 95.216.155.123 -> 65.21.5.75 :6784 0x5c9b66dd
INFO: 2021/01/27 10:25:24.940008 fastdp ->[95.216.155.123:6784|00:00:00:00:00:03(on3)]: IPSec init SA remote
INFO: 2021/01/27 10:25:24.940339 ipsec: InitSARemote: 65.21.5.75 -> 95.216.155.123 :6784 0x7dcf997d
INFO: 2021/01/27 10:25:24.941221 overlay_switch ->[00:00:00:00:00:03(on3)] using sleeve
INFO: 2021/01/27 10:25:24.941282 ->[95.216.155.123:58683|00:00:00:00:00:03(on3)]: connection fully established
INFO: 2021/01/27 10:25:24.942654 overlay_switch ->[00:00:00:00:00:03(on3)] using fastdp
INFO: 2021/01/27 10:25:24.944376 sleeve ->[95.216.155.123:6783|00:00:00:00:00:03(on3)]: Effective MTU verified at 1414
INFO: 2021/01/27 10:27:05.813397 Discovered remote MAC ce:87:a8:ef:84:15 at 00:00:00:00:00:03(on3)
INFO: 2021/01/27 10:42:07.681118 Discovered remote MAC 56:de:91:3e:64:ba at 00:00:00:00:00:01(on1)
INFO: 2021/01/27 10:45:30.100956 Discovered remote MAC de:21:59:4e:5f:2e at 00:00:00:00:00:01(on1)
INFO: 2021/01/27 10:47:12.119633 Creating container with WEAVE_CIDR ""
INFO: 2021/01/27 10:47:12.588950 Attaching container 4fd6f193bc0036003a06c5cde27e8435d28a1742b3d45a96a424726adc153e59 with WEAVE_CIDR "" to weave network
INFO: 2021/01/27 10:47:13.224678 [nameserver 00:00:00:00:00:02] adding entry for 4fd6f193bc0036003a06c5cde27e8435d28a1742b3d45a96a424726adc153e59: overnode-session-165e110586527e1e.weave.local. -> 10.42.0.0
INFO: 2021/01/27 10:47:13.572000 Discovered remote MAC 92:58:07:06:e0:ce at 00:00:00:00:00:03(on3)
INFO: 2021/01/27 10:47:13.662247 Discovered remote MAC f2:c0:4b:bc:41:a2 at 00:00:00:00:00:01(on1)
INFO: 2021/01/27 10:48:48.424997 [nameserver 00:00:00:00:00:02] container 4fd6f193bc0036003a06c5cde27e8435d28a1742b3d45a96a424726adc153e59 died; tombstoning entry overnode-session-165e110586527e1e.weave.local. -> 10.42.0.0
INFO: 2021/01/27 10:49:55.198669 Creating container with WEAVE_CIDR ""
INFO: 2021/01/27 10:49:55.672655 Attaching container 361d60f36da6b186342822451138abc0a4b4be265ddeae158812c851fccd879c with WEAVE_CIDR "" to weave network
INFO: 2021/01/27 10:49:56.272287 [nameserver 00:00:00:00:00:02] adding entry for 361d60f36da6b186342822451138abc0a4b4be265ddeae158812c851fccd879c: overnode-session-165e112b7ee1c426.weave.local. -> 10.42.0.0
INFO: 2021/01/27 10:49:57.919477 [nameserver 00:00:00:00:00:02] container 361d60f36da6b186342822451138abc0a4b4be265ddeae158812c851fccd879c died; tombstoning entry overnode-session-165e112b7ee1c426.weave.local. -> 10.42.0.0
INFO: 2021/01/27 10:50:26.797944 Creating container with WEAVE_CIDR ""
INFO: 2021/01/27 10:50:27.321665 Attaching container a907d245a95a457bd36a37bd898fa5df332d35ad7d8ef924da18284a403a65de with WEAVE_CIDR "" to weave network
INFO: 2021/01/27 10:50:27.940699 [nameserver 00:00:00:00:00:02] adding entry for a907d245a95a457bd36a37bd898fa5df332d35ad7d8ef924da18284a403a65de: overnode-session-165e1132d9ba8bc1.weave.local. -> 10.42.0.0
INFO: 2021/01/27 10:51:01.018630 [nameserver 00:00:00:00:00:02] container a907d245a95a457bd36a37bd898fa5df332d35ad7d8ef924da18284a403a65de died; tombstoning entry overnode-session-165e1132d9ba8bc1.weave.local. -> 10.42.0.0
INFO: 2021/01/27 10:51:25.912724 Discovered remote MAC 36:27:2e:89:8e:10 at 00:00:00:00:00:01(on1)
INFO: 2021/01/27 10:54:52.880932 Discovered remote MAC ce:87:a8:ef:84:15 at 00:00:00:00:00:03(on3)
INFO: 2021/01/27 10:55:06.640808 Discovered remote MAC 4e:bf:0e:f0:1a:bf at 00:00:00:00:00:01(on1)
INFO: 2021/01/27 11:04:09.614614 Discovered remote MAC 6e:0f:dd:62:34:bd at 00:00:00:00:00:01(on1)
INFO: 2021/01/27 11:05:51.479052 Creating container with WEAVE_CIDR ""
INFO: 2021/01/27 11:05:51.958744 Attaching container a59389920fbbaec33d5ed27e1078a71faa4c531a94774f2c065c8a65189ccf86 with WEAVE_CIDR "" to weave network
INFO: 2021/01/27 11:05:52.580546 [nameserver 00:00:00:00:00:02] adding entry for a59389920fbbaec33d5ed27e1078a71faa4c531a94774f2c065c8a65189ccf86: overnode-session-165e120a2458d6d3.weave.local. -> 10.42.0.0
INFO: 2021/01/27 11:05:53.798494 Discovered remote MAC 92:58:07:06:e0:ce at 00:00:00:00:00:03(on3)
INFO: 2021/01/27 11:05:53.798693 Discovered remote MAC f2:c0:4b:bc:41:a2 at 00:00:00:00:00:01(on1)
INFO: 2021/01/27 11:06:25.769866 [nameserver 00:00:00:00:00:02] container a59389920fbbaec33d5ed27e1078a71faa4c531a94774f2c065c8a65189ccf86 died; tombstoning entry overnode-session-165e120a2458d6d3.weave.local. -> 10.42.0.0
INFO: 2021/01/27 11:06:59.238935 Creating container with WEAVE_CIDR ""
INFO: 2021/01/27 11:06:59.705639 Attaching container 66e1e28c12ead861e56c5c3446fc7ace8a6e1d5fd0db3f75999ae927c609748c with WEAVE_CIDR "" to weave network
INFO: 2021/01/27 11:07:00.292623 [nameserver 00:00:00:00:00:02] adding entry for 66e1e28c12ead861e56c5c3446fc7ace8a6e1d5fd0db3f75999ae927c609748c: overnode-session-165e1219ec7306a9.weave.local. -> 10.42.0.0
INFO: 2021/01/27 11:07:33.533169 [nameserver 00:00:00:00:00:02] container 66e1e28c12ead861e56c5c3446fc7ace8a6e1d5fd0db3f75999ae927c609748c died; tombstoning entry overnode-session-165e1219ec7306a9.weave.local. -> 10.42.0.0

network addresses:

root@on2:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 96:00:00:91:b0:33 brd ff:ff:ff:ff:ff:ff
    inet 65.21.5.75/32 brd 65.21.5.75 scope global dynamic eth0
       valid_lft 82760sec preferred_lft 82760sec
    inet6 2a01:4f9:c010:8fd9::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::9400:ff:fe91:b033/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:e5:fe:36:79 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:e5ff:fefe:3679/64 scope link 
       valid_lft forever preferred_lft forever
4: datapath: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 62:95:4f:a0:98:da brd ff:ff:ff:ff:ff:ff
    inet6 fe80::6095:4fff:fea0:98da/64 scope link 
       valid_lft forever preferred_lft forever
6: weave: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue state UP group default qlen 1000
    link/ether 00:00:00:00:00:02 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::200:ff:fe00:2/64 scope link 
       valid_lft forever preferred_lft forever
8: vethwe-datapath@vethwe-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master datapath state UP group default 
    link/ether 26:7d:51:e1:98:bd brd ff:ff:ff:ff:ff:ff
    inet6 fe80::247d:51ff:fee1:98bd/64 scope link 
       valid_lft forever preferred_lft forever
9: vethwe-bridge@vethwe-datapath: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default 
    link/ether 8e:3c:ed:f2:eb:81 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::8c3c:edff:fef2:eb81/64 scope link 
       valid_lft forever preferred_lft forever
10: vxlan-6784: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65535 qdisc noqueue master datapath state UNKNOWN group default qlen 1000
    link/ether 56:75:9a:bd:45:14 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5475:9aff:febd:4514/64 scope link 
       valid_lft forever preferred_lft forever
14: veth1ac7899@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default 
    link/ether 4e:ce:40:31:ad:5b brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::4cce:40ff:fe31:ad5b/64 scope link 
       valid_lft forever preferred_lft forever
16: vethwepl5040@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default 
    link/ether 3e:c3:37:51:a5:88 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::3cc3:37ff:fe51:a588/64 scope link 
       valid_lft forever preferred_lft forever
22: vethwepl7709@if21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default 
    link/ether e2:12:3f:17:b6:85 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::e012:3fff:fe17:b685/64 scope link 
       valid_lft forever preferred_lft forever
26: vethwepl8669@if25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default 
    link/ether ea:73:5f:a2:34:fb brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::e873:5fff:fea2:34fb/64 scope link 
       valid_lft forever preferred_lft forever
30: vethwepl9597@if29: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default 
    link/ether 9a:13:56:14:cc:9d brd ff:ff:ff:ff:ff:ff link-netnsid 3
    inet6 fe80::9813:56ff:fe14:cc9d/64 scope link 
       valid_lft forever preferred_lft forever
34: vethwepl11351@if33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default 
    link/ether f6:72:06:88:26:dc brd ff:ff:ff:ff:ff:ff link-netnsid 4
    inet6 fe80::f472:6ff:fe88:26dc/64 scope link 
       valid_lft forever preferred_lft forever
38: vethwepl12293@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default 
    link/ether f6:97:ac:15:72:6b brd ff:ff:ff:ff:ff:ff link-netnsid 5
    inet6 fe80::f497:acff:fe15:726b/64 scope link 
       valid_lft forever preferred_lft forever

[vdo]

After redeploying the stack with Ubuntu 18.04, looks like is a problem specific to Debian 10

[avkonst]

Hi. Thanks for reporting. Looks like it is weave issue, but nothing suggests in the logs what might be wrong... Do you have docker ps status captured before running overnode up command?

[jtagcat]

I was planning to replicate the issue today, going to sleep atm.

I'll try tomorrow or the day after.

e: I can replicate; haven't debugged much so far

[vdo]

Hi. Thanks for reporting. Looks like it is weave issue, but nothing suggests in the logs what might be wrong... Do you have docker ps status captured before running overnode up command?

The output was basically the same, I could not connect to the Docker daemon