Runtime error: "host network interface down"

[marcwitasee]

I ran netrics debug run ping and received the following output:

Name: ping
Command: /Users/marcwitasee/test_netrics/venv/bin/netrics-ping

Status: Error (Exit code 71)

Result: -

Logged (standard error):

  <5> {"dest": "localhost", "status": "Error", "msg": "host network interface down"}

Running netrics.d --foreground also produces the error:

ℹ️  time="2023-04-05 12:57:39.960" level="info" event="3efEnxQRrqiA" session="3efEnxQRumoV" execution_count=0 scheduled_next=2023-04-05T12:58:00
☠️  time="2023-04-05 12:58:00.229" level="critical" event="3efEo7y1Xe3o" exc_line="base.py:27" exc_frame="verify" msg="fatal exception of type ProcessExecutionError"
netrics.d: fatal: ProcessExecutionError: Unexpected exit code: 71
Command line: | /Users/marcwitasee/test_netrics/venv/bin/netrics-ping
Stderr:       | <5> {"dest": "localhost", "status": "Error", "msg": "host network interface down"}

[marcwitasee]

Here is the traceback error:

ℹ️  time="2023-04-05 14:09:29.762" level="info" event="3efGlVv4iGOI" session="3efGlVv4lcU9" sched="tiered-tenancy" task="hops-scamper" msg="skipped: suppressed by if/unless condition"
ℹ️  time="2023-04-05 14:09:29.764" level="info" event="3efGlVvhSO3w" session="3efGlVv4lcU9" sched="tiered-tenancy" task="lml-scamper" msg="skipped: suppressed by if/unless condition"
Traceback (most recent call last):
  File "/Users/marcwitasee/test_netrics/venv/bin/netrics.d", line 8, in <module>
    sys.exit(daemon())
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/netrics/__main__.py", line 25, in entrypoint
    hook(
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/fate/cli/root.py", line 31, in entrypoint
    argcmdr.main(root, extend_parser=partial(extend_parser, **settings))
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/argcmdr.py", line 76, in main
    command.call()
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/argcmdr.py", line 377, in call
    return self.delegate('__call__', *additional)
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/argcmdr.py", line 397, in delegate
    return target_callable(*call_args[:param_count])
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/fate/cli/command/control.py", line 170, in __call__
    self.daemon(args.scheduler, background=background)
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/fate/cli/command/control.py", line 189, in daemon
    run_info = self.serve(scheduler)
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/fate/cli/command/control.py", line 204, in serve
    for completed_task in completed_tasks:
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/fate/util/iteration.py", line 52, in __next__
    return next(self.iterator)
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/fate/sched/base/scheduler.py", line 59, in __call__
    count = yield from self.exec_tasks(reset=reset)
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/fate/sched/tiered_tenancy.py", line 85, in exec_tasks
    count_ready = yield from pool.iter_ready(refill=queue.tenancy_tasks(pool.size))
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/fate/sched/base/pool.py", line 98, in iter_ready
    if task.ready():
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/fate/sched/base/scheduled_task.py", line 97, in poll
    if ready := self.__future__.poll():
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/plumbum/commands/modifiers.py", line 33, in poll
    self.wait()
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/plumbum/commands/modifiers.py", line 43, in wait
    self._returncode, self._stdout, self._stderr = run_proc(
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/plumbum/commands/processes.py", line 304, in run_proc
    return _check_process(proc, retcode, timeout, stdout, stderr)
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/plumbum/commands/processes.py", line 17, in _check_process
    proc.verify(retcode, timeout, stdout, stderr)
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/plumbum/machines/base.py", line 27, in verify
    raise ProcessExecutionError(
plumbum.commands.processes.ProcessExecutionError: Unexpected exit code: 71
Command line: | /Users/marcwitasee/test_netrics/venv/bin/netrics-dns-latency
Stderr:       | <5> {"dest": "localhost", "status": "Error", "msg": "host network interface down"}

[jesteria]

All right:

• The framework daemon shouldn't fail just because a measurement had an error; so, that was a regression. That should be handled now.
• The measurements should understand how to check that your device's networking is operational; and, they do that by ping localhost. However, it sounds like this doesn't work for your laptop.

So, (outside of Netrics), you can't just do ping localhost? What are the contents of your /etc/hosts? (Are you running an antivirus application or a special firewall?)

[marcwitasee]

• I can ping the localhost:

marcwitasee@Marcs-Workbook-Air ~ % ping localhost
PING localhost (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.135 ms
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.195 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.206 ms
64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.233 ms
64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=0.164 ms
64 bytes from 127.0.0.1: icmp_seq=5 ttl=64 time=0.204 ms
64 bytes from 127.0.0.1: icmp_seq=6 ttl=64 time=0.152 ms
64 bytes from 127.0.0.1: icmp_seq=7 ttl=64 time=0.224 ms
64 bytes from 127.0.0.1: icmp_seq=8 ttl=64 time=0.208 ms
64 bytes from 127.0.0.1: icmp_seq=9 ttl=64 time=0.207 ms

• Contents of /etc/hosts

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1	localhost
255.255.255.255	broadcasthost
::1             localhost
# Added by Docker Desktop
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal
# End of section

• I am running an antivirus software call Falcon

[marcwitasee]

Okay so I just uninstalled the antivirus software and tried again. Same error occurring.

Also, here is the output when I run the execute command:

(venv) marcwitasee@Marcs-Workbook-Air test_netrics % netrics --tb debug execute ping
Name: -
Command: /sbin/ping

Status: Error (Exit code 64)

Result: -

Logged (standard error):

  usage: ping [-AaDdfnoQqRrv] [-c count] [-G sweepmaxsize]
              [-g sweepminsize] [-h sweepincrsize] [-i wait]
              [-l preload] [-M mask | time] [-m ttl] [-p pattern]
              [-S src_addr] [-s packetsize] [-t timeout][-W waittime]
              [-z tos] host
         ping [-AaDdfLnoQqRrv] [-c count] [-I iface] [-i wait]
              [-l preload] [-M mask | time] [-m ttl] [-p pattern] [-S src_addr]
              [-s packetsize] [-T ttl] [-t timeout] [-W waittime]
              [-z tos] mcast-group
  Apple specific options (to be specified before mcast-group or host like all options)
              -b boundif           # bind the socket to the interface
              -k traffic_class     # set traffic class socket option
              -K net_service_type  # set traffic class socket options
              --apple-connect       # call connect(2) in the socket
              --apple-time          # display current time

[jesteria]

That looks like a different error.

Could you try again with the debug run command?

netrics debug run ping

[marcwitasee]

Yeah I was getting the same error with the debug run command. I also get a different error with the debug execute command.

(venv) marcwitasee@Marcs-Workbook-Air test_netrics % netrics debug run ping
Name: ping
Command: /Users/marcwitasee/test_netrics/venv/bin/netrics-ping

Status: Error (Exit code 71)

Result: -

Logged (standard error):

  <5> {"dest": "localhost", "status": "Error", "msg": "host network interface down"}

[jesteria]

Ok sure. (I only asked because the error you posted for the execute command was a different thing – it was telling you to specify a host to ping, such as localhost. But no matter.) Thanks for the info. This does indeed appear to be specific to the version of ping available on your system – that is, on MacOS – as opposed to on Linux. That appears to be why you can ping, but it fails in Netrics. In the latter case, we specify arguments which your ping presumably does not accept. To wit, I presume you can't: ping -c 1 -w 5 localhost But it looks like your system would be happy with the following argumentation: ping -c 1 -t 5 localhost This could and should be handled. But I will say it's likely not so high a priority as getting things working on our target platform. We could perhaps do what might be a relatively cheap check in Python against `sys.platform == 'darwin'` and then confirm that it's not the Linux version of ping with a system call `ping -V` (which I don't believe MacOS supports).

…

On Fri, Apr 7, 2023, 10:11 AM Marc Richardson ***@***.***> wrote: Yeah I was getting the same error with the debug run command. I also get a different error with the debug execute command. (venv) ***@***.*** test_netrics % netrics debug run ping Name: ping Command: /Users/marcwitasee/test_netrics/venv/bin/netrics-ping Status: Error (Exit code 71) Result: - Logged (standard error): <5> {"dest": "localhost", "status": "Error", "msg": "host network interface down"} — Reply to this email directly, view it on GitHub <#38 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEBUNRSYFQGU5B7FTLRUL3XAAOAVANCNFSM6AAAAAAWUOZAMQ> . You are receiving this because you commented.Message ID: ***@***.***>

[marcwitasee]

@jesteria yup I think you are right. I can run the ping command with the -t flag but not the -w flag. And you are also right that the MacOS ping does not support running ping -V.