NDT fails when reverse DNS timeout is too long

[pboothe]

When the reverse DNS for the host IP fails, all tests on a machine crash.

When the reverse DNS for a client fails, tests are likely to fail too.

All instances of reverse DNS in NDT should be removed, or at least made optional with a command-line flag.

[pboothe]

Reverse DNS is called via getnameinfo(), which is called in the I2Util library, so this bug was tricky to find.

[pboothe]

_I2AddrSetNodePort is the offending function. It is called by:

• I2AddrBySAddr
• I2AddrSetSAddr
• I2AddrNodeName
• I2AddrServName
• I2AddrNodeServName

Every use of each of those functions needs to be removed and replaced with one that does not end up calling getnameinfo()

[nkinkade]

This practical effects of this issue will hopefully be mitigated (not solved) by PR #46 in the nodebase-private repository. It should be in production sometime next week.

[pboothe]

This mitigation helps for the server problems. Clients with too-slow RDNS still might have issues. This makes me want to replace getnameinfo() with a function that just returns a string representation of the IP address, and doesn't touch reverse DNS, and to do it by using a shim library so that it doesn't matter what code is trying to call that function, it still never touches reverse DNS.

[mattmathis]

Another possible solution would be to add:
hints.ai_flags |= AI_NUMERICHOST|AI_NUMERICSERV
in front of 2 calls to getaddrinfo in I2util::addr.c.
I quote "The AI_NUMERICHOST flag suppresses any potentially lengthy network host address lookups."

[gfr10598]

Also found a reference to NI_NUMERICHOST in
https://github.com/ndt-project/ndt/blob/93976a3aeea92e91d1290dedd5fdf521ebc7e43c/README_WEB10G

[pboothe]

editing i2util code is the precise rabbithole I would not like to go down. :)

[mattmathis]

Right, if we were going to be supporting this for others it is a tar baby. If we don't send it upstream and only keep it as a 6 month bridge to nuNDT, it has minimal total cost. Yes there is risk that one plan may become the other, but if you say there are no other I2util users, it shouldn't matter. Rename the package I2utilNoRev if you want to manage future options.

[stephen-soltesz]

A proof of concept using LD_PRELOAD to unconditionally set the AI_NUMERICHOST and AI_NUMERICSERV flags for getnameinfo - https://github.com/stephen-soltesz/getnameinfo

Tested on mlab2.lga1t with IPv4. There are no test servers available atm with IPv6 for testing.