Feature Request: Add support for forward proxies in scripts

[adavault-ops]

Describe the bug
It is more secure to run BPNs with whitelisted outbound connectivity to specified URLs, and therefore it should be possible to use an outbound proxy (e.g. squid). This mostly works, but fails with non obvious errors for cncli sync and validate services without first creating ~/.curlrc with proxy and noproxy settings configured.

To Reproduce
Steps to reproduce the behavior:

1. Block outbound access on 80/443 from BPNs
2. Configure whitelisted access to URLs required via a proxy such as squid
3. On BPN export standard environment variables http_proxy, https_proxy, no_proxy
4. On BPN fail to set ~/.curlrc with proxy and no proxy values

Expected behavior
Error message from cncli sync or validate service to indicate unable to connect

Screenshots
If applicable, add screenshots to help explain your problem.

Version:

• OS: Ubuntu
• Product version: CNTools v13.4.1
• Cardano Node version: n/a
• Network you're connecting to: n/a

Additional context
Reporting this in case useful for others...may not be a bug as much a documentation issue.
Internet access via proxy is default behaviour on enterprise networks and this feature should be supported by default.

[rdlrt]

Is this about cncli or cntools?

[adavault-ops]

Cncli.sh uses curl (as does cntools.sh and a number of other scripts), so I assume it’s about both.

[rdlrt]

Thanks - in that case, I have reworded the title accordingly 👍🏼

[TrevorBenson]

Thanks - in that case, I have reworded the title accordingly 👍🏼

I suspect this is primarily about documentation as curl should be honoring properly exported http_proxy and https_proxy environment vars. At least the various proxy testing performed for Mithril I don't recall exposing any issues with cncli or cntools.

This does require no_proxy is well defined though, which should also be common for enterprises also using http_proxy or https_proxy to define. When it isn't then EKG_HOST, PROM_HOST, and any other loopback (localhost/127.0.0.1), or LAN subnets the proxy won't route "back" into, will incorrectly route through the proxy into limbo.

[adavault-ops]

Thanks very much for the response and agreed. Once .curlrc has the right proxy and noproxy settings (as you note above plus 0.0.0.0) and usual environment vars it all works fine. Guess the main feedback is that it took a little bit of extra messing about to realise that .curlrc was also needed along with 0.0.0.0. Otherwise it was standard. Can share the URLs that needed whitelisting, but pretty sure you already know those.

Edit: specifically most things worked without .curlrc being setup. Mithril is certainly ok, as are the version checks that the scripts run against GitHub. However the koios API calls to get stake and delegators details seem to fail (this info is not visible in gLiveview) as do the cncli-sync and cncli-validate services when run via systemd. Both of these functions do work when run directly via cncli e.g. ./cncli sync so perhaps the environment vars are not accessible when run by systemd and it needs to fall back to .curlrc?

[rdlrt]

Thanks - in that case, I have reworded the title accordingly 👍🏼

I suspect this is primarily about documentation as curl should be honoring properly exported http_proxy and https_proxy environment vars. At least the various proxy testing performed for Mithril I don't recall exposing any issues with cncli or cntools.

Documentation will certainly be a big part of it - though while it may start with that, there are likely gonna be unintended side-effects (beyond scripts that use curl for connectivity), for instance - it could have an overlay impact on gRest or one of the utility standalone scripts, which may still fail, as not everyone uses all tools and not all requests would go via curl. There are deployment scenarios with docker/kubernetes/systemd deployment or frankenstein wallets where reference to proxy may not be available - it's just some consideration to go through all scernarios and not end up with half tools supporting it 🙂

This does require no_proxy is well defined though, which should also be common for enterprises also using http_proxy or https_proxy to define. When it isn't then EKG_HOST, PROM_HOST, and any other loopback (localhost/127.0.0.1), or LAN subnets the proxy won't route "back" into, will incorrectly route through the proxy into limbo.

Yip - that would certainly a big highlight in the documentation and have to be user-managed