Fix downgrades_https? to detect any HTTPS→HTTP redirect

[Copilot]

The downgrades_https? method only checked the canonical endpoint for HTTPS downgrades. This failed to detect cases where an HTTPS endpoint redirects to HTTP but the canonical endpoint is already HTTP (e.g., https://nytimes.com → http://www.nytimes.com).

Changes

• lib/site-inspector/domain.rb: Changed logic from checking only canonical endpoint to checking all HTTPS endpoints

  # Before: only checks canonical
  canonical_endpoint.redirect? && canonical_endpoint.redirect.http?
  
  # After: checks any HTTPS endpoint
  endpoints.select(&:https?).any? { |e| e.redirect&.http? }

• spec/site_inspector/domain_spec.rb: Added test coverage for:

  • Canonical HTTPS endpoint downgrading
  • Non-canonical HTTPS endpoint downgrading (nytimes.com scenario)
  • No false positives when HTTPS unsupported or redirects to HTTPS

Example

# Domain with HTTPS endpoints that redirect to HTTP canonical endpoint
# Before: downgrades_https? => false (canonical is HTTP, doesn't redirect)
# After:  downgrades_https? => true (HTTPS endpoints redirect to HTTP)

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• 3.116.82.140.in-addr.arpa
  • Triggering command: /usr/bin/ruby3.2 ruby3.2 /home/REDACTED/work/site-inspector/site-inspector/vendor/bundle/ruby/3.2.0/bin/rspec --format documentation ec _64-linux-gnu/ru-I -plugin-opt=/usr/usr/include/ruby-3.2.0/ruby/backward vendor/bundle/ru-I -plugin-opt=-pasas vendor/bundle/ru-I -plugin-opt=-pas. x86_64-linux-gnu-I -o by-3.2.0 -I/usr/include/x/usr/include/ruby-3.2.0/ruby/backward vendor/bundle/ru-I -I/usr/include/rbash vendor/bundle/ru--norc -Wdate-time y-3.2.0 (dns block)
• example.com
  • Triggering command: /usr/bin/ruby3.2 ruby3.2 /home/REDACTED/work/site-inspector/site-inspector/vendor/bundle/ruby/3.2.0/bin/rspec --format documentation ec _64-linux-gnu/ru-I -plugin-opt=/usr/usr/include/ruby-3.2.0/ruby/backward vendor/bundle/ru-I -plugin-opt=-pasas vendor/bundle/ru-I -plugin-opt=-pas. x86_64-linux-gnu-I -o by-3.2.0 -I/usr/include/x/usr/include/ruby-3.2.0/ruby/backward vendor/bundle/ru-I -I/usr/include/rbash vendor/bundle/ru--norc -Wdate-time y-3.2.0 (dns block)
• whois.verisign-grs.com
  • Triggering command: /usr/bin/ruby3.2 ruby3.2 /home/REDACTED/work/site-inspector/site-inspector/vendor/bundle/ruby/3.2.0/bin/rspec --format documentation ec _64-linux-gnu/ru-I -plugin-opt=/usr/usr/include/ruby-3.2.0/ruby/backward vendor/bundle/ru-I -plugin-opt=-pasas vendor/bundle/ru-I -plugin-opt=-pas. x86_64-linux-gnu-I -o by-3.2.0 -I/usr/include/x/usr/include/ruby-3.2.0/ruby/backward vendor/bundle/ru-I -I/usr/include/rbash vendor/bundle/ru--norc -Wdate-time y-3.2.0 (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

---

This section details on the original issue you should resolve

<issue_title>downgrades_https logic doesn't make sense</issue_title>
<issue_description>As described in #83 (comment):

A site is only downgraded from HTTPS to HTTP when HTTPS is supported, but the canonical endpoint downgrades to HTTP.

This is not a useful definition of "downgrades HTTPS". Any site that supports HTTPS but redirects to HTTP should be considered to "downgrade HTTPS".

For example, take https://nytimes.com:

~> curl -v https://nytimes.com
* Rebuilt URL to: https://nytimes.com/
*   Trying 170.149.159.130...
* Connected to nytimes.com (170.149.159.130) port 443 (#0)
<snip>
> GET / HTTP/1.1
> Host: nytimes.com
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Server: Varnish
< Location: http://www.nytimes.com/
<snip>

Here, the "canonical" endpoint is http://www.nytimes.com. Requests to https://{www.,}nytimes.com always redirect to http://www.nytimes.com. Clearly the site downgrades HTTPS. However:

$ irb
irb(main):001:0> require 'site-inspector'
=> true
irb(main):002:0> site = SiteInspector.inspect "nytimes.com"
=> #<SiteInspector::Domain host="nytimes.com">
irb(main):003:0> site.https?
=> true
irb(main):004:0> site.downgrades_https?
=> false

</issue_description>

Comments on the Issue (you are @copilot in this section)

@konklone @garrettr Are you using 1.0.2 (which is what pulse.cio.gov uses), or 3.X (which is what the comment you link to refers to)? It looks like you're using 3.X, which I've not personally thoroughly tested or deployed.

1.0.2 uses the same documented rule, that it only considers the canonical endpoint: https://github.com/benbalter/site-inspector/blob/erics-mode/lib/site-inspector.rb#L346-L358

But as you demonstrated, nytimes.com's canonical endpoint redirects to HTTP, so it should be marked as downgrading HTTPS either way. And in 1.0.2, it does appear to do so:

$ site-inspector nytimes.com --http

Yields, among the JSON results:

"downgrade_https": true,

This seems to be a regression in 3.X.

I do think the definition of "downgrades HTTPS", when applied to a "domain", should only consider the canonical endpoint. If the NYT's canonical endpoint was https://www.nytimes.com and it didn't redirect down to HTTP, but for some reason https://nytimes.com redirected down to http://nytimes.com (perhaps as some intermediate redirect to the www version), then I would want to say that the nytimes.com domain "enforces HTTPS", even though it's possible to access it in a non-canonical way over HTTP.

@benbalter To be clear, the 3.x branch is the latest release, and is the only one currently supported. I'd be glad to get a fix together for the 3.x branch if there is a regression or should be a change in behavior.

@garrettr, @konklone what is the expected/preferred behavior here? Logically, if an endpoint redirects from HTTPS to HTTP, it by definition, wouldn't be the canonical endpoint. Perhaps the definition should be if any endpoint redirects from HTTPS to HTTP? Alternatively, we could only look if the HTTPS equivalent of the canonical domain redirects to HTTP.

What if it redirects from HTTPS to HTTPS, but with an HTTP intermediary? E.g.:

1. https://nytimes.com redirects to http://www.nytimes.com
2. http://www.nytime.com redirects to https://www.nytimes.com

Edit Wrong @garrettr

@benbalter Eek. Thanks @garethr. Sorry for the noise. @konklone For reference, DHS has begun a Python-based tool that tackles only this issue, of analyzing HTTPS behavior on domains to come up with the same conclusions that are currently on pulse.cio.gov:

https://github.com/dhs-ncats/pshtt

Our team at 18F is contributing, and hoping to end up using the same toolchain as DHS. The code isn't ready for use in Pulse yet, but it's seeing active work and should hopefully get there soon.

@konklone Some starting work on integrating to `domain-scan` is here: https://github.com/18F/domain-scan/pull/76

But as I say in it, there's still some more work to go before they're at feature and logical parity.

@benbalter > the same conclusions that are currently on pulse.cio.gov

@konklone could I trouble you to elaborate on those, in reference to the above question (regarding when HTTPS is downgraded)?

@benbalter > if a site has an HTTPS endpoint, but redirects it to HTTP, then it should be considered to "downgrade HTTPS".

👍

@konklone Our methodology i...

• Fixes downgrades_https logic doesn't make sense #84

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.