Subreddit listings intermittently fail with Reddit HTTP 500; add OAuth/HTML fallback

[aaajiao]

Summary

rdt sub <subreddit> and rdt feed --subs-only currently rely on Reddit public JSON listing endpoints such as:

• https://www.reddit.com/r/investing.json
• https://www.reddit.com/r/wallstreetbets.json

On 2026-05-18 these endpoints started returning repeated HTTP 500 for many subreddits, while auth and other endpoints still worked.

This causes rdt feed --subs-only to skip many subscribed subreddits and downstream systems to mark Reddit as missing.

Environment

• rdt-cli: 0.4.2
• Installed from GitHub main
• Auth: valid cookie session
• Account: authenticated successfully via rdt status and rdt whoami

What works

rdt status
# authenticated: true

rdt whoami
# returns current user data

curl https://www.reddit.com/r/all/hot.json?limit=1
# 200

curl https://www.reddit.com/r/popular/hot.json?limit=1
# 200

curl https://www.reddit.com/r/investing/
# 200 HTML page

What fails

rdt sub investing -n 1 --compact
rdt sub wallstreetbets -n 1 --compact
rdt sub economics -n 1 --compact
rdt sub CryptoCurrency -n 1 --compact

All repeatedly return:

ok: false
error:
  code: api_error
  message: Request failed after 3 retries

stderr shows repeated HTTP 500 retries.

Direct requests confirm the issue:

curl -A "aaajiaoisBot/0.1 by u/botisaaajiao" \
  "https://www.reddit.com/r/investing.json?limit=1&raw_json=1"
# {"message": "Internal Server Error", "error": 500}

The rate-limit headers still decrement, so the request reaches Reddit. This does not look like an expired cookie or local network failure.

Expected behavior

When public subreddit JSON listing endpoints return 500, rdt-cli should degrade more gracefully.

Possible fallbacks:

1. Try OAuth listing endpoint:
   • https://oauth.reddit.com/r/<subreddit>/hot
2. Try alternate listing sorts:
   • /new, /top?t=day
3. As last resort, parse the HTML/shreddit page when it returns 200.
4. Return partial feed data with a structured warning instead of making the whole source look unavailable.

Why this matters

For automation use, rdt feed --subs-only is used as a sentiment/narrative source. Partial subreddit 500s currently make downstream systems treat Reddit as a missing source, even when auth is valid and some Reddit endpoints still work.