fix(sources): twitter/youtube connector follow-ups (session config, rate limits, docs) - #180
Merged
Merged
Conversation
…ate limits, docs) Follow-ups on the X/Twitter (#175) and YouTube (#177) connectors after running both live, plus a shared cleanup. Twitter: - Session config (cookies/proxy/credentials dir) flows through Django settings (TWITTER_COOKIES_JSON / COOKIE_AUTH_TOKEN / COOKIE_CT0 / COOKIES_FILE / CREDENTIALS_DIR / PROXY), resolved per search call so a rotated cookie export applies without a restart; the documented-but-unwired JSON/file routes now work. CREDENTIALS_DIR defaults to an absolute path. - Skip timestamp-less tweets instead of minting now() (which would poison the watermark and strand every older-but-new tweet). - Honor X's rate-limit reset with an in-cycle retry loop (mirrors Reddit's 429 loop; X's x-rate-limit-reset is an absolute epoch), ticking the poll-lease heartbeat through the wait. - Watermark filter uses strict < not <= (same-second tweets were lost). - Pin twikit to the locked rev. Shared: - Promote sleep_with_heartbeat and rate_limit_delay to connectors/base so Reddit and Twitter share one definition each. - Rename ListenerError -> TwitterError (mirrors YouTubeError; drops the 'Listener' name AGENTS.md reserves). Docs/examples: - Twitter starter (feed + watch); list twitter + youtube in the examples README. - Per-connector credentials/ directory convention (gitignored) with a guide for generating the Twitter and YouTube cookie exports. - README: product mentions in 'What it does', YouTube in the listeners + diagram, and the changelog-style section replaced by a credentials/changelog pointer. Both connectors verified live end to end (feed -> watch -> judge). Full core suite, ruff, and ty green; reviewed via /review-pr with findings addressed.
joshpanka
force-pushed
the
fix/twitter-connector-followups
branch
from
September 1, 2026 18:17
b065b50 to
c2d83b6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-ups on the X/Twitter and YouTube connectors after running both live, plus a shared cleanup. Split into reviewable commits.
Twitter
TWITTER_COOKIES_JSON/TWITTER_COOKIES_FILEwere documented but never read from the env; all session config now flows throughconf/settings(mirroringYOUTUBE_COOKIES_FILE) and resolves per search call, so a rotated cookie export applies on the next poll without a restart.TWITTER_CREDENTIALS_DIRnow defaults to an absolute path instead of a CWD-relative one (which resolved differently under runserver vs gunicorn--chdir).now().from_tweetstampeddatetime.now()on a tweet with nocreated_at, which advances the source watermark past every older-but-new tweet and strands them. Such a tweet is malformed; the connector now skips just that one with a warning.rate_limitederror the client waits until X's window resets and retries, ticking the poll-lease heartbeat through the wait, rather than failing the source and burning a cycle. X sendsx-rate-limit-resetas an absolute epoch (not Reddit's relative seconds), handled accordingly.uv.lockalready resolved (the fork's default branch floats and this code handles auth).Shared
sleep_with_heartbeattobaseso the Reddit and Twitter backoff loops share one definition.Docs
apps/core/credentials/<connector>/, gitignored) with a README covering how to generate the Twitter and YouTube cookie exports, and the ToS caveat.socialEventcontract, per-branch framing) with a durable per-connector reference in the project's voice.Both connectors verified live end to end (feed → watch → judge). Full core suite, ruff, and ty green.
Builds on the connector work from #175 (@matthewdonsemail-lab) and #177.