feat(sources): X/Twitter social listening (twikit connector) + README update - #175
Open
matthewdonsemail-lab wants to merge 6 commits into
Open
feat(sources): X/Twitter social listening (twikit connector) + README update#175matthewdonsemail-lab wants to merge 6 commits into
matthewdonsemail-lab wants to merge 6 commits into
Conversation
…a parity Ports the listeningkit Twitter-search listener into openmagpie-core as a built-in source kind, with twikit added as an openmagpie-core library dependency (git pin, unclecode fork) per operator direction - not vendored from listeningkit. - schema: TwitterSearchSourceSpec (query, mode Top/Latest, count) added to the _BuiltinSourceSpec union; NewTweetPayload added to FeedItemData; schema.json regenerated (82 models); --check passes. - connector: TwikitClient with the live-cookie credential chain (TWITTER_COOKIES_JSON -> TWITTER_COOKIE_AUTH_TOKEN/CT0 -> cookies file -> login fallback) so the working listeningkit cookie JSONs keep functioning; error taxonomy translated to ConnectorParseError; tests use fakes only (no live I/O). - registration: twitter_search registered in sources/registry.py + connectors/__init__.py; payload self-registers; source-kind invariant test extended to the known five. - infra: Dockerfile builder gains git for the git-pinned dep; docker-compose.override.yml (local-only, gitignored) publishes Postgres on host 5433 to coexist with the buzz stack. - validation: 9 new connector tests + invariant tests green; CLI suite 145/145; ruff + ty clean. Co-authored-by: Matthew Don <matthewdonsemail@gmail.com> Signed-off-by: Matthew Don <matthewdonsemail@gmail.com>
…pping Live X polling through the connector exposed two issues (verified against live cookies from listeningkit/backend/credentials/personal.json, no proxy, no vendor API): 1. Event loop is closed on 2nd/3rd source in a multi-source feed poll. TwikitClient built one twikit Client at import time; twikit's Client creates an httpx.AsyncClient bound to the *currently running* event loop, and TwikitClient.search() drives it with a fresh asyncio.run loop per call. The shared client died with the first loop, so later sources in the same poll failed. Fix: construct the twikit Client inside _search_async (per call, inside the running loop); cookies are still resolved once in __init__. 2. X SearchTimeline intermittently 404s with an EMPTY body (reproduced: same query succeeds on immediate retry, session-independent). That is a transient upstream flake, not a deleted tweet/user; the old not_found/non-retryable mapping made ops treat a healthy source as dead. Fix: empty-body NotFound maps to search_timeline_unavailable (retryable=True, backoff); message-bearing NotFound stays not_found. - client.py: per-call twikit Client construction + docstring. - errors.py: empty-404 special case in map_twikit_error. - tests_twitter.py: regression test for the retryable empty-404 mapping. Validation: ruff check + format (416 files), ty, makemigrations --check, schema.json --check, full Django suite 587 tests OK, live poll of the x-buying-signals feed (4 twitter_search sources) green end-to-end. Signed-off-by: matthewdonsemail-lab <matthewdonsemail@gmail.com> Co-authored-by: Matthew Don <matthewdonsemail@gmail.com>
…ms, and a what-we've-done outline Brings the README in line with the twitter_search connector work shipped on this branch: - Diagram + 'Where it listens' now show X/Twitter as a shipped source, and Facebook, TikTok, and Instagram as 'soon to be added'. - 'What's shipped today' lists the twitter_search source kind. - New 'What we've done' section outlines the twikit connector, live-cookie auth, the live-poll reliability fixes, and the verified end-to-end feed -> watch -> webhook delivery (44/44 HTTP 200, payload matched against the Twenty socialEvent intake contract). - Roadmap's connector list updated to match (X shipped; FB/TikTok/IG next). Co-authored-by: Matthew Don <matthewdonsemail@gmail.com> Signed-off-by: Matthew Don <matthewdonsemail@gmail.com>
Mermaid treats unquoted edge labels with spaces as separate tokens; quote them to keep the diagram parseable on GitHub's renderer. Co-authored-by: Matthew Don <matthewdonsemail@gmail.com> Signed-off-by: Matthew Don <matthewdonsemail@gmail.com>
Author
|
@joshpanka alright there mate thought I'd throw together a PR as I was looking for a social listening tool myself thought about doing a whole redesign for one myself but found this and thought I'd contribute I like the concept of this here and I know other libraries to then intergrate I am specifically looking to only merge unofficial api clients into the product and if they're broken me and my team can maintain them aslong as this tools keep active maintainership! Really neat stuff here great work. If there's issues flag them for me and I'll look through them I got status 200 from web search without proxies I think that would also be a good addition into the project aswell but I'll continue to work off of this PR here. |
Adds optional pattern_id to the webhook body's per-item source object (WebhookSource), sourced from FeedItem.source_meta (operator-supplied Source.meta tags copied onto each item at record time). Receivers can now attribute yield by listening pattern - e.g. the Twenty socialEvent intake's patternId field - without deriving it by convention. - schema: WebhookSource.pattern_id (str | None, default None) - the legacy contract (label+kind only) still validates; untagged sources emit null - ActionItem gains source_meta (defaulted) so the action layer can read the tag - run_inputs populates source_meta from the FeedItem column - webhook _build_payload maps source_meta.pattern_id -> source.pattern_id - tests: tagged -> pattern_id present; untagged -> null; legacy payload valid Co-authored-by: Matthew Don <matthewdonsemail@gmail.com> Signed-off-by: Matthew Don <matthewdonsemail@gmail.com>
Author
|
Status update — 5 commits, contract closed end-to-end, CI awaiting the first-time-contributor gate. This PR now carries the full X/Twitter social-listening feature:
Verified live: 44 real tweets through feed → watch → webhook, 44/44 HTTP 200, payload matched against the receiving intake contract. Local gates green (ruff 416 files, ty, migrations, schema sync; full suite passed at the earlier commits). For maintainers: CI runs are parked in |
twikit 2.3.3 renders every HTTP error as `status: <code>, message: "<body>"` (client/client.py and guest/client.py), so str(exc) is never empty, not even for an empty body. The e611a6c guard `not str(exc).strip()` therefore never fired: live polls showed the transient X SearchTimeline empty-404 mapping to not_found (non-retryable), which made ops treat a healthy source as dead. Match the empty BODY instead: an empty body renders the trailing payload `message: ""`; a real 404 carries response text. - errors.py: empty-404 detection checks the rendered body suffix. - tests_twitter.py: connector-level empty-404 test now routes through a real twikit NotFound + map_twikit_error; added map-level tests pinning empty-body 404 -> search_timeline_unavailable (retryable) and message-bearing 404 -> not_found (non-retryable). Validation: ruff check + format, ty, whitespace check, twitter tests (12) and full core suite (590) OK. Co-authored-by: Matthew Don <matthewdonsemail@gmail.com> Signed-off-by: Matthew Don <matthewdonsemail@gmail.com>
Author
Additional Context of our usecase we use it inside of https://github.com/block/buzz and we've got it ingesting content every 300 seconds and passing out via a webhook this here works in production now. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP feature: X/Twitter social listening (twikit connector) + README update
This branch carries the social-listening feature work done for the
matthewdonsemail-labfork, presented as a PR ontomain:1.
feat(sources): addtwitter_searchconnector (twikit-based) with schema parity (1dcf731)TwitterSearchSourceSpec(query, mode Top/Latest, count) added to the builtin source-spec union;NewTweetPayloadadded toFeedItemData;schema.jsonregenerated (82 models),--checkpasses.TwikitClientwith the live-cookie credential chain (TWITTER_COOKIES_JSON→auth_token/ct0→ cookies file → login fallback) so existing session cookie JSONs keep working — no proxy, no vendor API.sources/registry.py+connectors/__init__.py; source-kind invariant extended to the known five kinds.gitfor the git-pinned twikit dep.2.
fix(sources/twitter): per-call twikit client + retryable empty-404 mapping (e611a6c)Verified against live cookies (no proxy, no vendor API), two issues fixed:
Validation: ruff/ty/migrations/schema checks + 587 tests OK; live poll of a 4-source
twitter_searchfeed → feed → watch → webhook delivered 44/44 HTTP 200, payload matched field-for-field against the TwentysocialEventintake contract.3.
docs(readme): platform diagram + what-we've-done outline (785cd4d)twitter_searchsource kind.Notes
matthewdonsemail-lab/openmagpie(public), upstream:obris-dev/openmagpie.semantic_filter(ENGINE_BASE_URL/ENGINE_MODEL), and the Twenty workflow webhook URL (server-generated at publish on deploy).