Repository navigation
feat(discovery): keyless mining scripts and the closed-platform notices - #16
Merged
Merged
Conversation
…Store, and Google Play The discovery mining skills reached every platform through site-scoped web search, which returns snippets with no vote, reply, or rating counts and so cannot meet the quote-with-engagement contract the skills state. Four scripts now read the free endpoints that still answer without a key and emit one JSON shape (url, date, author, title, text, engagement) with exit 0 full, 1 partial, 2 empty, and a --fixture flag so the tests never touch the network. - hn_search.py reads the Algolia HN index (https://hn.algolia.com/api), comments and stories, with a since filter and a permalink per hit. - bsky_search.py reads app.bsky.feed.searchPosts (https://docs.bsky.app/docs/api/app-bsky-feed-search-posts). Probed today: the public endpoint serves one page of 100 per query and answers every cursor with 403, so the script makes one request and records a limit above the cap as a note, not an error. - apple_reviews.py reads the customer-reviews RSS feed (50 per page, 10 pages, version and vote counts) and continues through the apps.apple.com catalog reviews endpoint by cursor past that cap or when the feed is empty. Live run returned 500 RSS items plus 30 catalog items for a 530 target. - google_play_reviews.mjs drives the google-play-scraper package (https://github.com/facundoolano/google-play-scraper), an optional dependency. Missing package and unknown app id both exit 2 with a clear message, and --verify checks the id against a store search because the package returns an empty list for a mistyped package. The instance allowlist admits the four script paths so a run never stalls on a permission prompt. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… the closed platforms Reddit closed its unauthenticated .json endpoints in late May 2026 (403 on every request, probed today), has announced RSS ends 2026-11-13, and closes the public Data API to new registrations on 2026-10-31 with access ending March 2027 (r/modnews, 2026-05-28 and 2026-09-30). G2 and Capterra answer every free fetch with 403. X has no free read path. Without these facts in the skill text a thin yield from any of them reads as a tooling fault and burns an escalation on the wrong cause. - spgr-mine-ugc-forums runs hn_search.py for Hacker News and follows a new reddit-access reference that records the closure schedule and the paths in order: last30days (optional, keyless, scrapes the web frontend), the Apify CLI behind APIFY_TOKEN (the path that outlives March 2027), RSS until 2026-11-13, then WebSearch as the floor. The escalation names the stage in effect. - spgr-mine-app-store-reviews runs apple_reviews.py and google_play_reviews.mjs, states the 500-review RSS cap and the catalog fallback, and verifies the Play package id first. - spgr-mine-social-media runs bsky_search.py per query (one page per query without auth), mines X only through Apify when the token is set and otherwise omits it with a decision-log line, and drops the handle the script emits before the artifact, per the no-PII rule. - spgr-mine-review-platforms names Jina Reader (r.jina.ai, no key) for Trustpilot and Product Hunt pages that block WebFetch, carries an Apify Actor table with a primary and a fallback per platform, and treats a missing Actor as an escalation, never a substitution. - spgr-search-web adds Jina Reader as the fetch fallback and forbids routing around a block with a session or cookies. Every path is optional. Each skill runs with none of the tools installed and records which path produced each source. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Area 8 of 11: discovery mining endpoints
Verified today: Reddit
.jsonreturns 403, RSS is announced to end 2026-11-13, the Data API ends March 2027. Four keyless scripts with one JSON shape and--fixtureoffline parsing: HN Algolia, Bluesky public search, Apple review RSS with the catalog fallback (500-review cap per country per sort), and Google Play through the npm package with app-id verification. Each ran live today. The mining skills route through them, name the closed platforms so a thin yield escalates with the policy reason, and cite last30days and the Apify CLI as optional external paths. Bluesky refuses cursor pages, so the script takes one page of up to 100 per query. 10 fixture tests.Stacked PR: merge in order, this one after its base branch. Merge with a merge commit, as the earlier PRs were.
🤖 Generated with Claude Code