This is a content extractor for Patreon, Pixiv Fanbox, Substack, and X (Twitter) photo media.
It leverages Playwright and the default extension bridge transport to open Chrome and download posts, images, and attachments automatically.
- Supported sites: Patreon, Pixiv Fanbox, Substack, X (Twitter photo media)
- Each post is downloaded as markdown file + images + attachments
- Even large attachments can be downloaded - up to 100s of MB
- HTML preview gallery generation - to view in the browser
- Local files cache for loading speed up - up to 95% hit rate
- Processed URLs are recorded to avoid extra traffic
- Substack only: epub generation
Existing account on Patreon with an active subscription to content creator. You can use free or any paid tier you need.
You should know what a shell/terminal is and how to execute commands in it.
Since Node.js is cross-platform, Windows/Linux/macOS are supported.
- Clone this repo into a folder. You can use
git cloneor just download source zip - Install Node.js. Used version is in the
.node-versionfile- You can use fnm to get correct version automatically.
- Install it, then
cdinto repository dir and it will install Node.js
- Install yarn - run
corepack enable - Run
yarn installto install packages - Run
yarn playwright install --with-depsto install browsers
Running any command listed below first time will open a login window.
Input your credentials and login, then press "Enter" in the shell to continue.
Next launches will reuse stored login info.
To force re-login on next scraping, delete flags.json file and retry.
yarn run launch --patreon <example> --year <year>yarn run launch --url https://<creator name>.fanbox.cc/posts --dir output/<creator name>Twitter extraction uses the default v2 bridge only; do not pass --v1. Provide a handle (with or without @):
yarn run launch --twitter <handle>The command opens https://x.com/<handle>/media?filter=photo, checks that an X account is logged in, and prompts for login if needed. Images are written to output/<handle>/. Add --update to stop at the first previously downloaded image.
yarn run launch --url <post url> --dir mypostAdd --update flag. It will stop fetching if processed posts are encountered.
yarn run launch --update --url https://<creator name>.fanbox.cc/posts --dir output/<creator name>Add --v1 to force the older browser automation path instead of the default extension bridge transport.
yarn run launch --v1 --patreon <example> --year <year>Add --trace to write sent bridge commands and arguments to output/<creator>/bridge.trace for debugging.
yarn run launch --trace --patreon <example> --year <year>toc:html is generated automatically by launch commands. Run gallery:html to replace gallery.html with the Twitter-style chronological feed, including each post's title, publication date, text, images, and attachment links.
yarn run gallery:html --dir "C:\patreon_extractor\output\creator\2025"yarn run epub --dir "C:\patreon_extractor\output\creator\"Gallery will open in the default system browser. If not, you can find gallery.html file in the directory you provided.
You can find files in the output/\<year>/ folder.
Html gallery file is always called gallery.html and can be viewed in the browser of your choice.
Scraping can be flaky and requests may fail due to network, increasing memory footprint etc.
Auto recovery is enabled by default. Use --no-recover to disable auto restarting browser on critical failures.
Cloudflare is a solution websites use to protect from bots and scrapers.
This is usually done by showing captcha on suspicious activity.
Either solve the captcha/challenge manually and press "Enter" in the shell afterward, or close and relaunch the command.
- Close the browser
- Delete
playwright-cachefolder,cookies.jsonfile andflags.jsonin the root of this repository folder - Start any command and login again
All js/css files and images are stored in the custom-cache folder.
Since there are no checks for the validity or staleness, old cache may break the scraping.
If you notice any weird problems, try to remove the folder and start scraping again.
🚨 Avoid deleting this folder unless necessary! Almost 90% of network requests are cached, which significantly speeds up the scraping process!
Use additional shell arguments:
yarn launch --url <your url> --year 2025 --dir output --server <your proxy server with port> --username <username> --password <password>Alternatively, use any system-wide proxy or VPN.
Playwright for scraping.
Rebrowser for working around Cloudflare detection.
Turndown to convert posts html into markdown.
Everything works locally.
The browser started for scraping is usual Chromium browser.
No credentials you enter are stored or sent anywhere externally.
You can examine built code in the dist directory after launching any commands for downloading.
Q: Why not use <name of other downloader> tool?
A: I tried some, and they didn't work for me. Also writing your own tool is fun.
Q: Will it give me access to unpaid content?
A: No, you need to have subscription to access content you want to download.
Q: Something broke and browser closed, will it need to download everything again?
A: No, processed URLs are stored in the \<output dir you provided>/visited.txt file.
Images and attachments are checked by their names before downloading.
Q: I'm stuck in a Cloudflare captcha infinite loop 😢
A: Open a new tab in automated browser with Google, wikipedia, etc. If you get any captchas there - solve them. Open some random urls more. Looks like it helps to build credibility with automated checks.
Alternatively, try another proxy/vpn.
Q: Low cache hit rate - less than 80%
A: You can uncomment logger in the cache.ts at the end of the file to log issues. Sometimes it may happen from SSL errors - if you're trying to avoid ISP restrictions. Disable said apps and try again.