Skip to content
BranchesTags

Repository files navigation

ig-archiver

Scrapes shared reels and posts from Instagram DMs, takes screenshots with Playwright, and generates AI summaries via GPT-4o vision. Any text message the sender wrote alongside the link is captured and used as intent context for the summary.

Current development version: 0.2.0. See CHANGELOG.md for release notes, ARCHITECTURE.md for system design and failure semantics, and PRODUCT_ROADMAP.md for the planned evolution from archive to creative-practice workflow.

ig-archiver/
├── extension/                    # chrome extension (React + TypeScript + Tailwind v4 + Vite 6)
│   ├── src/
│   │   ├── components/           # UI components (app, header, scan-button, progress-bar, status-feed)
│   │   ├── platform/             # platform abstraction (types, chromePlatform, electronPlatform)
│   │   ├── lib/                  # persistent archive jobs, scraper, config, truncate
│   │   ├── main.tsx
│   │   ├── types.ts
│   │   └── index.css
│   ├── public/
│   │   ├── manifest.json
│   │   └── content-script.js
│   ├── package.json
│   └── dist/                     # built output — load this folder in Chrome
└── server/                       # Express + TypeScript backend
    ├── server.ts
    ├── login.ts
    ├── tsconfig.json
    ├── lib/                      # jobs, archive runner, db, capture, summarize
    ├── public/                   # responsive archive dashboard
    ├── package.json
    ├── .env.example
    ├── screenshots/              # autogenerated — PNG captures
    ├── archive.sqlite            # autogenerated — archives and durable jobs
    └── database.json             # legacy archive, imported automatically once

Quick Start

1 — Set up the server

cd server
yarn install

First time only: install the Playwright browser binary:

yarn playwright install chromium

Copy the env template and add your OpenAI key:

cp .env.example .env
OPENAI_API_KEY=sk-...your-key-here...
PORT=3000

2 — Log in to Instagram

The server requires an authenticated Instagram session to visit and screenshot each post. Run this once:

yarn run login

A browser window will open. Log in to Instagram, then come back to the terminal and press Enter. Your session is saved to session.json and loaded automatically on every subsequent server start.

If Instagram ever logs you out, just run yarn run login again.

3 — Start the server

yarn run build
yarn start
# or
yarn run dev

You should see JSON log records for the server address and screenshot directory:

{"level":"info","event":"server.started","port":3000,"screenshots":".../server/screenshots"}

On the first 0.2.0 start, an existing database.json is imported automatically into archive.sqlite. The JSON file is left untouched as a migration backup.

4 — Build and load the extension

cd extension
yarn install
yarn run build

Then in Chrome:

  1. Navigate to chrome://extensions/
  2. Enable Developer mode (top-right toggle)
  3. Click Load unpacked
  4. Select the extension/dist/ folder

The extension does not require icons to work. Broken icon warnings can be ignored, or add your own 16×16, 48×48, and 128×128 PNGs to extension/icons/.

5 — Archive shared content from Instagram

  1. Open Instagram in Chrome — the extension's content script begins intercepting data immediately
  2. Navigate to the DM conversation you want to archive
  3. Click the IG Archiver toolbar icon
  4. Optionally try Load older messages or manually scroll upward to load more of the conversation
  5. Click Scan loaded messages to archive every loaded shared reel and post
  6. Pause, resume, or stop the server job from the popup as needed
  7. Close the popup or switch tabs safely; reopening it reconnects to the active job
  8. Open http://localhost:3000 (or use the popup link) to browse your archive

Instagram history loading is experimental because its private conversation API changes frequently. If it cannot load another batch, scroll upward manually and scan the messages currently loaded in the page.

How it works

Chrome Extension                        Node.js Server
────────────────────────────────        ────────────────────────────────
content-script.js (MAIN world)          POST /api/jobs  { urls: [...] }
  → patches XHR at document_start            │
  → captures get_slide_thread_nullable        ↓
    and fetch__SlideThread graphql      persistent job with bounded workers:
    responses as you browse                1. Playwright visits URL (authenticated)
  → stores SlideMessageXMAContent              - waitUntil: load (falls back to
    nodes in window.__igSlideThreads            domcontentloaded on timeout)
                                               - 1280×720 viewport
autoScrollOnce() (MAIN world)                  - SHA-1 filename → screenshots/
  → reads pageInfo cursor from           2. extract <title>, meta description,
    window.__igSlideThreads                  and post caption (article h1)
  → replays XHR to fetch older batch    3. GPT-4o vision → screenshot + caption
  → used only by the optional history     → summary, category + keywords
    action                              4. upsert entry in database.json
                                        5. expose polled progress plus
                                           pause/resume/stop controls
scrapeExternalLinks() (MAIN world)
  → reads window.__igSlideThreads
  → matches current thread via
    thread_key → thread_fbid mapping
  → extracts target_url from each
    XMA node (instagram.com/p/ or
    instagram.com/reel/ only)
  → extracts any user-typed message
    accompanying each link
    (node.text / node.message /
    xma.message, first non-empty wins)
  → sends URL list + message map
    to server and stores the returned
    job ID to reconnect after closure

Archive entry schema

{
  "url": "https://www.instagram.com/reel/ABC123/",
  "title": "Example post title",
  "metaDescription": "A brief description from the page.",
  "userMessage": "check this out!",
  "summary": "A one-sentence AI-generated overview of the post.",
  "category": "Learning",
  "keywords": "cooking, recipe, italian",
  "notes": "Try this recipe next weekend.",
  "aiConfidence": 86,
  "aiConfidenceReason": "Clear caption and visible cooking steps.",
  "intent": "learn",
  "workflowState": "up_next",
  "difficulty": "intermediate",
  "estimatedMinutes": 30,
  "priority": 4,
  "nextAction": "Recreate the texture treatment with my own lettering.",
  "mediums": ["Visual art"],
  "tools": ["Photoshop"],
  "skills": ["Texture", "Compositing"],
  "screenshotPath": "screenshots/3a9f12b04c1e.png",
  "archivedAt": "2026-03-03T10:00:00.000Z",
  "createdAt": "2026-03-03T10:00:00.000Z"
}

userMessage is the text the sender typed alongside the shared link. When present it is injected into the GPT-4o prompt as the primary intent signal, steering the summary and category toward what the user actually cared about rather than the page's generic content. Omitted from the entry when absent.

Categories: References · Memes · Inspiration · Tutorials · News · Ai · Tools · Music production · Movies and shows · Design · Music · Politics by default (one or two per entry). The list is editable from the dashboard.

Keywords: up to three comma-separated terms per entry, generated by the model

Creative workflow

Every new and migrated capture begins in Inbox and is never promoted automatically. From the dashboard it can be classified as Learn, Make, Reference, or Dismiss; enriched with medium, tool, skill, difficulty, effort, priority, and a concrete next action; then moved through Up Next, In Progress, Practiced, Applied, Published, or Cold Storage.

The active workflow deliberately allows only one In Progress item and five Up Next items. These limits are enforced transactionally by the server, including during bulk triage. Cold Storage remains searchable but is excluded from the active library and queue. Recapturing an existing Instagram URL refreshes capture provenance without erasing curated workflow metadata.

For large historical archives, Review three saves opens a bounded triage session instead of an infinite cleanup screen. Each save can be marked Learn, Make, Reference, or Cold Storage, or skipped without changing it. Learn and Make decisions remain outside the five-item queue until explicitly promoted and can be found through the Library intent filters.

Backlog overview summarizes unreviewed volume, original-message and screenshot coverage, age, and recurring metadata clusters. This first analysis is deterministic and local: it uses existing categories, keywords, mediums, tools, and skills without sending another AI request. Selecting a cluster filters Inbox to the matching captures.

Testing

Both packages ship a Vitest suite (132 tests total).

# server
cd server && npm test

# extension
cd extension && npm test

Watch mode: replace npm test with npm run test:watch.

Configuration

.env variable Default Description
OPENAI_API_KEY — Preferred API-key source; optional when saved in local Settings.
PORT 3000 Server listen port.
HOST 127.0.0.1 Listen address. Keep local unless protected by authentication.
SCREENSHOT_WIDTH 1280 Viewport / screenshot width.
SCREENSHOT_HEIGHT 720 Viewport / screenshot height.
TIMEOUT_MS 30000 Per-URL Playwright navigation timeout.
CONCURRENCY 3 Parallel capture workers (1–8).
RETRY_ATTEMPTS 3 Attempts for transient navigation and rate-limit failures.
RETRY_BASE_MS 750 Initial exponential-backoff delay in milliseconds.
DATA_DIR server directory SQLite, screenshots, session, and settings location.
OPENAI_MODEL gpt-4o OpenAI or compatible model name.
OPENAI_BASE_URL — Optional OpenAI-compatible API base URL.
MOCK — Set to true to skip OpenAI calls and return placeholder data.

SCROLL_LOADS (extension-side, in extension/src/lib/config.ts) controls how many scroll batches are fetched before scraping. Default is 5.

The dashboard Settings dialog can override concurrency, timeout, screenshot dimensions, existing-URL skipping, categories, model, API base URL, and API key. Environment configuration is preferred for secrets. A key entered in Settings is stored unencrypted in ignored server/config.json (or $DATA_DIR/config.json) using atomic replacement and owner-only 0600 permissions. It is write-only in the dashboard and is never returned by /api/config; a stored key takes precedence over OPENAI_API_KEY, and Use environment key removes it.

The server binds to 127.0.0.1 by default because the dashboard has no authentication. Docker listens on all interfaces inside the container but Compose publishes port 3000 only on the host loopback interface. Do not expose the service to a LAN or the public internet without adding authentication and TLS at a trusted reverse proxy.

The extension gear menu stores a custom archive-server URL in Chrome local storage. Remote server origins are requested as optional host permissions only when configured.

Dashboard

The server hosts a responsive dashboard at http://localhost:3000 with:

  • Full-text search across titles, summaries, notes, descriptions, and keywords
  • Dynamic category filters
  • Screenshot cards and an accessible detail dialog
  • Manual editing for titles, summaries, categories, keywords, and personal notes
  • AI interpretation confidence with a short evidence rationale; mock and legacy entries remain unscored
  • Archive deletion, including screenshot cleanup
  • Bulk selection with delete-selected and delete-all-visible actions
  • Runtime capture, category, and OpenAI-compatible provider settings

Archive jobs run in the Node.js process, so closing the extension popup or switching browser tabs does not interrupt them. Pause prevents new URLs from starting after active workers finish; Stop preserves completed results and cancels the remaining queue.

Archive entries, job snapshots, and job events are persisted in SQLite. Interrupted running jobs resume only unfinished URLs after a server restart; deliberately paused jobs remain paused until resumed. Archive writes are idempotent by URL, and transient capture failures use bounded exponential backoff.

Backup and restore

Use Export and Import in the dashboard for a portable JSON archive backup, or create one from the command line:

cd server
npm run backup -- ../ig-archiver-backup.json

Dashboard imports merge by URL. The API also supports an explicit replace mode through POST /api/archive/import. Screenshots remain in the configured screenshots/ directory, so include that directory alongside the JSON backup when moving the complete archive to another machine.

Docker

Create the persistent data directory and copy the authenticated Instagram session into it:

mkdir -p server/data
cp server/session.json server/data/session.json
# Optional when migrating an existing installation:
cp server/database.json server/data/database.json
docker compose up --build

The dashboard is available at http://localhost:3000. SQLite, screenshots, configuration, and the session live under server/data/. The container uses the same .env file as the local server.

Reliability model

  • SQLite schema migrations run automatically at startup.
  • Archive and job mutations are serialized and written through an atomic temporary-file rename.
  • Job events are persisted before the popup observes them.
  • Running jobs become resumable after a crash; completed URLs are not processed twice.
  • Logs are emitted as one JSON object per line with stable event names and contextual fields.
  • GitHub Actions runs server tests, extension tests, type checking, and the production build on every push and pull request.

The server only accepts cross-origin browser requests from Chrome extensions and its own localhost origin. It is designed as a personal local service, not a public multi-user application.

Troubleshooting

Symptom Fix
No session.json found Run yarn run login in /server.
OpenAI API key is not configured Add it in dashboard Settings or copy .env.example → .env.
Navigation failed for a URL Instagram may be unavailable or the saved session may have expired. Retry once, then run yarn run login if it persists.
Capture reports HTTP 429 or a blank page The server automatically retries through Instagram's embed view. If both views fail, wait before retrying and temporarily reduce concurrency to 1.
Existing records still show blank screenshots Disable Skip existing archived URLs, rescan them once, then re-enable it.
Extension shows "No shared posts found" Make sure the page was loaded with the extension active (reload the tab after installing).
Error: connect ECONNREFUSED in extension Make sure the server is running (yarn start in /server).
Playwright browser not found Run yarn playwright install chromium inside /server.
Fewer links than expected Scroll upward manually before scanning, or try Load older messages (experimental). Instagram's private history API is inconsistent.

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages