ChatGPT Conversation Scraper

Export all of your ChatGPT conversations to local Markdown files, organized by project. Includes images, project files, citations, memories, and custom instructions.

ChatGPT has no cross-conversation search within projects. This scraper creates a local, searchable archive of everything — so you can grep across your entire history.

Why This Exists

Your conversations with AI are some of the most valuable digital artifacts you produce — they contain your thinking process, your decisions, your creative work, and your problem-solving in raw form. Yet they live entirely on someone else's servers, subject to someone else's terms of service.

OpenAI can change their data retention policies, shut down access, or alter the export format at any time. The built-in "Export Data" feature gives you a bulk JSON dump that's barely human-readable. There's no way to search across conversations within a project — a glaring omission for a tool people use as an extension of their working memory.

This matters because:

• You should own your own thinking. If you've spent hundreds of hours working through problems with ChatGPT, that history has real intellectual value. It shouldn't be locked inside a platform you don't control.
• Services disappear. Companies pivot, get acquired, or go under. Your data shouldn't evaporate with them.
• Searchability is basic infrastructure. Being unable to search your own conversation history is not a missing feature — it's a missing right.
• Local copies are the only reliable backup. Cloud-only data is a single point of failure. A local archive in plain Markdown files will outlast any proprietary format.

This tool is a small act of digital self-determination: take what's yours, store it in a format you control, and make it actually useful.

What Gets Exported

Data	Format	Location
Conversations	Markdown (.md)	projects/{Project Name}/
Images from conversations	Original files (.jpg, .png, etc.)	projects/{Project Name}/_assets - {title}/
Project files (knowledge files)	Original format (.docx, .pdf, .txt, etc.)	projects/{Project Name}/_project_files/
ChatGPT memories	Markdown + JSON	projects/_account/memories.md
Custom instructions	Markdown + JSON	projects/_account/custom_instructions.md
Archived conversations	Markdown	projects/[Archived]/
Non-project conversations	Markdown	projects/[No Project]/

Output Structure

projects/
├── _account/
│   ├── memories.md
│   ├── memories.json
│   ├── custom_instructions.md
│   └── custom_instructions.json
├── [Archived]/
│   └── {title}.md
├── [No Project]/
│   └── {title}.md
├── My Project/
│   ├── _project_files/
│   │   ├── uploaded_doc.docx
│   │   └── reference.pdf
│   ├── _assets - Some Conversation/
│   │   ├── photo.jpg
│   │   └── screenshot.png
│   ├── Some Conversation.md
│   └── Another Conversation.md

Markdown Format

Each conversation is saved as a Markdown file with metadata and full message history:

# Conversation Title

**Created:** 2025-03-04 10:17:19 UTC
**Updated:** 2025-03-04 10:17:53 UTC
**Model:** gpt-4o
**Conversation ID:** 69a80685-0a40-838e-a26f-7bb5cad90173

---

## You

Your message here...

## ChatGPT

Response with **markdown**, `code`, and:

​```python
code_blocks_preserved()
​```

Some text with a citation[^1]

---

### References

[^1]: https://example.com/source

Prerequisites

• Node.js (v18+)
• curl (comes with most systems; on Windows, Git Bash includes it)
• Chrome with a cookie export extension (e.g., Cookie-Editor)

Setup

1. Export Your Cookies

1. Log in to chatgpt.com in Chrome
2. Open the Cookie-Editor extension
3. Click Export → choose JSON format
4. Save the output to cookies.txt in this directory

The file should look like this:

[
    {
        "domain": ".chatgpt.com",
        "name": "__Secure-next-auth.session-token",
        "value": "eyJhbGci...",
        ...
    },
    ...
]

2. Get Your Bearer Token

Use your exported cookies to request a session token:

curl -s 'https://chatgpt.com/api/auth/session' \
  -H "Cookie: $(cat cookies.txt | node -e "const c=JSON.parse(require('fs').readFileSync('/dev/stdin','utf8'));console.log(c.map(x=>x.name+'='+x.value).join('; '))")" \
  -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36" \
  -H "Accept: application/json" \
  -H "Referer: https://chatgpt.com/" \
  | node -e "process.stdin.on('data',d=>{const j=JSON.parse(d);console.log(j.accessToken)})"

Save the output (a long JWT string) to a file called .token:

# Paste the token into .token (no newline at end)
echo -n "eyJhbGci..." > .token

3. Test With a Single Conversation

Find any conversation ID from your ChatGPT URL (e.g., https://chatgpt.com/c/69a80685-0a40-838e-a26f-7bb5cad90173) and test:

node scraper.js --conv 69a80685-0a40-838e-a26f-7bb5cad90173

This saves one conversation to projects/[No Project]/. Check that it looks right before doing a full scrape.

4. Full Scrape

node scraper.js 2>&1 | tee scrape.log

This will:

1. Enumerate all your projects
2. Download all project files (knowledge files uploaded to projects)
3. Scrape every conversation in every project
4. Scrape all non-project conversations
5. Scrape archived conversations
6. Export your ChatGPT memories and custom instructions

Expect this to take a while depending on how many conversations you have. Progress is printed as it goes. The scraper uses a 1.2-second delay between API calls and backs off for 30 seconds if rate-limited.

5. Incremental Updates

After the initial full scrape, use the incremental scraper for subsequent runs:

node incremental.js

On the first run, it builds a manifest.json from your existing files (without re-downloading). On subsequent runs, it only fetches conversations that are new or have been modified since the last run.

The incremental scraper also tracks:

• Moved conversations — if you move a conversation to a different project, the old file is prefixed with [moved] - and a fresh copy is saved in the new project folder
• Deleted conversations — if a conversation is deleted from your account, the local file is prefixed with [deleted] - (nothing is ever removed from your archive)
• Project files — re-downloads if file size changes, skips if unchanged
• Memories & custom instructions — always refreshed

How It Works

Authentication

ChatGPT's web app uses internal API endpoints under https://chatgpt.com/backend-api/. Auth requires:

1. Session cookies exported from your browser
2. A Bearer token obtained from the /api/auth/session endpoint

The Bearer token expires periodically. If the scraper starts failing, re-export cookies and generate a new token.

API Endpoints Used

Endpoint	Purpose
/gizmos/snorlax/sidebar?owned_only=true	List all projects (paginated)
/gizmos/{project_id}/conversations?cursor=0	List conversations in a project
/conversations?offset=N&limit=N&order=updated	List non-project conversations
/conversations?is_archived=true	List archived conversations
/conversation/{id}	Full conversation with message tree
/files/download/{file_id}?inline=false	Get signed URL for conversation images
/files/{file_id}/download?gizmo_id={project_id}	Get signed URL for project files
/memories?limit=500	All stored memories
/user_system_messages	Custom instructions

Key Technical Details

• Projects are internally called "gizmos" with IDs prefixed g-p-
• Conversations contain a message tree (not a flat list) accessed via a mapping object
• Messages are walked backwards from current_node to root via parent pointers
• Message content is in message.content.parts[] as plain strings (already Markdown-formatted)
• Images use content_type: "image_asset_pointer" with asset_pointer: "sediment://file_XXXX"
• Image download is two-step: get a signed URL from the API, then download from that URL
• Project file download also two-step, but requires passing gizmo_id as a query parameter
• Citations contain invisible Unicode characters (PUA range U+E000–U+F8FF) that must be stripped before parsing
• Citation source URLs are in search_result_groups metadata, mapped by ref_index

Why curl Instead of fetch/axios?

Cloudflare's bot protection blocks requests from Node.js HTTP clients based on TLS fingerprinting. Using curl via child_process.execSync with browser-like headers bypasses this. This is the same approach used by tools like yt-dlp.

Rate Limiting

• 1.2-second delay between API calls
• If the API returns 429 (rate limited), the scraper waits 30 seconds and retries once
• Conversations are fetched sequentially, not in parallel
• In practice, a full scrape of ~1,500 conversations completes without hitting rate limits

Token Expiration

The Bearer token expires after some time (usually a few hours). If you see authentication errors mid-scrape:

1. Re-export cookies from Cookie-Editor (you may need to refresh chatgpt.com first)
2. Save the new cookies to cookies.txt
3. Generate a new token using the curl command above
4. Save to .token
5. Re-run the scraper — it will skip conversations that already have files on disk (for incremental.js) or overwrite them (for scraper.js)

Searching Your Archive

Once scraped, you can search across all conversations with standard tools:

# Search for a term across all conversations
grep -r "some topic" projects/

# Search within a specific project
grep -r "error message" projects/System\ Build/

# Find conversations mentioning a specific tool
grep -rl "Docker" projects/ | head -20

# Count conversations per project
for d in projects/*/; do echo "$(ls "$d"/*.md 2>/dev/null | wc -l) $d"; done | sort -rn

License

MIT