Skip to content

CLAUDE.md

Latest commit

 

History

History
196 lines (140 loc) · 7.8 KB

CLAUDE.md

File metadata and controls

196 lines (140 loc) · 7.8 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

TeraFetch is a batch downloader and IDM link exporter for TeraBox. It scrapes file lists from TeraBox share URLs, validates download links, and exports for IDM or downloads directly.

The project uses a simplified modular architecture with clear separation of concerns for maintainability and ease of understanding.

Development Commands

Setup

uv sync
uv run playwright install chromium

Run the application

# From TeraBox share URL with validation
uv run main.py -u "https://www.1024tera.com/sharing/link?surl=XXX" --idm --idm-check

# With retry enabled for more thorough validation (slower)
uv run main.py -u "https://www.1024tera.com/sharing/link?surl=XXX" --idm --idm-check --idm-check-retry

# Adjust retry attempts (default: 2)
uv run main.py -u "URL" --idm --idm-check --idm-check-retry --idm-check-max-retries 3

# From cache URL
uv run main.py -u "https://teradl.kingx.dev/cache?hash=XXX" --workers 5

# From saved JSON
uv run main.py --from-json downloads/file_list.json --limit 10

# Adjust validation workers (default: 3)
uv run main.py -u "URL" --idm --idm-check --idm-check-workers 5

Testing

# Syntax check all files
python -m py_compile main.py scrapers.py src/*.py

# Test specific module
python -m py_compile scrapers.py

Architecture

Simplified Structure

terafetch/
├── main.py              # CLI + orchestration
├── scrapers.py          # Scraping logic (share URLs + cache URLs)
├── src/
│   ├── downloader.py    # Download + validation
│   └── utils.py         # Helper utilities

main.py - CLI entry point with argument parsing, user interaction, download orchestration with link expiration retry logic, and IDM export handling.

Core Modules

scrapers.py - Unified scraping module

  • fetch_files() - Auto-detect URL type and fetch files
  • fetch_terabox_share() - Playwright-based scraping for share URLs
  • fetch_from_cache() - API-based scraping for cache URLs
  • convert_teraboxdownloader() - Normalize cache API responses
  • save_file_list() / load_file_list() - JSON persistence

src/downloader.py - Download implementation

  • download_batch() - Concurrent downloads with ThreadPoolExecutor
  • collect_download_links() - Parallel validation with configurable workers
  • _validate_download_url() - HEAD request optimization (10-100x faster), 15s timeout, smart fallback to GET
  • Performance optimizations: @lru_cache, connection pooling

src/utils.py - Helper utilities

  • validate_url() - Validates URL syntax only; scraper routing handles TeraBox support
  • is_terabox_share_url() - Distinguishes share URLs from cache URLs
  • setup_logging() - Logging configuration

Data Flow

  1. URL Detection: Check if TeraBox share URL or cache URL
  2. Fetching:
    • Share URLs → fetch_terabox_share() (Playwright)
    • Cache URLs → fetch_from_cache() → convert_teraboxdownloader()
  3. Normalization: Both produce same file record format
  4. Storage: Save to downloads/file_list.json
  5. Download/Export:
    • Download: download_batch() with curl or yt-dlp
    • IDM: collect_download_links() + save_links() → TXT files

File Record Format

{
    "name": "filename.mp4",
    "size": 1234567,
    "size_formatted": "1.2 MB",
    "dlink": "https://...",  # Direct download URL
    "quality": {"360p": "...", "720p": "...", "1080p": "..."},
    "streaming_url": "https://...m3u8",
    "caption": "https://...srt",
    "fs_id": 123456,  # For deduplication and re-scraping
    "thumbnail": "https://...",
    "folder": "/path/to/folder",
    "category": 1,
    "_source": "teraboxdownloader"
}

Performance Optimizations

  • HEAD Request Validation: Uses HEAD request first (10-100x faster than GET), only downloads content if Content-Type is suspicious
  • Reduced Timeout: 15 seconds for faster failure detection
  • Optional Retry Logic: Disabled by default for fastest validation. Failed links are exported to idm_links_failed.txt for manual retry in IDM. Enable with --idm-check-retry for more thorough validation at the cost of speed.
  • Caching: @lru_cache on _get_download_url() (2-5x speedup)
  • Connection Pooling: requests.Session with HTTPAdapter (30-50% faster)
  • Parallel Validation: ThreadPoolExecutor with configurable workers (default: 3, recommended: 5-10 for speed)
  • Parallel Downloads: Concurrent downloads with ThreadPoolExecutor

Output Files

  • downloads/file_list.json - Scraped file list
  • downloads/idm_links.txt - Valid links for IDM import (only when using --idm)
  • downloads/idm_links_failed.txt - Failed links that can be tried manually in IDM (only when using --idm --idm-check)
  • downloads/idm_links_all.txt - All links (valid + failed combined, only when using --idm --idm-check)
  • downloads/validation.log - Detailed validation logs (auto-cleared each run, only when using --idm-check)

Supported URLs (Tested)

TeraBox Share URLs:

  • https://www.1024tera.com/sharing/link?surl=XXX
  • https://1024terabox.com/s/XXX
  • https://terabox.app/sharing/link?surl=XXX

Cache URLs:

  • https://teradl.kingx.dev/cache?hash=XXX

Other domains may work but haven't been verified.

Link Expiration Handling

The downloader detects expired links by checking:

  • JSON responses with "link expired" error
  • HTML responses instead of file content
  • Small file sizes (< 1KB)

When detected, main.py automatically re-scrapes fresh links and retries (up to 2 attempts).

Validation Features

When using --idm-check:

  • HEAD Request First: Uses HEAD request (headers only) for fast validation
  • Smart Fallback: Falls back to GET request only if Content-Type is suspicious (JSON/HTML) or missing
  • 15s Timeout: Faster failure detection
  • Optional Retry Logic: By default, validation attempts each link once for speed. Use --idm-check-retry to enable retries (slower but more thorough). Failed links are exported to idm_links_failed.txt for manual retry in IDM.
  • Configurable Retries: Use --idm-check-max-retries N to set max retry attempts (default: 2, only applies when --idm-check-retry is enabled)
  • Parallel Validation: Configurable workers (default: 3, recommended: 5-10 for speed)
  • Connection Pooling: Reuses HTTP connections for better performance
  • Detailed Logging: Status codes, headers, and response content logged to validation.log

Performance: Valid links validate in ~0.1-0.5s. Without retries (default), failed links fail immediately (timeout or error). With retries enabled, failed links may take longer but have better chance of success.

Important: Links that fail validation aren't necessarily broken - they might have temporary server issues, rate limiting, or other transient problems. Always try importing idm_links_failed.txt into IDM as they often download successfully despite validation failures.

Retry Recommendation: Keep retries disabled (default) for fastest validation. Enable --idm-check-retry only if you experience many false negatives due to temporary network issues.

Dependencies

  • requests - HTTP client
  • playwright - Browser automation (required for share URLs)
  • yt-dlp - Video downloader (optional, for --quality flag)
  • curl - External command for downloads (must be in PATH)

Important Notes

  • Use --url instead of --from-json to enable auto re-scrape on expired links
  • Non-TeraBox URLs may pass basic syntax checks but will fail in the scraper
  • Validation workers: 1-3 (safe), 5-10 (fast but may trigger rate limits)
  • Links can fail validation but still work in IDM (server-side issues)
  • Referer headers are automatically set based on URL domain
  • Download speed depends on mirror/proxy server performance