Skip to content

scraper.md

Latest commit

 

History

History
562 lines (453 loc) · 13.3 KB

scraper.md

File metadata and controls

562 lines (453 loc) · 13.3 KB
Raw

Scraper API

Note: This API uses Server-Sent Events (SSE) for real-time progress monitoring.

Scraper

Trigger Manual Scrape

POST /api/scraper/trigger

Authentication: Required (Bearer token)

Request Body (optional):

{
  "theaterId": "C0153",  // Optional: scrape only this theater (must exist in database)
  "movieId": 12345       // Optional: scrape only this movie
}

Behavior:

  • No parameters → Full scrape (all theaters, all movies, all dates)
  • theaterId only → Scrape this theater (all movies, all dates for this theater)
  • movieId only → Scrape this movie (all theaters showing this movie)
  • Both theaterId and movieId → Scrape this movie at this specific theater only

Response (200 — started):

{
  "success": true,
  "data": {
    "reportId": 43,
    "message": "Scrape job queued for microservice",
    "queueDepth": 2
  }
}

Response Fields:

  • reportId - Unique scrape report ID (can be used to track progress in database)
  • message - Human-readable status message
  • queueDepth - Number of jobs in the Redis queue after this job was added

Response (404 — theater not found):

{
  "success": false,
  "error": "Theater not found: CXXXX"
}

Response (409 — already running):

{
  "success": false,
  "error": "A scrape is already in progress",
  "data": {
    "current_scrape": {
      "started_at": "2024-02-15T10:00:00.000Z",
      "trigger_type": "manual"
    }
  }
}

Examples:

# Get auth token first
TOKEN=$(curl -X POST http://localhost:3000/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"admin"}' | jq -r '.data.token')

# Full scrape (all theaters, all movies)
curl -X POST http://localhost:3000/api/scraper/trigger \
  -H "Authorization: Bearer $TOKEN"

# Theater-specific scrape (C-prefix)
curl -X POST http://localhost:3000/api/scraper/trigger \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"theaterId": "C0153"}'

# Theater-specific scrape (W-prefix)
curl -X POST http://localhost:3000/api/scraper/trigger \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"theaterId": "W7515"}'

# Movie-specific scrape
curl -X POST http://localhost:3000/api/scraper/trigger \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"movieId": 12345}'

# Combined: scrape specific movie at specific theater
curl -X POST http://localhost:3000/api/scraper/trigger \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"theaterId": "C0153", "movieId": 12345}'

Get Scraper Status

GET /api/scraper/status

Authentication: Not required (public endpoint)

Response (200 — success):

{
  "success": true,
  "data": {
    "isRunning": false,
    "currentSession": null,
    "latestReport": {
      "id": 42,
      "completed_at": "2024-02-15T10:15:23.000Z",
      "status": "success"
    }
  }
}

Response Fields:

  • isRunning - Whether a scrape is currently being processed by a worker
  • currentSession - Current scrape session details (null when no scrape is running)
  • latestReport - Most recent completed scrape report from database

Example:

curl http://localhost:3000/api/scraper/status

Watch Scrape Progress (SSE)

GET /api/scraper/progress

Opens a persistent Server-Sent Events connection. All previously accumulated events are replayed to new clients, then new events are streamed in real time. A heartbeat (: heartbeat) is sent every 15 seconds to keep the connection alive.

Response Headers:

  • Content-Type: text/event-stream
  • Cache-Control: no-cache
  • Connection: keep-alive
  • X-Accel-Buffering: no

Event Format:

All events are sent as plain data: lines (no named event: field). Each line is a JSON object with a type discriminator:

data: {"type":"started","total_theaters":3,"total_dates":7}

data: {"type":"theater_started","theater_name":"Épée de Bois","theater_id":"W7504","index":1}

data: {"type":"date_started","date":"2026-02-19","theater_name":"Épée de Bois"}

data: {"type":"movie_started","movie_title":"Mon Movie","movie_id":123456}

data: {"type":"movie_completed","movie_title":"Mon Movie","showtimes_count":5}

data: {"type":"movie_failed","movie_title":"Mon Movie","error":"HTTP 404"}

data: {"type":"date_completed","date":"2026-02-19","movies_count":12}

data: {"type":"date_failed","date":"2026-02-19","theater_name":"Épée de Bois","error":"HTTP 503"}

data: {"type":"theater_completed","theater_name":"Épée de Bois","total_movies":42}

data: {"type":"completed","summary":{"total_theaters":3,"successful_theaters":3,"failed_theaters":0,"total_movies":87,"total_showtimes":412,"total_dates":7,"duration_ms":34210,"errors":[]}}

data: {"type":"failed","error":"Fatal error message"}

Event Types:

Type Emitted Payload fields
started Once at start total_theaters, total_dates
theater_started Per theater theater_name, theater_id, index
date_started Per theater × date date, theater_name
movie_started Per movie movie_title, movie_id
movie_completed Per movie (success) movie_title, showtimes_count
movie_failed Per movie (error) movie_title, error
date_completed Per date (success) date, movies_count
date_failed Per date (error) date, theater_name, error
theater_completed Per theater (≥1 date ok) theater_name, total_movies
completed Once on success summary (ScrapeSummary object)
failed Once on fatal error error

Rate Limit Handling:

When the scraper detects an HTTP 429 (Too Many Requests) response from the source server:

  1. The scrape stops immediately to avoid further rate limiting
  2. Status is set to rate_limited instead of failed
  3. A theater_failed event is emitted with error_type http_429
  4. The final completed event includes "status": "rate_limited" in the summary

Example Rate Limited Summary:

{
  "type": "completed",
  "summary": {
    "total_theaters": 10,
    "successful_theaters": 3,
    "failed_theaters": 7,
    "total_movies": 45,
    "total_showtimes": 212,
    "total_dates": 7,
    "duration_ms": 12340,
    "status": "rate_limited",
    "errors": [{
      "theater_name": "Example Theater",
      "theater_id": "C0123",
      "date": "2026-03-24",
      "error": "HTTP 429 Too Many Requests",
      "error_type": "http_429",
      "http_status_code": 429
    }]
  }
}

Example:

curl -N http://localhost:3000/api/scraper/progress

JavaScript Example:

const eventSource = new EventSource('http://localhost:3000/api/scraper/progress');

// All events arrive via onmessage (no named event: field)
eventSource.onmessage = (e) => {
  const data = JSON.parse(e.data);
  console.log('Event:', data.type, data);

  if (data.type === 'completed') {
    console.log('Scraping complete:', data.summary);
    eventSource.close();
  }
  if (data.type === 'failed') {
    console.error('Scraping failed:', data.error);
    eventSource.close();
  }
};

eventSource.onerror = (err) => {
  console.error('SSE connection error:', err);
  eventSource.close();
};

Resume Rate-Limited Scrape

POST /api/scraper/resume/:reportId

Authentication: Required (Bearer token)

Parameters:

  • reportId (integer): ID of the parent report to resume (must have status rate_limited, failed, or partial_success)

Description: Creates a new scrape report that retries only the theater/date combinations that weren't successfully scraped in the parent report. The new report links to the parent via parent_report_id.

Response (200 — started):

{
  "success": true,
  "data": {
    "reportId": 124,
    "parentReportId": 123,
    "message": "Resume job queued for microservice",
    "queueDepth": 1,
    "pendingAttempts": [
      { "theater_id": "C0042", "date": "2026-03-26" },
      { "theater_id": "C0089", "date": "2026-03-25" }
    ]
  }
}

Response Fields:

  • reportId - New scrape report ID for the resume operation
  • parentReportId - Original report ID that is being resumed
  • message - Human-readable status message
  • pendingAttempts - List of theater/date combinations that will be retried
  • queueDepth - Number of jobs in queue after this job was added

Response (404 — report not found):

{
  "success": false,
  "error": "Report not found: 999"
}

Response (400 — no pending attempts):

{
  "success": false,
  "error": "No pending attempts found for report 123"
}

Response (409 — already running):

{
  "success": false,
  "error": "A scrape is already in progress"
}

Examples:

# Get auth token
TOKEN=$(curl -X POST http://localhost:3000/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"admin"}' | jq -r '.data.token')

# Resume rate-limited scrape
curl -X POST http://localhost:3000/api/scraper/resume/123 \
  -H "Authorization: Bearer $TOKEN"

# Check new report status
curl http://localhost:3000/api/reports/124 \
  -H "Authorization: Bearer $TOKEN"

Scrape Schedules

Schedule recurring scrapes using cron expressions.

List All Schedules

GET /api/scraper/schedules

Authentication: Required (Bearer token)
Permission: scraper:schedules:list

Response (200 — success):

{
  "success": true,
  "data": [
    {
      "id": 1,
      "name": "Daily Morning Scrape",
      "description": "Scrape all theaters every morning at 6 AM",
      "cron_expression": "0 6 * * *",
      "enabled": true,
      "target_theaters": ["W7504", "C0072"],
      "created_by": 1,
      "created_at": "2026-03-15T10:30:00Z",
      "updated_at": "2026-03-15T10:30:00Z"
    }
  ]
}

Get Schedule by ID

GET /api/scraper/schedules/:id

Authentication: Required (Bearer token)
Permission: scraper:schedules:list

Response (200 — success):

{
  "success": true,
  "data": {
    "id": 1,
    "name": "Daily Morning Scrape",
    "description": "Scrape all theaters every morning at 6 AM",
    "cron_expression": "0 6 * * *",
    "enabled": true,
    "target_theaters": ["W7504", "C0072"],
    "created_by": 1,
    "created_at": "2026-03-15T10:30:00Z",
    "updated_at": "2026-03-15T10:30:00Z"
  }
}

Response (404 — not found):

{
  "success": false,
  "error": "Schedule not found"
}

Create Schedule

POST /api/scraper/schedules

Authentication: Required (Bearer token)
Permission: scraper:schedules:create

Request Body:

{
  "name": "Daily Morning Scrape",
  "description": "Scrape all theaters every morning at 6 AM",
  "cron_expression": "0 6 * * *",
  "enabled": true,
  "target_theaters": ["W7504", "C0072"]
}

Request Fields:

  • name (string, required) - Schedule name (must be unique)
  • description (string, optional) - Human-readable description
  • cron_expression (string, required) - Cron expression (e.g., 0 6 * * * for 6 AM daily)
  • enabled (boolean, optional, default: true) - Whether schedule is active
  • target_theaters (array, optional) - List of theater IDs to scrape (empty = all theaters)

Response (201 — created):

{
  "success": true,
  "data": {
    "id": 1,
    "name": "Daily Morning Scrape",
    "description": "Scrape all theaters every morning at 6 AM",
    "cron_expression": "0 6 * * *",
    "enabled": true,
    "target_theaters": ["W7504", "C0072"],
    "created_by": 1,
    "created_at": "2026-03-15T10:30:00Z",
    "updated_at": "2026-03-15T10:30:00Z"
  }
}

Response (400 — validation error):

{
  "success": false,
  "error": "Schedule name is required"
}

Update Schedule

PUT /api/scraper/schedules/:id

Authentication: Required (Bearer token)
Permission: scraper:schedules:update

Request Body (all fields optional):

{
  "name": "Updated Schedule Name",
  "description": "Updated description",
  "cron_expression": "0 12 * * *",
  "enabled": false,
  "target_theaters": ["W7504"]
}

Response (200 — success):

{
  "success": true,
  "data": {
    "id": 1,
    "name": "Updated Schedule Name",
    "description": "Updated description",
    "cron_expression": "0 12 * * *",
    "enabled": false,
    "target_theaters": ["W7504"],
    "created_by": 1,
    "created_at": "2026-03-15T10:30:00Z",
    "updated_at": "2026-03-18T15:45:00Z"
  }
}

Delete Schedule

DELETE /api/scraper/schedules/:id

Authentication: Required (Bearer token)
Permission: scraper:schedules:delete

Response (204 — deleted):

(no body)

Response (404 — not found):

{
  "success": false,
  "error": "Schedule not found"
}

Trigger Schedule Immediately

POST /api/scraper/schedules/:id/trigger

Authentication: Required (Bearer token)
Permission: scraper:schedules:update

Description: Immediately trigger a schedule, bypassing the cron timing.

Response (200 — started):

{
  "success": true,
  "data": {
    "reportId": 43,
    "message": "Scrape job queued for microservice",
    "queueDepth": 2
  }
}

← Back to API Reference