🎬 Allo-Scrapper

Theater showtimes aggregator that scrapes and centralizes movie screening schedules from the source website theater pages. Built with Express.js, React, and PostgreSQL, fully containerized with Docker.

Latest Version: 4.3.0 | Status: Production Ready ✅

---

📋 Table of Contents

• Features
• Architecture
• White-Label Branding
• Quick Start
• Documentation
• Contributing
• License
• Support

---

✨ Features

• Automated Scraping: Scheduled scraping of theater showtimes from the source website
• Scraper Resilience: Automatic HTTP 429 rate limit detection with graceful shutdown
• RESTful API: Complete Express.js backend with TypeScript
• Modern UI: React SPA with Vite for fast development
• Real-time Progress: Server-Sent Events (SSE) for live scraping updates
• Weekly Reports: Track theater programs and identify new releases
• White-Label Branding: Complete customization (site name, logo, colors, fonts, footer) via admin panel
• User Management: Role-based access control with comprehensive user CRUD
• Role Management: Full CRUD for custom roles with granular permission assignment via admin panel
• JWT Authentication: Secure user authentication with token-based sessions
• Password Management: Change password functionality for authenticated users
• Content Security Policy: Strict CSP without unsafe-inline/unsafe-eval in script-src
• Session Expiry Handling: Expired JWTs are invalidated client-side and protected routes redirect to login
• Rate Limiting: Comprehensive rate limiting per endpoint type (auth, public, protected)
• Performance Optimized: JSON parse caching with LRU eviction (95-99% hit rate)
• Docker Ready: Full containerization with multi-stage builds (linux/amd64)
• CI/CD: GitHub Actions workflow for automated Docker image builds
• Scraper Microservice: Standalone scraper service communicates with the API via Redis job queue and pub/sub
• Observability: Prometheus metrics, Grafana dashboards, Loki log aggregation, Tempo distributed tracing
• Production Ready: Health checks, error handling, and database migrations

---

🏗 Architecture

┌─────────────────┐
│   React SPA     │  Port 80 (production) / 5173 (dev)
│   (Vite + TS)   │
└────────┬────────┘
         │ HTTP API / SSE
         ▼
┌─────────────────┐    Redis pub/sub    ┌───────────────────┐
│  Express.js API │◄───────────────────►│ Scraper           │
│  (TypeScript)   │   scrape:jobs queue │ Microservice      │
│  API + frontend │────────────────────►│ (ics-scraper)     │
│  only — no      │                    │                   │
│  scraping code  │                    │  ┌─────────────┐  │
│                 │                    │  │ Cron        │  │
│                 │                    │  │ (ics-scraper│  │
│                 │                    │  │  -cron)     │  │
└────────┬────────┘                    └────────┬──────────┘
         │ SQL                                  │ SQL
         └──────────────────┬───────────────────┘
                            ▼
              ┌─────────────────────────┐
              │   PostgreSQL  Port 5432 │
              │  theaters / movies /      │
              │  showtimes / reports    │
              └─────────────────────────┘

              ┌─────────────────────────┐
              │   Redis  (mandatory)    │  Job queue + progress pub/sub
              └─────────────────────────┘

  Monitoring (separate compose file — docker-compose.monitoring.yml):
  Prometheus :9090 → Grafana :3001
  Loki + Promtail (logs) → Grafana
  Tempo :3200 (traces, OTLP :4317) → Grafana
  Start with: docker compose --env-file .env --env-file .env.monitoring \
                -f docker-compose.yaml -f docker-compose.monitoring.yml up -d
  (see .env.monitoring.example for required variables)

Data Flow:

1. Client makes HTTP requests to Express API (/api/*)
2. API routes handle business logic and validate requests
3. API publishes scrape jobs to Redis (scrape:jobs queue) — the scraper microservice picks them up
4. Scraper fetches data from the source website and writes results directly to PostgreSQL
5. Progress events flow back to the API via Redis pub/sub → SSE → client
6. PostgreSQL stores structured theater, movie, and showtime data
7. Client receives JSON responses and renders UI

---

🎨 White-Label Branding

Allo-Scrapper supports complete white-label customization through a comprehensive admin panel. Transform the application to match your brand identity with custom colors, fonts, logo, and more.

Admin Panel Access

1. Navigate to /admin/settings (requires admin role)
2. Default credentials:
   • Username: admin
   • Password: admin

⚠️ Important: Change the default admin password immediately after first login (click your username → "Change Password").

Customization Options

The admin panel provides five tabs for complete branding control:

1. General Settings

• Site Name: Displayed in header, page title, and footer
• Logo: Custom logo image (PNG/JPG/SVG, max 200KB, min 100x100px)
• Favicon: Browser tab icon (ICO/PNG, max 50KB, 32x32 or 64x64px)

2. Color Scheme

Customize 9 color variables with live preview:

• Primary: Main brand color (buttons, links, highlights)
• Secondary: Header, footer background
• Accent: Call-to-action elements
• Background: Page background
• Surface: Card backgrounds
• Text Primary: Main text color
• Text Secondary: Muted text (labels, captions)
• Success: Success messages and indicators
• Error: Error messages and alerts

All colors must be valid hex codes (e.g., #FECC00, #1F2937, #FFF).

3. Typography

• Heading Font: Choose from 15+ Google Fonts for headings (h1-h6)
• Body Font: Choose font for body text and UI elements
• Live Preview: See font changes in real-time

Available Google Fonts: Inter, Roboto, Open Sans, Lato, Montserrat, Poppins, Raleway, Nunito, PT Sans, Source Sans Pro, Work Sans, Archivo, Manrope, DM Sans, Plus Jakarta Sans

4. Footer Customization

• Footer Text: Custom message with dynamic placeholders:
  • {site_name} → Site name from General settings
  • {year} → Current year
• Footer Links: Add unlimited custom links (e.g., Privacy Policy, Contact, About)
  • Each link: Label + URL
  • Drag to reorder

5. Email Branding

• From Name: Sender name for system emails
• From Address: Sender email address
• Header Color: Email template header background
• Footer Text: Email template footer message

Configuration Management

Export Settings (backup):

# Using admin panel: Click "Export Configuration" button

# Using API:
TOKEN=$(curl -X POST http://localhost:3000/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"admin"}' | jq -r '.data.token')

curl -H "Authorization: Bearer $TOKEN" \
  http://localhost:3000/api/settings/export > settings-backup.json

Import Settings (restore from backup):

# Using admin panel: Click "Import Configuration" button and select JSON file

# Using API:
curl -X POST -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d @settings-backup.json \
  http://localhost:3000/api/settings/import

Reset to Defaults:

• Click "Reset to Defaults" button in admin panel
• Restores original Allo-Scrapper branding

User Role Management

The system supports two roles:

Role	Permissions
admin	Full access: Settings, user management, reports, scraping control
user	Limited access: View theater schedules only

Safety Features:

• Cannot delete the last admin user
• Cannot demote the last admin to user role
• Admin authentication required to create new users
• Self-deletion prevention

Managing Users (admin only):

1. Navigate to /admin/users
2. Create, edit, delete users
3. Change user roles
4. Reset user passwords (generates secure random password)

API Access

Settings and user management are available via REST API:

• Settings API: /api/settings/* - See API.md
• Users API: /api/users/* - See API.md
• Theme CSS: /api/theme.css - Dynamically generated CSS with theme variables

For complete API documentation, see API.md.

Best Practices

1. Backup before major changes: Use export feature before making significant branding changes
2. Test in staging first: If using multi-environment setup
3. Use high contrast colors: Ensure accessibility (WCAG AA minimum)
4. Optimize images: Compress logo/favicon before upload to stay under size limits
5. Change default password: Critical security step for production deployments
6. Create backup admin: Have at least 2 admin users before deleting/demoting accounts

Troubleshooting

Images not uploading:

• Check file size (Logo: 200KB max, Favicon: 50KB max)
• Verify format (PNG, JPG, SVG for logo; ICO, PNG for favicon)
• Ensure dimensions meet minimums (Logo: 100x100px+, Favicon: 32x32 or 64x64)

Settings not saving:

• Check browser console for error messages
• Verify JWT token is valid (try logging out and back in)
• Check network tab for API errors

Theme not applying:

• Hard refresh browser (Ctrl+F5 or Cmd+Shift+R)
• Clear browser cache
• Check /api/theme.css endpoint directly

For detailed user guide, see Admin Panel Guide.

---

🚀 Quick Start

Prerequisites

• Docker and Docker Compose installed
• Ports 3000 and 5432 available
• OpenSSL installed (for JWT secret generation)
  • Linux/macOS: Pre-installed
  • Windows: Use Git Bash or WSL

Option A: Using Pre-built Images (Recommended)

The easiest way to deploy is using pre-built Docker images from GitHub Container Registry.

# Clone the repository (for configuration files)
git clone https://github.com/PhBassin/allo-scrapper.git
cd allo-scrapper

# Copy and edit the environment file (5 variables — see comments inside)
cp .env.example .env
# Fill in POSTGRES_PASSWORD and JWT_SECRET in .env

# Start all services (database auto-migrates on first startup)
docker compose up -d

# Trigger first scrape
curl -X POST http://localhost:3000/api/scraper/trigger

Access the application:

• Web UI: http://localhost:3000
• API: http://localhost:3000/api
• Health check: http://localhost:3000/api/health
  • Returns database connectivity status
  • Cached for 5 seconds to prevent connection pool exhaustion
  • Rate limited to 10 req/min per IP (localhost exempt for Docker/K8s probes)

Update to latest version:

# Pull the latest image and restart
docker compose pull
docker compose up -d

Option B: Building Locally

If you want to build the Docker image from source:

# Clone the repository
git clone https://github.com/PhBassin/allo-scrapper.git
cd allo-scrapper

# Copy and edit the environment file
cp .env.example .env
# Fill in POSTGRES_PASSWORD and JWT_SECRET in .env

# Build and start services (uses docker-compose.build.yml)
docker compose -f docker-compose.build.yml up --build -d

# Database auto-migrates on first startup — no manual migration needed

# Trigger first scrape
curl -X POST http://localhost:3000/api/scraper/trigger

Option C: Local Development (No Docker)

For active development without Docker:

Prerequisites:

• Node.js 20.19+ or 22.12+
• PostgreSQL 15+
• Redis (required for scraping operations)

Setup:

# Clone repository
git clone https://github.com/PhBassin/allo-scrapper.git
cd allo-scrapper

# Install server dependencies (IMPORTANT: run from server/ directory)
cd server
npm install

# Install client dependencies
cd ../client
npm install

# Setup environment and database
cd ..
cp .env.example .env && cat .env.dev.example >> .env

# Generate a secure JWT secret and set it in .env
# openssl rand -base64 64  →  paste into JWT_SECRET=
cd server
npm run db:migrate

# Start Redis (required — scraping will not work without it)
# e.g. via Docker: docker run -d -p 6379:6379 redis:7-alpine

# Run development servers (in separate terminals)
# Terminal 1 - API server
cd server && npm run dev    # API on http://localhost:3000

# Terminal 2 - Frontend dev server
cd client && npm run dev    # UI on http://localhost:5173

# Terminal 3 - Scraper microservice (for scraping to work locally)
cd scraper && npm run dev

⚠️ Important: Always run npm install from the server/ directory, not the root. The sharp image processing library requires native binaries that may not install correctly if run from the wrong directory.

Troubleshooting: If you encounter Cannot find package 'sharp' errors:

cd server
rm -rf node_modules
npm install

For production deployment and advanced configuration, see DEPLOYMENT.md and DOCKER.md.

---

⚙️ Configuration

Frontend Build Configuration

The frontend application name can be customized via Docker build arguments:

Variable	Description	Default	Used By
VITE_APP_NAME	Application name (browser tab, header, footer)	Allo-Scrapper	Docker build, GitHub Actions

Local Development:

# In .env file (root directory)
VITE_APP_NAME=Allo-Scrapper

# Run Vite dev server (reads from .env at runtime)
cd client
npm run dev

Docker Build:

# Using docker-compose.yaml (reads from .env file)
docker compose build

# Manual build with custom name
docker build --build-arg VITE_APP_NAME="My Theater App" .

GitHub Actions: The app name is set in .github/workflows/docker-build-push.yml as a build argument. All images built by CI/CD (develop, main, PR) use the configured name.

For complete environment variable documentation, see SETUP.md.

Dynamic White-Label Theme

The application supports runtime theme customization via the admin panel, allowing you to change branding, colors, fonts, and footer without rebuilding the Docker image.

Features:

• ✅ Site Name & Logo - Customize the site name and upload a custom logo
• ✅ Favicon - Upload a custom favicon
• ✅ Color Palette - Customize primary, secondary, accent, and UI colors
• ✅ Typography - Choose custom fonts from Google Fonts
• ✅ Footer - Custom footer text and links

How It Works:

1. Admin logs in and navigates to /admin/settings
2. Customizes branding in the admin panel (5 tabs: General, Colors, Typography, Footer, Email)
3. Changes apply immediately via /api/theme.css dynamic stylesheet
4. Settings stored in database (app_settings table)
5. Changes persist across container restarts

Loading Strategy:

• App shows loading screen while fetching settings from /api/settings
• Theme.css is injected dynamically via useTheme hook
• Hardcoded defaults used if API fails (graceful degradation)
• Logo, favicon, and document title update dynamically

For Admin Panel Documentation: See API.md for Settings API reference (/api/settings/* endpoints).

---

🛡️ Rate Limiting

The application includes comprehensive rate limiting to protect against abuse and ensure fair usage. Rate limits can be configured dynamically through the admin interface without server restarts.

Default Rate Limits

Endpoint Type	Default Limit	Window	Description
General API	100 requests	15 min	All /api/* routes
Authentication	5 attempts	15 min	Login endpoint (failed attempts only)
Registration	3 attempts	1 hour	New user registration
Protected Endpoints	60 requests	15 min	Authenticated user endpoints
Scraper Endpoints	10 requests	15 min	Expensive scraping operations
Public Endpoints	100 requests	15 min	Public read endpoints (theaters, movies)
Health Check	10 requests	1 min	/api/health endpoint (localhost exempt)

Dynamic Configuration

Rate limits can be managed through the admin interface:

1. Navigate to /admin?tab=ratelimits (admin-only)
2. Adjust limits based on your needs
3. Changes take effect immediately (no restart required)
4. All changes are logged in the audit trail

Features:

• ✅ Hot Reload - Changes apply immediately via in-memory config refresh
• ✅ Audit Trail - Complete history of all changes with user attribution
• ✅ Validation - Enforced min/max constraints prevent misconfiguration
• ✅ Backward Compatible - Falls back to environment variables if database unavailable
• ✅ Permission-Based - Granular permissions (read/update/reset/audit)

Environment Variables (Optional)

Rate limits can also be configured via environment variables (lower priority than database):

# Global window (milliseconds, 1 min to 1 hour)
RATE_LIMIT_WINDOW_MS=900000  # 15 minutes

# Per-endpoint limits (requests per window)
RATE_LIMIT_GENERAL_MAX=100
RATE_LIMIT_AUTH_MAX=5
RATE_LIMIT_REGISTER_MAX=3
RATE_LIMIT_PROTECTED_MAX=60
RATE_LIMIT_SCRAPER_MAX=10
RATE_LIMIT_PUBLIC_MAX=100
RATE_LIMIT_HEALTH_MAX=10

# Registration window (milliseconds, 5 min to 24 hours)
RATE_LIMIT_REGISTER_WINDOW_MS=3600000  # 1 hour

Priority Order:

1. Database configuration (managed via admin UI)
2. Environment variables (fallback)
3. Default values (hard-coded)

Key Features

Authenticated User Bucketing:

• Protected and scraper endpoints bucket by user ID (from JWT)
• Prevents a single user from exhausting rate limits for all users
• Falls back to IP-based bucketing for unauthenticated requests

Health Check Protection:

• Aggressive rate limiting (10 req/min) prevents resource exhaustion
• Localhost/Docker/Kubernetes IPs automatically exempted
• Response caching (5 seconds) reduces database load

Smart Skip Logic:

• Rate limiting automatically disabled in test environment
• Successful login attempts don't count toward auth limit

For API documentation, see API.md.

---

📚 Documentation

📖 Browse Full Documentation →

Tip for AI Agents: Use the @docs-writer OpenCode agent for all documentation tasks. It's specialized in maintaining our Divio-structured docs with automatic validation.

Our documentation is organized into the following categories:

🚀 Getting Started

New to Allo-Scrapper? Start here for quick setup and configuration.

• Quick Start - Get running in 5 minutes
• Installation - Detailed setup instructions
• Configuration - Environment variables

📖 Guides

Step-by-step tutorials for common tasks.

• Deployment - Production deployment, Docker, backups, monitoring
• Development - Local setup, testing, contributing, CI/CD
• Administration - Admin panel, white-label, user management

📋 Reference

Technical reference documentation.

• API - Complete REST API documentation
• Database - Schema and migrations
• Performance - Optimization and caching
• Scripts - Automation scripts
• Architecture - System design

🔧 Troubleshooting

Solutions to common issues.

• Common Issues
• Database | Docker | Networking | Scraper

📦 Project

Project meta-documentation.

• Changelog - Version history
• Security - Security policies
• AI Agents - Guidelines for AI coding agents

🔗 Backward Compatibility

All documentation has been migrated to the /docs/ directory. Root-level .md files are now symlinks that redirect to the new locations for backward compatibility with existing external links and bookmarks.

Original Files → New Locations:

• SETUP.md → docs/getting-started/installation.md
• DEPLOYMENT.md → docs/guides/deployment/production.md
• API.md → docs/reference/api/README.md
• AGENTS.md → docs/project/agents.md
• And more... (see full documentation structure)

---

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for:

• Code of Conduct
• Development workflow (Issue → Branch → TDD → PR)
• Coding standards
• Commit message conventions (Conventional Commits)
• Testing requirements

For AI coding agents, see AGENTS.md for mandatory workflow and TDD requirements.

Quick contribution checklist:

1. Create an issue first (bug/feature/task)
2. Create a feature branch from develop
3. Write tests before implementation (TDD)
4. Follow Conventional Commits format
5. Ensure all tests pass and Docker builds
6. Create PR referencing the issue
7. Add version label to PR (major, minor, or patch) when targeting main
8. Wait for review before merging

Automated versioning (PRs to main only):

• Add label major (breaking changes), minor (features), or patch (fixes)
• After merge + successful Docker build, version is auto-bumped and tagged
• CHANGELOG.md auto-generated from conventional commits
• GitHub release created automatically
• See AGENTS.md for details

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

💬 Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Documentation: See Documentation section above

---

Made with ❤️ for theater enthusiasts