A Node.js backend that tracks product prices across various fashion, streetwear, and lifestyle e-commerce platforms (including Nykaa Fashion, Bewakoof, Snitch, The Souled Store, and Culture Circle). It runs an Express API server alongside a polling scraper process that periodically fetches updated prices and writes them back to a shared Supabase database.
- Architecture Overview
- Project Structure
- How It Works
- Environment Variables
- Running Locally
- Adding a New Scraper
- API Reference
- Database Schema
Frontend (React/Vite)
│
│ REST API (JWT auth via Supabase)
▼
┌──────────────────┐ ┌──────────────────────────┐
│ Express API │◄──────►│ Supabase DB │
│ (index.js) │ │ products + user_products │
└──────────────────┘ └──────────────────────────┘
▲
│ writes scraped prices
┌────────────────┐
│ Scraper Loop │
│ (scrapper/ │
│ scrape.js) │
└────────────────┘
│
┌────────────────┐
│ scrapers/<site> │
│ handler.js │
└────────────────┘
Both the API server and the scraper run simultaneously via npm run dev (using concurrently). They communicate through the REST API — the scraper polls /api/scraper/products to get a list of URLs to scrape, then posts results to /api/scraper/update.
Price-Scraper-Backend/
│
├── index.js # Entry point — starts Express server
├── supabaseClient.js # Initialises and exports the Supabase client
├── package.json
│
├── routes/
│ ├── middleware/
│ │ └── auth.js # JWT auth middleware (validates Supabase tokens)
│ ├── productRoutes.js # /api/products — user-facing CRUD
│ └── scraperRoutes.js # /api/scraper — internal scraper endpoints
│
├── scrapper/
│ └── scrape.js # Scraper entry point — polling loop + dispatch
│
└── scrapers/ # One folder per supported site
├── bewakoof/
├── culturecircle/
├── myntra/ # (Python-based scraper implementation)
├── nykaa/
├── snitch/
└── souled_store/
├── client.js # HTTP client / API helper for this site
├── queries.js # GraphQL / API query strings
├── normalize.js # Maps raw API response → standard product shape
└── handler.js # Public interface: exports getProduct(url)
- Starts Express on
PORT(default5000). - Attaches the Supabase client to the app via
app.set("supabase", supabase)so all route handlers can access it viareq.app.get("supabase"). - Mounts two routers:
/api/products→productRoutes.js(authenticated user routes)/api/scraper→scraperRoutes.js(internal, secret-protected)
- Calls
GET /api/scraper/products(with thex-scraper-secretheader) to get all tracked product URLs. - For each URL, calls
detectSite(url)to determine which scraper to use. - Dispatches to the correct
getProduct(url)handler insidescrapers/<site>/. - On success, calls
POST /api/scraper/updateto persistcurrent_price,name, andimage. - Sleeps 10 seconds when there are no products, otherwise loops immediately after finishing a batch.
- Failed scrapes are retried up to 3 times with a random 1–3 second delay between attempts.
Every site folder must export a single async function from handler.js:
// Returns a normalized product object or throws
export async function getProduct(url) { ... }The returned object must include at minimum:
| Field | Type | Description |
|---|---|---|
lowestPrice |
number |
Lowest available price in INR |
name |
string |
Product name / title |
featuredImage |
string |
URL of the main product image |
Any additional fields (variants, listings, etc.) are fine to include — they are currently not persisted but can be extended later.
Create a .env file in the project root. Never commit this file.
# Supabase
SUPABASE_URL=https://xxxxxxxxxxxx.supabase.co
SUPABASE_SERVICE_ROLE_KEY=eyJhbG... # service role key (bypasses RLS — keep secret!)
# Scraper ↔ API shared secret
# Both the scraper and the API must have the same value
SCRAPER_SECRET=some-long-random-string
# URL the scraper uses to reach the Express API
# When running locally this is usually:
API_URL=http://localhost:5000
# Frontend origin allowed by CORS
FRONTEND_URL=http://localhost:5173
# Server port (optional, defaults to 5000)
PORT=5000The scraper (scrapper/) has its own .env file too, but it only needs API_URL and SCRAPER_SECRET. These are already in the root .env which dotenv.config() picks up automatically.
# 1. Install dependencies
npm install
# 2. Create your .env (see above)
# 3. Start both the API server and the scraper together
npm run devIndividual scripts:
npm run start # API server only
npm run scraper # Scraper onlyFollow these steps to add support for a new site (e.g. kickscrew).
scrapers/
└── kickscrew/
├── client.js
├── queries.js # (optional — only needed if the site has complex queries)
├── normalize.js
└── handler.js
This is the only file the rest of the codebase cares about. It must export getProduct:
// scrapers/kickscrew/handler.js
import { normalize } from './normalize.js';
export async function getProduct(url) {
const slug = extractSlug(url);
// fetch raw data from the site's API here...
const rawData = await fetchSomething(slug);
return normalize(rawData);
}
function extractSlug(url) {
// Parse the last path segment, or whatever identifier the site uses
const segments = new URL(url).pathname.split('/').filter(Boolean);
return segments[segments.length - 1];
}Map the site's raw response to the standard shape. At minimum include lowestPrice, name, and featuredImage:
// scrapers/kickscrew/normalize.js
export function normalize(raw) {
return {
site: 'kickscrew',
slug: raw.slug,
name: raw.title,
brand: raw.brand,
lowestPrice: raw.price, // ← required
featuredImage: raw.thumbnail, // ← required
currency: 'INR',
scrapedAt: new Date().toISOString(),
// add whatever else is useful
};
}Set up your HTTP client here (axios, fetch, GraphQL, etc.). Keep API base URLs and headers here so handler.js stays clean.
// scrapers/kickscrew/client.js
import axios from 'axios';
const client = axios.create({
baseURL: 'https://api.kickscrew.com',
headers: {
'user-agent': 'Mozilla/5.0 ...',
}
});
export async function get(path, params = {}) {
const res = await client.get(path, { params });
return res.data;
}Two small edits:
// --- 1. Import the new handler at the top ---
import { getProduct as getKickscrew } from '../scrapers/kickscrew/handler.js';
// --- 2. Add a detection case ---
const detectSite = (url) => {
if (url.includes('culture-circle.com')) return 'culturecircle';
if (url.includes('thesouledstore.com')) return 'souledstore';
if (url.includes('bewakoof.com')) return 'bewakoof';
if (url.includes('snitch.com') || url.includes('snitch.co.in')) return 'snitch';
if (url.includes('nykaafashion.com')) return 'nykaafashion';
if (url.includes('kickscrew.com')) return 'kickscrew'; // ← add this
return null;
};
// --- 3. Add a dispatch case ---
const scrapeProduct = async (url) => {
const site = detectSite(url);
if (site === 'culturecircle') return await getProduct(url);
if (site === 'souledstore') return await souledStoreHandler(url);
if (site === 'bewakoof') return await bewakoofGetProduct(url);
if (site === 'snitch') return await snitchGetProduct(url);
if (site === 'nykaafashion') return await nykaafashionGetProduct(url);
if (site === 'kickscrew') return await getKickscrew(url); // ← add this
throw new Error(`No handler for URL: ${url}`);
};That's it — the scraper loop, retry logic, and database update are handled automatically.
All user-facing routes require a valid Supabase JWT in the Authorization: Bearer <token> header.
| Method | Path | Auth | Description |
|---|---|---|---|
POST |
/api/products |
✅ | Start tracking a product URL |
GET |
/api/products |
✅ | List all products the logged-in user is tracking |
DELETE |
/api/products/:productId |
✅ | Stop tracking a product |
POST /api/products body:
{
"url": "https://www.culture-circle.com/products/all/cloudtilt-black-eclipse",
"name": "optional initial name",
"image": "optional initial image url",
"price": 12000
}If the product URL already exists in the database, it is reused (no duplicate rows). If the user is already tracking it, the request succeeds silently.
Protected by the x-scraper-secret header (value must match SCRAPER_SECRET env var). These are called only by the scraper process, not by the frontend.
| Method | Path | Description |
|---|---|---|
GET |
/api/scraper/products |
Returns all product { id, url } rows |
POST |
/api/scraper/update |
Updates price, name, image for a product |
POST /api/scraper/update body:
{
"productId": "uuid",
"price": 11500,
"name": "On Cloudtilt Black Eclipse",
"image": "https://cdn.culture-circle.com/..."
}Two tables in Supabase (Postgres):
| Column | Type | Notes |
|---|---|---|
id |
uuid |
Primary key |
url |
text |
Unique product URL |
name |
text |
Product name (filled by scraper) |
image |
text |
Featured image URL (filled by scraper) |
current_price |
numeric |
Latest scraped price |
previous_price |
numeric |
Price before the last update |
last_checked |
timestamp |
When the scraper last ran for this product |
| Column | Type | Notes |
|---|---|---|
user_id |
uuid |
References Supabase auth.users |
product_id |
uuid |
References products.id |
The user_products table has a unique constraint on (user_id, product_id) to prevent duplicates.