Incubator Fund Scraper

A small tool that turns any portfolio-style website into a clean CSV. Built first to answer one question for myself: which Indian incubator still has grant money left?

Why this exists

I was looking for an incubator to apply into for my startup, and the Government of India runs the Startup India Seed Fund Scheme (SISFS). The official portfolio page lists ~200 incubators, each with their own allocated, disbursed, and remaining funds, but the page is built as a React app — you cannot Ctrl-F across all of them, you cannot sort by "who has the most grant left", and you definitely cannot export it.

Pitching to an incubator that has already burnt through its corpus is a waste of everyone's time. So I wrote this:

1. Paste the portfolio URL.
2. Hit Scrape.
3. Get a CSV with every incubator, the rupees still sitting in their grant pool, what sectors they fund, where they are, their evaluation score, and a direct link to their profile.

That alone made the project worth building. Then I generalised the backend so the same UI works for any other public listing site — job boards, grant portals, university directories, government registries, anything that's basically a list with structured rows.

What you get out of the box

For SISFS specifically, every row in the CSV has:

Column	What it means
incubator_name	Full legal name
state, city	Location
sectors	Sectors the incubator funds
first_total_approved_inr	First-round corpus approved by SISFS
reapply_total_approved_inr	Top-up corpus approved on re-application
total_approved_inr	Sum of the two above
remaining_total_inr	Total amount still unallocated
remaining_grant_inr	Grant rupees still available to startups
remaining_grant_lakhs	Same number in lakhs (₹1L = 100,000)
remaining_grant_crores	Same number in crores (₹1Cr = 10,000,000)
evaluation_score	SISFS evaluation score for the incubator
image_url, portfolio_url	Links back to the SISFS profile

Last run when I was building this: 193 incubators, ₹755.25 cr total approved across the scheme, about ₹51.23 cr of grant money still unallocated — concentrated in a surprisingly small set of incubators. That's the kind of insight that took me 30 seconds with this tool and would have taken hours of manual scrolling otherwise.

How it works for any other site

The backend dispatches by hostname:

• If the URL matches a registered site adapter, that adapter runs. Adapters call the underlying JSON API directly — fast, structured, no HTML parsing tax. The bundled SISFS adapter is one of these.
• If no adapter matches, the generic adapter kicks in:
  1. Spins up a headless Chromium via Playwright.
  2. Waits for the page to settle (networkidle + a small grace window) so client-rendered apps have time to draw.
  3. Tries to extract HTML tables first — <thead> / <tbody> rows become {column: value} objects.
  4. If there are no real tables, looks for repeating card / list patterns ([class*="card"], [class*="listing"], <article>, <li>) and emits {title, link, text} rows.
  5. If even that fails, falls back to a section-by-section dump of headings and paragraphs.

That covers the long tail: roughly anything you'd want to "just spreadsheet" — directories, leaderboards, tender lists, university fellowship pages, scheme dashboards, grant programmes. Plug in a new adapter for the cases where the generic strategy isn't structured enough; the rest of the app stays the same.

Tech

Layer	Stack
Frontend	React 18 + Vite
Backend	Node.js + Express + Playwright (headless Chromium) + Cheerio
Output	UTF-8 CSV via json2csv

No database, no queues, no auth — pure paste-and-go. Frontend talks to backend over /api, backend streams the CSV back as a download.

Project structure

incubator-fund-scraper/
├── client/                      # React + Vite app (port 5173)
│   ├── index.html
│   ├── vite.config.js           # proxies /api → :5174
│   ├── package.json
│   └── src/
│       ├── main.jsx
│       ├── App.jsx              # paste URL · scrape · summary cards · CSV
│       └── styles.css
├── server/                      # Express API (port 5174)
│   ├── package.json
│   └── src/
│       ├── index.js             # /api/health · /api/scrape · /api/csv
│       └── adapters/
│           ├── index.js         # hostname → adapter dispatcher
│           ├── startupindia.js  # SISFS portfolio adapter
│           └── generic.js       # Playwright + Cheerio fallback
├── docs/
│   └── screenshot.png
├── .gitignore
└── README.md

Install

You need:

• Node.js 20+ (tested on 24)
• npm 10+
• A few hundred MB of free disk for the headless Chromium binary that Playwright downloads on first install

git clone git@github.com:tuttucodes/incubator-fund-scraper.git
cd incubator-fund-scraper

# 1) backend — installs Express, Playwright, Cheerio, json2csv,
#    and downloads Chromium via the postinstall hook
cd server
npm install

# 2) frontend — React + Vite
cd ../client
npm install

The Playwright Chromium download runs automatically through the postinstall script in server/package.json. If you ever need to do it manually:

cd server
npx playwright install chromium

Run

Two terminals.

# terminal 1 — backend on :5174
cd server
npm run dev

# terminal 2 — frontend on :5173
cd client
npm run dev

Open http://localhost:5173, paste a URL, hit Scrape, then Download CSV.

Production-ish run for the backend:

cd server
npm start              # plain `node src/index.js`, no watcher

API

POST /api/scrape

{ "url": "https://seedfund.startupindia.gov.in/portfolio" }

Returns:

{
  "adapter": "startupindia-seedfund",
  "title": "SISFS Portfolio — Incubators",
  "columns": ["id", "incubator_name", "state", "city", "..."],
  "rows": [{ "...": "..." }],
  "count": 193,
  "summary": {
    "incubators": 193,
    "total_approved_inr": 7552500000,
    "total_remaining_grant_inr": 512336601,
    "total_remaining_grant_crores": 51.2337
  }
}

For unknown URLs, adapter is "generic" and the response includes a strategy field — "table", "list", or "text" — telling you which extraction path was used.

POST /api/csv

{ "rows": [ { "...": "..." } ], "filename": "data.csv" }

Returns the CSV as text/csv; charset=utf-8 with a Content-Disposition: attachment header.

Adding a new site adapter

When the generic strategy doesn't capture the structure you want, write a dedicated adapter. The contract is small:

1. Create server/src/adapters/yoursite.js. Export an async function(url) that returns:

   {
     source,       // string — the URL that was scraped
     title,        // string — human-readable page title
     columns,      // string[] — column order for the CSV
     rows,         // object[] — one row per item, keys must match columns
     count,        // number  — rows.length
     summary       // optional aggregate stats shown in the UI cards
   }

2. Register it in server/src/adapters/index.js:

   {
     name: "yoursite",
     match: (u) => /(^|\.)yoursite\.com$/i.test(u.hostname),
     run: scrapeYoursite,
   }

3. Restart the server.

Look at startupindia.js for a clean example — it hits the upstream JSON API, normalises raw paise amounts into INR / lakhs / crores, and computes a portfolio-level summary in one pass.

Configuration

Env var	Default	What it does
PORT	5174	Backend HTTP port

The frontend proxy in client/vite.config.js points to 5174 — change both if you change the port.

Notes on responsible use

• The generic adapter sets a datascraper/1.0 User-Agent. Swap it for something descriptive of your project before pointing it at sites that ask you to identify yourself.
• It does not crawl. One URL in, one page out. No link-following, no recursion.
• It reuses a single Chromium instance across requests to keep latency down. Restart the server to release it.
• Respect each target site's robots.txt and Terms of Service before hitting it. This is a personal-research tool, not a load test.

Roadmap

• Adapter for the Startup India recognised startups registry
• Adapter for state-level startup mission portals (Kerala, Karnataka, Maharashtra)
• Server-side pagination for very large generic-adapter results
• Streaming CSV download (skip the round-trip through JSON)
• Optional persistence so the same URL doesn't get re-scraped within an interval

License

MIT.