Skip to content

Restrict Devpost Scraper to Open-Source Hackathons Only #4

Description

@anmolsah
# Limit Scraper to Open-Source Hackathons Only

## Context
Contribly is a developer-first directory specifically focused on **open-source hackathons**. However, the current CLI feed scraper (`scripts/scrape.ts`) pulls a general list of all active hackathons from Devpost, regardless of whether they are open-source.

We need to tweak the scraper so that it only fetches and imports hackathons that are open-source.

## Suggested Approach
The scraper currently queries the Devpost public API inside `scripts/scrape.ts`. We can restrict this in one of two ways (or both):

### Option A: API Query Filtering (Recommended to test first)
Check if the Devpost API supports filtering by the "Open Source" theme directly in the query parameters. For example:

https://devpost.com/api/hackathons?status[]=upcoming&status[]=open&themes[]=Open+Source&per_page=15


### Option B: Code-Level Filtering
If the API does not support query filtering, filter the results inside the parsing loop. 

Inside `scripts/scrape.ts`, each hackathon object `h` contains a `themes` array of type `{ id: number; name: string }[]`. We can filter the parsed listings by checking if any theme matches `"Open Source"`.

For example, in the loop:
```typescript
const isOpenSource = h.themes?.some((theme: any) => theme.name.toLowerCase() === 'open source');
if (!isOpenSource) continue; // Skip non-open-source hackathons

Files to Tweak

How to Test

  1. Make sure you have installed dependencies:
    npm install
  2. Run the scraper:
    npm run scrape
  3. Run typescript checks to ensure no type issues:
    npm run typecheck
  4. Verify in the UI (npm run dev) or in app/lib/mockData.ts that only open-source hackathons are present.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions