Skip to content

Commit dc4f00c

Browse files
committed
feat(adapter): add rescue flights adapter (El Al + Israir)
Adds @browserkit/adapter-rescue-flights — a public adapter that queries real-time rescue flight availability from El Al and Israir. El Al scraper: - Scrapes the Angular CDK virtual-scroll seat availability page - Collects flight data incrementally during scroll to handle DOM recycling - Returns flight#, departure time, exact seat count for next 8 days Israir scraper (Phase 1+2): - Reads listing page DOM for all announced rescue flights - Calls Israir's internal priceBar API per unique route for seat counts and prices — one API call per route instead of one browser nav per flight - Intercepts FLIGHTS API responses via page.route() for exact flight#, departure time, and per-flight seat count on available dates - All API calls via page.evaluate() (Imperva TLS-fingerprint bound) Also adds: - Makefile with agent-check target - .claude/commands/agent-check.md — agent visual verification workflow - Updated README, CLAUDE.md, AGENTS.md, progress.md, browserkit.config.js - adapter-rescue-flights lives at jonzarecki/adapter-rescue-flights (not browserkit-dev org) Made-with: Cursor
1 parent fb899a3 commit dc4f00c

24 files changed

Lines changed: 1303 additions & 14 deletions

.claude/commands/agent-check.md

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# /agent-check
2+
3+
Verify the rescue-flights adapter scrapers against the live sites.
4+
Use the cursor-ide-browser MCP tools directly for all browser interaction.
5+
Loop until all checks pass.
6+
7+
## Tools to use
8+
9+
- `browser_navigate` — load each live URL
10+
- `browser_take_screenshot` — capture the rendered page for visual comparison
11+
- `browser_snapshot` — get the ARIA tree for structural checks
12+
- `browser_scroll` — scroll the El Al list to expose virtual-scroll rows
13+
14+
## Loop
15+
16+
### Step 1 — Run scrapers
17+
18+
```
19+
make agent-check
20+
```
21+
22+
This builds the adapter, runs all four scraper calls, and writes
23+
`agent-check-results.json` to the repo root.
24+
25+
Read the file. Note any entries where `availableSeats > 0` (El Al) and the
26+
full flight count (Israir).
27+
28+
### Step 2 — El Al visual check (`to_israel`)
29+
30+
1. `browser_navigate``https://www.elal.com/heb/seat-availability?d=1`
31+
2. Dismiss any cookie banner if present.
32+
3. Scroll down slowly (the list uses Angular CDK virtual scroll — rows only
33+
render when in viewport). Take screenshots as you scroll.
34+
4. For each flight row visible in the screenshots, verify the scraper JSON
35+
agrees on the seat count, especially any row showing a number > 0.
36+
37+
**Failure condition:** a flight row visible on screen shows seats > 0 but the
38+
scraper JSON says 0 or is missing that entry entirely.
39+
40+
### Step 3 — El Al visual check (`from_israel`)
41+
42+
Same as Step 2 with `?d=0`.
43+
44+
### Step 4 — Israir visual check (`to_tel_aviv`)
45+
46+
1. `browser_navigate``https://www.israir.co.il/Flights/Rescue_Flights/To_Tel_Aviv`
47+
2. Take a screenshot. Count the visible flight cards.
48+
3. Use `browser_snapshot` to confirm button count.
49+
4. Verify scraper JSON count matches and all `origin` fields are IATA codes
50+
(3 uppercase letters), not raw Hebrew text.
51+
52+
### Step 5 — Israir visual check (`from_tel_aviv`)
53+
54+
Same as Step 4 with the From_Tel_Aviv URL.
55+
**Important:** this page loads flight cards asynchronously — wait for them to
56+
appear before counting. If the screenshot shows skeleton/placeholder UI, wait
57+
and retry the screenshot.
58+
59+
### Step 6 — Evaluate
60+
61+
- **All checks pass** → report PASS and stop.
62+
- **Any check fails** → identify the exact discrepancy, fix the relevant
63+
source file in `packages/adapter-rescue-flights/src/`, then go back to Step 1.
64+
65+
## Notes
66+
67+
- Do not stop until the loop exits cleanly on a fresh run.
68+
- Transient page load failures are expected; just retry that step.
69+
- For El Al, the "show only flights with available seats" checkbox can be
70+
ticked to reduce noise and make the screenshot easier to read.

.context/progress.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,23 @@
11
# Progress
22

3+
## Session 6 — Rescue Flights Adapter (2026-03-30)
4+
5+
### `packages/adapter-rescue-flights` — new public adapter
6+
7+
Built a complete Phase 1+2 rescue flights adapter covering El Al and Israir:
8+
9+
- **El Al scraper** (`elal.ts`): Angular CDK virtual-scroll page. Scrolls in 300px steps and collects data at each position to handle DOM node recycling. Returns flight#, departure time, and exact seat count (0 = sold out) for all flights in the next 8 days.
10+
- **Israir scraper** (`israir.ts`): Two-phase approach.
11+
- Phase 1: reads listing page DOM for all announced rescue flights (58+ cards, two directions).
12+
- Phase 2: seeds a browser session, then calls Israir's internal `priceBar` API (`/api/results/priceBar`) per unique route for seat counts + prices across all dates. For available flights, intercepts the `FLIGHTS` API (`/api/search/FLIGHTS`) via `page.route()` to get exact flight#, departure time, and per-flight seat count.
13+
- Total time: ~45s for from_tel_aviv (51 flights), ~15s for to_tel_aviv listing-only.
14+
- All API calls via `page.evaluate()` — Imperva/TLS fingerprint bound, Node.js fetch rejected.
15+
- **`run-check.ts`** and **`Makefile`**: `make agent-check` calls adapter tools over MCP, dumps `agent-check-results.json`. Agent does visual verification using browser screenshots.
16+
- **14 unit tests** passing; integration tests updated for Phase 2 assertions.
17+
- **`browserkit.config.js`** updated: port 52746.
18+
- **`pnpm-workspace.yaml`** updated.
19+
- **README.md** updated with rescue flights entry in Available Adapters table.
20+
321
## Session 0 — Project Setup (2026-03-22)
422

523
- Synthesized SPEC.md from ChatGPT brainstorm conversation about authenticated MCP wrappers

.cursor/hooks/state/continual-learning-index.json

Lines changed: 12 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -53,21 +53,25 @@
5353
"mtimeMs": 1774450242000,
5454
"lastProcessedAt": "2026-03-25T16:00:00.000Z"
5555
},
56-
"/Users/jzarecki/.cursor/projects/Users-jzarecki-Projects-session-mcp/agent-transcripts/672ffd14-9bc9-4c91-9eb7-9759b40e8893/672ffd14-9bc9-4c91-9eb7-9759b40e8893.jsonl": {
57-
"mtimeMs": 1774522118000,
58-
"lastProcessedAt": "2026-03-26T00:00:00.000Z"
59-
},
6056
"/Users/jzarecki/.cursor/projects/Users-jzarecki-Projects-session-mcp/agent-transcripts/d03e9a64-5187-4bee-88df-279763fa977e/d03e9a64-5187-4bee-88df-279763fa977e.jsonl": {
6157
"mtimeMs": 1774474691000,
6258
"lastProcessedAt": "2026-03-26T00:00:00.000Z"
6359
},
60+
"/Users/jzarecki/.cursor/projects/Users-jzarecki-Projects-session-mcp/agent-transcripts/672ffd14-9bc9-4c91-9eb7-9759b40e8893/672ffd14-9bc9-4c91-9eb7-9759b40e8893.jsonl": {
61+
"mtimeMs": 1774790896000,
62+
"lastProcessedAt": "2026-03-30T13:25:00.000Z"
63+
},
6464
"/Users/jzarecki/.cursor/projects/Users-jzarecki-Projects-session-mcp/agent-transcripts/8511b06d-07c8-42f2-8f30-5b6762c2fdc6/8511b06d-07c8-42f2-8f30-5b6762c2fdc6.jsonl": {
65-
"mtimeMs": 1774523258000,
66-
"lastProcessedAt": "2026-03-26T00:00:00.000Z"
65+
"mtimeMs": 1774771011000,
66+
"lastProcessedAt": "2026-03-30T13:25:00.000Z"
6767
},
6868
"/Users/jzarecki/.cursor/projects/Users-jzarecki-Projects-session-mcp/agent-transcripts/9bec9e6f-a179-401c-90cc-6d64720fbfee/9bec9e6f-a179-401c-90cc-6d64720fbfee.jsonl": {
69-
"mtimeMs": 1774534587000,
70-
"lastProcessedAt": "2026-03-26T00:00:00.000Z"
69+
"mtimeMs": 1774769808000,
70+
"lastProcessedAt": "2026-03-30T13:25:00.000Z"
71+
},
72+
"/Users/jzarecki/.cursor/projects/Users-jzarecki-Projects-session-mcp/agent-transcripts/9e65043f-de65-4412-8de4-cb92e2d2d3be/9e65043f-de65-4412-8de4-cb92e2d2d3be.jsonl": {
73+
"mtimeMs": 1774901830000,
74+
"lastProcessedAt": "2026-03-30T15:00:00.000Z"
7175
}
7276
}
7377
}
Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
{
22
"version": 1,
3-
"lastRunAtMs": 1774534549568,
4-
"turnsSinceLastRun": 6,
5-
"lastTranscriptMtimeMs": 1774534549508,
6-
"lastProcessedGenerationId": "785251f5-f6dd-46fd-bd02-6b91246cd354",
3+
"lastRunAtMs": 1774901809573,
4+
"turnsSinceLastRun": 3,
5+
"lastTranscriptMtimeMs": 1774901809436,
6+
"lastProcessedGenerationId": "3000fdf6-5678-4931-8b00-e64206fb26f8",
77
"trialStartedAtMs": null
88
}

.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,9 @@ __pycache__/
1313
*.pyc
1414
.venv/
1515

16+
# agent-check output — runtime, not for commits
17+
agent-check-results.json
18+
1619
# OS
1720
.DS_Store
1821

AGENTS.md

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ Durable facts and correction patterns for this workspace. Updated by continual-l
3030
- Browser mode switching (`headless` / `watch` / `paused`), screenshot, page state, and navigate are consolidated into a single `browser` MCP tool with an `action` parameter — user explicitly asked to reduce tool count ("too many tools"); do NOT revert to 5 separate management tools
3131
- Management tools bypass the LockManager; regular automation tools go through it
3232
- "Raw" Playwright access means exposing the CDP WebSocket URL (`wsEndpoint()`) of each adapter's browser — external agents (Claude Code, Cursor) attach to the already-authenticated session and write their own Playwright scripts via shell
33-
- The Playwright skill pattern: AI writes a script to `/tmp`, executes it via shell — primary approach for shell-capable clients (Cursor, Claude Code); opt-in `run_script` MCP tool planned for clients without shell access (Claude Desktop)
33+
- The Playwright skill pattern: AI writes a script to `/tmp`, executes it via shell — primary approach for shell-capable clients (Cursor, Claude Code); `run_script` MCP tool **implemented** using `quickjs-emscripten` (sandboxed QuickJS) for clients without shell access (Claude Desktop)
3434
- MCP resources use `page://${site}/snapshot` (site name dynamic) — user pushed back when the URI appeared to hardcode the adapter name
3535
- Testing utilities (`createTestAdapterServer`, `createTestMcpClient`) live at `@browserkit/core/testing` subpath — a separate harness package was explicitly rejected ("I don't think we need it, it should be in either adapter or in core")
3636
- Real Chrome (`channel: "chrome"`) is required for Google-based adapters — Playwright's bundled Chromium is blocked by Google's login with "This browser or app may not be secure". `isLoggedIn` must NOT navigate during login polling or it redirects the user away from the sign-in page.
@@ -41,7 +41,8 @@ Durable facts and correction patterns for this workspace. Updated by continual-l
4141
- Framework navigates to `adapter.loginUrl` before calling `isLoggedIn()` when browser is at `about:blank` — adapters do NOT need to handle this themselves
4242
- `warm_up_browser()` (visiting google/wiki/github before login) was evaluated from stickerdaniel's code — decided as "nice to have" for first-time login, not adopted yet
4343
- `browserkit login <site>` is blocked by the `CI=1` env var that Cursor sets — must run as `CI="" node packages/core/dist/cli.js login <site>` to open a headed browser from within Cursor terminal
44-
- `browser` tool `snapshot` action is planned — returns incremental aria-snapshot diff, more token-efficient than screenshots; `page-snapshot` MCP resource already exists, the action adds diff support. Inspired by `SawyerHood/dev-browser`.
44+
- `browser` tool `snapshot` action is **implemented** — returns incremental aria-snapshot diff (`track: boolean` returns only lines changed since last call via set-difference); `page-snapshot` MCP resource shares the same `captureSnapshot()` helper. Inspired by `SawyerHood/dev-browser`.
45+
- `browserkit login --connect` flag is **implemented** — attaches to an existing CDP endpoint via `autoDiscoverCdpEndpoint()` in `session-manager.ts`; useful when Chrome is already running.
4546

4647
## Design Process Preferences
4748

@@ -61,6 +62,14 @@ Durable facts and correction patterns for this workspace. Updated by continual-l
6162
- Adapter roadmap priority: Reddit → Twitter/X (flagship, hardest bot detection) → Amazon (no consumer API at all) → Airbnb, Google Maps, Booking.com — all documented in the main README as planned adapters.
6263
- Reddit adapter is two-phase: Phase 1 = unauthenticated `old.reddit.com` (stable HTML class names, no login required); Phase 2 = authenticated (separate plan file). Target `old.reddit.com` exclusively — new Reddit is a React SPA with aggressive DOM churn.
6364
- Booking.com adapter was started (plan + Phase 1 architecture); user preference is to plan architecture first, then implement.
65+
- Booking.com adapter has **6 tools**: `get_upcoming_bookings`, `get_past_bookings`, `get_booking_details` (account tools) + `search_hotels`, `get_property`, `get_availability` (search/property tools). All 6 work in **headless mode** via CloakBrowser — no watch mode required.
66+
- **CloakBrowser** (`cloakbrowser` npm package) is in `@browserkit/core` `optionalDependencies` — lazily imported only when `antiDetection: { useCloakBrowser: true }` is set in adapter config. Provides 33 C++-level stealth patches that bypass DataDome. Do NOT use `channel: "chrome"` for Booking.com; use `useCloakBrowser: true` instead.
67+
- Global anti-detection patches applied to **all adapters** (unconditionally): `viewport: null` (avoids 1280×720 bot signal), `--blink-settings=primaryPointerType=4,...` (declares mouse at Blink level), `--window-size=1920,1080` (non-zero headless dimensions).
6468
- Phased adapter development pattern: Phase 1 is unauthenticated/mock, Phase 2 is authenticated/live. Add verification gates between phases.
6569
- Live scraping tests should run in GitHub CI via an external browser service (desired; specific service not yet chosen).
6670
- The main browserkit README doubles as the project's public-facing "blogpost" — user refers to it interchangeably; keep it polished and up-to-date with available + planned adapters
71+
- `adapter-rescue-flights` **has been implemented** (Phase 1 complete): scrapes El Al (`elal.com/heb/seat-availability?d=0`/`d=1`) via Angular DOM (`.seat-availability-list > .flight-group > .flight-item`) and Israir (`israir.co.il/Flights/Rescue_Flights/To_Tel_Aviv` / `/From_Tel_Aviv`) via internal APIs; both sites are public/unauthenticated.
72+
- Israir listing page shows flights as `button.flight-promotion.manual` elements — buttons have **no URLs and no data attributes**; clicking them opens a SPA booking widget, NOT a new page. Do NOT try to extract URLs from them.
73+
- Israir internal APIs discovered: `GET /api/results/priceBar?origin=TLV&destination=LCA&isOneWay=false&adults=1` (per-date seat counts + cheapest price) and `POST /api/search/FLIGHTS?origin=...&destination=...&startDate=...` with body `{"ignoredResults":0,"siteId":"isra2023"}` (flight number, departure/arrival times, exact seat count per segment).
74+
- Israir APIs require Imperva/Incapsula session cookies (`visid_incap_*`, `incap_ses_*`, `reese84`) bound to the browser's TLS fingerprint — Node.js `fetch` with copied cookies is rejected. Must call via `page.evaluate()` from within the browser context.
75+
- Israir scraping pattern (45s total vs 4 min naive): (1) read listing page DOM for all `button.flight-promotion.manual` cards, (2) seed-navigate to one search URL to establish Imperva session, (3) call `priceBar` API for all 6 destination cities in parallel via `page.evaluate()`, (4) intercept `POST /api/search/FLIGHTS` via route interception on 4 concurrent browser context pages for available dates only.

CLAUDE.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,20 @@ TBD — stack not decided yet (TypeScript + Playwright or Python + Playwright).
3535
- Run `/plan` to decide what to work on next
3636
- Run `/review` before committing
3737

38+
## Adapter verification (rescue-flights)
39+
40+
After **any** change to `packages/adapter-rescue-flights/src/`, run the `/agent-check`
41+
command. The agent will:
42+
43+
1. Run `make agent-check` — builds the adapter and dumps scraper output to
44+
`agent-check-results.json`.
45+
2. Open each live URL in the browser, take screenshots, and visually compare the
46+
screenshots against the scraper JSON.
47+
3. Fix any discrepancy in the scraper source, then repeat from step 1.
48+
49+
**Do not stop or commit until the visual comparison passes on all four checks.**
50+
The full workflow is documented in `.claude/commands/agent-check.md`.
51+
3852
## Forbidden patterns
3953

4054
- No `any` types (TS) or untyped functions (Python)

Makefile

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
PKG_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
2+
NODE := node
3+
PNPM := pnpm
4+
5+
.PHONY: build test agent-check
6+
7+
build:
8+
$(PNPM) --filter @browserkit/adapter-rescue-flights build
9+
10+
test: build
11+
$(PNPM) --filter @browserkit/adapter-rescue-flights test
12+
13+
## Call adapter tools over MCP HTTP (port 52746), dump results to agent-check-results.json.
14+
## Requires: browserkit start (adapter must already be running).
15+
agent-check: build
16+
@echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
17+
@echo " Rescue Flights — agent-check"
18+
@echo " Adapter must be running: browserkit start"
19+
@echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
20+
$(NODE) $(PKG_DIR)packages/adapter-rescue-flights/dist/run-check.js

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@ Configure your MCP client (Cursor, Claude Desktop, etc.):
4949
| [`@browserkit/adapter-hackernews`](https://github.com/browserkit-dev/adapter-hackernews) | Hacker News | none | `get_top`, `get_new`, `get_ask`, `get_show`, `get_comments` |
5050
| [`@browserkit/adapter-linkedin`](https://github.com/browserkit-dev/adapter-linkedin) | LinkedIn | required | `get_person_profile`, `get_company_profile`, `get_company_posts`, `search_people`, `search_jobs`, `get_job_details`, `get_feed` |
5151
| [`@browserkit/adapter-reddit`](https://github.com/browserkit-dev/adapter-reddit) | Reddit | none | `get_subreddit`, `get_thread`, `search`, `get_user` |
52+
| `@browserkit/adapter-rescue-flights` ([jonzarecki](https://github.com/jonzarecki/adapter-rescue-flights)) | El Al + Israir | none | `get_elal_flights`, `get_israir_flights` |
5253

5354
---
5455

browserkit.config.js

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ export default {
1111
channel: "chrome",
1212
},
1313
"/Users/jzarecki/Projects/session-mcp/packages/adapter-linkedin/dist/index.js": { port: 52744, channel: "chrome" },
14+
"@browserkit/adapter-rescue-flights": { port: 52746 },
1415
"/Users/jzarecki/Projects/browserkit-adapter-booking/dist/index.js": {
1516
port: 52745,
1617
// channel: "chrome" not used — CloakBrowser uses its own Chromium binary

0 commit comments

Comments
 (0)