diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 8a9f681..2e85875 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -1,4 +1,4 @@ -{"_type":"issue","id":"FlatPilot-3mt","title":"Add pagination to WG-Gesucht + Kleinanzeigen scrapers","description":"Currently both scrapers fetch only page 1, silently truncating ~80% of new listings on busy days. This is the #1 cause of \"missing flats from kleinanzeigen and wg gesucht\" reported by the user.\n\nConcrete evidence:\n- src/flatpilot/scrapers/wg_gesucht.py:146 hardcodes the URL to end with `.2.1.0.html` — the trailing `0` is the page index.\n- src/flatpilot/scrapers/wg_gesucht.py:8 docstring admits \"Phase 1 MVP only reads the search page.\"\n- src/flatpilot/scrapers/kleinanzeigen.py:155 `_search_url()` builds `/s-wohnung-mieten/{slug}/c203l{loc_id}{suffix}` with no `/seite:N` segment.\n- src/flatpilot/scrapers/inberlinwohnen.py:121 already paginates with `?page=N` — pattern exists, just not applied here.\n\nThe `known_external_ids: frozenset[str]` parameter is already threaded through both `fetch_new()` signatures (wg_gesucht.py:110, kleinanzeigen.py:113) but is unused.\n\nAcceptance criteria:\n- Both scrapers loop through pages until any of: (a) a page returns fewer than expected card count, (b) every card's external_id is in `known_external_ids`, or (c) a hard safety cap (suggest 5 pages).\n- WG-Gesucht: replace trailing `0` with page index 0, 1, 2, …\n- Kleinanzeigen: verify URL pattern — likely `/s-wohnung-mieten/{slug}/seite:N/c203l{loc_id}{suffix}` or query-string variant. Probe before coding.\n- Per-page polite delay (reuse the existing session machinery; do not bypass rate limits).\n- Tests: extend tests/test_wg_gesucht_scraper.py and tests/test_kleinanzeigen_scraper.py with multi-page fixture HTML to assert pagination terminates correctly on each of the three conditions above.","status":"in_progress","priority":1,"issue_type":"feature","assignee":"Mukhammad Ibrokhimov","owner":"ibrohimovmuhammad2020@gmail.com","created_at":"2026-05-15T18:02:28Z","created_by":"Mukhammad Ibrokhimov","updated_at":"2026-05-15T19:52:59Z","started_at":"2026-05-15T19:52:58Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"FlatPilot-3mt","title":"Add pagination to WG-Gesucht + Kleinanzeigen scrapers","description":"Currently both scrapers fetch only page 1, silently truncating ~80% of new listings on busy days. This is the #1 cause of \"missing flats from kleinanzeigen and wg gesucht\" reported by the user.\n\nConcrete evidence:\n- src/flatpilot/scrapers/wg_gesucht.py:146 hardcodes the URL to end with `.2.1.0.html` — the trailing `0` is the page index.\n- src/flatpilot/scrapers/wg_gesucht.py:8 docstring admits \"Phase 1 MVP only reads the search page.\"\n- src/flatpilot/scrapers/kleinanzeigen.py:155 `_search_url()` builds `/s-wohnung-mieten/{slug}/c203l{loc_id}{suffix}` with no `/seite:N` segment.\n- src/flatpilot/scrapers/inberlinwohnen.py:121 already paginates with `?page=N` — pattern exists, just not applied here.\n\nThe `known_external_ids: frozenset[str]` parameter is already threaded through both `fetch_new()` signatures (wg_gesucht.py:110, kleinanzeigen.py:113) but is unused.\n\nAcceptance criteria:\n- Both scrapers loop through pages until any of: (a) a page returns fewer than expected card count, (b) every card's external_id is in `known_external_ids`, or (c) a hard safety cap (suggest 5 pages).\n- WG-Gesucht: replace trailing `0` with page index 0, 1, 2, …\n- Kleinanzeigen: verify URL pattern — likely `/s-wohnung-mieten/{slug}/seite:N/c203l{loc_id}{suffix}` or query-string variant. Probe before coding.\n- Per-page polite delay (reuse the existing session machinery; do not bypass rate limits).\n- Tests: extend tests/test_wg_gesucht_scraper.py and tests/test_kleinanzeigen_scraper.py with multi-page fixture HTML to assert pagination terminates correctly on each of the three conditions above.","notes":"Shipped via PR #68 alongside bd-4p2.","status":"closed","priority":1,"issue_type":"feature","assignee":"Mukhammad Ibrokhimov","owner":"ibrohimovmuhammad2020@gmail.com","created_at":"2026-05-15T18:02:28Z","created_by":"Mukhammad Ibrokhimov","updated_at":"2026-05-24T15:42:44Z","started_at":"2026-05-15T19:52:58Z","closed_at":"2026-05-24T15:42:44Z","close_reason":"Shipped via PR #68 (merged 2026-05-23): both scrapers paginate up to MAX_PAGES=5 with the inberlinwohnen termination pattern; session.goto_with_retry wraps every pg.goto with jittered 1s/3s retries on null/5xx/exception. 48 new tests pass on main; CI green.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"FlatPilot-tgw","title":"auto-apply: classify expired listings, stop them poisoning cooldown queue","description":"When auto-apply hits a listing that's been deleted from the platform, the filler raises FormNotFoundError -\u003e apply.py records status=failed -\u003e cooldown_remaining_sec treats it as a real platform-throttle event -\u003e 120s cooldown locks the queue. Every subsequent matching flat is skipped while real platform-side throttling never even happened. Fix: introduce ListingExpiredError, raise it on HTTP 404/410, on Kleinanzeigen redirect away from /s-anzeige/, and on WG-Gesucht missing CTA. apply.py records as 'auto_skipped: listing_expired' so existing cooldown/failures-for-flat exclusions cover it. run_pipeline_apply also excludes such flats from selection for 7 days (TTL chosen over permanent exclusion so a future selector regression self-heals).","status":"closed","priority":1,"issue_type":"bug","assignee":"Mukhammad Ibrokhimov","owner":"ibrohimovmuhammad2020@gmail.com","created_at":"2026-05-06T15:57:55Z","created_by":"Mukhammad Ibrokhimov","updated_at":"2026-05-06T16:20:50Z","started_at":"2026-05-06T15:58:07Z","closed_at":"2026-05-06T16:20:50Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"FlatPilot-a9l","title":"doctor crashes on legacy DBs missing matched_saved_searches_json","description":"The user_id rebuild migration (758a31a) assumes matches has matched_saved_searches_json and applications has triggered_by_saved_search, but the COLUMNS forward-migration that added those was deleted in the same commit. DBs that predate the saved-searches feature crash with OperationalError: no such column: matched_saved_searches_json on init_db. Fix: restore COLUMNS entries and call ensure_columns() before _rebuild_user_scoped_tables.","notes":"Fixed in PR #42. Root cause: the user_id rebuild migration's INSERT SELECT references columns the COLUMNS forward-migration used to add — but that COLUMNS entry was deleted in the same commit. Fix moves ensure_columns() before _rebuild_user_scoped_tables and restores the COLUMNS entries.","status":"closed","priority":1,"issue_type":"bug","assignee":"Mukhammad Ibrokhimov","owner":"ibrohimovmuhammad2020@gmail.com","created_at":"2026-05-06T11:41:57Z","created_by":"Mukhammad Ibrokhimov","updated_at":"2026-05-06T12:50:55Z","started_at":"2026-05-06T11:42:08Z","closed_at":"2026-05-06T12:50:55Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"FlatPilot-k17","title":"WG-Gesucht filler blocked by #sec_advice modal on submit","description":"Reproduced 2026-05-05 against the live WG-Gesucht 'messenger' contact form (`form#messenger_form`). After filling the message, the submit click times out because a security-advice modal — `\u003cdiv id=\"sec_advice\" class=\"modal fade noprint mt50 mt-xs-10 in\"\u003e…\u003c/div\u003e` — is rendered above the page and intercepts pointer events. Playwright's auto-retry loop hits 58+ retries over 30s, then raises TimeoutError.\n\nTwo bugs:\n1. The filler (src/flatpilot/fillers/wg_gesucht.py) does not detect or dismiss `#sec_advice` before clicking `button[type=submit]`. The modal appears to fire on the first submit per session, possibly only for newly-logged-in accounts or accounts that haven't accepted the advice yet.\n2. The filler does not catch playwright.TimeoutError from `.click()`. The raw TimeoutError bubbles past auto_apply._try_flat's `except FillError`, past run_pipeline_apply, and is caught only at pipeline.run_pipeline_once as a generic 'apply failed: TimeoutError'. Because no FillError was raised, apply.py:298 does not write a `status='failed'` row to applications — the failure is invisible in the audit trail. Worse, the bubble-out aborts the rest of the auto-apply queue: flats #34 and #46 in the same run were never attempted.\n\nRepro:\n- Fresh `flatpilot login wg-gesucht` (newly authenticated cookies)\n- `flatpilot apply \u003cflat_id\u003e` against any active WG-Gesucht listing → submit click times out\n- applications table stays empty; apply_locks correctly released by `finally`\n\nFix scope:\n- Add CONSENT-style selector tuple to dismiss `#sec_advice` before submit (probably `#sec_advice button:has-text('Verstanden')` or similar — verify in headed mode).\n- Wrap submit click in try/except PlaywrightTimeoutError → raise SubmitVerificationError(...) so the apply pipeline records a failed row and continues to the next candidate.\n\nWithout (1) auto-apply is non-functional. Without (2) the failure mode is silent and corrupts queue ordering. Both should land together. Blocks the entire 'WG-Gesucht auto-apply works' claim — currently the only platform with a working filler at all.","status":"closed","priority":1,"issue_type":"bug","assignee":"Mukhammad Ibrokhimov","owner":"ibrohimovmuhammad2020@gmail.com","created_at":"2026-05-05T21:04:36Z","created_by":"Mukhammad Ibrokhimov","updated_at":"2026-05-05T21:57:25Z","started_at":"2026-05-05T21:50:16Z","closed_at":"2026-05-05T21:57:25Z","close_reason":"Fixed in branch fix/wg-gesucht-sec-advice-modal: pre-emptive #sec_advice dismissal + dismiss-and-retry on click timeout, with PlaywrightTimeoutError translated to SubmitVerificationError so apply pipeline records failed rows.","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -51,7 +51,7 @@ {"_type":"issue","id":"FlatPilot-rdss","title":"A. Project Foundation (Phase 0)","description":"New clean-room repo scaffolding for FlatPilot. Prereq for every other sub-epic.","status":"closed","priority":1,"issue_type":"epic","created_at":"2026-04-20T15:52:56Z","updated_at":"2026-04-22T10:30:09Z","closed_at":"2026-04-22T10:30:09Z","close_reason":"All child tasks merged; Phase 1 MVP complete","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"FlatPilot-33h","title":"Filter short-term / Zwischenmiete listings via title keywords","description":"User reports auto-applying to flats whose titles clearly say \"2 Wochen\", \"Zwischenmiete\", \"befristet\" etc. — wasting cap slots and sending obviously-wrong applications.\n\nRoot cause: `title` IS already captured on every flat (src/flatpilot/scrapers/base.py:35 — required field; populated at wg_gesucht.py:182 and kleinanzeigen.py:215), but no matcher filter inspects it. The existing `filter_contract` (src/flatpilot/matcher/filters.py:119-127) reads `flat[\"min_contract_months\"]`, which is on the detail page — and neither WG-Gesucht nor Kleinanzeigen scrapers fetch detail pages, so `min_contract_months` is `None` everywhere and the filter silently passes everything.\n\nAcceptance criteria:\n- New profile field: `exclude_short_term: bool = True` (default True for new wizards; ask in wizard/init.py).\n- New matcher filter `filter_short_term` in src/flatpilot/matcher/filters.py that scans `flat[\"title\"]` and `flat.get(\"description\", \"\")` (lowercased) for any of:\n - Keywords: `zwischenmiete`, `befristet`, `temporär`, `temporary`, `auf zeit`, `untermiete`, `short.?term`, `kurzzeit`, `möbliert auf zeit`\n - Numeric patterns: `\\b\\d+\\s*(woche|wochen|tage)\\b`, `\\b[1-5]\\s*monate?\\b` (1–5 months counts as short-term)\n - Date-range pattern: `(\\d{2}\\.\\d{2}\\.\\d{4})\\s*(bis|-)\\s*(\\d{2}\\.\\d{2}\\.\\d{4})` where the span \u003c `profile.min_contract_months * 30` days\n- Reason string returned on reject: `\"short_term_listing\"`.\n- Wire into `evaluate()` in matcher/runner.py alongside the other filters.\n- Tests (tests/test_matcher_filters.py): one positive + one negative per keyword class, one date-range edge case.\n- Keep total LOC ≤ 60 to honor CLAUDE.md matcher size budget.\n\nOut of scope (separate issue): URL-level WG-Gesucht category gating and detail-page enrichment for definitive `min_contract_months`.","status":"closed","priority":2,"issue_type":"feature","assignee":"Mukhammad Ibrokhimov","owner":"ibrohimovmuhammad2020@gmail.com","created_at":"2026-05-15T18:20:07Z","created_by":"Mukhammad Ibrokhimov","updated_at":"2026-05-19T09:47:49Z","started_at":"2026-05-16T20:20:08Z","closed_at":"2026-05-19T09:47:49Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"FlatPilot-nkb","title":"Make Nominatim negative-cache short-lived so geocode misses are recoverable","description":"A flaky Nominatim hit permanently buries a real flat. When Nominatim returns no results, the miss is cached with the same 180-day TTL as successful lookups. Future scrapes of the same address never retry, and the matcher silently rejects every such flat with `location_unknown`.\n\nEvidence:\n- src/flatpilot/matcher/distance.py:122 caches `{\"lat\": None, \"lng\": None}` on empty Nominatim results with default TTL.\n- src/flatpilot/matcher/filters.py:144 — `location_unknown` is a hard reject; user gets no audit trail.\n\nOptions (pick one in the implementation):\nA. Split TTL: positive hits keep 180 days; negative hits get ~24h. Simple, no schema change.\nB. Surface `geocode_failed` as a distinct match decision (not `reject`) so the doctor / dashboard can show a count and the user knows the flat exists but is in geocoding purgatory. Requires a small reclassifier on next pass.\n\nAcceptance criteria:\n- Geocode misses retry within 24h (option A) or are stored as deferred (option B).\n- Cache schema migration is forward-compatible (don't break the existing JSON file format).\n- Test: a cached miss older than the negative-cache TTL triggers a fresh Nominatim call.","status":"closed","priority":2,"issue_type":"bug","assignee":"Mukhammad Ibrokhimov","owner":"ibrohimovmuhammad2020@gmail.com","created_at":"2026-05-15T18:02:47Z","created_by":"Mukhammad Ibrokhimov","updated_at":"2026-05-19T09:47:49Z","started_at":"2026-05-18T21:02:02Z","closed_at":"2026-05-19T09:47:49Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"FlatPilot-4p2","title":"Retry transient HTTP failures in scrapers instead of yielding zero flats","description":"A single null response or HTTP ≥400 on the first page wipes the entire platform from a scrape pass. No retry, no partial result — the scraper returns early yielding nothing.\n\nEvidence:\n- src/flatpilot/scrapers/wg_gesucht.py:130-136 — null response or HTTP ≥400 → `return` (zero flats).\n- src/flatpilot/scrapers/kleinanzeigen.py:134-142 — same pattern.\n\nAcceptance criteria:\n- 2 retries with jittered backoff (suggest 1s, 3s) before giving up.\n- Distinguish transient (5xx, null, network error) from hard failures (4xx auth required, captcha, hard block) — only retry the transient class.\n- Existing block-detection raise path (kleinanzeigen.py:172) must still propagate to backoff machinery, not retry.\n- Add a test that asserts: first attempt 503, second attempt 200 → flats are yielded.\n- Refactor candidate: pull the retry+rate-limit-check wrapper into `scrapers/session.py` so future scrapers inherit it.","status":"in_progress","priority":2,"issue_type":"bug","assignee":"Mukhammad Ibrokhimov","owner":"ibrohimovmuhammad2020@gmail.com","created_at":"2026-05-15T18:02:38Z","created_by":"Mukhammad Ibrokhimov","updated_at":"2026-05-15T19:53:00Z","started_at":"2026-05-15T19:52:59Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"FlatPilot-4p2","title":"Retry transient HTTP failures in scrapers instead of yielding zero flats","description":"A single null response or HTTP ≥400 on the first page wipes the entire platform from a scrape pass. No retry, no partial result — the scraper returns early yielding nothing.\n\nEvidence:\n- src/flatpilot/scrapers/wg_gesucht.py:130-136 — null response or HTTP ≥400 → `return` (zero flats).\n- src/flatpilot/scrapers/kleinanzeigen.py:134-142 — same pattern.\n\nAcceptance criteria:\n- 2 retries with jittered backoff (suggest 1s, 3s) before giving up.\n- Distinguish transient (5xx, null, network error) from hard failures (4xx auth required, captcha, hard block) — only retry the transient class.\n- Existing block-detection raise path (kleinanzeigen.py:172) must still propagate to backoff machinery, not retry.\n- Add a test that asserts: first attempt 503, second attempt 200 → flats are yielded.\n- Refactor candidate: pull the retry+rate-limit-check wrapper into `scrapers/session.py` so future scrapers inherit it.","notes":"Shipped via PR #68 alongside bd-3mt.","status":"closed","priority":2,"issue_type":"bug","assignee":"Mukhammad Ibrokhimov","owner":"ibrohimovmuhammad2020@gmail.com","created_at":"2026-05-15T18:02:38Z","created_by":"Mukhammad Ibrokhimov","updated_at":"2026-05-24T15:42:44Z","started_at":"2026-05-15T19:52:59Z","closed_at":"2026-05-24T15:42:44Z","close_reason":"Shipped via PR #68 (merged 2026-05-23): both scrapers paginate up to MAX_PAGES=5 with the inberlinwohnen termination pattern; session.goto_with_retry wraps every pg.goto with jittered 1s/3s retries on null/5xx/exception. 48 new tests pass on main; CI green.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"FlatPilot-rv2","title":"Auto-apply: --drain loops until all reachable platforms hit cap, prints failure summary on exit","description":"Currently `flatpilot run --drain` exhausts the matched-flat queue and exits, often well below daily caps (user just saw 5/20 wg-gesucht submits in one run, then quit because the queue was empty).\n\nGoal: a single invocation that scrape→match→apply repeatedly, sleeping cooldowns, until every \"reachable\" platform's daily cap is met (or 2 consecutive passes add zero new applyable flats). On any exit path — caps reached, empty-streak, SIGINT/SIGTERM — print a grouped, deduped summary of per-flat failures so the user can fix filler bugs.\n\nOut of scope: changing `flatpilot run` (no flags) one-pass behavior, changing `--watch` semantics, file/Telegram summary delivery (terminal-only for now).\n\nDesign: `docs/superpowers/specs/2026-05-14-auto-apply-drain-loop-design.md`","status":"closed","priority":2,"issue_type":"feature","assignee":"Mukhammad Ibrokhimov","owner":"ibrohimovmuhammad2020@gmail.com","created_at":"2026-05-14T12:04:27Z","created_by":"Mukhammad Ibrokhimov","updated_at":"2026-05-14T12:10:15Z","started_at":"2026-05-14T12:04:30Z","closed_at":"2026-05-14T12:10:15Z","close_reason":"Implemented in PR #57 (feat/auto-apply-drain-loop): --drain now loops until all reachable platforms hit cap, prints failure summary on exit","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"FlatPilot-289","title":"auto-apply: stop spamming 'filler not registered' rows on every pass","description":"## Symptom\n\nThe applications table on the dev box has **310 rows** for inberlinwohnen of the form:\n\n status=failed notes='auto_skipped: filler not registered for platform '\\''inberlinwohnen'\\'''\n\n(observed 2026-05-14). Every \\`run_pipeline_apply\\` pass re-queries every matched flat, hits the \\`completeness_ok\\` → \\`get_filler(platform)\\` LookupError branch, and writes a fresh \\`auto_skipped: filler not registered\\` row. The exclusion clauses in the queue-selector SQL (\\`auto_apply.py:188–218\\`) cover submitted rows and recent listing_expired rows but NOT this case, so the spam grows unbounded.\n\n## Why this is wrong\n\n- 310 rows × N future passes = unbounded growth for a state that is permanent (until a filler is implemented). FlatPilot-6pq is filed as backlog with \"probably not worth it\" — the filler is not coming soon.\n- Dashboard / status command have to filter past this noise on every render.\n- Each pass burns a tiny amount of work (SQL row + console line) for zero diagnostic value past the first occurrence.\n\n## Plan\n\nIn \\`src/flatpilot/auto_apply.py::run_pipeline_apply\\`, extend the queue-selector SQL with one more \\`NOT EXISTS\\` clause analogous to the listing_expired exclusion — but with no TTL (a missing filler is permanent until code lands):\n\n\\`\\`\\`sql\nAND NOT EXISTS (\n SELECT 1 FROM applications a\n WHERE a.flat_id = m.flat_id\n AND a.user_id = ?\n AND a.method = 'auto'\n AND a.status = 'failed'\n AND a.notes LIKE 'auto_skipped: filler not registered%'\n)\n\\`\\`\\`\n\nResult: one row per flat per missing-filler platform (preserves first-time visibility); zero growth thereafter. If \\`FlatPilot-6pq\\` is ever picked up and a filler lands, a one-off cleanup query can delete those rows to re-enable the flats — out of scope here.\n\n## Acceptance\n\n1. New pipeline pass over the existing dataset does NOT add any new \\`auto_skipped: filler not registered\\` rows for inberlinwohnen flats that already have one.\n2. New inberlinwohnen matches that have never been processed still get exactly one \\`auto_skipped: filler not registered\\` row (for visibility).\n3. Unit / integration test in \\`tests/test_run_pipeline_apply.py\\` asserting: pre-existing filler-not-registered row → flat excluded from the queue scan.\n4. Existing listing_expired exclusion test stays green.\n\n## Out of scope\n\n- Backfill cleanup of the 310 existing rows. They are already correctly classified as auto_skipped; if the dashboard becomes too cluttered, a one-shot delete query is a follow-up.\n- Implementing the inberlinwohnen filler itself — that is FlatPilot-6pq (backlog).\n\n## Related\n\n- FlatPilot-6pq — inberlinwohnen / immoscout24 filler decision (backlog).\n- FlatPilot-tgw — same exclusion pattern, for listing_expired (with TTL).\n- FlatPilot-8kt — out-of-scope note mentioned this row spam as a separate bead candidate.","status":"closed","priority":2,"issue_type":"bug","assignee":"Mukhammad Ibrokhimov","owner":"ibrohimovmuhammad2020@gmail.com","created_at":"2026-05-14T10:47:13Z","created_by":"Mukhammad Ibrokhimov","updated_at":"2026-05-14T10:53:10Z","started_at":"2026-05-14T10:47:58Z","closed_at":"2026-05-14T10:53:10Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"FlatPilot-b13","title":"kleinanzeigen filler: surface HTTP status / server-error code on submit failure","description":"## Symptom (observed 2026-05-09)\n\nThree identical failures on flat 590 (Terrassenwohnung Berlin-Dahlem) in `applications`:\n\n 196 failed kleinanzeigen: neither success nor error indicator appeared within 15000ms after submit ...\n 171 failed kleinanzeigen: neither success nor error indicator appeared within 7000ms after submit ...\n 146 failed ... same\n\nRow 196 is *after* PR #52 bumped the success timeout from 7s to 15s, so the timeout bump alone did not resolve it.\n\n## Screenshot evidence\n\nFailure screenshot at ~/.flatpilot/screenshots/kleinanzeigen/3394499312-203-26318-20260509T222001.png shows a **full-page kleinanzeigen server-error page**:\n\n Fehler [400]\n Huch, da ist ein Fehler (400) aufgetreten\n\nNOT inline-validation errors on un-filled structured fields (which was the hypothesis behind FlatPilot-ic1). The submit POST is reaching the kleinanzeigen server and being rejected with HTTP 400; the filler sees neither the success banner nor the inline error banner because the page has navigated away to a server-side error template.\n\n## Why this matters\n\nThe existing `SubmitVerificationError` message (\"neither success nor error indicator appeared\") is misleading for this failure mode — it implies the page is silent, when in fact it is loudly showing a 400 page. That misleading signal pushed scoping toward FlatPilot-ic1 (Profile schema rewrite to fill 15 structured fields) as the cause; FlatPilot-ic1 may or may not be the right fix, but we cannot tell until we know whether the 400 is from missing form data, CSRF, anti-bot, or something else.\n\n## Plan\n\nMirror the FlatPilot-8kt pattern: ship the diagnostic first, then commit to a fix from evidence.\n\nIn `flatpilot/fillers/kleinanzeigen.py::_verify_submitted`, after the success-marker timeout and before the catch-all \"neither indicator\" branch, detect the full-page server-error state and surface the status code:\n\n1. Add a `server_error_marker` selector to `SELECTORS` that matches the error-page heading (text-based locator on `Fehler [\\d+]`).\n2. If the marker is visible, extract the bracketed status code (regex `Fehler \\[(\\d+)\\]`) from the page text and raise `SubmitVerificationError` with a message that includes both the code and the form URL.\n3. Failure screenshot continues to capture regardless (existing best-effort helper).\n\n## Acceptance\n\n1. When the post-submit page renders `Fehler [400]`, the filler raises `SubmitVerificationError` whose message includes `HTTP 400` (or `Fehler [400]`) and the contact URL — not the generic \"neither success nor error indicator\" message.\n2. Unit test in `tests/test_filler_kleinanzeigen.py` covers the new branch: submit click → page text contains `Fehler [400]` → `SubmitVerificationError` with the status code in the exception message.\n3. Existing branches (success banner; error banner; neither) remain green.\n4. The reclassified failure note lets us decide between FlatPilot-ic1 and other root-cause fixes on the next live failure.\n\n## Out of scope\n\n- Capturing the HTTP response body from the submit POST via Playwright `page.on(\"response\", ...)` — possible follow-up if the page-text marker proves unreliable.\n- The actual fix for whatever causes the 400. That is FlatPilot-ic1 or a sibling bead, decided post-diagnostic.\n\n## Related\n\n- FlatPilot-ic1 — premised on missing-required-fields; needs this diagnostic to confirm or refute.\n- FlatPilot-8kt — same diagnostic-first pattern.","status":"closed","priority":2,"issue_type":"bug","assignee":"Mukhammad Ibrokhimov","owner":"ibrohimovmuhammad2020@gmail.com","created_at":"2026-05-14T10:37:12Z","created_by":"Mukhammad Ibrokhimov","updated_at":"2026-05-14T10:53:10Z","started_at":"2026-05-14T10:37:16Z","closed_at":"2026-05-14T10:53:10Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -74,7 +74,7 @@ {"_type":"issue","id":"FlatPilot-t2p3","title":"Test Epic","description":"parent","status":"closed","priority":2,"issue_type":"epic","created_at":"2026-04-20T15:48:34Z","updated_at":"2026-04-20T15:48:53Z","closed_at":"2026-04-20T15:48:53Z","close_reason":"test graph schema","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"FlatPilot-l27z","title":"Test","description":"test","status":"closed","priority":2,"issue_type":"task","created_at":"2026-04-20T15:48:12Z","updated_at":"2026-04-20T15:48:23Z","closed_at":"2026-04-20T15:48:23Z","close_reason":"test cleanup","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"FlatPilot-997r","title":"Test","description":"test","status":"closed","priority":2,"issue_type":"task","created_at":"2026-04-20T15:47:25Z","updated_at":"2026-04-20T15:48:05Z","closed_at":"2026-04-20T15:48:05Z","close_reason":"test cleanup","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"FlatPilot-569","title":"ic1 Phase 2: iterate kleinanzeigen contact form and fill structured fields from profile.contact_details","description":"Follow-up to bd-ic1 Phase 1 (PR shipping the schema + wizard + Filler protocol thread-through). Phase 2 is the actual filler change.\n\nScope:\n- After _reveal_contact_form, iterate every visible input/select/textarea under form#viewad-contact-form.\n- For each field, read its name / id / associated \u003clabel for=...\u003e text.\n- Match against a configurable label-keyword map (Anrede, Vorname, Nachname, Telefon, Straße, PLZ, SCHUFA, Haushalt, ...) → profile.contact_details attribute.\n- Fill the field if it's empty and the profile defines a value. Skip otherwise. For \u003cselect\u003e, use select_option().\n- Add fixture HTML for the 15-field variant and the message-only variant (capture from a real listing).\n- Tests assert each variant is filled correctly and that fields not in the profile stay untouched.\n\nPrerequisite: empirical capture of real kleinanzeigen contact-form HTML for both variants. Without that, the label keywords and selector strategy are guesses.","status":"in_progress","priority":3,"issue_type":"feature","assignee":"Mukhammad Ibrokhimov","owner":"ibrohimovmuhammad2020@gmail.com","created_at":"2026-05-19T13:42:03Z","created_by":"Mukhammad Ibrokhimov","updated_at":"2026-05-20T20:50:51Z","started_at":"2026-05-20T20:50:51Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"FlatPilot-569","title":"ic1 Phase 2: iterate kleinanzeigen contact form and fill structured fields from profile.contact_details","description":"Follow-up to bd-ic1 Phase 1 (PR shipping the schema + wizard + Filler protocol thread-through). Phase 2 is the actual filler change.\n\nScope:\n- After _reveal_contact_form, iterate every visible input/select/textarea under form#viewad-contact-form.\n- For each field, read its name / id / associated \u003clabel for=...\u003e text.\n- Match against a configurable label-keyword map (Anrede, Vorname, Nachname, Telefon, Straße, PLZ, SCHUFA, Haushalt, ...) → profile.contact_details attribute.\n- Fill the field if it's empty and the profile defines a value. Skip otherwise. For \u003cselect\u003e, use select_option().\n- Add fixture HTML for the 15-field variant and the message-only variant (capture from a real listing).\n- Tests assert each variant is filled correctly and that fields not in the profile stay untouched.\n\nPrerequisite: empirical capture of real kleinanzeigen contact-form HTML for both variants. Without that, the label keywords and selector strategy are guesses.","status":"closed","priority":3,"issue_type":"feature","assignee":"Mukhammad Ibrokhimov","owner":"ibrohimovmuhammad2020@gmail.com","created_at":"2026-05-19T13:42:03Z","created_by":"Mukhammad Ibrokhimov","updated_at":"2026-05-20T21:16:38Z","started_at":"2026-05-20T20:50:51Z","closed_at":"2026-05-20T21:16:38Z","close_reason":"Shipped via PR #67: kleinanzeigen_form planner, executor wiring with empty-field guard, full + message-only fixtures.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"FlatPilot-k1v","title":"ko1 Part B: WG-Gesucht/Kleinanzeigen detail-page enrichment for min_contract_months","description":"Follow-up to bd-ko1 Part A (URL gating shipped via PR #63). Part B was scoped out of that PR to keep the diff small.\n\nWhat's still needed (verbatim from the original bd-ko1):\n\nFor flats that pass all cheap filters, fetch the detail page (politely, with the existing session machinery) and parse the explicit fields:\n- WG-Gesucht: Mindestmietdauer, Befristet: Ja/Nein, Verfügbar bis\n- Kleinanzeigen: Mietdauer, the 'Befristet' toggle if present\n\nPopulate flat['min_contract_months'] so the existing filter_contract (filters.py:119-127) actually fires. Cap the per-run detail-fetch budget (e.g. 30 flats/run) to keep latency and politeness in check.\n\nAcceptance:\n- At least one new test asserts that a detail page with 'Mindestmietdauer: 24 Monate' populates min_contract_months=24 on the flat row.\n- New flat field available_until populated when present (cross-checks the date-range short-term heuristic from bd-33h).\n- Detail fetch failure must not poison the flat — fall back to the bd-33h keyword filter.","status":"closed","priority":3,"issue_type":"feature","assignee":"Mukhammad Ibrokhimov","owner":"ibrohimovmuhammad2020@gmail.com","created_at":"2026-05-19T10:03:16Z","created_by":"Mukhammad Ibrokhimov","updated_at":"2026-05-20T20:49:21Z","started_at":"2026-05-20T17:44:50Z","closed_at":"2026-05-20T20:49:21Z","close_reason":"Shipped via PR #66: enrich pipeline stage, parse_detail/fetch_detail for both scrapers, available_until column, filter_short_term structured-date path, flatpilot enrich CLI.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"FlatPilot-ko1","title":"Definitive long-term gating: WG-Gesucht category URL + detail-page enrichment","description":"Follow-up to the title-keyword short-term filter (cheap heuristic). For a definitive signal, fetch detail-page data so the existing `filter_contract` (filters.py:119-127) actually has `min_contract_months` to compare against.\n\nTwo parts:\n\nPART A — WG-Gesucht URL-level category gating (low risk, no detail fetch)\n- WG-Gesucht search URL is `/wohnungen-in-\u003cslug\u003e.\u003ccity_id\u003e.2.1.0.html` (src/flatpilot/scrapers/wg_gesucht.py:146).\n- The `2` is the \"Wohnung / full flat\" category. WG-Gesucht has a separate URL space for \"Befristete WG / Zwischenmiete\". Verify in `scripts/wg_probe.py` whether the current URL already excludes Zwischenmiete or if there's a finer-grained flag (e.g. `rent_type=long_term`) we should add.\n- If the current URL already filters this out: document it in the scraper docstring and close part A.\n- If not: add the appropriate query param or path segment, gated by `profile.exclude_short_term`.\n\nPART B — detail-page enrichment for matched flats only\n- For flats that pass all cheap filters, fetch the detail page (politely, with the existing session machinery).\n- Parse the explicit fields:\n - WG-Gesucht: `Mindestmietdauer`, `Befristet: Ja/Nein`, `Verfügbar bis`\n - Kleinanzeigen: `Mietdauer`, the \"Befristet\" toggle if present\n- Populate `flat[\"min_contract_months\"]` so the existing `filter_contract` filter actually fires.\n- Cap the per-run detail-fetch budget (e.g. 30 flats/run) to keep latency and politeness in check.\n- This is the same enrichment the wg_gesucht.py:8 docstring already flags as a future pass.\n\nAcceptance criteria:\n- Part A: a probe run confirms long-term-only URL behavior OR a URL change is shipped + tested.\n- Part B: at least one new test asserts that a detail page with \"Mindestmietdauer: 24 Monate\" populates `min_contract_months=24` on the flat row.\n- New flat field `available_until` populated when present (for the date-range short-term heuristic in the keyword-filter issue to cross-check).\n- Detail fetch failure must not poison the flat — fall back to the keyword filter from the sibling issue.","status":"closed","priority":3,"issue_type":"feature","assignee":"Mukhammad Ibrokhimov","owner":"ibrohimovmuhammad2020@gmail.com","created_at":"2026-05-15T18:20:21Z","created_by":"Mukhammad Ibrokhimov","updated_at":"2026-05-19T10:03:28Z","started_at":"2026-05-19T09:59:45Z","closed_at":"2026-05-19T10:03:28Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"FlatPilot-ykg","title":"Log matcher reject reasons at DEBUG with external_id for audit","description":"Matcher rejections are persisted to `matches.decision_reasons_json` but never logged. `flatpilot doctor` shows no per-flat reason; users have to query SQLite to figure out why a specific flat was dropped.\n\nEvidence:\n- src/flatpilot/matcher/filters.py — every reject reason (rent_too_high, rooms_unknown, district_unknown, location_unknown, no_pets_allowed, move_in_too_late, etc.) returns a string but no log line.\n- src/flatpilot/matcher/runner.py:76-80 — the decision is computed and persisted but not surfaced.\n\nAcceptance criteria:\n- Single `logger.debug` line per flat with `external_id`, `platform`, decision, and `reasons`.\n- Goes through the existing `flatpilot.log` config — `--verbose` or `FLATPILOT_LOG_LEVEL=DEBUG` switches it on.\n- No new dependency, no perf hit at INFO level.\n- Optional follow-up: `flatpilot doctor --explain \u003cexternal_id\u003e` to print the same info from DB without needing logs replayed.","status":"closed","priority":3,"issue_type":"feature","assignee":"Mukhammad Ibrokhimov","owner":"ibrohimovmuhammad2020@gmail.com","created_at":"2026-05-15T18:03:18Z","created_by":"Mukhammad Ibrokhimov","updated_at":"2026-05-19T09:47:49Z","started_at":"2026-05-16T20:20:09Z","closed_at":"2026-05-19T09:47:49Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -148,12 +148,12 @@ {"_type":"issue","id":"FlatPilot-u5gh","title":"O4. Docker-compose + deployment docs","description":"One-command local stack; deployment instructions for a VPS.","status":"open","priority":4,"issue_type":"task","created_at":"2026-04-20T15:52:57Z","updated_at":"2026-04-20T15:52:57Z","dependencies":[{"issue_id":"FlatPilot-u5gh","depends_on_id":"FlatPilot-x0pq","type":"blocks","created_at":"2026-04-20T17:52:57Z","created_by":"Mukhammad Ibrokhimov","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"FlatPilot-x0pq","title":"O1. Architecture decision record","description":"Decide: FastAPI + Next.js; magic-link auth; SQLite -\u003e Postgres migration path; deployment story (docker-compose starter).","status":"open","priority":4,"issue_type":"task","assignee":"Mukhammad Ibrokhimov","created_at":"2026-04-20T15:52:57Z","updated_at":"2026-05-05T21:08:19Z","started_at":"2026-05-03T15:38:59Z","dependency_count":0,"dependent_count":3,"comment_count":0} {"_type":"issue","id":"FlatPilot-z3me","title":"O2. Per-user isolation in data model","description":"All tables gain user_id; row-level scoping; migration from single-user state.","status":"open","priority":4,"issue_type":"task","created_at":"2026-04-20T15:52:57Z","updated_at":"2026-04-20T15:52:57Z","dependencies":[{"issue_id":"FlatPilot-z3me","depends_on_id":"FlatPilot-x0pq","type":"blocks","created_at":"2026-04-20T17:52:57Z","created_by":"Mukhammad Ibrokhimov","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} -{"_type":"memory","key":"after-a-flatpilot-pr-merges-do-not-delete","value":"After a FlatPilot PR merges, DO NOT delete the local or remote feature branch, and DO NOT drop stashes, unless the user explicitly asks. Why: user's standing preference to preserve merged branches for historical reference (e.g. feat/l2-anschreiben-composer was called out as hands-off in the handoff). How to apply: after merge, only sync main (git checkout main \u0026\u0026 git pull --ff-only) and stop. Do not run 'git branch -d', 'git push --delete', 'git stash drop', or any similar cleanup without an explicit request. Treat 'cleanup' as scope the user opts into, not a default step."} -{"_type":"memory","key":"clean-room-policy-flatpilot-is-independent-mit-licensed","value":"Clean-room policy: FlatPilot is independent MIT-licensed. Never copy code from any other project (including any job-hunting agent that may have inspired the high-level architecture). Read outside code only as a design reference; always rewrite from scratch."} -{"_type":"memory","key":"flatpilot-no-credential-storage","value":"Never store user passwords for rental platforms (WG-Gesucht, Kleinanzeigen, ImmoScout24, future) anywhere in FlatPilot — not in profile.json, .env, or any other file. Reasons: (1) Docker container can't reach host OS keyring, leaving only file storage which is unsafe for credentials; (2) programmatic credential login is a strong anti-bot signal that risks getting accounts flagged; (3) 2FA / captchas still require human intervention even with stored passwords; (4) ToS posture is much worse than cookie reuse from a real human session. Solution: cookies-only auth. CLI version (FlatPilot-5b8) uses headed Playwright + 'press Enter when logged in' UX. Future multi-user Web UI (FlatPilot-8jx) uses browser extension or bookmarklet to capture cookies from the user's normal browser session — never asks for or stores passwords. None of WG-Gesucht / Kleinanzeigen / ImmoScout24 publish OAuth endpoints, so a literal 'Sign in with Platform' redirect-and-callback flow is structurally impossible. Decided in PR #18 session 2026-04-22."} +{"_type":"memory","key":"no-ai-co-author-trailers-in-commits-no","value":"No AI co-author trailers in commits. No 'Co-Authored-By: Claude ...', no '🤖 Generated with Claude Code'. Commits are authored by the human only."} {"_type":"memory","key":"flatpilot-product-direction","value":"FlatPilot is currently single-user CLI (Phase 3). Long-term direction: convert to small multi-user web app — 'Option C' scope — for friends-of-friends, no public marketing, no payments. Strict sequencing: complete Phase 3 CLI for the developer's own use FIRST, then validate by using it personally on real WG-Gesucht searches for 1-2 weeks, THEN start Epic O (Web UI / FlatPilot-0wfb). Do NOT start FastAPI / React / Postgres work until Phase 3 is shipped end-to-end and personally validated. The CLI work is not throwaway — every Python module becomes a backend module the FastAPI app calls later. Phase 5 will need: Hetzner/Fly hosting, Postgres migration, FastAPI backend, React frontend, magic-link email auth, browser extension OR bookmarklet for per-user platform cookie capture. Decided in PR #18 session 2026-04-22."} +{"_type":"memory","key":"commit-author-for-this-repo-mukhammad-ibrokhimov-ibrohimovmu","value":"Commit author for this repo: Mukhammad Ibrokhimov \u003cibrohimovmuhammad2020@gmail.com\u003e. In a fresh clone: git config user.email ibrohimovmuhammad2020@gmail.com \u0026\u0026 git config user.name 'Mukhammad Ibrokhimov'. Any other email on the machine is NOT for commits here."} {"_type":"memory","key":"flatpilot-prompt-ride-along-checklist","value":"When the user's prompt has a 'Possible ride-alongs' section that explicitly lists candidates for inline inclusion, treat each listed item as a TASK ITEM, not just background reading. Add it to the working checklist before starting implementation, and verify each one is either (a) included in the PR or (b) explicitly justified as skipped before declaring the work done. Why: in PR #30 (FlatPilot-etu) the user explicitly suggested 'A README note about what flatpilot scrape --platform inberlinwohnen now ingests on fresh install → ride along inline' and I dropped it; only caught it when the user asked 'do we have any more related PR tasks'. How to apply: when reading any FlatPilot prompt, scan for 'ride-along' / 'inline' / 'fix inline' suggestions, list them in your initial pitch back to the user, and tick them off as you go."} -{"_type":"memory","key":"no-ai-co-author-trailers-in-commits-no","value":"No AI co-author trailers in commits. No 'Co-Authored-By: Claude ...', no '🤖 Generated with Claude Code'. Commits are authored by the human only."} {"_type":"memory","key":"for-flatpilot-batch-several-small-beads-tasks-into","value":"For FlatPilot: one PR per epic. If an epic has \u003e10 tasks, split into multiple PRs (thematic split, topologically ordered). Otherwise the whole epic lands in one PR — one commit per task, each commit message referencing its beads ID. Small leftover tasks from completed sub-areas can ride along if they fit the theme. Why: the user set this rule after observing that per-task PRs were noise and mixed-theme mega-PRs were hard to review. How to apply: before claiming the first task in an epic, count the epic's open tasks with 'bd list' filtered to that epic; if \u003e10, decide on a thematic split (e.g. 'matcher core' vs 'matcher CLI') and declare it in the first PR description. Each PR closes every beads task it contains."} +{"_type":"memory","key":"after-a-flatpilot-pr-merges-do-not-delete","value":"After a FlatPilot PR merges, DO NOT delete the local or remote feature branch, and DO NOT drop stashes, unless the user explicitly asks. Why: user's standing preference to preserve merged branches for historical reference (e.g. feat/l2-anschreiben-composer was called out as hands-off in the handoff). How to apply: after merge, only sync main (git checkout main \u0026\u0026 git pull --ff-only) and stop. Do not run 'git branch -d', 'git push --delete', 'git stash drop', or any similar cleanup without an explicit request. Treat 'cleanup' as scope the user opts into, not a default step."} +{"_type":"memory","key":"flatpilot-no-credential-storage","value":"Never store user passwords for rental platforms (WG-Gesucht, Kleinanzeigen, ImmoScout24, future) anywhere in FlatPilot — not in profile.json, .env, or any other file. Reasons: (1) Docker container can't reach host OS keyring, leaving only file storage which is unsafe for credentials; (2) programmatic credential login is a strong anti-bot signal that risks getting accounts flagged; (3) 2FA / captchas still require human intervention even with stored passwords; (4) ToS posture is much worse than cookie reuse from a real human session. Solution: cookies-only auth. CLI version (FlatPilot-5b8) uses headed Playwright + 'press Enter when logged in' UX. Future multi-user Web UI (FlatPilot-8jx) uses browser extension or bookmarklet to capture cookies from the user's normal browser session — never asks for or stores passwords. None of WG-Gesucht / Kleinanzeigen / ImmoScout24 publish OAuth endpoints, so a literal 'Sign in with Platform' redirect-and-callback flow is structurally impossible. Decided in PR #18 session 2026-04-22."} {"_type":"memory","key":"never-push-directly-to-main-every-change-goes","value":"Never push directly to main. Every change goes through a PR: git fetch origin; git checkout -b \u003ctype\u003e/\u003cslug\u003e origin/main; commit; git push -u origin \u003cbranch\u003e; gh pr create --base main --head \u003cbranch\u003e --fill. Return the PR URL and stop. Force-push allowed only on personal feature branches that nobody else has pulled (--force-with-lease)."} -{"_type":"memory","key":"commit-author-for-this-repo-mukhammad-ibrokhimov-ibrohimovmu","value":"Commit author for this repo: Mukhammad Ibrokhimov \u003cibrohimovmuhammad2020@gmail.com\u003e. In a fresh clone: git config user.email ibrohimovmuhammad2020@gmail.com \u0026\u0026 git config user.name 'Mukhammad Ibrokhimov'. Any other email on the machine is NOT for commits here."} +{"_type":"memory","key":"clean-room-policy-flatpilot-is-independent-mit-licensed","value":"Clean-room policy: FlatPilot is independent MIT-licensed. Never copy code from any other project (including any job-hunting agent that may have inspired the high-level architecture). Read outside code only as a design reference; always rewrite from scratch."}