Project Specification — Electronics Competitor Product Matching

This document is a machine-readable specification for agents working on this project. It consolidates the hackathon challenge, our team's architecture, task ownership, interface contracts, and technical decisions.

0. Quick Summary — What We Built

STATUS: ALL 3 CATEGORIES COMPLETE AND READY FOR SUBMISSION

How to Run the Matching System:

# Navigate to matching_system directory
cd matching_system

# Run a specific category (REQUIRED --category flag)
python main.py --category tv_audio --stats
python main.py --category large_appliances --stats
python main.py --category small_appliances --stats

# Full run with scraped data and LLM
python main.py --category tv_audio --include-scraped --split-output --enable-llm --stats

# Override LLM model (use Pro instead of Flash)
python main.py --category tv_audio --enable-llm --llm-model gemini-1.5-pro --stats

# Test specific passes only
python main.py --category tv_audio --passes 1 2 3 4 --stats

Output location: output/ directory — files named submission_{category}.json

Configuration: Edit matching_system/config.py to adjust thresholds, product profiles, field mappings, LLM settings

API Key: Set GEMINI_API_KEY in .env file at project root

What Works Right Now:

Complete Multi-Category Matching System — Profile-based matching across 3 product categories with 100% coverage on all:
- 9 product profiles (tv, microwave, cooktop, dishwasher, washing_machine, fridge_freezer, vacuum, small_kitchen, generic)
- Pass 1-4: Traditional matching (EAN, model number, profile-aware attributes, profile-aware weighted scoring)
- Pass 5: LLM-enhanced matching using Google Gemini API for difficult orphan cases
- Pass 6: Profile-aware aggressive fallback for remaining orphans
Scraped Data Integration — Successfully scraped and integrated 2 hidden retailers (TV & Audio):
- e-tec.at: 1,365 candidate products scraped via discover mode
- electronic4you.at: 387 candidate products (from cached data; site currently returning 503)
- Combined target pool: 2,313 products (561 visible + 1,752 scraped)
Submission Files Generated — Ready to upload to hackathon platform:
- output/submission_combined_tv_audio.json — 1,215 matches (829 visible + 386 hidden), 100% coverage
- output/submission_visible_tv_audio.json — 829 matches from Amazon AT & MediaMarkt AT
- output/submission_hidden_tv_audio.json — 386 matches from e-tec.at & electronic4you.at
- output/submission_large_appliances.json — 1,597 matches, 100% source coverage
- output/submission_small_appliances.json — 319 matches, 100% source coverage
Production-Quality Code — Modular, configurable, extensible architecture:
- CLI interface with --category flag and multiple options
- Declarative PRODUCT_PROFILES config — add new product types without changing code
- Proper error handling and logging

Results (All Categories):

Category	Sources	Coverage	Total Matches	Visible	Hidden	Avg Confidence
TV & Audio	17	100% (17/17)	1,215	829	386	64.8%
Large Appliances	44	100% (44/44)	1,597	1,597	0	67.1%
Small Appliances	29	100% (29/29)	319	319	0	75.4%

What's Missing:

2 hidden retailers not yet scraped: expert.at, cyberport.at (Person B's responsibility)
Scrapers not yet extended to Large/Small Appliances categories

1. Challenge Context

Hackathon: 8-hour, team of 3.
Objective: Match 91 source products (Austrian electronics retailer) to equivalent products sold by competitors.
Categories: Large Appliances (44 source, ~3500 target), Small Appliances (30 source, ~1700 target), TV & Audio (17 source, ~560 target). Released progressively by organizers.
Currently released: All 3 categories (TV & Audio, Large Appliances, Small Appliances).
Currently completed: All 3 categories — multi-category profile-based matching system. Submission files generated for all categories.
Scoring: 100 points per category. Total score = sum of best score per category across all submissions.
- Matching Score (50 pts) — visible retailers (Amazon AT, MediaMarkt AT): Recall 30, Precision 10, Coverage 10.
- Scraping Score (50 pts) — hidden retailers (Expert AT, Cyberport AT, electronic4you.at, E-Tec): Recall 30, Precision 10, Coverage 10.
Jury evaluation (separate from score): System maturity, UX, reusability/flexibility, creativity.
Submission format: One JSON file per category uploaded to https://hackathon-production-49ca.up.railway.app/. See section 8 for schema.
Rules: No manual lookups. Scraping must be rate-limited (1-2 req/sec). LLMs allowed. Any language/library allowed.

1.5 Key Technical Innovations

Recall Improvement: Graded Similarity Scoring (Pass 4 & 6)

Problem: The original weighted matcher (Pass 4) treated display_type and resolution as binary (1.0 for exact match, 0.0 for mismatch). This meant LED vs QLED scored 0.0 even though they are compatible display types. Similarly, missing attributes on one side scored 0.0, unfairly penalizing products with sparse data.

Solution: Three changes to improve recall without sacrificing precision:

Graded display type similarity — Instead of binary, uses a similarity table:
- LED vs QLED: 0.7 (compatible), LED vs LCD: 0.8, QLED vs OLED: 0.5, etc.
- Previously OLED was completely isolated; now loosely compatible with QLED (0.5)
Graded resolution similarity — Instead of binary, uses proximity scoring:
- UHD_4K vs FULL_HD: 0.3, FULL_HD vs HD_READY: 0.4, UHD_4K vs HD_READY: 0.1
Improved missing attribute handling — When an attribute is:
- Missing on BOTH sides: excluded from weighting entirely (no penalty)
- Missing on ONE side: scores 0.3 (neutral) instead of 0.0 (harsh penalty)
Threshold adjustment — Pass 4 threshold lowered from 0.60 to 0.55; aggressive pass size tolerance widened from +/-2" to +/-5"

Impact: Total matches increased from 394 to 832 (+111%) while average confidence only dropped from 66.0% to 64.3% (-2.6%).

LLM-Enhanced Matching (Pass 5)

Problem: Traditional rule-based matching misses products with inconsistent naming, missing attributes, or ambiguous specifications.

Solution: Pass 5 uses Google Gemini API to intelligently match orphan products (sources with no matches from Passes 1-4).

Implementation Details:

Trigger condition: Only runs on orphan sources (unmatched after Passes 1-4)
Strategy: One-to-many comparison (source vs all targets in single API call)
Context window: ~200-300 tokens per product (key fields: name, brand, category, price, important specs)
Model: gemini-1.5-flash (default, ~$0.01-0.015 per orphan) or gemini-1.5-pro (optional, ~$0.13-0.25 per orphan)
Confidence scoring: LLM determines confidence on 0-1 scale, configurable minimum threshold (default: 0.4)
Error handling: Graceful fallback — if Pass 5 fails, system continues to Pass 6

Cost Analysis (TV & Audio):

Current cost: $0.00 (no orphans to process — all 17 sources matched in Passes 1-4)
Expected cost per orphan: ~$0.01-0.015 (flash) or ~$0.13-0.25 (pro)
Token usage estimate: ~100K-150K tokens per source (comparing against 566 targets)

Benefits:

Catches difficult matches that rule-based systems miss
Provides intelligent confidence scoring
Explains matching reasoning (can be logged for debugging)
Scales well with larger target pools

Scraped Data Integration

Problem: Hackathon scoring includes 50 points for hidden retailers, but they're not in the provided target pool.

Solution: Built scraper-to-target-pool converter that seamlessly integrates scraped data into the matching pipeline.

Implementation Details:

scraped_to_target_converter.py — Converts scraped intermediate format to target pool format
data_loader.load_combined_target_pool() — Merges visible retailers + scraped retailers into unified target pool
Scraped products get unique references: SCRAPED_{RETAILER}_{ARTICLE_ID} (e.g., SCRAPED_ETEC_106973)
Pipeline treats scraped products identically to visible retailer products

Benefits:

Single unified pipeline for all retailers (no separate code paths)
Easy to add new scraped retailers (just drop JSON file in results/ directory)
Submission files automatically split by visible/hidden for separate scoring

Split Submission Output

Problem: Hackathon scores visible retailers (50 pts) and hidden retailers (50 pts) separately.

Solution: Generate 3 submission files automatically: visible-only, hidden-only, and combined.

Implementation Details:

submission.split_matches_by_retailer_type() — Categorizes matches by retailer type
Visible retailers: Amazon AT, MediaMarkt AT
Hidden retailers: e-tec.at, electronic4you.at, expert.at, cyberport.at
Files: submission_visible_*.json, submission_hidden_*.json, submission_combined_*.json

Benefits:

Upload combined file for full scoring
Or upload split files if platform allows separate submissions
Clear visibility into visible vs hidden match distribution

2. Team Structure & Task Ownership

Person	Role	Responsibility	Status
Person A	Matching pipeline	Match source products to visible retailer target pool (~5800 products). Implements the 6-pass matching algorithm with multi-category support. Produces matched `competitors[]` entries with `reference` fields from the target pool.	COMPLETED - Multi-category matching system with profile-based detection, LLM integration
Person B	Scraping (expert.at + cyberport.at)	Build scrapers for 2 of the 4 hidden retailers. Discovers products, extracts details, produces `competitors[]` entries with scraped references, URLs, prices.	PENDING - Scrapers not yet completed
Person C (UI owner)	Scraping (electronic4you.at + e-tec.at) + UI	Build scrapers for the remaining 2 hidden retailers. Build the demo application that integrates ALL team members' outputs.	COMPLETED - Both scrapers operational, producing valid results

3. Architecture Overview

                         +------------------+
                         |   SOURCE PRODUCTS |  (JSON input)
                         +--------+---------+
                                  |
                    +-------------+-------------+
                    |             |             |
              +-----v-----+ +---v----+ +------v-------+
              | MATCHING   | | SCRAPE | | SCRAPE       |
              | (Person A) | | expert | | electronic4you|
              | visible    | | cyber  | | e-tec        |
              | retailers  | | (B)    | | (C)          |
              +-----+------+ +---+----+ +------+-------+
                    |             |             |
                    +-------------+-------------+
                                  |
                         +--------v---------+
                         |     MERGER       |  (matching/merger.py)
                         |  Dedup + combine |
                         +--------+---------+
                                  |
                    +-------------+-------------+
                    |                           |
              +-----v------+          +--------v--------+
              | SUBMISSION |          | REACT UI + API  |
              | JSON file  |          | (ui/ + ui_api.py)|
              +------------+          +-----------------+

Pipeline Flow (from workflow.md)

Graded Task:

Receive source products JSON as input.
Parallelize into 5 branches: 1 for visible retailer matching + 4 for scraping (one per hidden retailer website).
Scraping branches (in parallel): Identify most distinguishing feature of each source product (e.g., brand + category) to scope the scrape. Then format scraped data to match source JSON structure.
All branches (in parallel): Run the 6-pass matching algorithm (see section 5).
Merge: Reunite outputs from all 5 branches, deduplicate, validate.
Submit: Generate submission JSON and upload.

Open-ended Task (UI):

User types a product name; show autocomplete/suggestions from source DB.
User selects a product and optionally specifies priorities/filters.
Run the same pipeline (steps 1-5 above) with user's specifications influencing matching and scraping.
Display results; allow user to filter/sort by price, brand, retailer, etc.

4. Tech Stack

Participant Environment

Python 3.12 — runtime (virtualenv at hack_venv/)
pandas — data manipulation
requests — HTTP for scraping (electronic4you.at, e-tec.at)
beautifulsoup4 + lxml — HTML parsing
google-generativeai — Gemini API for LLM-based matching (Pass 5)
scikit-learn — ML matching utilities
sentence-transformers — semantic similarity / vector embeddings
playwright — browser automation (fallback for JS-rendered sites)
rapidfuzz — fuzzy string matching (used by Person A, listed in agent.md)
fastapi + uvicorn — lightweight UI adapter API (ui_api.py)
react + vite + react-router — UI in ui/
python-dotenv — environment variable loading for local .env

Hackathon Platform (pre-built, read-only)

Frontend: React 19, TypeScript, Vite 7, TailwindCSS 4, React Router 7, TanStack React Query 5
Backend: FastAPI, PostgreSQL (psycopg v3), GitHub OAuth + JWT
Deployment: Docker, Railway.app
Web app URL: https://hackathon-production-49ca.up.railway.app/

5. Matching Algorithm

Person A implements this. Scrapers (B & C) must format their output so it can feed into the same pipeline.

6-Pass Strategy (recall-optimized) — FULLY IMPLEMENTED

Pass	Method	Confidence	Description
1	EAN Exact	95-100%	`source.ean == target.ean` + brand similarity >= 0.6
2	Model Number	80-95%	Substring or 90% fuzzy match + brand >= 0.6
3	Profile-Aware Attributes	70-85%	Brand>=0.8 + same profile + profile-specific strict criteria (numeric tolerance, category match)
4	Profile-Aware Weighted Score	60-75%	Profile-specific weights and threshold (e.g., brand 30% + capacity 25% + power 20% + ... >= profile threshold)
5	LLM Semantic	40-90%	For orphaned sources only. Google Gemini API, one-to-many comparison.
6	Profile-Aware Aggressive	40-60%	Brand>=0.6 + same profile + name similarity >= 0.65, or relaxed weighted score

Product Profile System (NEW — Multi-Category Support)

The system uses declarative PRODUCT_PROFILES in config.py to drive all matching behavior per product sub-type:

Profile	Category	Detection Keywords	Key Attributes
`tv`	TV & Audio	Fernseher, TV, Smart TV, QLED, OLED	screen_size, display_type, resolution
`microwave`	Large Appliances	Mikrowelle, Microwave	capacity_liters, power_watts
`cooktop`	Large Appliances	Kochfeld, Kochplatte, Ceranfeld	num_zones, cooktop_type, power_watts
`dishwasher`	Large Appliances	Geschirrspüler, Spülmaschine	place_settings, energy_class
`washing_machine`	Large Appliances	Waschmaschine, Wäschetrockner	capacity_kg, spin_speed, energy_class
`fridge_freezer`	Large Appliances	Kühlschrank, Gefrierschrank	capacity_liters, energy_class
`vacuum`	Small Appliances	Staubsauger, Saugroboter	power_watts, vacuum_type
`small_kitchen`	Small Appliances	Toaster, Fritteuse, Sandwichmaker	power_watts, num_slots
`generic`	Any	(fallback)	brand + name similarity only

Each profile defines: detection_keywords, detection_spec_keys, extraction_map, weights, match_threshold, strict_criteria, target_filter_include, target_filter_exclude.

Attribute Extraction

TV profile (backward compatible):

Brand: specs['Hersteller'] > specs['Marke'] > specs['Brand'] > specs['Manufacturer'] > first word of name
Screen Size: Parse from "32 Zoll", "32\"", "80 cm". Tolerance: +/-2"
Display Type: LED, QLED, OLED, LCD, Mini LED. Graded similarity table.
Resolution: 4K UHD, Full HD, HD Ready. Graded proximity scoring.

Appliance profiles (NEW):

Brand: Same extraction + expanded KNOWN_BRANDS (~85 brands)
Numeric attributes: Extracted from German spec keys (e.g., Garraum-Volumen, Schleuderdrehzahl), unit parsing, ratio-based similarity (±25-30% tolerance)
Energy class: Ordinal distance scoring using ENERGY_CLASS_ORDER (A+++ through G)
Integer attributes: place_settings, num_zones — ordinal similarity
Categorical attributes: cooktop_type, vacuum_type — fuzzy string match

Similarity Scoring

Numeric (ratio): 1.0 - abs(a - b) / max(a, b) within tolerance (25-30%), else 0.0
Energy class (ordinal): 1.0 - distance / max_distance where distance = position difference in A+++...G scale
Integer (ordinal): 1.0 - abs(a - b) / max(a, b, 1) for place_settings, zones, slots
Missing attributes: Missing on BOTH sides = excluded; missing on ONE side = 0.3 (neutral)

6. Scraper Specifications

6.1 electronic4you.at (Person C)

Technology: requests + beautifulsoup4 (no JS rendering needed).
Search endpoint: https://www.electronic4you.at/catalogsearch/result/?q={query}
EAN search: Works. Tested with EAN 5901292526665 — returned correct TCL 65Q6C match with price (649 EUR), article number (270451), stock info.
Anti-bot behavior discovered: endpoint intermittently returns 429 Too Many Requests during full-run scraping.
Search strategy (ordered):
1. Search by EAN/GTIN (highest reliability).
2. Search by model number (e.g., "TCL 65Q6C", "QE55Q7FAAUXXN").
3. Search by brand + key specs (e.g., "Samsung QLED 50 Zoll").
Data to extract from results: product name, price (EUR), product URL, article number (Art.Nr.), availability status.
Rate limiting: 1 request per second minimum; use exponential backoff on 429/503 and continue per-product if a query fails.
User-Agent: Set a proper browser-like User-Agent string.
Operational note: Full-batch runs can be slow because each source product may trigger multiple search requests and detail-page requests.

6.2 e-tec.at (Person C)

Current status: API discovered and working; Playwright fallback not required for now.
Search endpoint in production use: POST https://www.e-tec.at/xsite/endpoint/loadMoreData
Required request metadata:
- scope=xsite
- namespace=xs_shop
- classx=search\\xs_shop_search_magic
- fn=loadMoreData
- p_id=42
- lang=DE
- data field contains JSON payload with config_id=-69
- urlParams.search must be JSON string: {"query":"...","filters":{},"sorter":"_score desc"}
Parsing approach: parse product objects from inline window.V_PRODUCTS[...] JSON snippets in HTML_CONTENT_AJAX.
Search strategy: Same as electronic4you — EAN first, model number second.
Category browsing URL: https://www.e-tec.at/de/tv-audio.html — can be used for category-level scraping if search fails.
Rate limiting: 1 request per second minimum.

6.3 expert.at (Person B)

Owned by Person B. No specs here; they will produce output in the shared interface format (section 7).

6.4 cyberport.at (Person B)

Owned by Person B. No specs here; they will produce output in the shared interface format (section 7).

7. Interface Contracts

7.1 Source Products JSON (input)

Location: source_products_tv_&_audio.json (root directory, also data/ for future categories).

{
  "ean": "5901292526665" | null,
  "name": "TCL 65Q6C 4K QD-Mini LED TV mit Google TV 164 cm (65\")",
  "brand": "" | null,
  "category": "TV & Audio",
  "image_url": "https://...",
  "price_eur": 599 | null,
  "reference": "P_A7DDBB5E",
  "specifications": { ... } | null
}

7.2 Scraper Output (intermediate — per scraper)

Each scraper produces a JSON list. This is the interface between scrapers and the merger.

[
  {
    "source_reference": "P_A7DDBB5E",
    "match": {
      "reference": "SCRAPED_E4Y_270451",
      "competitor_retailer": "electronic4you.at",
      "competitor_product_name": "TCL 65Q6C 4K QD-Mini LED TV mit Google TV 164",
      "competitor_url": "https://www.electronic4you.at/...",
      "competitor_price": 649.00
    },
    "match_method": "ean_exact",
    "confidence": 0.95
  }
]

Reference format for scraped products: SCRAPED_{RETAILER_CODE}_{ARTICLE_ID} where RETAILER_CODE is one of: E4Y (electronic4you), ETEC (e-tec), EXP (expert), CYB (cyberport).

7.3 Submission JSON (final output — uploaded to hackathon platform)

[
  {
    "source_reference": "P_A7DDBB5E",
    "competitors": [
      {
        "reference": "P_43E3D659",
        "competitor_retailer": "Amazon AT",
        "competitor_product_name": "...",
        "competitor_url": "https://...",
        "competitor_price": 599.00
      },
      {
        "reference": "SCRAPED_E4Y_270451",
        "competitor_retailer": "electronic4you.at",
        "competitor_product_name": "...",
        "competitor_url": "https://...",
        "competitor_price": 649.00
      }
    ]
  }
]

Required fields for scoring: source_reference, competitors[].reference. All other fields are optional but useful for debugging and UI display.

7.4 Teammates' Output Files (for merger integration)

Person A and Person B should save their results as JSON files in results/ directory:

results/matching_visible_{category}.json — Person A's visible retailer matches
results/scraping_expert_{category}.json — Person B's expert.at results
results/scraping_cyberport_{category}.json — Person B's cyberport.at results
results/scraping_electronic4you_{category}.json — Person C's electronic4you.at results
results/scraping_etec_{category}.json — Person C's e-tec.at results

All use the intermediate format from section 7.2. The merger combines them into the final submission format (section 7.3).

8. File Structure

claude_hack/
├── README.md                          # Hackathon challenge description
├── workflow.md                        # Pipeline workflow description
├── agent.md                           # 6-pass matching algorithm spec
├── summary.md                         # THIS FILE — project specification
├── requirements.txt                   # Python dependencies
├── app.py                             # Streamlit demo UI (Person C)
├── scrapers/
│   ├── utils.py                       # Shared utilities: EAN extraction, rate limiter, output formatting
│   ├── electronic4you.py              # electronic4you.at scraper (Person C)
│   └── etec.py                        # e-tec.at scraper (Person C)
├── matching_system/                   # COMPLETE MULTI-CATEGORY MATCHING PIPELINE (Person A)
│   ├── __init__.py
│   ├── main.py                        # CLI entry point with --category, --enable-llm, --include-scraped, --split-output flags
│   ├── config.py                      # Configuration: CATEGORIES, PRODUCT_PROFILES, KNOWN_BRANDS, thresholds, LLM settings
│   ├── pipeline.py                    # 6-pass pipeline orchestrator with profile logging
│   ├── data_loader.py                 # Load source/target JSON, combines visible + scraped data
│   ├── attribute_extractor.py         # Profile-aware extraction: brand, model, sub-type detection, numeric/energy/categorical attrs
│   ├── submission.py                  # Format and save submission JSON (split by visible/hidden)
│   ├── llm_client.py                  # Google Gemini API client for Pass 5
│   ├── utils.py                       # Utility functions (normalize_text, fuzzy_match, size/price scoring)
│   └── matchers/
│       ├── __init__.py
│       ├── base_matcher.py            # Abstract matcher interface
│       ├── ean_matcher.py             # Pass 1: EAN exact match
│       ├── model_matcher.py           # Pass 2: Model number match
│       ├── attribute_matcher.py       # Pass 3: Profile-aware strict attribute match
│       ├── weighted_matcher.py        # Pass 4: Profile-aware weighted score match
│       ├── llm_matcher.py             # Pass 5: LLM-based semantic match (orphans only)
│       └── aggressive_matcher.py      # Pass 6: Profile-aware aggressive fallback match
├── scraped_to_target_converter.py     # Converts scraped intermediate format to target pool format
├── ui/                                # React/Vite UI workspace (Person C)
│   ├── src/app/pages/                 # Search / All Products / Output pages
│   ├── src/app/api/client.ts          # Frontend API client
│   └── .env.example                   # VITE_API_URL example
├── ui_api.py                          # FastAPI adapter for UI endpoints
├── data/
│   ├── source_products_tv_&_audio.json
│   ├── target_pool_tv_&_audio.json
│   ├── source_products_large_appliances.json
│   ├── target_pool_large_appliances.json
│   ├── source_products_small_appliances.json
│   └── target_pool_small_appliances.json
├── results/                           # Output directory for all team members
│   ├── candidates_etec_tv_audio.json         # e-tec discover mode output (1,364 products)
│   ├── candidates_electronic4you_tv_audio.json # electronic4you candidates from cache (387 products)
│   ├── scraping_etec_tv_audio.json           # e-tec best-match output (3 products)
│   ├── cache_electronic4you.json             # Request cache for electronic4you scraper
│   ├── cache_feature_scope_llm.json          # LLM feature classification cache
│   ├── scraping_expert_tv_audio.json         # MISSING - Person B task
│   └── scraping_cyberport_tv_audio.json      # MISSING - Person B task
├── output/                            # FINAL SUBMISSION FILES
│   ├── submission_combined_tv_audio.json     # 1,215 matches (829 visible + 386 hidden)
│   ├── submission_visible_tv_audio.json      # 829 matches, 100% coverage
│   ├── submission_hidden_tv_audio.json       # 386 matches from hidden retailers
│   ├── submission_large_appliances.json      # 1,597 matches, 100% coverage
│   └── submission_small_appliances.json      # 319 matches, 100% coverage
├── .env                               # Local secrets (GEMINI_API_KEY) - gitignored
├── .gitignore                         # Ignores .env, venv, pycache, etc.
├── ui-description.md                  # Open-ended UI specification and implementation notes
├── webapp/                            # Hackathon platform (pre-built, DO NOT MODIFY)
│   ├── backend/                       # FastAPI + PostgreSQL
│   └── frontend/                      # React + Vite + TailwindCSS
└── hack_venv/                         # Python 3.12 virtualenv

9. React UI Specification

Purpose

Demonstrate a production-quality product matching tool for the jury. Integrates all team members' pipelines into a unified interface.

Stack and runtime

Frontend: React + Vite in ui/
API adapter: FastAPI in ui_api.py
Data sources: source_products_tv_&_audio.json, results/*.json
Scraper and merge execution: existing Python modules (scrapers/*, matching/merger.py)

Pages / Sections

Product Search & Browse
- Sidebar with all source products, searchable/filterable by name, brand, category.
- As user types, show autocomplete suggestions from the source product database.
- Clicking a product selects it and shows details (name, brand, price, image, specs).
Match Results Panel
- For the selected source product, display all matched competitors in a table.
- Columns: Retailer, Product Name, Price, Price Difference, Match Method, Confidence, URL link.
- Color-coded confidence badges (green >= 80%, yellow 50-80%, red < 50%).
- Group by retailer or sort by price/confidence.
On-demand Actions
- "Scrape this product" button — triggers scraping for the selected product across electronic4you.at and e-tec.at.
- "Run full pipeline" button — runs matching + scraping for all source products.
- User can specify priorities/filters (e.g., "only brand X", "max price Y") that influence the matching algorithm.
Data Management
- Upload new source products JSON (drag-and-drop).
- Upload teammates' result files for integration.
- Upload new category data when released by organizers.
Export & Submission
- "Generate Submission JSON" button — merges all results, deduplicates, outputs submission file.
- Download button for the generated JSON.
- Display submission stats: total matches, per-retailer breakdown, coverage percentage.
Dashboard / Stats
- Coverage: X/17 source products have at least 1 match.
- Matches per retailer bar chart.
- Match method breakdown pie chart (EAN, model number, fuzzy, LLM).
- Price comparison summary.

API contract for frontend

GET /api/products — source product cards
GET /api/search/suggestions?q=... — autocomplete list
POST /api/pipeline/run — run selected or full pipeline
GET /api/pipeline/results?query=...&specs=... — output cards + stats

Trigger semantics

Search page suggestions do not run full pipeline.
Output page search triggers pipeline run for selected query.
Output page feature filter filters within current output results only.
"Run all" triggers full scrape+merge cycle.

10. Source Product Data Summary

TV & Audio (17 products)

13 TVs, 1 power cable, 3 others
EAN availability: 6/17 have top-level ean. Additional in specs (GTIN, EAN-Code).
Brands: Samsung (3), TCL (4), Sharp (3), LG (1), PEAQ (2), XIAOMI (1), CHIQ (1), sonero (1), unknown (1)
Price range: 149-669 EUR. Screen sizes: 24"-65".

Large Appliances (44 products)

Sub-types: Microwave (10), Cooktop (7), Dishwasher (5), Washing Machine (4), Fridge/Freezer (3), Dryer (1), Oven Set (1), Accessories (13)
EAN: 89% null in source, 100% null in target pool
Brand field always empty — extracted from name/specs (Brand, Marke, Hersteller)
Spec keys: German (Garraum-Volumen, Schleuderdrehzahl, Nennkapazität Maßgedecke, etc.)

Small Appliances (29 products)

Sub-types: Vacuum (5), Toaster (3), Air Fryer (2), Sandwichmaker (2), Heizkissen (2), 8 single-product types, Accessories (6)
EAN: 69% null in source, 100% null in target pool
Same brand extraction approach as large appliances

11. Key Decisions Made

Work split: Person C handles electronic4you.at + e-tec.at scrapers AND the demo UI. This contributes to both the 50-point scraping score and the jury evaluation.
UI framework: React + Vite frontend with a FastAPI adapter, based on the provided zip UI.
e-tec.at approach: Try API discovery first before falling back to Playwright.
EAN extraction: Aggressively extract identifiers from all available fields (ean, GTIN, EAN-Code, Modellnummer, Hersteller Artikelnummer).
Full pipeline integration: The React UI integrates ALL team members' outputs, not just Person C's scrapers.
Rate limiting: 1 request per second for all scrapers. Proper User-Agent header.
Resilience over speed for electronic4you: prioritize retries/backoff and partial progress over aggressive request rates because of observed 429 responses.
Batch optimization direction: reduce request count by query deduplication, cache detail pages, and only fetch details for shortlisted candidates.
Declarative product profiles: Instead of hardcoding product-type logic, use PRODUCT_PROFILES config dict that drives detection, extraction, weighting, filtering, and matching — add new product types by adding config, not code.
Generic fallback for accessories: Products that don't match any profile (accessories, cables, etc.) use the generic profile — match on brand + name similarity only, avoiding false positives from irrelevant attribute comparisons.
Brand extraction from specs: Appliance data has empty brand field — extract from specifications['Brand'], specifications['Marke'], specifications['Hersteller'], or first word of product name against KNOWN_BRANDS list.

12. API Keys & Environment

Google Gemini API key: stored in .env as GEMINI_API_KEY — used by Pass 5 (LLM matcher).
OpenRouter/OpenAI-compatible key: optional, can be stored in .env as OPENROUTER_API_KEY and OPENAI_API_KEY if needed for future enhancements.
Security rule: .env is gitignored; never print or commit API keys.
Brave Search API key: optional fallback (not currently used by Person C scrapers).
Python virtualenv: hack_venv/ — activate with source hack_venv/bin/activate (Linux/Mac) or hack_venv\Scripts\activate (Windows).
Installed in venv: rapidfuzz, python-dotenv, lxml, fastapi, uvicorn, google-generativeai.
Frontend runtime: Node/npm in ui/ (npm install, npm run dev, npm run build).

13. Current Implementation Status

COMPLETED — Person A (Multi-Category Matching Pipeline)

Complete multi-category, profile-based 6-pass matching system:

matching_system/main.py — CLI with --category (tv_audio, large_appliances, small_appliances), --enable-llm, --include-scraped, --split-output, --passes, --stats
matching_system/config.py — CATEGORIES (per-category paths), PRODUCT_PROFILES (9 profiles), expanded KNOWN_BRANDS (~85), ENERGY_CLASS_ORDER, PROFILE_DETECTION_ORDER, LLM settings
matching_system/pipeline.py — 6-pass orchestrator with category param, profile distribution logging, profile-aware summary
matching_system/data_loader.py — Generic data loading for any category
matching_system/attribute_extractor.py — detect_product_profile(), profile-aware extraction, extract_profile_attributes(), numeric/energy/categorical helpers
matching_system/submission.py — Submission formatting with visible/hidden split
matching_system/llm_client.py — Google Gemini API client (Pass 5)
matching_system/utils.py — Utility functions (normalize, fuzzy match, scoring)
matching_system/matchers/ — All 6 profile-aware matchers:
- ean_matcher.py (Pass 1) — EAN exact match
- model_matcher.py (Pass 2) — Model number match
- attribute_matcher.py (Pass 3) — Profile-aware strict attribute match (TV mode + profile mode)
- weighted_matcher.py (Pass 4) — Profile-aware weighted score with ratio/ordinal/energy similarity
- llm_matcher.py (Pass 5) — LLM-based semantic match (orphans only)
- aggressive_matcher.py (Pass 6) — Profile-aware aggressive fallback (3 strategies)
scraped_to_target_converter.py — Converts scraped data to target pool format

Testing Results (All 3 Categories — Visible Retailers Only):

Category	Sources	Coverage	Matches	Avg Conf	Pass 1	Pass 2	Pass 3	Pass 4
TV & Audio	17	100%	829	64.5%	3	11	47	768
Large Appliances	44	100%	1,597	67.1%	13	36	331	1,217
Small Appliances	29	100%	319	75.4%	6	18	193	102

Zero runtime errors across all categories
Pass 5 (LLM) and Pass 6 (Aggressive) correctly inactive (0 matches — full coverage achieved by Passes 1-4)
TV & Audio results unchanged (backward compatible)

Submission Files (READY TO UPLOAD):

output/submission_tv_audio.json — 829 matches, 100% coverage
output/submission_large_appliances.json — 1,597 matches, 100% coverage
output/submission_small_appliances.json — 319 matches, 100% coverage

COMPLETED — Person C (Scrapers)

scrapers/utils.py — shared helpers (EAN/model extraction, rate limiter, similarity, JSON I/O)
scrapers/electronic4you.py — scraper with search/detail parsing, scoring, CLI output
scrapers/etec.py — scraper via loadMoreData endpoint, parses window.V_PRODUCTS payloads
app.py — Streamlit dashboard
Results: scraping_etec_tv_audio.json (3 products), scraping_electronic4you_tv_audio.json (2 products)

PENDING — Person B (Scrapers)

scrapers/expert.py — not yet completed
scrapers/cyberport.py — not yet completed

FilesExpand file tree

summary.md

Latest commit

History

summary.md

File metadata and controls

Project Specification — Electronics Competitor Product Matching

0. Quick Summary — What We Built

How to Run the Matching System:

What Works Right Now:

Results (All Categories):

What's Missing:

1. Challenge Context

1.5 Key Technical Innovations

Recall Improvement: Graded Similarity Scoring (Pass 4 & 6)

LLM-Enhanced Matching (Pass 5)

Scraped Data Integration

Split Submission Output

2. Team Structure & Task Ownership

3. Architecture Overview

Pipeline Flow (from workflow.md)

4. Tech Stack

Participant Environment

Hackathon Platform (pre-built, read-only)

5. Matching Algorithm

6-Pass Strategy (recall-optimized) — FULLY IMPLEMENTED

Product Profile System (NEW — Multi-Category Support)

Attribute Extraction

Similarity Scoring

6. Scraper Specifications

6.1 electronic4you.at (Person C)

6.2 e-tec.at (Person C)

6.3 expert.at (Person B)

6.4 cyberport.at (Person B)

7. Interface Contracts

7.1 Source Products JSON (input)

7.2 Scraper Output (intermediate — per scraper)

7.3 Submission JSON (final output — uploaded to hackathon platform)

7.4 Teammates' Output Files (for merger integration)

8. File Structure

9. React UI Specification

Purpose

Stack and runtime

Pages / Sections

API contract for frontend

Trigger semantics

10. Source Product Data Summary

TV & Audio (17 products)

Large Appliances (44 products)

Small Appliances (29 products)

11. Key Decisions Made

12. API Keys & Environment

13. Current Implementation Status

COMPLETED — Person A (Multi-Category Matching Pipeline)

COMPLETED — Person C (Scrapers)

PENDING — Person B (Scrapers)