Updated: 2026-03-07 | Duration: 8h | Strategy: 5-pass modular system (no LLM)
Objective: Match 91 source products to competitor products from 6 Austrian retailers
Scoring: 100 pts (50 visible + 50 scraped) - Recall (30) > Precision (10) > Coverage (10)
Web App: hackathon-production-49ca.up.railway.app
- Visible: Amazon AT | Hidden: Expert AT, Cyberport AT, electronic4you.at, E-Tec
- EAN Coverage: 5/17 (29.4%) - requires fuzzy matching
- Brands: Samsung, LG, TCL, Sharp, PEAQ, XIAOMI, CHIQ, Cello
| Pass | Method | Criteria | Confidence | Expected |
|---|---|---|---|---|
| 1 | EAN Exact | source.ean == target.ean + brand ≥0.6 |
95-100% | 4-5 |
| 2 | Model Number | Substring/90% fuzzy + brand ≥0.6 | 80-95% | 6-8 |
| 3 | Strict Attr | Brand≥0.8 + Size±2" + Display + Resolution | 70-85% | 3-5 |
| 4 | Weighted Score | 25%Brand + 20%Size + 15%Display + 15%Res + 10%Model + 10%Price + 5%Name ≥0.60 | 60-75% | 1-2 |
| 5 | Aggressive | Brand≥0.6 + Size±2" (orphans) OR score≥0.5 | 40-60% | 0-1 |
| TOTAL | Avg 70% | 14-21 |
Match Policy: Keep all valid matches per source (maximize recall), allow target reuse across sources
Approach: Extract ALL specifications dynamically - work directly with German spec keys
Data Analysis (from actual dataset):
- Source products: 152 unique spec keys, most common: Hersteller (14), WLAN (12), Displaytyp (11)
- Target products: 534 unique spec keys, most common: Marke (289), Modellname (257), Konnektivitätstechnologie (247)
- Only 62.6% of target products have specs (351/561)
Strategy:
- Extract critical attributes (brand, size, display, resolution, model, ean, price) with robust fallback logic
- Store ALL other specs as-is (no normalization/translation needed)
- Store raw specs for flexibility
- Build spec index showing coverage across products
- Allow Pass 4 weighted matcher to use ANY spec with configurable weights
Brand:
- Priority:
specs['Hersteller']→specs['Marke']→ first word matching KNOWN_BRANDS → first word in name - Normalize: "Samsung Electronics GmbH" → "SAMSUNG"
Model Number:
- Priority:
specs['Hersteller Modellnummer']→specs['Herstellernummer']→specs['Modellnummer']→ regex from name - Clean: Remove
.AEU,/XXN,FUXXNsuffixes, uppercase - Example:
32LQ63806LC.AEU→32LQ63806LC
Screen Size:
- Check:
specs['Bildschirmdiagonale in cm, Zoll']→specs['Bildschirmgröße']→ name patterns - Patterns:
32 Zoll,32",81 cm(÷2.54) - Returns: int in inches
Display Type:
- Check:
specs['Displaytyp']→specs['Produkttyp']→specs['Bildschirmtechnologie']→ name - Normalize: LED, QLED, OLED, LCD, MINI_LED
- Compatibility: QLED⊃LED, Mini LED⊃LED, OLED distinct
Resolution:
- Check:
specs['Bildqualität']→specs['Auflösung']→specs['Bildschirmauflösung']→ name - Normalize: UHD_4K (4K/3840x2160), FULL_HD (1920x1080), HD_READY (1366x768/720p)
EAN:
- Priority:
product['ean']→specs['EAN-Code']→specs['GTIN']
Price:
product['price_eur']
Note: No pre-filtering or post-validation initially - implement if needed based on results
Source Products (17):
- EANs: 5/17 (29.4%) - low coverage requires fuzzy matching
- Specs: 152 unique keys, all products have specs
- Top keys: Hersteller (14), WLAN (12), Displaytyp (11), Energieeffizienzklasse (10)
- Price range: €149-€669 | Sizes: 24"-65"
- Brands: Samsung, Sharp, LG, XIAOMI, PEAQ, TCL, CHIQ
Target Products (561):
- Specs: 534 unique keys, only 351/561 (62.6%) have specs
- Top keys: Marke (289), Modellname (257), Konnektivitätstechnologie (247), Hersteller (228)
- Mix: TVs + accessories (headphones, cables, adapters) - no pre-filtering
- Retailer: Amazon AT (visible)
- Data quality: Many missing EAN/specs/prices
Expected Results: 14-21 matches (82-100% coverage) at ~70% avg confidence
matching_system/
├── config.py # Paths, thresholds, DEFAULT_WEIGHTS, known brands, display compatibility
├── utils.py # Fuzzy matching (rapidfuzz), text normalization
├── data_loader.py # Load/validate JSONs, data summaries
├── attribute_extractor.py # Dynamic extraction of ALL specs + critical attributes
├── matchers/
│ ├── base_matcher.py # Abstract class: match() → [(source_ref, target_ref, conf, pass)]
│ ├── ean_matcher.py # Pass 1: EAN exact
│ ├── model_matcher.py # Pass 2: Model fuzzy/substring
│ ├── attribute_matcher.py # Pass 3: Strict attributes
│ ├── weighted_matcher.py # Pass 4: Weighted score ≥0.60
│ └── aggressive_matcher.py # Pass 5: Fallback for orphans
├── pipeline.py # Orchestrate passes, merge, dedupe, format
├── submission.py # Generate submission JSON
└── main.py # CLI entry point
Note: No scrapers - scraping is handled separately by team
Load → Extract Attrs → Pass 1-5 → Validate → Dedupe → Format → Submit
# Core attributes with default weights:
DEFAULT_WEIGHTS = {
'brand': 0.25,
'size': 0.20,
'display_type': 0.15,
'resolution': 0.15,
'model_number': 0.10,
'price': 0.10,
'name': 0.05
# Users can add ANY spec with custom weights via CLI
}
# Scoring logic:
score = sum(similarity(source[spec], target[spec]) * weight
for spec, weight in weights.items()
if spec in source and spec in target)
# Price scoring: ±50%=1.0, ±100%=0.5, >100%=0.0 (neutral if missing)
# Threshold: ≥0.60 for match (configurable via CLI)# Contents:
- Paths: SOURCE_DATA, TARGET_DATA, OUTPUT_DIR
- Pass thresholds: EAN_BRAND_VALIDATION, MODEL_FUZZY_THRESHOLD, etc.
- DEFAULT_WEIGHTS dict (7 core, extensible for any spec via CLI)
- WEIGHTED_SCORE_THRESHOLD, AGGRESSIVE_SCORE_THRESHOLD
- KNOWN_BRANDS list (15 brands)
- DISPLAY_COMPATIBILITY mapping
- PRICE_RATIO_EXACT, PRICE_RATIO_ACCEPTABLE
- NAME_FUZZY_THRESHOLDStatus: ✅ Implemented and tested
- Weights sum to 1.00
- All paths verified
- Focused on matching only (no scraping config)
- Works with German spec keys directly (no translation)
# Functions (8 total - simplified from 10):
1. normalize_text(text) → str # Uppercase, German chars (ä→AE, ö→OE, ü→UE), trim
2. clean_model_number(model) → str # Remove .AEU, XXN, FUXXN suffixes
3. fuzzy_match_text(t1, t2) → float # 0-1 using rapidfuzz.fuzz.ratio
4. fuzzy_match_brand(b1, b2) → float # 0-1 using token_sort_ratio
5. substring_match(t1, t2) → bool
6. calculate_size_score(s1, s2, max_diff=10) → float # Linear decay
7. calculate_price_score(p1, p2) → float or None # ±50%/±100% thresholds
8. extract_size_from_text(text) → int or None # Patterns: 32", 32 Zoll, 81cm
Note: Removed normalize_spec_key() - working with German keys directly
Note: Removed extract_model_from_text() - will be in attribute_extractor# Functions (5 total):
1. load_json(filepath, encoding='utf-8') → list # Error handling
2. load_source_products(filepath=None) → list # Defaults to config.SOURCE_DATA
3. load_target_products(filepath=None) → list # Validate required fields
4. validate_product(product, required_fields) → bool
5. get_data_summary(products) → dict # Stats: count, with_ean, with_specs, etc.Test:
sources = load_source_products()
targets = load_target_products()
print(f"Sources: {len(sources)}, Targets: {len(targets)}")
print(get_data_summary(sources))# Functions (8 total - simplified):
# Critical attribute extractors (with fallback logic):
1. extract_brand(product) → str or None
2. extract_model_number(product) → str or None # Includes regex extraction
3. extract_screen_size(product) → int or None
4. extract_display_type(product) → str or None
5. extract_resolution(product) → str or None
# Main extraction function:
6. extract_all_specs(product) → dict
# Returns:
{
# Critical fields (extracted with fallbacks):
'brand': str, 'model_number': str, 'ean': str,
'size': int, 'display_type': str, 'resolution': str,
'price': float, 'name': str,
# ALL other specs stored as-is (German keys):
# All keys from product['specifications'] copied directly
# Metadata:
'_raw_specs': dict, # Original specs for debugging
'_source_type': 'source' or 'target'
}
7. extract_all_specs_batch(products) → list[dict]
# Extract specs for all products
8. display_types_compatible(d1, d2) → bool
# Check compatibility using config.DISPLAY_COMPATIBILITY
Note: No build_spec_index() needed initially - can add later if neededTest Script:
from data_loader import *
from attribute_extractor import *
# Load data
sources = load_source_products()
targets = load_target_products()
print("=== Testing Extraction on 3 Source Products ===")
for i, product in enumerate(sources[:3], 1):
specs = extract_all_specs(product)
print(f"\nProduct {i}: {product['name'][:50]}")
print(f" Brand: {specs['brand']}")
print(f" Model: {specs['model_number']}")
print(f" Size: {specs['size']}\"")
print(f" Display: {specs['display_type']}")
print(f" Resolution: {specs['resolution']}")
print(f" EAN: {specs['ean']}")
print(f" Total specs: {len([k for k in specs if not k.startswith('_')])}")
print("\n=== Testing Display Compatibility ===")
print(f"LED vs QLED: {display_types_compatible('LED', 'QLED')}")
print(f"OLED vs LED: {display_types_compatible('OLED', 'LED')}")
print("\n=== Phase 1 Complete! ===")Deliverables:
- ✅ config.py: Complete data loading infrastructure
- ⏳ utils.py: Text processing utilities
- ⏳ data_loader.py: JSON loading and validation
- ⏳ attribute_extractor.py: Dynamic extraction with German keys (no translation)
- Result: Ready for matcher implementation
-
matchers/base_matcher.py -
matchers/ean_matcher.py(Pass 1) -
matchers/model_matcher.py(Pass 2) -
matchers/attribute_matcher.py(Pass 3) -
matchers/weighted_matcher.py(Pass 4) - Test: Run each matcher independently
-
pipeline.py(orchestrate passes 1-4, merge, dedupe) -
submission.py(format JSON per README:155-188) -
main.py(CLI with full configurability - see below) - SUBMIT v1 for feedback
main.py CLI Interface:
python main.py [options]
# I/O
--source PATH # Source JSON (default: config)
--target PATH # Target JSON (default: config)
--output PATH # Output JSON (default: output/submission.json)
# Pass selection
--passes 1,2,3,4,5 # Which passes to run (default: 1,2,3,4)
# Pass 4: Fully configurable weights (can add ANY spec)
--weight brand=0.25 # Override default weight
--weight screen_size_inches=0.20
--weight display_type=0.15
--weight resolution=0.15
--weight model_number=0.10
--weight price=0.10
--weight name=0.05
--weight wifi=0.02 # Add custom spec weight
--weight hdr=0.03 # Add another custom weight
# Thresholds
--weighted-threshold 0.60 # Pass 4 cutoff (default: 0.60)
--aggressive-threshold 0.50 # Pass 5 cutoff (default: 0.50)
# Output control
--verbose # Print detailed match info
--stats # Print statistics
--save-specs PATH # Save extracted specs to JSON (debug)-
matchers/aggressive_matcher.py(Pass 5 - orphans) - Integrate into pipeline
- SUBMIT v2 (improved recall)
Note: Phase 5 (Web Scraping) is handled by another team member
Total Est: 4-6h (matching only - no scraping)
- Architecture: Modular structure (not single script/notebook)
- LLM: Skip for now - focus on rule-based matching (5 passes instead of 6)
- Match Policy: Keep all valid matches per source, allow target reuse (maximize recall)
- Dynamic Attributes: Extract ALL specs, store with original German keys (no translation)
- Fully Configurable Weights: Pass 4 weights for ANY spec via CLI (not just 7 core attributes)
- No Pre-filtering/Post-validation: Test all targets initially, add if needed based on results
- Scope: Matching algorithm only - scraping handled by another team member
- Web App: https://hackathon-production-49ca.up.railway.app/
- Submission Format: README:155-188 (JSON with source_reference + competitors[])
- Scoring: README:99-128 (Recall 30pts > Precision 10pts > Coverage 10pts)
- Dependencies: rapidfuzz (for fuzzy matching)
- ✅ Step 1.1: config.py - Complete and tested
- All paths verified
- Weights sum to 1.00
- Focused on matching only (no scraping)
- Works with German spec keys directly
- ✅ Step 1.2: utils.py - Complete and tested
- 8 utility functions implemented: normalize_text, clean_model_number, fuzzy_match_text, fuzzy_match_brand, substring_match, calculate_size_score, calculate_price_score, extract_size_from_text
- All tests passing (47/47 assertions passed)
- German character handling (ä→AE, ö→OE, ü→UE, ß→SS)
- Model number suffix removal (UXXN, .AEU, XXN, etc.)
- Multi-pattern size extraction (Zoll, inch, cm, model numbers)
- Fuzzy matching with rapidfuzz (ratio and token_sort_ratio)
- ✅ Step 1.3: data_loader.py - Complete and tested
- 5 functions implemented: load_json, load_source_products, load_target_products, validate_product, get_data_summary
- All functions tested and working correctly
- Proper error handling for missing files and invalid JSON
- Data summary stats: Sources (17 total, 10 with EAN/GTIN 58.8%, 16 with specs 94.1%, 16 with price 94.1%)
- Data summary stats: Targets (561 total, 86 with EAN/GTIN 15.3%, 351 with specs 62.6%, 184 with price 32.8%)
- Handles both 'ean' field and 'GTIN' in specifications
- Handles both 'price' and 'price_eur' fields
- ✅ Step 1.4: attribute_extractor.py - Complete and tested
- 8 functions implemented: extract_brand, extract_model_number, extract_screen_size, extract_display_type, extract_resolution, extract_all_specs, extract_all_specs_batch, display_types_compatible
- All tests passing (41/41 assertions passed)
- Intelligent multi-strategy extraction with fallbacks for all critical fields
- Brand extraction: brand field → Hersteller → Marke → known brands in name → first word
- Model extraction: specifications → regex patterns (Samsung/LG/Sony/Generic) → reference field
- Size extraction: specifications → name → model number (supports Zoll, inch, cm, model codes)
- Display type: specifications → name (normalized to LED/QLED/OLED/LCD/MINI_LED)
- Resolution: specifications → name (normalized to UHD_4K/FULL_HD/HD_READY)
- Preserves ALL additional specs with German keys
- Extraction success rates: brand 100%, model 100%, size 94.1%, display 94.1%, resolution 94.1%
-
✅ matchers/base_matcher.py - Abstract base class
- Abstract match() method - all matchers must implement
- Helper methods: _get_brand_similarity(), _get_size_similarity(), _validate_brand(), _validate_size()
- Statistics tracking: _update_stats(), get_stats(), print_stats()
- Verbose logging: _log() method (prints if verbose=True)
- All tests passing (test_base_matcher.py)
-
✅ matchers/ean_matcher.py (Pass 1: EAN Exact Match)
- Logic: Exact EAN match + brand validation ≥0.6
- Confidence: 95-100% (0.95 + brand_sim * 0.05)
- Uses EAN index for fast lookups
- Test results: 3 matches found (Samsung F6000, PEAQ 32GF/43GQU)
- Match rate: 30.0% of sources with EAN
- All tests passing (test_ean_matcher.py)
-
✅ matchers/model_matcher.py (Pass 2: Model Number Match)
- Logic: Substring OR fuzzy ≥90% match + brand validation ≥0.6
- Confidence: 85-95% for substring, 80-90% for fuzzy
- Test results: 9 matches found (exceeded expected 6-8)
- Match rate: 52.9% of sources
- Note: Many matches use reference as model number fallback
- All tests passing (test_model_matcher.py)
-
✅ matchers/attribute_matcher.py (Pass 3: Strict Attribute Match)
- Logic: Brand≥0.8 + Size±2" + Display compatible + Resolution matches
- Confidence: 70-85% based on attribute alignment
- Test results: 60 matches found (exceeded expected 3-5, good for recall!)
- Match rate: 375.0% (multiple matches per source)
- Note: Common attributes (32" Full HD TVs) lead to many compatible matches
- All tests passing (test_attribute_matcher.py)
-
✅ matchers/weighted_matcher.py (Pass 4: Weighted Score Match)
- Logic: Weighted similarity score ≥0.60 with configurable weights
- Default weights: brand (25%), size (20%), display (15%), resolution (15%), model (10%), price (10%), name (5%)
- Confidence: 60-75% based on score above threshold
- Supports custom weights for ANY attribute via constructor
- Normalizes score by applied weights (missing attributes don't unfairly lower score)
- All tests passing (test_weighted_matcher.py)
-
✅ matchers/aggressive_matcher.py (Pass 5: Aggressive Match - Fallback for Orphans)
- Logic: Two strategies for orphan sources:
- Brand≥0.6 AND Size±2" → confidence 50-60%
- Weighted score ≥0.50 → confidence 40-55%
- Only processes orphan sources (unmatched in previous passes)
- Confidence: 40-60% based on match quality
- Designed to maximize recall for difficult cases
- All tests passing (test_aggressive_matcher.py)
- Logic: Two strategies for orphan sources:
Phase 2 Summary:
- All 5 matchers implemented and tested
- Total test files: 6 (base + 5 matchers)
- All matchers follow consistent interface via BaseMatcher
- Confidence ranges properly distributed: Pass 1 (95-100%) → Pass 5 (40-60%)
- Ready for pipeline integration
-
✅ pipeline.py - Orchestrates the 5-pass matching system
- Runs all matchers sequentially (Pass 1 → Pass 5)
- Merges results from all passes
- Deduplicates matches (keeps highest confidence per source-target pair)
- Handles orphan detection for Pass 5 (only runs on unmatched sources)
- Runtime configuration of thresholds (weighted_threshold, aggressive_threshold)
- Statistics and logging (get_stats(), print_summary())
- Test results with real data: 394 matches, 100% coverage, avg confidence 0.660
- Pass breakdown: Pass 1 (3), Pass 2 (6), Pass 3 (52), Pass 4 (333), Pass 5 (0)
- All tests passing (test_pipeline.py)
-
✅ submission.py - Formats output to submission JSON structure
- format_submission() - Converts matches to official format per README:155-188
- save_submission() - Saves official submission JSON
- save_submission_with_confidence() - Extended format with confidence/pass metadata for analysis
- print_submission_summary() - Statistics display
- Handles both required fields (source_reference, competitors[].reference) and optional metadata (retailer, name, url, price)
- Groups matches by source, includes all competitors per source
- All tests passing (test_submission.py)
-
✅ main.py - CLI entry point with full configurability
- Full CLI interface with argparse
- I/O options: --source, --target, --output-dir, --output-name
- Pass selection: --passes 1 2 3 4 5 (choose which passes to run)
- Custom weights: --weight brand=0.30 --weight wifi=0.10 (configure ANY attribute)
- Thresholds: --weighted-threshold, --aggressive-threshold
- Output control: --verbose, --stats, --save-specs, --save-extended, --no-metadata
- Integrates pipeline, submission formatter, and data loaders
- Test results: All configurations working (default, custom weights, selective passes)
- Produces official submission JSON ready for scoring
Phase 3 Summary:
- Complete matching system end-to-end
- 3 new files: pipeline.py, submission.py, main.py
- 3 test files: test_pipeline.py, test_submission.py
- CLI fully functional with comprehensive options
- Real data results: 394 matches, 100% source coverage, 23.18 avg matches/source
- Ready for submission and scoring
Session 2 (2026-03-07): Detailed pipeline design complete.
Session 3 (2026-03-07): Phase 1 detailed plan with dynamic attribute extraction strategy.
Session 4 (2026-03-07): Simplified config.py - matching only, no translation. Using agent.md for context.
Session 5 (2026-03-07): Implemented and tested utils.py - all 8 functions working correctly.
Session 5 (2026-03-07): Implemented and tested data_loader.py - all 5 functions working with accurate data insights.
Session 5 (2026-03-07): Implemented and tested attribute_extractor.py - Phase 1 complete! All 8 functions with intelligent fallback logic working perfectly.
Session 6 (2026-03-07): Implemented and tested base_matcher.py - Abstract base class complete. Starting Phase 2: Matchers.
Session 6 (2026-03-07): Implemented and tested ean_matcher.py (Pass 1) - 3 matches found with real data.
Session 6 (2026-03-07): Implemented and tested model_matcher.py (Pass 2) - 9 matches found (exceeded expected 6-8).
Session 6 (2026-03-07): Implemented and tested attribute_matcher.py (Pass 3) - 60 matches found (exceeded expected 3-5, high recall).
Session 6 (2026-03-07): Implemented and tested weighted_matcher.py (Pass 4) - Flexible weighted scoring with custom weight support.
Session 6 (2026-03-07): Implemented and tested aggressive_matcher.py (Pass 5) - Orphan fallback with 2 strategies. Phase 2 COMPLETE!
Session 7 (2026-03-07): Implemented and tested pipeline.py - Orchestrates all 5 passes with deduplication and stats. 394 matches, 100% coverage.
Session 7 (2026-03-07): Implemented and tested submission.py - Formats matches to official JSON per README. Handles required and optional fields.
Session 7 (2026-03-07): Implemented and tested main.py - Full CLI with custom weights, pass selection, thresholds. Phase 3 COMPLETE! System ready for submission.