🏠 PropMarker: Semantic Property Key-Tag Tagger & Evaluator

An enterprise-grade, LLM-powered engine for semantic key-tag extraction from UK residential property descriptions. This project acts as a drop-in replacement for traditional, brittle regex/keyword-matching pipelines—leveraging advanced Language Models (such as OpenAI's gpt-4o-mini or Google's gemini-2.0-flash) via structured schema parsing to tag properties with absolute precision.

---

📖 Table of Contents

1. 🌟 Why PropMarker is a Game-Changer
2. ⚙️ System Architecture
3. 📂 Directory Structure
4. 🚀 Quick Start Guide
5. 🖥️ Interactive Web Dashboard
6. 📊 Evaluation & Analysis Suite
7. 🔍 Visual Manual Review Tool
8. 🕷️ Deep Dive: Web Scraping Protocol
9. 🧠 Deep Dive: Structured LLM Schema & Disambiguation

---

🌟 Why PropMarker is a Game-Changer

• Semantic Cognition vs. Brittle Regex: Traditional keyword-matching pipelines fail on context and phrasing. For example, recognizing a property is offered with "no onward chain", "chain-free", or "no upper chain" requires maintaining dozens of complex regular expressions. PropMarker uses LLMs to grasp semantic equivalence out-of-the-box.
• Strict Disambiguation Guardrails: The LLM prompts incorporate strict property guidelines to distinguish tricky overlapping concepts (e.g., separating notice_of_offer from in_receipt_of_offer, and filtering out false positives for ex_local_authority caused by common mentions of "Council Tax Band").
• Pydantic Structured Output Control: PropMarker guarantees JSON shapes conform to strict validation criteria by mapping all extracted property attributes to a typed Pydantic class (KeyTagsLLMOutput). All labels are outputted as binary flags (0 or 1), accompanied by textual justifications (reason_* fields) for complete auditability.
• Fail-safe Resilience: In production, the tagger automatically falls back to an all-zero schema in the event of rate-limiting, authentication failures, or formatting issues. The surrounding pipeline never crashes.
• Cost & Token Transparency: Real-time callback listeners monitor token usage and exact financial costs of API calls, providing comprehensive telemetry.

---

⚙️ System Architecture

The following diagram illustrates how PropMarker scrapes, normalizes, extracts, evaluates, and reports on property data:

graph TD
    A[Rightmove URL / Raw Text] --> B[Rightmove Scraper Engine]
    B -->|Regex PAGE_MODEL Parser| C[JS Page Model JSON Extraction]
    C -->|Deref Reference Arrays| D[Normalized Property Data]
    D -->|Description, Features, Tenure, Lease| E[Structured LLM Tagger]
    E -->|LangChain + Pydantic Schema| F[Structured KeyTagsLLMOutput]
    F -->|Validation & Disambiguation Guardrails| G{Extraction Valid?}
    G -->|Yes| H[Semantic Key-Tags Dict]
    G -->|No / API Error| I[Zero-filled Fail-safe Schema]
    H --> J[Output Channels]
    I --> J
    J --> K[Real-time Web Dashboard]
    J --> L[CSV Evaluation Datasets]
    L --> M[Analytics & Comparison Suite]
    L --> N[Visual Manual Review Tool]

Loading

---

📂 Directory Structure

The project layout is divided into decoupled, portable modules:

key_tags/
│
├── LLM_Key_Tags_2.py            # Core LLM tagger engine & batch evaluator
├── server.py                    # Web server for the interactive tagging dashboard
├── .env.example                 # Environment configuration template
├── .gitignore                   # Version control exclusion file
├── README.md                    # Visual system documentation
│
├── public/                      # Web dashboard frontend static assets
│   ├── index.html               # Modern UI layout
│   ├── index.css                # Premium styling (vibrant dark glassmorphism)
│   └── index.js                 # Event handlers & API communications
│
├── data/                        # Project datasets & evaluation directories
│   ├── input/                   # Source CSV property datasets
│   │   └── rightmove_data.rightmove_outcodes_sale_for_interns.csv
│   └── output/                  # Raw evaluation results & statistical summaries
│       ├── eval_output.csv      # Baseline comparison dataset (200 properties)
│       └── eval_output_summary.csv
│
├── tools/                       # Multi-purpose evaluation & metrics analysis scripts
│   ├── analyze.py               # Generates accuracy metrics, token logs, & cost statistics
│   ├── analyze_review.py        # Compiles metrics based on human feedback CSVs
│   ├── generate_review_tool.py  # Compiles reports/review_tool.html visual report
│   ├── compare.py               # Diffs accuracy changes between two specific runs
│   ├── compare2.py              # Compares different baseline models
│   ├── compare_25.py            # Runs quick comparative metrics for subsets
│   ├── generate_compare_table.py# Generates markdown comparison tables
│   └── generate_table.py        # Generates basic markdown reports
│
├── scripts/                     # Automated utility & maintenance scripts
│   ├── fix_exclude.ps1          # PowerShell tag exclusion utility
│   └── fix_prompt.ps1           # PowerShell prompt adjustment utility
│
└── reports/                     # Visual metrics and debugging assets
    ├── table.md                 # Markdown comparison summary table
    └── review_tool.html         # Interactive color-coded visual debugger

---

🚀 Quick Start Guide

1. Prerequisites

Ensure you have Python 3.9 or higher installed.

2. Setup Virtual Environment & Install Dependencies

Run the following commands in your terminal to initialize the environment:

# Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate      # On macOS/Linux
.venv\Scripts\activate         # On Windows

# Upgrade pip and install requirements
pip install -r requirements.txt
# Alternatively, install standard dependencies:
pip install langchain langchain-openai langchain-core pydantic python-dotenv requests word2number langchain-community

3. Configure API Keys

1. Copy the environment configuration template:

   cp .env.example .env

2. Open .env and fill in your keys:

   OPENAI_API_KEY=sk-proj-yourActualOpenAiKeyHere
   GOOGLE_API_KEY=AIzaSy-yourActualGoogleKeyHere # (Optional, for Gemini model testing)

---

🖥️ Interactive Web Dashboard

PropMarker hosts a modern web interface that allows users to paste Rightmove URLs and witness the scraper, parser, and structured LLM tagger operate in real-time.

python server.py

• Dashboard URL: Open http://localhost:8002 in your web browser.
• Offline Demo Mode: To experiment without incurring LLM cost or requiring an API key, use URL https://www.rightmove.co.uk/properties/159407213#/?channel=RES_BUY (Caroline's Cottage) which falls back to offline stubbed results.
• UI Features: Serves categorized tag breakdowns, visual active badges, token tallies, call costs, and specific textual justifications matching the listing.

---

📊 Evaluation & Analysis Suite

For batch processing and model optimization, PropMarker comes equipped with a comprehensive evaluation and command-line diagnostics system:

A. Run Batch Evaluation

Evaluate the tagging engine across your CSV datasets to measure parsing performance against Ground Truth (GT):

# Evaluate the complete input dataset
python LLM_Key_Tags_2.py

# Evaluate a small subset (e.g., first 5 properties)
python LLM_Key_Tags_2.py --limit 5

# Evaluate a specific listing using its property ID (PID)
python LLM_Key_Tags_2.py --pid 171376361

B. Compute Statistical Metrics

Analyze run outputs to view average accuracy, overall token expenses, active ambiguities, worst-performing properties, and classification metrics:

python tools/analyze.py

C. Differential Run Comparison

Compare the accuracy impact of modifications to prompts, guidelines, or models:

# Compare two specific run output CSV files
python tools/compare.py

# Compare baseline model results against newer test files
python tools/compare2.py

---

🔍 Visual Manual Review Tool

Reviewing raw CSV datasets is tedious and error-prone. The Manual Review Tool compiles your evaluation logs into a rich interactive HTML visual debugger:

• 🟩 Green Badges (✓): Correct match (GT and LLM agree the tag is present).
• 🟥 Red Badges (FP): False Positive (LLM tagged it, but Ground Truth says 0).
• 🟨 Yellow Badges (FN): False Negative (LLM missed it, but Ground Truth says 1).

Generate and Open:

1. Compile the HTML report:

   python tools/generate_review_tool.py

2. Double-click reports/review_tool.html to open it in any web browser.
3. Filter properties instantly by Tag, Error Type (FP/FN), or search by PID.

---

🕷️ Deep Dive: Web Scraping Protocol

The Rightmove scraping mechanism resides in LLM_Key_Tags_2.py and server.py. Rather than relying on fragile DOM queries (which break as soon as Rightmove updates its class names), it uses an elegant page model parser:

1. Protocol Headers: Requests are made with realistic User-Agent and Accept headers to mimic a web browser and avoid bot blocking.
2. PAGE_MODEL Parsing: Rightmove embeds the entire page metadata as a single JSON object inside a script block. The scraper targets this via regex:

   re.search(r"window\.PAGE_MODEL\s*=\s*(.*?)</script>", html_content, re.DOTALL)

3. De-referencing Index Tables: To optimize bandwidth, Rightmove uses pointer indices (integers) mapping back to a shared array. PropMarker recursively dereferences these indices to rebuild the true descriptive strings, property types, list of features, and lease metadata.

---

🧠 Deep Dive: Structured LLM Schema & Disambiguation

Structured Output Schema

The schema mapping is enforced by LangChain's .with_structured_output(KeyTagsLLMOutput) utilizing Pydantic constraints:

class KeyTagsLLMOutput(BaseModel):
    quick_sale: int = Field(description="1 if property listing explicitly targets quick buyers, investment buyers, cash buyers...")
    no_chain: int = Field(description="1 if property has no onward chain...")
    # ... additional tag declarations
    reason_quick_sale: str = Field(description="Verbatim quote or short explanation of the quick_sale tag decision")
    # ... additional justification strings

Prompt Guardrails

Guidelines are strictly defined to combat common classification pitfalls:

• Ex-Local Authority: Triggers only if the description specifies it was ex-council/LA. Does not trigger simply because council tax details are mentioned.
• Repossession / Notice of Offer: Triggers only if the listing declares repossession status. Under UK guidelines, listings must publish a "Notice of Offer" if they receive a bid on a repossessed asset; PropMarker distinguishes this from normal offer processes.
• Shared Ownership: Ensures that general marketing phrases (e.g. "shares amenities") are not misconstrued as shared equity ownership.