Skip to content

Data_Sources.md

Latest commit

 

History

History
76 lines (50 loc) · 4.38 KB

Data_Sources.md

File metadata and controls

76 lines (50 loc) · 4.38 KB
Raw

Data Sources Reference

The Football Market Intelligence Engine integrates a diverse array of data sources, spanning historical match archives, financial market valuations, global ranking systems, and real-time APIs. This document catalogs the provenance of the core datasets used to drive the machine learning models and market evaluations.

1. Historical Match Data

The foundation of the prediction models relies on comprehensive historical international football results.

  • Fjelstul World Cup Database

    • Link: Fjelstul World Cup Database on GitHub
    • Usage: Serves as the primary source of truth for all historical FIFA World Cup matches, including granular details on tournament stages, lineups, and specific match events.

  • International Football Results (1872 to 2026)

    • Link: Kaggle Dataset
    • Usage: Provides the vast historical context of all international matches (friendlies, qualifiers, regional tournaments) outside of the World Cup finals. This dataset is crucial for generating continuous Elo ratings and team form metrics over long historical windows.

2. Ratings & Rankings Data

To quantify the relative strength of international teams, the engine utilizes both official and mathematical ranking systems.

  • World Football Elo Ratings

    • Link: eloratings.net
    • Usage: The core objective measure of team strength used heavily by the ML pipeline. The Elo rating system acts as a zero-sum mechanism for evaluating team quality based on match outcomes and opponent strength.

  • Official FIFA Men's World Ranking

    • Source: Scraped from the official FIFA website.
    • Local File: data/Elo/fifa_mens_rank.csv
    • Usage: Included as a secondary rating feature (fifa_rank_difference, fifa_points_difference) in the Modern XGBoost model to capture the official administrative view of team strength.

3. Player & Financial Data

In the modern era (post-2004), a team's aggregated financial value and individual player pedigree is one of the strongest predictors of success.

  • Transfermarkt

    • Link: transfermarkt.com
    • Usage: The definitive source for player market values, club affiliations (identifying "elite club" players), and transfer history.

  • Comprehensive Football Datasets (Kaggle)

    • Link: Kaggle Football Datasets
    • Local Directory: The entirety of the CSV files residing in the data/Player/ directory (e.g., player_national_performances, player_profiles, player_market_value).
    • Usage: These datasets power the deep player-level analytics and provide the necessary data to compute metrics like squad_total_market_value, squad_total_international_caps, and elite_club_players_diff.

  • Current World Cup Squads

    • Source: Scraped from the official FIFA website.
    • Local File: data/Current World Cup Teams/worldcup_squads.json
    • Usage: Provides the official starting rosters and 26-man squads for the upcoming/current tournament, ensuring the models evaluate the exact players available for selection.

4. Live APIs & Feeds

To transition from historical backtesting to live market evaluation, the system requires continuous streams of real-world operational data.

  • API-Football
    • Link: api-football.com
    • Usage: The primary operational heartbeat of the application. Used via poll_api_football.py to continuously fetch live fixtures, match results, starting lineups, in-play events, and injury updates.

5. Prediction Markets & Odds

The engine evaluates its predictions against actual market prices to find arbitrage and value betting (EV+) opportunities.

  • Kalshi

    • Link: kalshi.com
    • Usage: Polled by poll_betting_markets.py to ingest regulated prediction market data, extracting implied probabilities and tracking market liquidity and open interest.

  • Polymarket

    • Link: polymarket.com
    • Usage: Polled by poll_betting_markets.py to ingest decentralized prediction market data, providing a high-volume crypto-native perspective on tournament futures and match odds.