AMFI NAV Loader

Downloads mutual fund NAV data from the AMFI India portal and loads it into MySQL.

License

Apache License 2.0 — see http://www.apache.org/licenses/LICENSE-2.0. Copyright 2025.

Setup

1. Install uv, then install dependencies:

   uv sync

2. Create a .env file (see .env.sample) with your MySQL and, optionally, Telegram config.

3. Create the database and load the schema:

   mysql -u root -p -e "CREATE DATABASE IF NOT EXISTS nav_data;"
   mysql -u root -p nav_data < schema.sql
   mysql -u root -p -e "SET GLOBAL event_scheduler = ON;"

   The table (nav_data) is partitioned by year and indexed on nav_date and scheme_code; the event scheduler lets MySQL auto-create next year's partition. To keep it on across restarts, add event_scheduler = ON under [mysqld] in your MySQL config.

Usage

Command	What it does
uv run nav_loader.py	Daily job: fills in any missing/incomplete business days since the last run
uv run nav_loader.py --date YYYY-MM-DD	Downloads and inserts one specific date (e.g. to fill a gap or reprocess a day)
uv run nav_loader.py --months 3	Backfills the past 3 months
uv run nav_loader.py --yearly 5	Backfills the past 5 years
uv run nav_loader.py --dry-run [--date YYYY-MM-DD]	Downloads and parses one day's data and prints a summary — no MySQL or Telegram needed. Defaults to the latest business day. Use this to sanity-check the script, e.g. after AMFI changes the navall.txt format.

Add --notify to any job to send a Telegram summary on completion (requires TELEGRAM_BOT_TOKEN/TELEGRAM_CHAT_ID).

All jobs skip weekends and Indian public holidays.

Output

• data/ — downloaded raw NAV files (navall_YYYY-MM-DD.txt)
• logs/nav_loader.log — detailed run logs
• Temporary CSVs are written during processing and cleaned up automatically

Data Validation

Rows are dropped (and logged) if the NAV isn't numeric or the date can't be parsed. The parser reads AMFI's column layout from each file's header, so it keeps working if AMFI reorders or adds columns — but if AMFI ever removes a column this script needs (e.g. NAV or Date), parsing fails loudly rather than silently inserting bad data.

Troubleshooting

• Can't connect to MySQL — check .env credentials and that the DB server is reachable. Use --dry-run to test the download/parse path without a DB at all.
• Downloads failing — check AMFI portal accessibility and the logs for the response content; the script retries 3x with a delay.
• Unexpected row drops — run --dry-run for the date in question; it reports how many rows were parsed vs. valid and why.

Tests

uv run --with pytest pytest tests/