Magpie Ecommerce Data API — Python client & pipeline recipes

Production-ready Python pipelines for Southeast Asian ecommerce data — Shopee, Tokopedia, TikTok Shop, Lazada, Blibli and Amazon.

Monthly GMV, units sold, market share, pricing and ratings across six countries — plus on-demand scraping of product, search and merchant pages. One Python client, one credit wallet, copy-paste pipelines that land data in S3, GCS or your warehouse.

pip install magpie-data

New accounts get free trial credits — enough to run every recipe here. Sign up at data.magpieiq.com →

---

60-second quickstart

from magpie_data import MagpieClient, Exports

client = MagpieClient()  # reads MAGPIE_API_KEY

# 1. Price it first — estimates are always FREE and charge nothing.
exports = Exports(client)
estimate = exports.estimate(
    country="ID",
    category_3=["Facial Serum"],
    date_from="2025-05",
    date_to="2025-05",
)
print(estimate)  # → 289,621 rows · 2,896 credits

# 2. Only then commit — with a hard spend ceiling.
job = exports.submit(
    country="ID",
    category_3=["Facial Serum"],
    date_from="2025-05",
    date_to="2025-05",
    format="parquet",
    max_credits=5_000,  # refuses to run if it would cost more
)

# 3. Land it straight in your lake — streamed, never buffered in memory.
files = job.wait().download_to("s3://my-data-lake/magpie/facial-serum/2025-05/")
print(files)

That is the whole pattern: estimate → submit → land. No cost surprises, no manual polling, no temp files.

---

Two APIs, one wallet

	Data API	Scraping API
What	Aggregated monthly metrics — GMV, units, market share, price, rating	On-demand collection from live product, search and merchant pages
Grain	Brand · category · merchant · SKU	Individual products, keywords or shops
Freshness	Monthly, check on Data Estimator page	Live, on request
Shape	Query (JSON/CSV) or bulk export (Parquet/CSV)	Async job per product → gzipped JSONL
Use for	Market sizing, share tracking, category analysis	Price monitoring, rank tracking, competitor watching

Both bill from the same prepaid credit balance, so one API key works for both and you're not reconciling two invoices.

Scraping quickstart

from magpie_data import MagpieClient, Scraping

scraping = Scraping(MagpieClient())

job = scraping.submit(
    "tiktok_pdp",
    [
        {"product_id": "1729...", "country": "id"},
        {"product_id": "1730...", "country": "id"},
    ],
)
print(job.job_id, job.cost)  # persist job_id — jobs outlive your process

for row in job.wait().rows():  # streams gzipped JSONL, never buffers it all
    print(row["sku_name"], row["price"])

Available products — each has its own item limit, enforced client-side so a mistake costs nothing:

Product key	What it collects	Max items/job
tokopedia_pdp · blibli_pdp · tiktok_pdp · amazon_pdp	Product detail pages	2,000
tokopedia_search · amazon_search	Search / category listings	50
tokopedia_merchant · blibli_merchant	Merchant catalogues	50

Two things to design around:

• You poll; nothing is pushed. There are no webhooks or callbacks. wait() polls with backoff, but for long jobs persist job_id and reattach later with resume() — that keeps a scheduler worker free instead of blocking it.
• Partial success is normal. A job can finish partial_complete with some items failed. That is a data-quality signal, not an error: check job.counts and decide your own tolerance.

The interesting part is combining both APIs: use the Data API to find who matters in a category, then point the Scraping API at exactly those SKUs every day. That's recipe 00 — and it's the cheapest way to run daily monitoring, because the long tail of a category contributes very little GMV.

---

Recipes

Each folder is a self-contained, runnable pipeline with its own README, cost estimate and expected runtime.

Recipe	What it does	Stack
00 · Market share → daily price monitoring	Rank SKUs by GMV, then track those prices daily	Both APIs · S3
01 · Monthly export to S3	Land a category's raw SKU rows in S3 every month	Data API · S3 · GitHub Actions
02 · Daily price & stock monitoring	Scrape your own SKU list daily, diff prices	Scraping API · S3
03 · Airflow export DAG	The same export, production-shaped	Data API · Airflow
04 · Load into BigQuery / Snowflake	Export → object store → warehouse	Data API · BigQuery · Snowflake
05 · Share of search	Who owns the first page for your keywords	Scraping API
06 · Amazon cross-market	Rank a keyword across Amazon US/UK/DE… storefronts	Scraping API

Start with 01 if you just want data in your lake; 00 if you want the argument for using both APIs together.

---

Coverage

Aggregated metrics + exports (Data API) — monthly. For the date range currently available, check the Data Estimator:

Marketplace	ID	TH	VN	SG	PH	MY
Shopee	✅	✅	✅	✅	✅	✅
Tokopedia	✅
TikTok Shop	✅	✅	✅	✅	✅	✅
Lazada	✅	✅	✅	✅	✅	✅
Blibli	✅

Check what is currently exportable with Exports(client).catalog() — only baked data is sellable.

Live scraping (Scraping API) covers Tokopedia, Blibli, TikTok Shop and Shopee across Southeast Asia, plus Amazon on its own country set (sg, jp, us, uk, de, fr, au, it, es, ca) — Amazon is scraping-only and is not part of the aggregated metrics.

---

Why not just build your own scraper?

An honest comparison, because sometimes rolling your own is right.

	Your own scraper	This API
Anti-bot handling	You maintain it, forever	Handled
Layout changes	Breaks silently, you find out from stale dashboards	Handled
Proxy cost & rotation	Your problem, and it's the biggest line item	Included
Historical data	You only have data from the day you started	Already collected — query it today
Market share	Needs full-category coverage — very expensive to scrape	Precomputed
Time to first row	Weeks	Minutes
Build your own when	you need a site we don't cover, or logic so bespoke no API fits

If you only need a handful of URLs occasionally, a simple script is genuinely fine. This is for when you need it reliable, historical, and on a schedule.

---

Credits & cost control

Pricing is prepaid credits, shared across both APIs. Two habits keep it predictable:

1. Estimate first. estimate=True on metrics, or Exports.estimate(), returns the row count and price and charges nothing.
2. Set a ceiling. max_credits= on submit() refuses to run an unexpectedly expensive scope — worth setting in anything automated, where a widened filter could quietly cost a lot.

Retries are safe: every export submit carries an idempotency key, so a pipeline task that retries after a timeout reattaches to the original job instead of paying twice.

Full pricing →

---

Installation

pip install magpie-data              # core
pip install "magpie-data[s3]"        # + S3
pip install "magpie-data[gcs]"       # + Google Cloud Storage
pip install "magpie-data[parquet]"   # + local Parquet reading
pip install "magpie-data[all]"       # everything

Set your key:

export MAGPIE_API_KEY="..."          # or copy .env.example → .env

Python 3.9+.

---

Responsible use

This repo is for legitimate commercial data collection, and there are lines we hold:

• Public catalog data only — no personal data, no logged-in or private content.
• No evasion techniques. You won't find anti-bot circumvention here and we won't accept PRs adding it.
• Respect rate limits. The recipes are paced deliberately; please don't "optimise" that away.
• Your responsibility. You are responsible for complying with each marketplace's terms of service and the laws that apply to you. Nothing here is legal advice.

---

Documentation & support

• API documentation — full endpoint reference
• Data dictionary — every field explained
• Pricing — credit costs
• Service levels

Issues and PRs are welcome on this repo. Support here is best-effort; for commercial support use the contact route in the docs.

License

MIT — see LICENSE. Use it in commercial work freely.