Skip to content

Latest commit

 

History

2 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Shopee Scraper

shopee-scraper

checks license

This shopee scraper runs on a dedicated endpoint, which means no selectors, no rendering and no proxy tier to pick. It also means the response is a fixed shape, and the useful question is not how to call it but which of its seventeen fields actually contain anything.

So this README is a field audit. Three tiers: what comes back populated, what is documented as permanently null, and what the landing page promises that the response does not carry at all.

Built on ScrapingBee's Shopee endpoint. Every value below is from a live call on 2026-09-15.

Indonesia only. Everything else is a 400.

The single most important constraint, and it is easy to miss:

{"errors": {"query": {"url": ["Only shopee.co.id URLs are supported."]}}}

That is the live response for shopee.sg, shopee.com.my and shopee.ph, tested individually. All three returned HTTP 400 at 0 credits. Shopee operates across Southeast Asia and Latin America, and this endpoint covers the Indonesian marketplace and nothing else.

A missing url gives the same treatment: {"errors": {"query": {"url": ["Missing data for required field."]}}}, also free. Malformed requests cost you nothing, so validate the host before you spend.

The call

GET https://app.scrapingbee.com/api/v1/shopee?url=<shopee.co.id product URL>

75 credits per successful call. Four parameters and that is the whole surface:

Parameter Required Notes
api_key yes Send as Authorization: Bearer YOUR_API_KEY
url yes A shopee.co.id product URL
add_html no Default false. Adds the raw page to the response
tag no Your own label, echoed back in the response headers
import os, requests

r = requests.get(
    "https://app.scrapingbee.com/api/v1/shopee",
    headers={"Authorization": f"Bearer {os.environ['SCRAPINGBEE_API_KEY']}"},
    params={"url": "https://shopee.co.id/...-i.93014939.1881883105"},
    timeout=200,
)
product = r.json()

Two operational notes from the documentation, both worth respecting. Failed requests are retried automatically inside the API, so a call can take a while to return and your client timeout needs headroom. And this endpoint is not available in the ScrapingBee CLI, unlike most of the others.

add_html=true is free. It added 660,071 characters of page HTML to the response and the call still billed 75.

Tier one: the fields that work

Live response for an Epson ink listing:

{
  "title": "Tinta Epson 664 / T664 Original (L120 L121 L1300)",
  "price": 103000.0,
  "currency": "IDR",
  "item_id": 1881883105,
  "shop_id": 93014939,
  "rating": 4.96,
  "reviews_count": 9263,
  "seller_name": "Kreasindo Retail Official Store",
  "category": ["Komputer & Aksesoris", "Printer & Scanner", "Tinta Printer"],
  "images": ["https://down-id.img.susercontent.com/file/..."],
  "variants": [{"model_id": 3236230061, "name": "Black (T6641)", "price": null}],
  "url": "https://shopee.co.id/...-i.93014939.1881883105"
}

Three more products pulled through the same call, all populated:

Price (IDR) Rating Reviews Variants Seller
75,000 4.95 414 10 CIPTA KARYA Shop
103,000 4.96 13,440 4 Kreasindo Retail Official Store
179,500 4.93 891 1 Epson Official Store Indonesia

price is a float in IDR, not a formatted string, so no parsing. item_id and shop_id come back as integers and are the stable keys to build a table on. category is a breadcrumb array in Indonesian, because the endpoint returns whatever language Shopee uses for the listing.

The discount fields, verified

original_price and discount_percent are null unless the product is currently discounted. Four products in a row came back null, which made it look like the fields never populate. They do:

{"price": 189000.0, "original_price": 599000.0, "discount_percent": 68}

189,000 against 599,000 is a 68.4 percent cut, returned as 68. The percentage is derived and rounded to a whole number, so treat it as a label and recompute from the two prices if you need precision.

Because these fields are null on undiscounted listings, discount_percent is not None is a clean promotion flag. That is the cheapest way to build a Shopee promo tracker.

Tier two: fields that are always null

The documentation is unusually straight about this, and the live calls agree.

sold_count is null on effectively every request. Shopee stopped exposing a sold count on product pages, and the field is kept for forward compatibility. Null on all five products tested. Do not build a demand model on it.

variants[].price is null in practice. Shopee publishes one price per listing rather than one per variant, so the per variant price is reserved for a future improvement. Confirmed null across 4, 10 and 1 variant listings. Use the top level price, which is the default variant's price. You still get model_id and name per variant, which is enough to enumerate SKUs even without individual prices.

html is null unless you ask for it. The key is always present, so test the value rather than the key.

Tier three: what is not in the response at all

Worth being explicit, because the Shopee scraper page names three workflows and the response supports one and a half of them.

Stock is not returned. Probing the live payload for stock, inventory, available, quantity, sold_out, in_stock and normal_stock returned false on all seven. There is no inventory field, so "monitor stock availability per SKU" is not something this endpoint does. The closest signal is that a delisted product stops resolving.

Seller metrics beyond the name are not returned. You get seller_name and shop_id. There is no shop rating, no response rate and no shop level review total. The rating and reviews_count you do get are per product, not per seller. Aggregating product ratings by shop_id across many calls is the workaround, and at 75 credits each that is a real cost to plan for.

Price tracking, which is the third workflow, works well.

Discovery: the part that needs solving

The endpoint takes a product URL, so where do the URLs come from? Not from Shopee. Two routes tested:

Route Credits Product URLs found
shopee.co.id/search?keyword= via the HTML API 10 0
shopee.co.id/...-cat.<id> category page 10 0
shopee.co.id/sitemap.xml 10 Returns an HTML page, not XML

Shopee's own search and category pages render behind a sign in prompt and expose no product links to an anonymous fetch. That is a dead end, and paying 10 credits to confirm it is a waste.

What does work is search engine discovery, at 1 credit:

curl -G "https://app.scrapingbee.com/api/v1/" \
  -H "Authorization: Bearer $SCRAPINGBEE_API_KEY" \
  --data-urlencode "url=https://html.duckduckgo.com/html/?q=site%3Ashopee.co.id+tinta+epson" \
  --data-urlencode 'extract_rules={"results":{"selector":"div.result","type":"list","output":{"url":{"selector":"a.result__a","output":"@href"}}}}' \
  -d mode=auto

That returned 12 results containing 6 real product URLs, every one of which then resolved through the Shopee endpoint. One catch: DuckDuckGo wraps results in a redirect, so the real destination is in the uddg query parameter and needs unwrapping before you match on it.

The product URL shape hands you the keys for free:

https://shopee.co.id/<slug>-i.<shop_id>.<item_id>

Both IDs are right there in the path, so you can extract them without a call, deduplicate a candidate list, and only spend 75 credits on URLs you have not already seen. The DuckDuckGo search API covers that discovery step in its own right.

Cost model

Measured from spb-cost response headers:

Call Credits
Shopee endpoint, successful 75
Shopee endpoint with add_html=true 75, no surcharge
Wrong host or missing url 0
DuckDuckGo discovery query 1

The asymmetry is the whole planning problem: discovery is 1 and extraction is 75. A pipeline that discovers 10 URLs for 1 credit and extracts all of them for 750 is spending 99 percent of its budget on extraction, so the deduplication step earns its keep immediately.

At the entry paid tier of 250,000 credits that is roughly 3,300 product pulls a month. ScrapingBee does not cache, so store every response and pull again on the cadence prices actually move rather than on a schedule.

Plan tiers are on the pricing page.

Scope

Public Shopee Indonesia product pages. Buyer accounts, order history, the chat system, Seller Centre and anything requiring a signed in session are out of reach, and scraping under login credentials is prohibited by ScrapingBee's terms of service. Shopee's own search and category pages sit behind a sign in prompt, which is why discovery runs through a search engine instead.

seller_name often identifies a small business or an individual trader, and product descriptions and images are the seller's own content, so treat both accordingly rather than as anonymous rows. Shopee Indonesia's Terms of Service govern the marketplace, and Shopee publishes an Open Platform API for authorised sellers and partners, which is the supported route when you have that relationship.

Reference: Shopee API documentation, AI web scraping, data extraction, markdown output.

Adjacent marketplace endpoints: Meesho scraper API, Amazon search API, AliExpress product API, Alibaba API, eBay scraper, Etsy API, Walmart price API, Flipkart API, marketplace API, OLX scraper.

FAQ

Does this work for Shopee Singapore, Malaysia or the Philippines? No. Only shopee.co.id. Every other host returns HTTP 400 with Only shopee.co.id URLs are supported. at 0 credits.

Why is sold_count always null? Because Shopee no longer publishes a sold count on product pages. The field is kept for forward compatibility and was null on every product tested.

Why is my variant price null? Shopee exposes one price per listing, not per variant. Use the top level price, which is the default variant. Variants still give you model_id and name.

Can I monitor stock levels? Not from this response. Seven different stock field probes came back empty. There is no inventory data in the payload.

How do I find product URLs to scrape? Through a search engine. Shopee's own search and category pages return zero product links to an anonymous fetch. A site:shopee.co.id query costs 1 credit and returned 6 usable URLs.

Do I get charged for a bad URL? No. Wrong host and missing parameter both return 400 at 0 credits.

Is the price a string or a number? A float, in IDR. 103000.0, not "Rp103.000".

License

MIT. See LICENSE.