Overriding suffix_list_urls

[leeprevost]

Hope you are well @john-kurkowski.

I'm finding some strange behaviour where tldextract returns a null string using "top_domain_under_public_suffix" property from urls such as:

urls that end with ".er"

urls with [hostname].jp

urls with [hostname].sch.uk

extract("test.kobe.jp").top_domain_under_public_suffix
""

I think this is likely due to erroneous use of the * in the public suffix list such as:
*.er
*.sch.uk
*.kobe.jp

if this is the case and you agree, I'm now considering taking the original public suffix list and cleansing it slightly to remove the erroneous problems. I'm using tldextract to operate on billions of nodes in the common crawl corpus.

so, considering:

1. Rules regex such as (r"^*.(\w+.jp)$", r"\1") to replace "*.kobe.jp" with "kobe.jp" in a my_custom_list
2. Reinitialize with tldextract.TLDExtract(suffix_list_urls = my_custom_list)

Does this sound right? Agree these are problems in the https://publicsuffix.org/list/public_suffix_list.dat ?

[john-kurkowski]

Hi @leeprevost, good to hear from you again, and thanks for the clear examples. Your approach on this sounds right to me!

I reproduced the issue, but I don’t think it’s a tldextract bug or necessarily a PSL error. For wildcard PSL entries like *.kobe.jp, *.sch.uk, and *.er, a hostname such as test.kobe.jp or foo.sch.uk is itself treated as the public suffix. In that case, top_domain_under_public_suffix in this repo is expected to be empty, because there isn’t an additional label above the public suffix to form a registrable domain. So I’d treat this as PSL-defined behavior.

If you want different behavior for your use case, a custom suffix list is a reasonable local workaround, yep. One small heads up: suffix_list_urls expects URLs or file:// paths to a PSL-format file, not an in-memory list.

If you still suspect those wildcard entries are wrong, the best next stop would be the Public Suffix List project upstream. That said, PSL rules often reflect historical browser/cookie-boundary decisions, so the rationale for an entry is not always obvious.