[Discussion] Create additional list with simplified (non wildcard) format for piHole etc.

[bluetardis]

Hi @popcar2

I have done some testing with current (v6.6...)

You can add regex but its one line at a time.
Pihole does prefer subscribing to lists in .txt

Have some initial testing with pihole.
The best solution would be to get the list with the explanations but simplified formatting. eg

#block low effort ai spam tech blog
badaisite.com
anotherpainai.com

Its import will ignore the other notes etc so hopefully this can just be parsed as an output.
Doesn't need to have the extra /blog etc

A simple example is here but the lists can be massive. The gravity database engine would prefer thousands of entries vs wildcards as its tuned to be significantly faster for that.
https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt

Bought you some tea too !

[popcar2]

Hi, thanks for your donation :) I did see it earlier. I don't have alternate versions of the list since it can be a bit of a pain to maintain multiple files instead of the one .txt, but you can do this all at once using Regex.

1. Use something like https://regex101.com/
2. Since you want only the domains you can use this regex: ^[^:]+:\/\/.*?\.([^\/]*)(?:\/.*)?$
3. Put in the list, then click on export matches
4. Export for plaintext (make sure to turn off "include full match".
5. All done. You should have a list of domains you can copy to your pihole.

Although keep in mind some parts of the blocklist are made with subdomains in mind, it's not my intention to entirely block some domains. Hope this helps.

[bluetardis]

Thanks for that - it worked fine but will have to be manually updated.
For anyone else reading this - you have to load the file eg badlist.txt to the Pihole itself and then you can add it manually.

To import to pihole
go to the subscribed lists

You can add local files in the admin panel as well. Under Group management > Adlists you can add files like file:///home/pi/adlists/list1.txt

The file should only contain one domain per line. Hosts format is fine too but Pi-hole will just ignore the IP addresses anyway.

Not sure if I can persuade you with a few more coffees to keep a regexed parsed copy of your file with just the hosts available online so it stays up to date?

[popcar2]

I'll look into whether it can be automated without much effort on my end - but no promises there 🙃

[dezza]

@popcar2 in the long run it could be beneficial to have a basic format, a GET-query to filter them (filter by block type, explicitly exclude, output in different format).

I have the same considerations for my own list. It would also be nice to have some shellscript that verifies that the hostnames are still up and running - if they are dead for more than X time mark them dead and consider pruning them from the list.

[VinceCarbone]

Just here to say I'd also like a pi-hole compatible list. I mean I could probably make something that downloads your list, uses regex to instead make it pi-hole compatible, and then re-upload said list to another repo, but that seems a bit silly if the pi-hole compatible list could just exist here :/

[VinceCarbone]

Decided to just proactively do it because it's easy enough. Here's the pi-hole compatible list, generated based on the content in popcar2's repo

https://raw.githubusercontent.com/VinceCarbone/BadWebsiteBlocklist-For-Pi-hole/refs/heads/main/BadWebsiteBlocklist-For-Pi-Hole.txt