Question: AI bots, do we need to add them?

[redimongo]

So I know that bots are bad, and happy to see if I can find a full list of AI bot user-agents. Just wondering before I do all that, are we wanting to add them to bots? Some may already be in bots file (btw)

Note this image comes from Cloudflare.

[johnspurlock]

Hi Russell, most of those agents already have entries in bots.json. Note most podcast analytics systems categorize any unknown user-agent containing the text 'bot' to be a bot anyway, so even though GPTBot was not explicitly in the bots file, no one was counting downloads from them.

I did just add an explicit entry for GPTBot however, since it definately fits the definition of well-known.

Happy to add explicit entries for any other bots not already classified, if you see them hitting your media files (I usually double check to make sure they appear in OP3 logs as well).