Bots are overwhelming websites with their hunger for AI data

HELLFIRE

https://www.theregister.com/2025/06/17/bot_overwhelming_websites_report/
https://www.glamelab.org/products/are-ai-bots-knocking-cultural-heritage-offline/

Galleries, Libraries, Archives, and Museums (GLAMs) say they’re being overwhelmed by AI bots – web crawling scripts that visit websites and download data to be used for training AI models – according to a report issued on Tuesday by the GLAM-E Lab, which studies issues affecting GLAMs. … Based on an anonymized survey of 43 organizations, the report indicates that cultural institutions are alarmed by the aggressive harvesting of their content, which shows no regard for the burden that data-harvesting places on websites. …

Regards

SpHeRe31459

Surprised this is only making news now. It’s been a thing for a few months.

For those curious at what this kind of bot abuse looks like, tech podcaster and sys admin Jim Salter documented how insane these bots are on a small site back in April. It’s quite the thread. The AI scrapers change IP addresses, make fake UserAgent strings, etc. to get around being blocked. The bots sloppily scrape pages many times a second, there doesn’t seem to be any logic or caching, it’s seems purely “just because”, in effect DoS’ing small sites.
https://fosstodon.org/@jimsalter/114270715367978012