To keep #OpenStreetMap.org up and running while we're being deluged by scrapers, we've blocked 320,000+ primarily residential IPv4 addresses in the last 24 hours (+ 100,000 IPv6) involved in scraping.
If you need OSM data, please don't scrape the website - use the official downloads at https://planet.openstreetmap.org
🙏🌍 #AI #Bots #Abuse
@osm_tech FWIW, if the robots attacking you are anything like the ones attacking me (~790k unique IP addresses at peak, about half of them residential), you can mitigate a large majority of them with three ifs in a trenchcoat, implementable in nginx and Caddy, and probably apache and others too.
I'm able to mitigate ~60 million requests / day on a €4/month VPS. I had to scale that up once during a larger wave to a €11/month VPS, and that barely blinked when I was hit by a 2.5k request/sec wave that lasted ~4 days. (My bottleneck on the cheaper VPS was TLS, the defense mechanisms I employ are very lightweight.)
Happy to help if you need a hand, just give me a shout.
@algernon FYI Mull on android just got me mazed - x-request-id: 4J6to06jHMROFKYFdO0GL butyou have more important things on the go
@arichtman WTF are Mull doing. Chrome, but no sec-ch-ua.
I'm not having much luck in finding their Android browser... I'm seeing Mullvad VPN, and the browser in alpha for win/mac/linux, but not android. Can you point me in the right direction?
Not going to dive into it now, but I'd like to save it for my records.
@arichtman Meanwhile: I found a way to identify this particular browser, at least as long as they add an extra /<version number> part to the Chrome component.
I'll try to deploy that... uhh... sometime soon.
@osm_tech and we can tell the scrapers are AI built because a cursory glance at the documentation on the "coders" part would've prevented this problem.
@ClariNerd @osm_tech Because their IP ranges are increasingly being blocked by servers following their harmful scraping habits, AI companies are now releasing "browsers" so they can scrape from residential IPs instead and circumvent blocks. Oh, sorry, I meant "so they can empower users with AI insight in this new era of information".
