Is AI Killing Information Access?
AI scrapers are crushing sites, and OA targets are especially soft
One of the biggest philosophical challenges about AI — one of its greatest opportunities or dangers, depending on your perspective (i.e., whether you’re a monster or not) — is that it has the potential to centralize information in a way that is anti-democratic and temptingly authoritarian. There is a reason Elon Musk wants access to all the information inside the US government, after all.
But Elon is not alone, it is full of aspirants to the crown. As a result, the world of AI is killing web sites with mindless crawling protocols clogging servers and taking down sites. This is causing many sites to raise their security game, leaving OA sites — which are premised on openness — particularly vulnerable, as Eric Hellman writes: