AI Bots Are Now a Signifigant Source of Web Traffic
The viral virtual assistant OpenClaw-formerly known as Moltbotand before that Clawdbot-is a symbol of a broader revolution underway that could fundamentally change how the Internet functions. Instead of a place that is mainly inhabited by people, the web can very quickly be dominated by autonomous AI bots.
IN new report measuring bot activity on the web, as related data shared with WIRED by internet infrastructure company Akamai, shows that AI bots already account for a meaningful share of web traffic. The findings also shed light on an increasingly sophisticated arms race unfolding as bots deploy clever tactics to bypass website defenses meant to keep them out.
“The majority of the Internet will be bot traffic in the future,” says Toshit Pangrahi, co-founder and CEO of TollBit, a company that tracks web scraping activity and published the new report. “It's not just a copyright issue, there's a new visitor emerging on the Internet.”
Most major websites try to limit what content bots can scrape and feed to AI systems training goals. (WIRED's parent company, Condé Nast, as well as other publishers, are currently suing several AI companies for alleged copyright infringement related to AI training.)
But another type of AI-related website scraping is now also on the rise. Many chatbots and other AI tools can now retrieve real-time information of the web and use it to increase and improve their outputs. This may include up-to-the-minute product prices, movie theater schedules, or summaries of the latest news.
According to Akamai's data, bot traffic related to training has steadily increased since last July. Meanwhile, the global activity of bots retrieving web content for AI agents is also on the rise.
“AI is changing the web as we know it,” Robert Blumofe, Akamai's chief technology officer, tells WIRED “The next arms race will determine the future look, feel and functionality of the web, as well as the basis of doing business.”
In the fourth quarter of 2025, TollBit estimates that an average of one in every 50 visits to its customers' websites was from an AI scraping bot. In the first three months of 2025, that number was just one in every 200. The company says that in the fourth quarter, more than 13 percent of bot requests involved robots.txt, a file that some websites use to indicate which pages bots should avoid. TollBit says the share of AI bots ignoring robots.txt increased 400 percent from the second quarter to the fourth quarter of last year.
TollBit also reported a 336 percent increase in the number of websites attempting to block AI bots over the past year. Pangrahi says scraping techniques are becoming more sophisticated as sites try to control how bots access their content. Some bots disguise themselves by making their traffic look like it's coming from a normal web browser or send requests designed to mimic how people normally interact with websites. TollBit's study notes that the behavior of some AI agents is now almost indistinguishable from human web traffic.
TollBit brands tools that website owners can use to charge AI scrapers to access their content. Other companies, including Cloudflare, offer similar tools. “Anyone who relies on human web traffic—starting with publishers, but basically anyone—is going to be affected,” Pangrahi says. “There has to be a faster way to have this machine-to-machine, programmatic exchange of value.”
