AI just met a new kind of barrier, and this time, it comes with a price tag.
Artificial intelligence (AI) bot blocker
Cloudflare, which underpins around 20% of the internet, has announced a new system to block AI bots from accessing website content without permission. It’s a big deal for publishers, especially those who feel that AI companies have been building powerful tools using their work, without credit, consent, or compensation.
Prominent sites like Sky News, Buzzfeed, and The Associated Press are among the first to use the tool, which eventually could include a “Pay Per Crawl” option, allowing content owners to charge AI firms for scraping their content. For creators, journalists, and digital publishers, it feels like a long-overdue line in the sand.
As Condé Nast’s CEO put it: “This is a critical step toward creating a fair value exchange on the Internet that protects creators, supports quality journalism and holds AI companies accountable.”
But not everyone is convinced this is the silver bullet.
From a technical perspective, it may kick off another arms race. As our Head of Web noted:
“Bots will get good at avoiding this, then this too will catch up, and so on. It’s cat and mouse. Although if anyone’s going to build the best version of this, it’s Cloudflare.”
We’ve seen similar moves before, like the paywalls introduced by major publishers, which were often easily bypassed or undermined by open access to alternative sources. Some predict that AI scraping restrictions will follow a similar path, unless there’s stronger legal backing to support them.
Still, the debate points to a larger truth:
AI is currently useless without human-created data.
It relies on vast pools of knowledge, content, and creativity, the majority of which comes from people, not machines. And if the people behind that content want to opt out, be paid, or demand attribution, shouldn’t that be their right?
Interestingly, not all creators may want to block bots. Some may prefer the opposite, making their content more accessible, as long as it’s referenced properly and drives visibility. Especially for thought leadership and marketing content, it could be a win-win: more reach, more citations, more credibility.
That’s where this becomes less a binary issue of “yes or no to scraping,” and more a nuanced question of control, value exchange, and long-term sustainability.
As we explored in our recent blog reflecting on an AI panel discussion, we believe the future of AI must be Human + AI, not Human vs AI. Cloudflare’s move supports that philosophy, reinforcing the value of human-created content and introducing early frameworks that begin to re-establish fairness, transparency, and control. It’s a tangible example of the kind of ethical leadership we called for: designing systems that protect creativity while enabling innovation, and rebalancing the relationship between creators and the tools trained on their work.
We’re entering an era where creators need to think about:
- Where their content lives
- Who’s allowed to access it
- What value they get in return
Whether this turns out to be a landmark moment or just another temporary fix, one thing is clear: The economics of the web are being rewritten in real time, and content creators need a seat at the table.
