Content Independence Day: no AI crawl without compensation!

Almost 30 years ago, two graduate students at Stanford University — Larry Page and Sergey Brin — began working on a research project they called Backrub. That, of course, was the project that resulted in Google. But also something more: it created the business model for the web.

The deal that Google made with content creators was simple: let us copy your content for search, and we'll send you traffic. You, as a content creator, could then derive value from that traffic in one of three ways: running ads against it, selling subscriptions for it, or just getting the pleasure of knowing that someone was consuming your stuff.

Google facilitated all of this. Search generated traffic. They acquired DoubleClick and built AdSense to help content creators serve ads. And acquired Urchin to launch Google Analytics to let you measure just who was viewing your content at any given moment in time.

For nearly thirty years, that relationship was what defined the web and allowed it to flourish.

But that relationship is changing. For the first time in its history, the number of searches run on Google is declining. What's taking its place? AI.

If you're like me, you've been amazed Continue reading

The crawl before the fall… of referrals: understanding AI’s impact on content providers

Content publishers welcomed crawlers and bots from search engines because they helped drive traffic to their sites. The crawlers would see what was published on the site and surface that material to users searching for it. Site owners could monetize their material because those users still needed to click through to the page to access anything beyond a short title.

Artificial Intelligence (AI) bots also crawl the content of a site, but with an entirely different delivery model. These Large Language Models (LLMs) do their best to read the web to train a system that can repackage that content for the user, without the user ever needing to visit the original publication.

The AI applications might still try to cite the content, but we’ve found that very few users actually click through relative to how often the AI bot scrapes a given website. We have discussed this challenge in smaller settings, and today we are excited to publish our findings as a new metric shown on the AI Insights page on Cloudflare Radar.

Visitors to Cloudflare Radar can now review how often a given AI model sends traffic to a site relative to how often it crawls that site. We Continue reading

Control content use for AI training with Cloudflare’s managed robots.txt and blocking for monetized content

Cloudflare is giving all website owners two new tools to easily control whether AI bots are allowed to access their content for model training. First, customers can let Cloudflare create and manage a robots.txt file, creating the appropriate entries to let crawlers know not to access their site for AI training. Second, all customers can choose a new option to block AI bots only on portions of their site that are monetized through ads.

The new generation of AI crawlers

Creators that monetize their content by showing ads depend on traffic volume. Their livelihood is directly linked to the number of views their content receives. These creators have allowed crawlers on their sites for decades, for a simple reason: search crawlers such as Googlebot made their sites more discoverable, and drove more traffic to their content. Google benefitted from delivering better search results to their customers, and the site owners also benefitted through increased views, and therefore increased revenues.

But recently, a new generation of crawlers has appeared: bots that crawl sites to gather data for training AI models. While these crawlers operate in the same technical way as search crawlers, the relationship is no longer symbiotic. AI Continue reading

Introducing pay per crawl: enabling content owners to charge AI crawlers for access

A changing landscape of consumption 

Many publishers, content creators and website owners currently feel like they have a binary choice — either leave the front door wide open for AI to consume everything they create, or create their own walled garden. But what if there was another way?

At Cloudflare, we started from a simple principle: we wanted content creators to have control over who accesses their work. If a creator wants to block all AI crawlers from their content, they should be able to do so. If a creator wants to allow some or all AI crawlers full access to their content for free, they should be able to do that, too. Creators should be in the driver’s seat.

After hundreds of conversations with news organizations, publishers, and large-scale social media platforms, we heard a consistent desire for a third path: They’d like to allow AI crawlers to access their content, but they’d like to get compensated. Currently, that requires knowing the right individual and striking a one-off deal, which is an insurmountable challenge if you don’t have scale and leverage. 

What if I could charge a crawler? 

We believe your choice need not be binary — Continue reading

From Googlebot to GPTBot: who’s crawling your site in 2025

Web crawlers are not new. The World Wide Web Wanderer debuted in 1993, though the first web search engines to truly use crawlers and indexers were JumpStation and WebCrawler. Crawlers are part of one of the backbones of the Internet’s success: search. Their main purpose has been to index the content of websites across the Internet so that those websites can appear in search engine results and direct users appropriately. In this blog post, we’re analyzing recent trends in web crawling, which now has a crucial and complex new role with the rise of AI.

Not all crawlers are the same. Bots, automated scripts that perform tasks across the Internet, come in many forms: those considered non-threatening or “good” (such as API clients, search indexing bots like Googlebot, or health checkers) and those considered malicious or “bad” (like those used for credential stuffing, spam, or scraping content without permission). In fact, around 30% of global web traffic today, according to Cloudflare Radar data, comes from bots, and even exceeds human Internet traffic in some locations.

A new category, AI crawlers, has emerged in recent years. These bots collect data from across the web to train Continue reading

Message Signatures are now part of our Verified Bots Program, simplifying bot authentication

As a site owner, how do you know which bots to allow on your site, and which you’d like to block? Existing identification methods rely on a combination of IP address range (which may be shared by other services, or change over time) and user-agent header (easily spoofable). These have limitations and deficiencies. In our last blog post, we proposed using HTTP Message Signatures: a way for developers of bots, agents, and crawlers to clearly identify themselves by cryptographically signing requests originating from their service. 

Since we published the blog post on Message Signatures and the IETF draft for Web Bot Auth in May 2025, we’ve seen significant interest around implementing and deploying Message Signatures at scale. It’s clear that well-intentioned bot owners want a clear way to identify their bots to site owners, and site owners want a clear way to identify and manage bot traffic. Both parties seem to agree that deploying cryptography for the purposes of authentication is the right solution.     

Today, we’re announcing that we’re integrating HTTP Message Signatures directly into our Verified Bots Program. This announcement has two main parts: (1) for bots, crawlers, and agents, we’re simplifying enrollment into the Verified Continue reading

IS-IS 3-Way Handshake and the Power of SHOULD

Yesterday, I mentioned that a Cisco router running pre-standard IS-IS 3-way handshake (this is why you need it) interoperates with multiple implementations of RFC 5303. How’s that possible, and does it matter whether you configure the ancient Cisco routers (release 15.x) to use IETF 3-way handshake instead of the “proprietary” one?

TL&DR: It SHOULD NOT matter, but the more I explore the RFCs, the more I’m amazed anything works at all.

I took a trip to the Wireshark land to figure out the details (you can download the capture file):

Read more …

Start netlab Tools without Changing Topology File

Dan Partelly figured out that we have to configure the standard (IETF) 3-way IS-IS handshake on old IOSv images. On the other hand, all IS-IS integration tests pass for IOSv and IOSvL2. I wondered what was going on.

Fortunately, a few months ago, I spent some time installing the client-side Edgeshark components on my laptop. All I needed to do was enable the edgeshark tool in my lab topology and restart the lab.

Read more …

🚀 A Guide for Aspiring B.Tech CS Students: Navigating Your Journey & Preparing for the Real World

👋 Introduction My daughter will be starting her B.Tech in Computer Science at MIT, Manipal this year. As a huge AI proponent, I often share the latest AI trends and tools with my family. When my daughter decided to pursue CS, she asked me several questions about AI, which inspired this blog. I hope this … Continue reading 🚀 A Guide for Aspiring B.Tech CS Students: Navigating Your Journey & Preparing for the Real World →

Celebrate Micro-Small, and Medium-sized Enterprises Day with Cloudflare

On June 27, the United Nations celebrates Micro-, Small, and Medium-sized Enterprises Day (MSME) to recognize the critical role these businesses play in the global economy and economic development. According to the World Bank and the UN, small and medium-sized businesses make up about 90 percent of all businesses, between 50-70 percent of global employment, and 50 percent of global GDP. They not only drive local and national economies, but also sustain the livelihoods of women, youth, and other groups in vulnerable situations. 

As part of MSME Day, we wanted to highlight some of the amazing startups and small businesses that are using Cloudflare to not only secure and improve their websites, but also build, scale, and deploy new serverless applications (and businesses) directly on Cloudflare's global network. 

A startup for startups

Cloudflare started as an idea to provide better security and performance tools for everyone. Back in 2010, if you were a large enterprise and wanted better performance and security for your website, you could buy an expensive piece of on-premise hardware or contract with a large, global Content Delivery Network (CDN) provider. Those same types of services were not only unaffordable for most website owners Continue reading

Russian Internet users are unable to access the open Internet

Since June 9, 2025, Internet users located in Russia and connecting to web services protected by Cloudflare have been throttled by Russian Internet Service Providers (ISPs).

As the throttling is being applied by local ISPs, the action is outside of Cloudflare’s control and we are unable, at this time, to restore reliable, high performance access to Cloudflare products and protected websites for Russian users in a lawful manner. 

Internal data analysis suggests that the throttling allows Internet users to load only the first 16 KB of any web asset, rendering most web navigation impossible.

Cloudflare has not received any formal outreach or communication from Russian government entities about the motivation for such an action. Unfortunately, the actions are consistent with longstanding Russian efforts to isolate the Internet within its borders and reduce reliance on Western technology by replacing it with domestic alternatives. Indeed, Russian President Vladimir Putin recently publicly threatened to throttle US tech companies operating inside Russia. 

External reports corroborate our analysis, and further suggest that a number of other service providers are also affected by throttling or other disruptive actions in Russia, including at least Hetzner, DigitalOcean, and OVH.

The impact

Cloudflare is seeing disruptions across Continue reading

Hedge 272: Are we addicted to the CLI?

Is the CLI the best way to configure, manage, and troubleshoot routers and other networking gear? Or should we move past the CLI towards automation and (possibly even) GUI-based tools? Mark Posser joins Russ and Tom to discuss on this episode of the Hedge.
 

 
download
 
For more reading on this topic, please check out this post by Chris Grundemann.

Orange Me2eets: We made an end-to-end encrypted video calling app and it was easy

Developing a new video conferencing application often begins with a peer-to-peer setup using WebRTC, facilitating direct data exchange between clients. While effective for small demonstrations, this method encounters scalability hurdles with increased participants. The data transmission load for each client escalates significantly in proportion to the number of users, as each client is required to send data to every other client except themselves (n-1).

In the scaling of video conferencing applications, Selective Forwarding Units (SFUs) are essential.  Essentially a media stream routing hub, an SFU receives media and data flows from participants and intelligently determines which streams to forward. By strategically distributing media based on network conditions and participant needs, this mechanism minimizes bandwidth usage and greatly enhances scalability. Nearly every video conferencing application today uses SFUs.

In 2024, we announced Cloudflare Realtime (then called Cloudflare Calls), our suite of WebRTC products, and we also released Orange Meets, an open source video chat application built on top of our SFU.

We also realized that use of an SFU often comes with a privacy cost, as there is now a centralized hub that could see and listen to all the media contents, even though its sole job is Continue reading

Worth Reading 062625


If you’ve worried that AI might take your job, deprive you of your livelihood, or maybe even replace your role in society, it probably feels good to see the latest AI tools fail spectacularly.


The Virginia Supreme Court issued a ruling against Cox Communications that should trouble anybody building a fiber network that must cross railroad tracks. The case involves a dispute brought by the Norfolk Southern Railroad that challenged a new right-of-way law related to railroads.


Julia Angwin’s opinion piece clutches at courtroom verdicts and minor regulatory wins like a child gripping a plastic sword in the middle of an actual war. Yes, there are lawsuits.


This makes a huge difference to the way ChatGPT works: it can now behave as if it has recall over prior conversations, meaning it will be continuously customized based on that previous history.


Traditionally, Cilium’s BGP implementation required users to explicitly specify peer IP addresses in BGP cluster configurations to establish BGP sessions with Top-of-Rack (ToR) switches. While this approach functions adequately in small environments, it becomes difficult to manage for large-scale deployments involving thousands of Kubernetes nodes distributed across numerous racks.

SwiNOG 40: A Day of Awesomeness

A few days ago, I attended a SwiNOG meeting for the first time and realized what a mistake I was making — I should have been there years ago.

Not only was the event impeccably organized (what else would you expect in Switzerland) and at the best event location I have ever experienced (it’s hard to beat this view), it was also full of short, interesting, up-to-the-point presentations (you can already view the slide decks, YouTube videos should be available shortly). Plus, I met so many old friends I haven’t seen in years, and people I communicated with for years but never met before.

It’s not like the organizers would need any more publicity (the event was sold out), but if you happen to be near Switzerland in time for the next meeting, make sure to be there.

Thanks again to the wonderful SwiNOG core team for a fantastic experience! I hope we’ll meet again at the next SwiNOG meeting!

Switching to eBPF One Step at a Time with Calico DNS Inline Policy

Calico Enterprise lets users write network policies using domain names instead of IP addresses. This is done by dynamically mapping domain names to IP addresses and matching the egress traffic against these IPs. We have discussed this feature in detail when we introduced the Inline mode for the eBPF data plane in Calico Enterprise 3.20 release! It addresses the latency and performance issues of the various modes used by Calico in iptables/nftables data planes. It is a shame that Calico users who are not yet ready to switch completely to eBPF would miss out on this big DNS policy improvement. Don’t worry! We found a way to port it to iptables to enhance our users’ experience without forcing users to make a huge leap.

In Calico Enterprise v3.21, we have extended the Inline DNS policy mode to iptables. In this mode, DNS policies are updated in real time as DNS responses are parsed by eBPF within the data plane, thus improving the performance.

Calico iptables – DNS Inline policy

In all the existing modes in the iptables data plane, the DNS response packets are sent to Felix – Calico’s userspace agent. It parses the packets and updates the Continue reading

Worth Reading 062425


There has been a major change in the landscape of the internet over the past few years with the progressive introduction of the QUIC transport protocol. Here I’d like to look at where we are up to with the deployment of QUIC on the public Internet. But first, a review of the QUIC protocol.


The PCI Special Interest Group (PIC-SIG) just released official specs for PCIe 7.0, doubling the bandwidth again for high-performance kit such as network cards, while hinting that PCIe 8.0 may not achieve the same.


Our model training with adversarial data significantly enhanced our defenses against indirect prompt injection attacks in Gemini 2.5 models (technical details).


Apple has a new paper; it’s pretty devastating to LLMs, a powerful followup to one from many of the same authors last year.


Good intentions don’t always result in good outcomes. This is especially the case with recent suggestions regarding end-to-end-encryption adaptability requirements for number independent communication services.

Building agents with OpenAI and Cloudflare’s Agents SDK

What even is an Agents SDK?

The AI landscape is evolving at an incredible pace, and with it, the tools and platforms available to developers are becoming more powerful and interconnected than ever. Here at Cloudflare, we're genuinely passionate about empowering you to build the next generation of applications, and that absolutely includes intelligent agents that can reason, act, and interact with the world.

When we talk about "Agents SDKs", it can sometimes feel a bit… fuzzy. Some SDKs (software development kits) described as 'agent' SDKs are really about providing frameworks for tool calling and interacting with models. They're fantastic for defining an agent's "brain" – its intelligence, its ability to reason, and how it uses external tools. Here’s the thing: all these agents need a place to actually run. Then there's what we offer at Cloudflare: an SDK purpose-built to provide a seamless execution layer for agents. While orchestration frameworks define how agents think, our SDK focuses on where they run, abstracting away infrastructure to enable persistent, scalable execution across our global network.

Think of it as the ultimate shell, the place where any agent, defined by any agent SDK (like the powerful new OpenAI Agents SDK), Continue reading

1 … 4 5 6 7 8 … 3,798