Latest copyright decision in Germany rejects blocking through global DNS resolvers
This post is also available in Deutsch.
A recent decision from the Higher Regional Court of Cologne in Germany marked important progress for Cloudflare and the Internet in pushing back against misguided attempts to address online copyright infringement through the DNS system. In early November, the Court in Universal v. Cloudflare issued its decision rejecting a request to require public DNS resolvers like Cloudflare’s 1.1.1.1. to block websites based on allegations of online copyright infringement. That’s a position we’ve long advocated, because blocking through public resolvers is ineffective and disproportionate, and it does not allow for much-needed transparency as to what is blocked and why.
What is a DNS resolver?
To see why the Universal decision matters, it’s important to understand what a public DNS resolver is, and why it’s not a good place to try to moderate content on the Internet.
The DNS system translates website names to IP addresses, so that Internet requests can be routed to the correct location. At a high-level, the DNS system consists of two parts. On one side sit a series of nameservers (Root, TLD, and Authoritative) that together store information mapping domain names to IP addresses; on the other Continue reading
Cloudflare Gen 12 Server: Bigger, Better, Cooler in a 2U1N form factor
Two years ago, Cloudflare undertook a significant upgrade to our compute server hardware as we deployed our cutting-edge 11th Generation server fleet, based on AMD EPYC Milan x86 processors. It's nearly time for another refresh to our x86 infrastructure, with deployment planned for 2024. This involves upgrading not only the processor itself, but many of the server's components. It must be able to accommodate the GPUs that drive inference on Workers AI, and leverage the latest advances in memory, storage, and security. Every aspect of the server is rigorously evaluated — including the server form factor itself.
One crucial variable always in consideration is temperature. The latest generations of x86 processors have yielded significant leaps forward in performance, with the tradeoff of higher power draw and heat output. In this post we will explore this trend, and how it informed our decision to adopt a new physical footprint for our next-generation fleet of servers.
In preparation for the upcoming refresh, we conducted an extensive survey of the x86 CPU landscape. AMD recently introduced its latest offerings: Genoa, Bergamo, and Genoa-X, featuring the power of their innovative Zen 4 architecture. At the same time, Intel unveiled Sapphire Rapids as Continue reading
Cyber Week: Analyzing Internet traffic and e-commerce trends
Throughout the year, special events lead to changes in Internet traffic. We observed this with Thanksgiving in the US last week, where traffic dipped, and during periods like Black Friday (November 24, 2023) and Cyber Monday (November 27, 2023), where traffic spiked.
But how significant are these Cyber Week days on the Internet? Is it a global phenomenon? Does e-commerce interest peak on Black Friday or Cyber Monday, and are attacks increasing during this time? These questions are important to retailers and stakeholders around the world. At Cloudflare, we manage substantial traffic for our customers, which gives us a unique vantage from which to analyze traffic and attack patterns across large swaths of the Internet.
As we'll explore next, we observed varying trends. From a global perspective, there was a clear Internet traffic winner: Cyber Monday was the highest overall traffic day of 2023 (as it was for 2022), followed by Black Friday, and then Monday, November 21 from the same week. But zooming in, this pattern didn’t hold in some countries.
For this analysis, we examined anonymized samples of HTTP requests crossing our network, as well as DNS queries. Cloudflare's global data shows that peak request traffic occurred on Continue reading
Better debugging for Cloudflare Workers, now with breakpoints
As developers, we’ve all experienced times when our code doesn’t work like we expect it to. Whatever the root cause is, being able to quickly dive in, diagnose the problem, and ship a fix is invaluable.
If you’re developing with Cloudflare Workers, we provide many tools to help you debug your applications; from your local environment all the way into production. This additional insight helps save you time and resources and provides visibility into how your application actually works — which can help you optimize and refactor code even before it goes live.
In this post, we’ll explore some of the tools we currently offer, and do a deep dive into one specific area — breakpoint debugging — looking at not only how to use it, but how we recently implemented it in our runtime, workerd.
Available Debugging Tools
Logs
console.log. It might be the simplest tool for a developer to debug, but don’t underestimate it. Built into the Cloudflare runtime is node-like logging, which provides detailed, color-coded logs. Locally, you can view these logs in a terminal window, and they will look like this:
Outside local development, once your Worker is deployed, console.log statements are visible via Continue reading
Steve Bray: Why I joined Cloudflare
I am excited to announce that I joined Cloudflare last month as Head of Australia & New Zealand, to continue to build on Cloudflare’s success in the region through extending our valuable relationships with our customers and partners.
My journey to Cloudflare
I’ve been fortunate over my 25-year career in the IT industry to have worked for some of the most recognised and innovative organisations such as Oracle, Salesforce, and Zendesk. It’s been exciting to be inside these businesses as they’ve taken new ideas about how software can be developed and delivered to solve real world problems for any organisation’s customers. I’ve learned a lot by being a part of the industry, but probably more importantly, I’ve learned the most from the smart, experienced, diverse groups of talented people that I’ve had the pleasure to work with in ANZ and across Asia Pacific. I have always been interested in the problems that organisations are trying to solve through technology — for example, responding to strategic challenges, reducing cost, improving revenue, reducing risk — and joining Cloudflare is an opportunity to stay focussed on addressing those critical issues with our customers and partners using Cloudflare’s innovative solutions.
So why Cloudflare?
Cloudflare’s Continue reading
Cloudflare named a leader in Forrester Edge Development Platforms Wave, Q4 2023
This post is also available in Español, Deutsch, Português, 日本語, 한국어.
Forrester has recognized Cloudflare as a leader in The Forrester Wave™: Edge Development Platforms, Q4 2023 with the top score in the current offering category.
According to the report by Principal Analyst, Devin Dickerson, “Cloudflare’s edge development platform provides the building blocks enterprises need to create full stack distributed applications and enables developers to take advantage of a globally distributed network of compute, storage and programmable security without being experts on CAP theorem.“
Over one million developers are building applications using the Developer Platform products including Workers, Pages, R2, KV, Queues, Durable Objects, D1, Stream, Images, and more. Developers can easily deploy highly distributed, full-stack applications using Cloudflare’s full suite of compute, storage, and developer services.
Workers make Cloudflare’s network programmable
“ A key strength of the platform is the interoperability with Cloudflare’s programmable global CDN combined with a deployment model that leverages intelligent workload placement.”
– The Forrester Wave™: Edge Development Platforms, Q4 2023
Workers run across Cloudflare’s global network, provide APIs to read from and write directly to the local cache, Continue reading
Do hackers eat turkey? And other Thanksgiving Internet trends
Thanksgiving is a tradition celebrated by millions of Americans across six time zones and 50 states, usually involving travel and bringing families together. This year, it was celebrated yesterday, on November 23, 2023. With the Internet so deeply enmeshed into our daily lives, anything that changes how so many people behave is going to also have an impact on online traffic. But how big an impact, exactly?
At a high level: a 10% daily decrease in Internet traffic in the US (compared to the previous week). That happens to be the exact same percentage decrease we observed in 2022. So, Thanksgiving in the US, at least in the realm of Internet traffic, seems consistent with last year.
Let’s dig into more details about how people deal with cooking (or online ordering!) and whether family gatherings are less online, according to our Cloudflare Radar data. We’ll also touch on whether hackers stop for turkey, too.
The Thanksgiving hour: around 15:00 (local time)
While we can see a 10% overall daily drop in US traffic due to Thanksgiving, the drop is even more noticeable when examining traffic on an hour-by-hour basis. Internet activity began to decrease significantly after 12:00 EST, persisting Continue reading
Workers AI Update: Stable Diffusion, Code Llama + Workers AI in 100 cities
Thanksgiving might be a US holiday (and one of our favorites — we have many things to be thankful for!). Many people get excited about the food or deals, but for me as a developer, it’s also always been a nice quiet holiday to hack around and play with new tech. So in that spirit, we're thrilled to announce that Stable Diffusion and Code Llama are now available as part of Workers AI, running in over 100 cities across Cloudflare’s global network.
As many AI fans are aware, Stable Diffusion is the groundbreaking image-generation model that can conjure images based on text input. Code Llama is a powerful language model optimized for generating programming code.
For more of the fun details, read on, or head over to the developer docs to get started!
Generating images with Stable Diffusion
Stability AI launched Stable Diffusion XL 1.0 (SDXL) this past summer. You can read more about it here, but we’ll briefly mention some really cool aspects.
First off, “Distinct images can be prompted without having any particular ‘feel’ imparted by the model, ensuring absolute freedom of Continue reading
Workers AI Update: Hello Mistral 7B
This post is also available in Deutsch.
Today we’re excited to announce that we’ve added the Mistral-7B-v0.1-instruct to Workers AI. Mistral 7B is a 7.3 billion parameter language model with a number of unique advantages. With some help from the founders of Mistral AI, we’ll look at some of the highlights of the Mistral 7B model, and use the opportunity to dive deeper into “attention” and its variations such as multi-query attention and grouped-query attention.
Mistral 7B tl;dr:
Mistral 7B is a 7.3 billion parameter model that puts up impressive numbers on benchmarks. The model:
- Outperforms Llama 2 13B on all benchmarks
- Outperforms Llama 1 34B on many benchmarks,
- Approaches CodeLlama 7B performance on code, while remaining good at English tasks, and
- The chat fine-tuned version we’ve deployed outperforms Llama 2 13B chat in the benchmarks provided by Mistral.
Here’s an example of using streaming with the REST API:
curl -X POST \
“https://api.cloudflare.com/client/v4/accounts/{account-id}/ai/run/@cf/mistral/mistral-7b-instruct-v0.1” \
-H “Authorization: Bearer {api-token}” \
-H “Content-Type:application/json” \
-d '{ “prompt”: “What is grouped query attention”, “stream”: true }'
API Response: { response: “Grouped query attention is a technique used in natural language processing (NLP) and machine learning Continue reading2024, the year of elections
2024 is a year of elections, with more than 70 elections scheduled in 40 countries around the world. One of the key pillars of democracy is trust. To that end, ensuring that the Internet is trusted, secure, reliable, and accessible for the public and those working in the election space is critical to any free and fair election.
Cloudflare has considerable experience in gearing up for elections and identifying how our cyber security tools can be used to help vulnerable groups in the election space. In December 2022, we expanded our product set to include Zero Trust products to assist these groups against new and emerging threats. Over the last few years, we’ve reported on our work in protecting a range of election entities and as we prepare for the 2024 elections, we want to provide insight into attack trends we’ve seen against these groups to understand what to expect in the next year.
For this blog post, we identified cyber attack trends for a variety of groups in the elections space based in the United States, as many of our Cloudflare Impact projects provide services to these groups. These include U.S. state and local government websites protected under Continue reading
How to execute an object file: Part 4, AArch64 edition
Translating source code written in a high-level programming language into an executable binary typically involves a series of steps, namely compiling and assembling the code into object files, and then linking those object files into the final executable. However, there are certain scenarios where it can be useful to apply an alternate approach that involves executing object files directly, bypassing the linker. For example, we might use it for malware analysis or when part of the code requires an incompatible compiler. We’ll be focusing on the latter scenario: when one of our libraries needed to be compiled differently from the rest of the code. Learning how to execute an object file directly will give you a much better sense of how code is compiled and linked together.
To demonstrate how this was done, we have previously published a series of posts on executing an object file:
- How to execute an object file: Part 1
- How to execute an object file: Part 2
- How to execute an object file: Part 3
The initial posts are dedicated to the x86 architecture. Since then the fleet of our working machines has expanded to include a large and growing number of ARM CPUs. This Continue reading
Introducing advanced session audit capabilities in Cloudflare One
The basis of Zero Trust is defining granular controls and authorization policies per application, user, and device. Having a system with a sufficient level of granularity to do this is crucial to meet both regulatory and security requirements. But there is a potential downside to so many controls: in order to troubleshoot user issues, an administrator has to consider a complex combination of variables across applications, user identity, and device information, which may require painstakingly sifting through logs.
We think there’s a better way — which is why, starting today, administrators can easily audit all active user sessions and associated data used by their Cloudflare One policies. This enables the best of both worlds: extremely granular controls, while maintaining an improved ability to troubleshoot and diagnose Zero Trust deployments in a single, simple control panel. Information that previously lived in a user’s browser or changed dynamically is now available to administrators without the need to bother an end user or dig into logs.
A quick primer on application authentication and authorization
Authentication and Authorization are the two components that a Zero Trust policy evaluates before allowing a user access to a resource.
Authentication is the process of verifying the identity Continue reading
Introducing hostname and ASN lists to simplify WAF rules creation
If you’re responsible for creating a Web Application Firewall (WAF) rule, you’ll almost certainly need to reference a large list of potential values that each field can have. And having to manually manage and enter all those fields, for numerous WAF rules, would be a guaranteed headache.
That’s why we introduced IP lists. Having a separate list of values that can be referenced, reused, and managed independently of the actual rule makes for a better WAF user experience. You can create a new list, such as $organization_ips, and then use it in a rule like “allow requests where source IP is in $organization_ips”. If you need to add or remove IPs, you do that in the list, without touching each of the rules that reference the list. You can even add a descriptive name to help track its content. It’s easy, clean, and organized.
Which led us, and our customers, to ask the next natural question: why stop at IPs?
Cloudflare’s WAF is highly configurable and allows you to write rules evaluating a set of hostnames, Autonomous System Numbers (ASNs), countries, header values, or values of JSON fields. But to do so, you’ve to input a list of Continue reading
Streaming and longer context lengths for LLMs on Workers AI
Workers AI is our serverless GPU-powered inference platform running on top of Cloudflare’s global network. It provides a growing catalog of off-the-shelf models that run seamlessly with Workers and enable developers to build powerful and scalable AI applications in minutes. We’ve already seen developers doing amazing things with Workers AI, and we can’t wait to see what they do as we continue to expand the platform. To that end, today we’re excited to announce some of our most-requested new features: streaming responses for all Large Language Models (LLMs) on Workers AI, larger context and sequence windows, and a full-precision Llama-2 model variant.
If you’ve used ChatGPT before, then you’re familiar with the benefits of response streaming, where responses flow in token by token. LLMs work internally by generating responses sequentially using a process of repeated inference — the full output of a LLM model is essentially a sequence of hundreds or thousands of individual prediction tasks. For this reason, while it only takes a few milliseconds to generate a single token, generating the full response takes longer, on the order of seconds. The good news is we can start displaying the response as soon as the first tokens are generated, Continue reading
Post Mortem on Cloudflare Control Plane and Analytics Outage
Beginning on Thursday, November 2, 2023 at 11:43 UTC Cloudflare's control plane and analytics services experienced an outage. The control plane of Cloudflare consists primarily of the customer-facing interface for all of our services including our website and APIs. Our analytics services include logging and analytics reporting.
The incident lasted from November 2 at 11:44 UTC until November 4 at 04:25 UTC. We were able to restore most of our control plane at our disaster recovery facility as of November 2 at 17:57 UTC. Many customers would not have experienced issues with most of our products after the disaster recovery facility came online. However, other services took longer to restore and customers that used them may have seen issues until we fully resolved the incident. Our raw log services were unavailable for most customers for the duration of the incident.
Services have now been restored for all customers. Throughout the incident, Cloudflare's network and security services continued to work as expected. While there were periods where customers were unable to make changes to those services, traffic through our network was not impacted.
This post outlines the events that caused this incident, the architecture we had in place to prevent issues Continue reading
Cloudflare incident on October 30, 2023
Multiple Cloudflare services were unavailable for 37 minutes on October 30, 2023. This was due to the misconfiguration of a deployment tool used by Workers KV. This was a frustrating incident, made more difficult by Cloudflare’s reliance on our own suite of products. We are deeply sorry for the impact it had on customers. What follows is a discussion of what went wrong, how the incident was resolved, and the work we are undertaking to ensure it does not happen again.
Workers KV is our globally distributed key-value store. It is used by both customers and Cloudflare teams alike to manage configuration data, routing lookups, static asset bundles, authentication tokens, and other data that needs low-latency access.
During this incident, KV returned what it believed was a valid HTTP 401 (Unauthorized) status code instead of the requested key-value pair(s) due to a bug in a new deployment tool used by KV.
These errors manifested differently for each product depending on how KV is used by each service, with their impact detailed below.
What was impacted
A number of Cloudflare services depend on Workers KV for distributing configuration, routing information, static asset serving, and authentication state globally. These services instead received Continue reading
Introducing notifications for HTTP Traffic Anomalies
When it comes to managing Internet properties, the difference between a small technical hiccup and major incident is often a matter of speed. Proactive alerting plays a crucial role, which is why we were excited when we released HTTP Error Rate notifications — giving administrators visibility into when end users are experiencing errors.
But what if there are issues that don't show up as errors, like a sudden drop in traffic, or a spike?
Today, we're excited to announce Traffic Anomalies notifications, available to enterprise customers. These notifications trigger when Cloudflare detects unexpected changes in traffic, giving another valuable perspective into the health of your systems.
Unexpected changes in traffic could be indicative of many things. If you run an ecommerce site and see a spike in traffic that could be great news — maybe customers are flocking to your sale, or you just had an ad run on a popular TV show. However, it could also mean that something is going wrong: maybe someone accidentally turned off a firewall rule, and now you’re seeing more malicious traffic. Either way, you might want to know that something has changed.
Similarly, a sudden drop in traffic could mean many things. Perhaps Continue reading
Introducing HAR Sanitizer: secure HAR sharing
On Wednesday, October 18th, 2023, Cloudflare’s Security Incident Response Team (SIRT) discovered an attack on our systems that originated from an authentication token stolen from one of Okta’s support systems. No Cloudflare customer information or systems were impacted by the incident, thanks to the real-time detection and rapid action of our Security Incident Response Team (SIRT) in tandem with our Zero Trust security posture and use of hardware keys. With that said, we’d rather not repeat the experience — and so we have built a new security tool that can help organizations render this type of attack obsolete for good.
The bad actor in the Okta breach compromised user sessions by capturing session tokens from administrators at Cloudflare and other impacted organizations. They did this by infiltrating Okta’s customer support system and stealing one of the most common mechanisms for troubleshooting — an HTTP Response Archive (HAR) file.
HAR files contain a record of a user’s browser session, a kind of step-by-step audit, that a user can share with someone like a help desk agent to diagnose an issue. However, the file can also contain sensitive information that can be used to launch an attack.
As a follow-up to the Continue reading
Email Routing subdomain support, new APIs and security protocols
It's been two years since we announced Email Routing, our solution to create custom email addresses for your domains and route incoming emails to your preferred mailbox. Since then, the team has worked hard to evolve the product and add more powerful features to meet our users' expectations. Examples include Route to Workers, which allows you to process your Emails programmatically using Workers scripts, Public APIs, Audit Logs, or DMARC Management.
We also made significant progress in supporting more email security extensions and protocols, protecting our customers from unwanted traffic, and keeping our IP space reputation for email egress impeccable to maximize our deliverability rates to whatever inbox upstream provider you chose.
Since leaving beta, Email Routing has grown into one of our most popular products; it’s used by more than one million different customer zones globally, and we forward around 20 million messages daily to every major email platform out there. Our product is mature, robust enough for general usage, and suitable for any production environment. And it keeps evolving: today, we announce three new features that will help make Email Routing more secure, flexible, and powerful than ever.
New security protocols
The SMTP email protocol Continue reading
DDoS threat report for 2023 Q3
Welcome to the third DDoS threat report of 2023. DDoS attacks, or distributed denial-of-service attacks, are a type of cyber attack that aims to disrupt websites (and other types of Internet properties) to make them unavailable for legitimate users by overwhelming them with more traffic than they can handle — similar to a driver stuck in a traffic jam on the way to the grocery store.
We see a lot of DDoS attacks of all types and sizes, and our network is one of the largest in the world spanning more than 300 cities in over 100 countries. Through this network we serve over 64 million HTTP requests per second at peak and about 2.3 billion DNS queries every day. On average, we mitigate 140 billion cyber threats each day. This colossal amount of data gives us a unique vantage point to understand the threat landscape and provide the community access to insightful and actionable DDoS trends.
In recent weeks, we've also observed a surge in DDoS attacks and other cyber attacks against Israeli newspaper and media websites, as well as financial institutions and government websites. Palestinian websites have also seen a significant increase in DDoS attacks. View Continue reading