Automatic Platform Optimization post-launch report
Last year during Birthday Week, we announced Automatic Platform Optimization for WordPress (APO): smart HTML caching for WordPress sites using Cloudflare. Initial testing across various WordPress sites demonstrated significant improvements in performance metrics like Time to First Byte (TTFB), First Contentful Paint (FCP), and Speed Index. We wanted to measure how APO impacted web performance for our customers since the launch.
In the blog post, we answer the following questions:
- How fast is Automatic Platform Optimization? Can you demonstrate it with data?
We will show real-world improvements for several performance metrics.
- Is Automatic Platform Optimization flexible enough to integrate smoothly with my WordPress site?
We have added and improved lots of features since the initial launch.
- Will Automatic Platform Optimization work when used with other plugins?
We will cover the most common use cases and explain how Automatic Platform Optimization could be fined-tuned.
Measuring performance with WebPageTest
We use WebPageTest as a go-to tool for synthetic testing at Cloudflare. It measures web performance metrics in real browsers, is highly programmable, and could scale to test millions of sites per day. Among the benefits of synthetic testing are easy to produce results and their relatively high reproducibility.
Automatic Platform Optimization Continue reading
Enhancing privacy-focused Web Analytics to better meet your metrics needs
Last December we opened up our brand new privacy-first Web Analytics platform to everyone. Today, we’re excited to announce the release of three of the most requested features: adding multiple websites to an account, supporting Single-page Applications (SPA) as well as showing Core Web Vitals in Web Analytics.
Bringing privacy-first analytics to everyone
Since we launched two months ago, we’ve received a lot of feedback from our users. We are really happy that we are able to provide our privacy-first analytics to so many of you.
Popular analytics vendors have business models driven by ad revenue. Using them implies a bargain: they track visitor behavior and create buyer profiles to retarget your visitors with ads; in exchange, you get free analytics.
Our mission is to help build a better Internet, and part of that is to deliver essential web analytics to everyone with a website without compromising user privacy. We’ve never been interested in tracking users or selling advertising. We don’t want to know what you do on the Internet — it’s not our business.
You now can measure multiple sites
When we launched Web Analytics, each account was only able to measure one website. We are happy to announce Continue reading
Lessons Learned from Scaling Up Cloudflare’s Anomaly Detection Platform
Introduction to Anomaly Detection for Bot Management
Cloudflare’s Bot Management platform follows a “defense in depth” model. Although each layer of Bot Management has its own strengths and weaknesses, the combination of many different detection systems — including Machine Learning, rule-based heuristics, JavaScript challenges, and more — makes for a robust platform in which different detection systems compensate for each other’s weaknesses.
One of these systems is Anomaly Detection, a platform motivated by a simple idea: because bots are made to accomplish specific goals, such as credential stuffing or content scraping, they interact with websites in distinct and difficult-to-disguise ways. Over time, the actions of a bot are likely to differ from those of a real user. Anomaly detection aims to model the characteristics of legitimate user traffic as a healthy baseline. Then, when automated bot traffic is set against this baseline, the bots appear as outlying anomalies that can be targeted for mitigation.
An anomaly detection approach is:
- Resilient against bots that try to circumvent protections by spoofing request metadata (e.g., user agents)
- Able to catch previously unseen bots without being explicitly trained against them.
So, how well does this work?
Today, Anomaly Detection processes more than Continue reading
ARMs Race: Ampere Altra takes on the AWS Graviton2
Over three years ago, we embraced the ARM ecosystem after evaluating the Qualcomm Centriq. The Centriq and its Falkor cores delivered a significant reduction in power consumption while maintaining a comparable performance against the processor that was powering our server fleet at the time. By the time we completed porting our software stack to be compatible with ARM, Qualcomm decided to exit the server business. Since then, we have been waiting for another server-grade ARM processor with hopes to improve our power efficiencies across our global network, which now spans more than 200 cities in over 100 countries.
ARM has introduced the Neoverse N1 platform, the blueprint for creating power-efficient processors licensed to institutions that can customize the original design to meet their specific requirements. Ampere licensed the Neoverse N1 platform to create the Ampere Altra, a processor that allows companies that own and manage their own fleet of servers, like ourselves, to take advantage of the expanding ARM ecosystem. We have been working with Ampere to determine whether Altra is the right processor to power our first generation of ARM edge servers.
The AWS Graviton2 is the only other Neoverse N1-based processor publicly accessible, but only made Continue reading
About the March 8 & 9, 2021 Verkada camera hack
Cloudflare uses a vendor called Verkada for cameras in our offices in San Francisco, Austin, New York, London and Singapore. These cameras are used at the entrances, exits and main thoroughfares of our offices and have been part of maintaining the security of offices that have been closed for almost a year.
Yesterday, we were notified of a breach of Verkada that allowed a hacker to access Verkada’s internal support tools to manage those cameras remotely, as well as access them through a remote root shell. As soon as we were notified of the breach, we proceeded to shut down the cameras in all our office locations to prevent further access.
To be clear: this hack affected the cameras and nothing else. No customer data was accessed, no production systems, no databases, no encryption keys, nothing. Some press reports indicate that we use a facial recognition feature available in Verkada. This is not true. We do not.
Our internal systems follow the same Zero Trust model that we provide to our customers, and as such our corporate office networks are not implicitly trusted by our other locations or data centers. From a security point of view connecting from one of Continue reading
Control web applications with two-clicks in Cloudflare Gateway
Nearly a year ago, we announced Cloudflare for Teams, Cloudflare’s platform for securing users, devices, and data. With Cloudflare for Teams, our global network becomes your team’s network, replacing on-premise appliances and security subscriptions with a single solution delivered closer to your users — wherever they work. Cloudflare for Teams centers around two core products: Cloudflare Access and Cloudflare Gateway.
Cloudflare Gateway protects employees from security threats on the Internet and enforces appropriate use policies. We built Gateway to help customers replace the pain of backhauling user traffic through centralized firewalls. With Gateway, users instead connect to one of Cloudflare’s data centers in 200 cities around the world where our network can apply consistent security policies for all of their Internet traffic.
In March 2020, we launched Gateway’s first feature, a secure DNS filtering solution. With Gateway’s DNS filtering, administrators can click a single button to block known threats, like sources of malware or phishing sites. Policies can also be used to block specific risky categories, like gambling or social media. When users request a filtered site, Gateway stops the DNS query from resolving and prevents the device from connecting to a malicious destination or hostname with blocked material.
The benefits of serving stale DNS entries when using Consul
Introduction
We use Consul for service discovery, and we’ve deployed a cluster that spans several of our data centers. This cluster exposes HTTP and DNS interfaces so that clients can query the Consul catalog and search for a particular service and the majority of the clients use DNS. We were aware from the start that the DNS query latencies were not great from certain parts of the world that were furthest away from these data centers. This, together with the fact that we use DNS over TLS, results in some long latencies. The TTL of these names being low makes it even more impractical when resolving these names in the hot path.
The usual way to solve these issues is by caching values so that at least subsequent requests are resolved quickly, and this is exactly what our resolver of choice, Unbound, is configured to do. The problem remains when the cache expires. When it expires, the next client will have to wait while Unbound resolves the name using the network. To have a low recovery time in case some service needs to failover and clients need to use another address we use a small TTL (30 seconds) Continue reading
Happy International Women’s Day!
Here at Cloudflare, we’re thrilled to celebrate International Women’s Day today! We have tons of events planned throughout the month of March, which is our way of honoring Women’s Empowerment Month. We’ll be making sure we acknowledge women’s achievements, raise awareness about women’s equality, and lobby for accelerated gender parity — Cloudflare style.
We take the International Women’s Day initiatives and its calls to action seriously. Then again, how could we not? The latest 2020 Global Gender Gap Report from the World Economic Forum indicates that it’ll take another 257 years to close the gender gap, if we continue at our current pace of progress. It’s going to take all of us to make a positive impact and accelerate the reality of a gender equal world.
Introducing Womenflare
Before we dive further into how we’re planning to celebrate International Women’s Day and Women’s Empowerment Month, we’d like to introduce ourselves. We’re Womenflare — Cloudflare’s Employee Resource Group (ERG) for all who identify as and advocate for women (Talea and Angela are the global Womenflare leads and John is the Womenflare executive advocate). We launched Womenflare on International Women’s Day in 2020, and it was one of the last things we Continue reading
Protecting against recently disclosed Microsoft Exchange Server vulnerabilities: CVE-2021-26855, CVE-2021-26857, CVE-2021-26858, and CVE-2021-27065
Enabling the Cloudflare WAF and Cloudflare Specials ruleset protects against exploitation of unpatched CVEs: CVE-2021-26855, CVE-2021-26857, CVE-2021-26858, and CVE-2021-27065.
Cloudflare has deployed managed rules protecting customers against a series of remotely exploitable vulnerabilities that were recently found in Microsoft Exchange Server. Web Application Firewall customers with the Cloudflare Specials ruleset enabled are automatically protected against CVE-2021-26855, CVE-2021-26857, CVE-2021-26858, and CVE-2021-27065.
If you are running Exchange Server 2013, 2016, or 2019, and do not have the Cloudflare Specials ruleset enabled, we strongly recommend that you do so. You should also follow Microsoft’s urgent recommendation to patch your on-premise systems immediately. These vulnerabilities are actively being exploited in the wild by attackers to exfiltrate email inbox content and move laterally within organizations’ IT systems.
Edge Mitigation
If you are running the Cloudflare WAF and have enabled the Cloudflare Specials ruleset, there is nothing else you need to do. We have taken the unusual step of immediately deploying these rules in “Block” mode given active attempted exploitation.
If you wish to disable the rules for any reason, e.g., you are experiencing a false positive mitigation, you can do so by following these instructions:
- Login to the Cloudflare Dashboard Continue reading
The Teams Dashboard: Finding a Product Voice
My name is Alice Bracchi, and I’m the technical and UX writer for Cloudflare for Teams, Cloudflare's Zero Trust and Secure Web Gateway solution.
Today I want to talk about product voice — what it is, why it matters, and how I set out to find a product voice for Cloudflare for Teams.
On the Cloudflare for Teams Dashboard (or as we informally call it, “the Teams Dash”), our customers have full control over the security of their network. Administrators can replace their VPN with a solution that runs on Zero Trust rules, turning Cloudflare's network into their secure corporate network. Customers can secure all traffic by configuring L7 firewall rules and DNS filtering policies, and organizations have the ability to isolate web browsing to suspicious sites.
All in one place.
As you can see, a lot of action takes place on the Teams Dash. As an interface, it grows and changes at a rapid pace. This poses a lot of interesting challenges from a design point of view — in our early days, because we were focused on solving problems fast, many of our experiences ended up feeling a bit disjointed. Sure, users were able to Continue reading
Conntrack turns a blind eye to dropped SYNs
Intro
We have been working with conntrack, the connection tracking layer in the Linux kernel, for years. And yet, despite the collected know-how, questions about its inner workings occasionally come up. When they do, it is hard to resist the temptation to go digging for answers.
One such question popped up while writing the previous blog post on conntrack:
“Why are there no entries in the conntrack table for SYN packets dropped by the firewall?”
Ready for a deep dive into the network stack? Let’s find out.
We already know from last time that conntrack is in charge of tracking incoming and outgoing network traffic. By running conntrack -L we can inspect existing network flows, or as conntrack calls them, connections.
So if we spin up a toy VM, connect to it over SSH, and inspect the contents of the conntrack table, we will see…
$ vagrant init fedora/33-cloud-base
$ vagrant up
…
$ vagrant ssh
Last login: Sun Jan 31 15:08:02 2021 from 192.168.122.1
[vagrant@ct-vm ~]$ sudo conntrack -L
conntrack v1.4.5 (conntrack-tools): 0 flow entries have been shown.
… nothing!
Even though the conntrack kernel Continue reading
Cloudflare recognized as a ‘Leader’ in The Forrester Wave for DDoS Mitigation Solutions
We’re thrilled to announce that Cloudflare has been named a leader in The Forrester WaveTM: DDoS Mitigation Solutions, Q1 2021. You can download a complimentary copy of the report here.
According to the report, written by, Forrester Senior Analyst for Security and Risk, David Holmes, “Cloudflare protects against DDoS from the edge, and fast… customer references view Cloudflare’s edge network as a compelling way to protect and deliver applications.”
Unmetered and unlimited DDoS protection for all
Cloudflare was founded with the mission to help build a better Internet — one where the impact of DDoS attacks is a thing of the past. Over the last 10 years, we have been unwavering in our efforts to protect our customers’ Internet properties from DDoS attacks of any size or kind. In 2017, we announced unmetered DDoS protection for free — as part of every Cloudflare service and plan including the Free plan — to make sure every organization can stay protected and available.
Thanks to our home-grown automated DDoS protection systems, we’re able to provide unmetered and unlimited DDoS protection for free. Our automated systems constantly analyze traffic samples asynchronously as to avoid impact to performance. They scan for Continue reading
How to execute an object file: Part 1
Calling a simple function without linking
When we write software using a high-level compiled programming language, there are usually a number of steps involved in transforming our source code into the final executable binary:
First, our source files are compiled by a compiler translating the high-level programming language into machine code. The output of the compiler is a number of object files. If the project contains multiple source files, we usually get as many object files. The next step is the linker: since the code in different object files may reference each other, the linker is responsible for assembling all these object files into one big program and binding these references together. The output of the linker is usually our target executable, so only one file.
However, at this point, our executable might still be incomplete. These days, most executables on Linux are dynamically linked: the executable itself does not have all the code it needs to run a program. Instead it expects to "borrow" part of the code at runtime from shared libraries for some of its functionality:
This process is called runtime linking: when our executable is being started, the operating system will invoke the dynamic Continue reading
The Teams Dashboard: Behind the Scenes
Back in 2010, Cloudflare was introduced at TechCrunch Disrupt as a security and performance solution that took the tools of the biggest service providers and made them available to anyone online. But simply replicating these tools wasn’t enough — we needed to make them ridiculously easy to use.
When we launched Cloudflare for Teams almost ten years later, the vision was very much the same — build a secure and powerful Zero Trust solution that is ridiculously easy to use. However, while we talk about what we’re building with a regular cadence, we often gloss over how we are designing Cloudflare for Teams to make it simple and easy to use.
In this blog post we’ll do just that — if that sounds like your jam, keep scrolling.
Building a house
First, let's back up a bit and introduce Cloudflare for Teams.
We launched Cloudflare for Teams in January, 2020. With Teams, we wanted to alleviate the burden Cloudflare customers were feeling when trying to protect themselves and their infrastructure from threats online. We knew that continuing to rely on expensive hardware would be difficult to maintain and impractical to scale.
At its core, Teams joins two products together — Continue reading
Flow-based monitoring for Magic Transit
Network-layer DDoS attacks are on the rise, prompting security teams to rethink their L3 DDoS mitigation strategies to prevent business impact. Magic Transit protects customers’ entire networks from DDoS attacks by placing our network in front of theirs, either always on or on demand. Today, we’re announcing new functionality to improve the experience for on-demand Magic Transit customers: flow-based monitoring. Flow-based monitoring allows us to detect threats and notify customers when they’re under attack so they can activate Magic Transit for protection.
Magic Transit is Cloudflare’s solution to secure and accelerate your network at the IP layer. With Magic Transit, you get DDoS protection, traffic acceleration, and other network functions delivered as a service from every Cloudflare data center. With Cloudflare’s global network (59 Tbps capacity across 200+ cities) and <3sec time to mitigate at the edge, you’re covered from even the largest and most sophisticated attacks without compromising performance. Learn more about Magic Transit here.
Using Magic Transit on demand
With Magic Transit, Cloudflare advertises customers’ IP prefixes to the Internet with BGP in order to attract traffic to our network for DDoS protection. Customers can choose to use Magic Transit always on or on demand. With always Continue reading
Creating serendipity with Python
We've been experimenting with breaking up employees into random groups (of size 4) and setting up video hangouts between them. We're doing this to replace the serendipitous meetings that sometimes occur around coffee machines, in lunch lines or while waiting for the printer. And also, we just want people to get to know each other.
Which lead to me writing some code. The core of which is divide n elements into groups of at least size g minimizing the size of each group. So, suppose an office has 15 employees in it then it would be divided into three groups of sizes 5, 5, 5; if an office had 16 employees it would be 4, 4, 4, 4; if it had 17 employees it would be 4, 4, 4, 5 and so on.
I initially wrote the following code (in Python):
groups = [g] * (n//g)
for e in range(0, n % g):
groups[e % len(groups)] += 1
The first line creates n//g (// is integer division) entries of size g (for example, if g == 4 and n == 17 then groups == [4, 4, 4, 4]). The for loop deals with the 'left over' parts that Continue reading
Through the eyes of a Cloudflare Technical Support Engineer
This post originally appeared on Landing Jobs under the title Mission: Protect the Internet where you can find open positions at Cloudflare Lisbon.
Justina Wong, Technical Support Team Lead in Lisbon, talks about what it’s like working at Cloudflare, and everything you need to know if you want to join us.
Justina joined Cloudflare about three years ago in London as a Technical Support Engineer. Currently, she’s part of their Customer Support team working in Lisbon as a team lead.
I can’t speak for others, but I love the things you can learn from the others. There are so many talented individuals who are willing and ready to teach/share. They are my inspiration and I want to become them!
On a Mission to Protect the Internet
Justina’s favourite Cloudflare products are firewall-related ones. The company’s primary care is for the customers and they want to make attack mitigation as easy as possible. As she puts it, “the fact that these protections are on multiple layers, like L7, L3/4, is very important, and I’m proud to be someone who can help our customers when they face certain attacks.”.
Cloudflare is constantly releasing new products to help build a better Internet, Continue reading
Using HPKE to Encrypt Request Payloads
The Managed Rules team was recently given the task of allowing Enterprise users to debug Firewall Rules by viewing the part of a request that matched the rule. This makes it easier to determine what specific attacks a rule is stopping or why a request was a false positive, and what possible refinements of a rule could improve it.
The fundamental problem, though, was how to securely store this debugging data as it may contain sensitive data such as personally identifiable information from submissions, cookies, and other parts of the request. We needed to store this data in such a way that only the user who is allowed to access it can do so. Even Cloudflare shouldn't be able to see the data, following our philosophy that any personally identifiable information that passes through our network is a toxic asset.
This means we needed to encrypt the data in such a way that we can allow the user to decrypt it, but not Cloudflare. This means public key encryption.
Now we needed to decide on which encryption algorithm to use. We came up with some questions to help us evaluate which one to use:
- What requirements do we have for Continue reading
Introducing: Smarter Tiered Cache Topology Generation
Caching is a magic trick. Instead of a customer’s origin responding to every request, Cloudflare’s 200+ data centers around the world respond with content that is cached geographically close to visitors. This dramatically improves the load performance for web pages while decreasing the bandwidth costs by having Cloudflare respond to a request with cached content.
However, if content is not in cache, Cloudflare data centers must contact the origin server to receive the content. This isn’t as fast as delivering content from cache. It also places load on an origin server, and is more costly compared to serving directly from cache. These issues can be amplified depending on the geographic distribution of a website’s visitors, the number of data centers contacting the origin, and the available origin resources for responding to requests.
To decrease the number of times our network of data centers communicate with an origin, we organize data centers into tiers so that only upper-tier data centers can request content from an origin and then they spread content to lower tiers. This means content that loads faster for visitors, is cheaper to serve, and reduces origin resource consumption.
Today, I’m thrilled to announce a fundamental improvement to Argo Continue reading
Tiered Cache Smart Topology
A few years ago, we released Argo to help make the Internet faster and more efficient. Argo observes network conditions and finds the optimal route across the Internet for origin server requests, avoiding congestion along the way.
Tiered Cache is an Argo feature that reduces the number of data centers responsible for requesting assets from the origin. With Tiered Cache active, a request in South Africa won’t go directly to an origin in North America, but, instead, look in a large, nearby data center to see if the data requested is cached there first. The number and location of the data centers used by Tiered Cache is controlled by a piece of configuration called the topology. By default, we use a generic topology for every customer that strikes a balance between cache hit ratios and latency that is suitable for most users.
Today we’re introducing Smart Topology, which maximizes cache hit ratios by building on Argo’s internal infrastructure to identify the single best data center for making requests to the origin.
Standard Cache
The standard method for caching assets is to let each data center be a reverse proxy for the origin server. In this scheme, a miss in any Continue reading