Health Check Analytics and how you can use it

At the end of last year, we introduced Standalone Health Checks - a service that lets you monitor the health of your origin servers and avoid the need to purchase additional third party services. The more that can be controlled from Cloudflare decreases maintenance cost, vendor management, and infrastructure complexity. This is important as it ensures you are able to scale your infrastructure seamlessly as your company grows. Today, we are introducing Standalone Health Check Analytics to help decrease your time to resolution for any potential issues. You can find Health Check Analytics in the sub-menu under the Traffic tab in your Cloudflare Dashboard.

As a refresher, Standalone Health Checks is a service that monitors an IP address or hostname for your origin servers or application and notifies you in near real-time if there happens to be a problem. These Health Checks support fine-tuned configurations based on expected codes, interval, protocols, timeout and more. These configurations enable you to properly target your checks based on the unique setup of your infrastructure. An example of a Health Check can be seen below which is monitoring an origin server in a staging environment with a notification set via email.

Once you set Continue reading

Project Galileo’s 6th year Anniversary: The Impact of COVID-19 on the most vulnerable groups on the Internet

Consistent with our mission to “help build a better Internet,” Cloudflare believes that one of the most important roles for the Internet is to empower marginalized voices that may not be heard, or bring together oppressed groups of people that may otherwise find themselves isolated and alone. Six years ago, Cloudflare started Project Galileo to provide free services to vulnerable nonprofits, journalism and independent media voices online who might otherwise be in danger of being silenced by cyberattacks. Much has changed in the past couple of months as the COVID-19 pandemic has transformed the world while the United States faces a wave of protests addressing racial violence and inequality. These events have put further strain on vulnerable groups working in these spaces, and we have seen many organizations step up to ensure that those who are most affected by these circumstances are protected. At Cloudflare, we believe that protecting these groups from attack is essential to helping build a better Internet.

We are excited to mark the 6th anniversary of the project this month, and it is a good time for us to reflect, talk to participants, and see how the Project has grown and changed over the course of Continue reading

High Availability Load Balancers with Maglev

Background

We run many backend services that power our customer dashboard, APIs, and features available at our edge. We own and operate physical infrastructure for our backend services. We need an effective way to route arbitrary TCP and UDP traffic between services and also from outside these data centers.

Previously, all traffic for these backend services would pass through several layers of stateful TCP proxies and NATs before reaching an available instance. This solution worked for several years, but as we grew it caused our service and operations teams many issues. Our service teams needed to deal with drops of availability, and our operations teams had much toil when needing to do maintenance on load balancer servers.

Goals

With the experience with our stateful TCP proxy and NAT solutions in mind, we had several goals for a replacement load balancing service, while remaining on our own infrastructure:

1. Preserve source IPs through routing decisions to destination servers. This allows us to support servers that require client IP addresses as part of their operation, without workarounds such as X-Forwarded-For headers or the PROXY TCP extension.
2. Support an architecture where backends are located across many racks and subnets. This prevents solutions that cannot Continue reading

7 Layers: IoT Part 2 — IoT Devices are Dangerously Insecure

This week is the second in a two-part series on the Internet of Things. We cover IoT security...

Read More »

UtahFS: Encrypted File Storage

Encryption is one of the most powerful technologies that everyone uses on a daily basis without realizing it. Transport-layer encryption, which protects data as it’s sent across the Internet to its intended destination, is now ubiquitous because it’s a fundamental tool for creating a trustworthy Internet. Disk encryption, which protects data while it’s sitting idly on your phone or laptop’s hard drive, is also becoming ubiquitous because it prevents anybody who steals your device from also being able to see what’s on your desktop or read your email.

The next improvement on this technology that’s starting to gain popularity is end-to-end encryption, which refers to a system where only the end-users are able to access their data -- not any intermediate service providers. Some of the most popular examples of this type of encryption are chat apps like WhatsApp and Signal. End-to-end encryption significantly reduces the likelihood of a user’s data being maliciously stolen from, or otherwise mishandled by a service provider. This is because even if the service provider loses the data, nobody will have the keys to decrypt it!

Several months ago, I realized that I had a lot of sensitive files on my computer (my diary, if Continue reading

Ladies and Gentlemen… Cloudflare TV!

I'm excited to announce the upcoming launch of Cloudflare TV. A 24x7 live television broadcast, streamed globally via the Cloudflare network. You can tune in to the pre-broadcast station and check out the upcoming schedule at: cloudflare.tv

I'm kicking off the first live broadcast starting at 12:00pm Pacific (1900 UTC) on Monday, June 8 with a conversation with Chris Young (add to calendar). Chris was most recently the CEO of McAfee and has had a career defining the cyber security industry, from his own startup Cyveillance in the 1990s, to leadership positions at AOL, RSA, VMWare, Cisco, and Intel. I hope you'll tune in and then stay tuned for all the content our team has in store.

Which leaves the question: why on earth is Cloudflare launching a 24x7 television station?

The Uniting Power of Television and Tech Conferences

I was born in the 70's, am a child of the 80's, and got started in my career in the 90's. In the background, throughout much of it, was linear television we watched together. Over the last few months I've learned that Michelle Zatlyn, my co-founder and Cloudflare's COO, and I shared a love of Children's Television Network's Continue reading

How we use HashiCorp Nomad

In this blog post, we will walk you through the reliability model of services running in our more than 200 edge cities worldwide. Then, we will go over how deploying a new dynamic task scheduling system, HashiCorp Nomad, helped us improve the availability of services in each of those data centers, covering how we deployed Nomad and the challenges we overcame along the way. Finally, we will show you both how we currently use Nomad and how we are planning on using it in the future.

Reliability model of services running in each data center

For this blog post, we will distinguish between two different categories of services running in each data center:

• Customer-facing services: all of our stack of products that our customers use, such as caching, WAF, DDoS protection, rate-limiting, load-balancing, etc.
• Management services: software required to operate the data center, that is not in the direct request path of customer traffic.

Customer-facing services

The reliability model of our customer-facing services is to run them on all machines in each data center. This works well as it allows each data center’s capacity to scale dynamically by adding more machines.

Scaling is especially made easy thanks to Continue reading

Cyberattacks since the murder of George Floyd

As we’ve often seen in the past, real world protest and violence is usually accompanied by attacks on the Internet. This past week has been no exception. The shocking murder of George Floyd on May 25 was followed, over the weekend of May 30/31, by widespread protests and violence in the US. At the same time, Cloudflare saw a large uptick in cyberattacks, particularly cyberattacks on advocacy organizations fighting racism.

This chart shows the number of cyberattack HTTP requests blocked by Cloudflare over the last week (blue line) compared to the corresponding week in April a month before (green line). Cloudflare’s scale means that we are blocking attacks in the many 10s of billions per day, but even with that scale it’s clear that during the last week there have been even more attacks than before. And those attacks grew over the weekend.

Digging in a little deeper we can compare the attacks over this past weekend with a corresponding weekend a month before. Over the weekend of April 25/26, Cloudflare blocked a total of 116,317,347,341 (a little over 116 billion cyberattack HTTP requests performing DDoS or trying to break into websites, apps or APIs were blocked).

Since 116,317,347,341 can Continue reading

Export logs from Cloudflare Gateway with Logpush

Like many people, I have spent a lot more time at home in the last several weeks. I use the free version of Cloudflare Gateway, part of Cloudflare for Teams, to secure the Internet-connected devices on my WiFi network. In the last week, Gateway has processed about 114,000 DNS queries from those devices and blocked nearly 100 as potential security risks.

I can search those requests in the Cloudflare for Teams UI. The logs capture the hostname requested, the time of the request, and Gateway’s decision to allow or block. This works fine for one-off investigations into a block, but does not help if I want to analyze the data more thoroughly. The last thing I want to do is click through hundreds or thousands of pages.

That problem is even more difficult for organizations attempting to keep hundreds or thousands of users and their devices secure. Whether they secure roaming devices with DoH or a static IP address, or keep users safe as they return to offices, deployments at that scale need a better option for auditing tens or hundreds of millions of queries each week.

Starting today, you can configure the automatic export of logs from Cloudflare Gateway Continue reading

Tanium’s endpoint security meets Cloudflare for Teams

When Cloudflare first launched in 2010, network security still relied heavily on physical security. To connect to a private network, most users simply needed to be inside the walls of the office. Once on that network, users could connect to corporate applications and infrastructure.

When users left the office, a Virtual Private Network (VPN) became a bandaid to let users connect back into that office network. Administrators poked holes in their firewall that allowed traffic to route back through headquarters. The backhaul degraded user experience and organizations had no visibility into patterns and events that occurred once users were on the network.

Cloudflare Access launched two years ago to replace that model with an identity-based solution built on Cloudflare’s global network. Instead of a private network, teams secure applications with Cloudflare’s network. Cloudflare checks every request to those applications for identity, rather than IP ranges, and accelerates those connections using the same network that powers some of the world’s largest web properties.

In this zero-trust model, Cloudflare Access checks identity on every request - not just the initial login to a VPN client. Administrators build rules that Cloudflare’s network continuously enforces. Each request is evaluated for permission and logged for Continue reading

Test your home network performance

With many people being forced to work from home, there’s increased load on consumer ISPs. You may be asking yourself: how well is my ISP performing with even more traffic? Today we’re announcing the general availability of speed.cloudflare.com, a way to gain meaningful insights into exactly how well your network is performing.

We’ve seen a massive shift from users accessing the Internet from busy office districts to spread out urban areas.

Although there are a slew of speed testing tools out there, none of them give you precise insights into how they came to those measurements and how they map to real-world performance. With speed.cloudflare.com, we give you insights into what we’re measuring and how exactly we calculate the scores for your network connection. Best of all, you can easily download the measurements from right inside the tool if you’d like to perform your own analysis.

We also know you care about privacy. We believe that you should know what happens with the results generated by this tool. Many other tools sell the data to third parties. Cloudflare does not sell your data. Performance data is collected and anonymized and is governed by the terms of Continue reading

Making DNS record changes more reliable

DNS is the very first step in accessing any website, API, or pretty much anything on the Internet, which makes it mission-critical to keeping your site up and running. This week, we are launching two significant changes that allow our customers to better maintain and update their DNS records. For customers who use Cloudflare as their authoritative DNS provider, we’ve added a much asked for feature: confirmation to DNS record edits. For our secondary DNS customers, we’re excited to provide a brand new onboarding experience.

Confirm and Commit

One of the benefits of using Cloudflare DNS is that changes quickly propagate to our 200+ data centers. And I mean very quickly: DNS propagation typically takes <5 seconds worldwide. Our UI was set up to allow customers to edit records, click out of the input box, and boom! The record has propagated!

There are a lot of advantages to fast DNS, but there's also one clear downside – it leaves room for fat fingering. What if you accidentally toggle the proxy icon, or mistype the content of your DNS record? This could result in users not being able to access your website or API and could cause a significant outage. To Continue reading

Secondary DNS — A faster, more resilient way to serve your DNS records

What is secondary DNS, and why is it important?

In DNS, nameservers are responsible for serving DNS records for a zone. How the DNS records populate into the nameservers differs based on the type of nameserver.

A primary master is a nameserver that manages a zone’s DNS records. This is where the zone file is maintained and where DNS records are added, removed, and modified. However, relying on one DNS server can be risky. What if that server goes down, or your DNS provider has an outage? If you run a storefront, then your customers would have to wait until your DNS server is back up to access your site. If your website were a brick and mortar store, this would be effectively like boarding up the door while customers are trying to get in.This type of outage can be very costly.

Now imagine you have another DNS server that has a replica of your DNS records. Wouldn’t it be great to have it as a back-up if your primary nameserver went down? Or better yet, what if both served your DNS records at all times— this could help decrease the latency of DNS requests, distribute the load between Continue reading

Releasing Cloudflare Access’ most requested feature

Cloudflare Access, part of Cloudflare for Teams, replaces legacy corporate VPNs with Cloudflare’s global network. Instead of starting a VPN client to backhaul traffic through an office, users visit the hostname of an internal application and login with your team’s SSO provider. While the applications feel like SaaS apps for end users, your security and IT departments can configure granular controls and audit logging in a single place.

Since Access launched two years ago, customers have been able to integrate multiple SSO providers at the same time. This MultiSSO option makes it seamless for teams to have employees login with Okta or Azure AD while partners and contractors use LinkedIN or GitHub.

The integrations always applied globally. Users would see all SSO options when connecting to any application protected by Cloudflare Access. As more organizations use Cloudflare Access to connect distributed and mixed workforces to resources, listing every provider on every app no longer scales.

For example, your team might have an internal GitLab instance that only employees need to access using your corporate G Suite login. Meanwhile, the marketing department needs to share QA versions of new sites with an external agency who authenticates with LinkedIn. Asking both Continue reading

Resolve internal hostnames with Cloudflare for Teams

Phishing attacks begin like any other visit to a site on the Internet. A user opens a suspicious link from an email, and their DNS resolver looks up the hostname, then connects the user to the origin.

Cloudflare Gateway’s secure DNS blocks threats like this by checking every hostname query against a constantly-evolving list of known threats on the Internet. Instead of sending the user to the malicious host, Gateway stops the site from resolving. The user sees a “blocked domain” page instead of the malicious site itself.

As teams migrate to SaaS applications and zero-trust solutions, they rely more on the public Internet to do their jobs. Gateway's security works like a bouncer, keeping users safe as they navigate the Internet. However, some organizations still need to send traffic to internal destinations for testing or as a way to make the migration more seamless.

Starting today, you can use Cloudflare Gateway to direct end user traffic to a different IP than the one they originally requested. Administrators can build rules to override the address that would be returned by a resolver and send traffic to a specified alternative.

Like the security features of Cloudflare Gateway, the redirect function is Continue reading

DeepLinks and ScrollAnchor

What are DeepLinks?

To directly quote Wikipedia:

“Deep linking is the use of a hyperlink that links to a specific, generally searchable or indexed, piece of web content on a website (e.g. http://example.com/path/page), rather than the website's home page (e.g., http://example.com). The URL contains all the information needed to point to a particular item.”

Why DeepLinks in Dashboard?

There are many user experiences in Cloudflare’s Dashboard that are enhanced by the use of deep linking, such as:

• We’re able to direct users from marketing pages directly into the Dashboard so they can interact with new/changed features.
• Troubleshooting docs can have clearer, more intently directions. e.g. “Enable SSL encryption here” vs “Log into the Dashboard, choose your account and zone, navigate to the security tab, change SSL encryption level, blah blah blah”.

One of the interesting challenges with deep linking in the Dashboard is that most interesting resources are “locked” behind the context of an account and a zone/domain/website. To illustrate this, look at a tree of possible URL paths into Cloudflare’s Dashboard:

dash.cloudflare.com/ -> root-level resources: login, sign-up, forgot-password, two-factor

dash.cloudflare.com/<accountId>/ -> account-level resources: analytics, workers,  Continue reading

Network-Layer DDoS Attack Trends for Q1 2020

As we wrapped up the first quarter of 2020, we set out to understand if and how DDoS attack trends have shifted during this unprecedented time of global shelter in place. Since then, traffic levels have increased by over 50% in many countries, but have DDoS attacks increased as well?

Traffic increases are often observed during holiday seasons. During holidays, people may spend more time online; whether shopping, ordering food, playing online games or a myriad of other online activities. This higher usage translates into higher revenue per minute for the companies that provide those various online services.

Downtime or service degradation during these peak times could result in user churn and loss of significant revenue in a very short time. ITIC estimates that the average cost of an outage is $5,600 per minute, which extrapolates to well over $300K per hour. It is therefore no surprise that attackers capitalize on the opportunity by launching a higher number of DDoS attacks during the holiday seasons.

The current pandemic has a similar cause and effect. People are forced to stay home. They have become more reliant on online services to accomplish their daily tasks which has generated a surge in the Continue reading

Backblaze B2 and the S3 Compatible API on Cloudflare

In May 2020, Backblaze, a founding Bandwidth Alliance partner announced S3 compatible APIs for their B2 Cloud Storage service. As a refresher, the Bandwidth Alliance is a group of forward-thinking cloud and networking companies that are committed to discounting or waiving data transfer fees for shared customers. Backblaze has been a proud partner since 2018. We are excited to see Backblaze introduce a new level of compatibility in their Cloud Storage service.

History of the S3 API

First let’s dive into the history of the S3 API and why it’s important for Cloudflare users.

Prior to 2006, before the mass migration to the Cloud, if you wanted to store content for your company you needed to build your own expensive and highly available storage platform that was large enough to store all your existing content with enough growth headroom for your business. AWS launched to help eliminate this model by renting their physical computing and storage infrastructure.

Amazon Simple Storage Service (S3) led the market by offering a scalable and resilient tool for storing unlimited amounts of data without building it yourself. It could be integrated into any application but there was one catch: you couldn’t use any existing standard Continue reading

Making Video Intuitive: An Explainer

On the Stream team at Cloudflare, we work to provide a great viewing experience while keeping our service affordable. That involves a lot of small tweaks to our video pipeline that can be difficult to discern by most people. And that makes the results of those tweaks less intuitive.

In this post, let's have some fun. Instead of fine-grained optimization work, we’ll do the opposite. Today we’ll make it easy to see changes between different versions of a video: we’ll start with a high-quality video and ruin it. Instead of aiming for perfection, let’s see the impact of various video coding settings. We’ll go on a deep dive on how to make some victim video look gloriously bad and learn on the way.

Everyone agrees that video on the Internet should look good, start playing fast, and never rebuffer regardless of the device they’re on. People can prefer one version of a video over another and say it looks better. Most people, though, would have difficulty elaborating on what ‘better’ means. That’s not an issue when you’re just consuming video. However, when you’re storing, encoding, and distributing it, how that video looks determines how happy your viewers are.

To determine Continue reading

CUBIC and HyStart++ Support in quiche

quiche, Cloudflare's IETF QUIC implementation has been running CUBIC congestion control for a while in our production environment as mentioned in Comparing HTTP/3 vs. HTTP/2 Performance). Recently we also added HyStart++  to the congestion control module for further improvements.

In this post, we will talk about QUIC congestion control and loss recovery briefly and CUBIC and HyStart++ in the quiche congestion control module. We will also discuss lab test results and how to visualize those using qlog which was recently added to the quiche library as well.

QUIC Congestion Control and Loss Recovery

In the network transport area, congestion control is how to decide how much data the connection can send into the network. It has an important role in networking so as not to overrun the link but also at the same time it needs to play nice with other connections in the same network to ensure that the overall network, the Internet, doesn’t collapse. Basically congestion control is trying to detect the current capacity of the link and tune itself in real time and it’s one of the core algorithms for running the Internet.

QUIC congestion control has been written based on many years of TCP Continue reading