L4Drop: XDP DDoS Mitigations

Efficient packet dropping is a key part of Cloudflare’s distributed denial of service (DDoS) attack mitigations. In this post, we introduce a new tool in our packet dropping arsenal: L4Drop.

We've written about our DDoS mitigation pipeline extensively in the past, covering:

• Gatebot: analyzes traffic hitting our edge and deploys DDoS mitigations matching suspect traffic.
• bpftools: generates Berkeley Packet Filter (BPF) bytecode that matches packets based on DNS queries, p0F signatures, or tcpdump filters.
• Iptables: matches traffic against the BPF generated by bpftools using the xt_bpf module, and drops it.
• Floodgate: offloads work from iptables during big attacks that could otherwise overwhelm the kernel networking stack. Incoming traffic bypasses the kernel to go directly to a BPF interpreter in userspace, which efficiently drops packets matching the BPF rules produced by bpftools.

Both iptables and Floodgate send samples of received traffic to Gatebot for analysis, and filter incoming packets using rules generated by bpftools. This ends up looking something like this:

This pipeline has served us well, but a lot has changed since we implemented Floodgate. Our new Gen9 and ARM servers use different network Continue reading

Working out Backend Knots and Building Routers (Austin Real World Serverless Video Recap)

I work in Developer Relations at Cloudflare and I'm fortunate to have top-notch developers around me who are willing to share their knowledge with the greater developer community. I produced a series of events this autumn called Real World Serverless at multiple locations around the world and I want to share the recorded videos from these events.

Our Austin Real World Serverless event (in partnership with the ATX Serverless User Group Meetup) included two talks about Serverless technology featuring Victoria Bernard and Preston Pham from Cloudflare. They spoke about working out backend knots with Workers and building a router for the great good.

About the talks

Working out Backend Knots with Workers - Victoria Bernard (0:00-15:19)

Cloudflare Workers is a platform the makes serverless development and deployment easier than ever. A worker is a script running between your clients' browsers and your site's origin that can intercept requests. Victoria went over some popular use cases of how proxy workers can dramatically improve a site's performance and add functionality that would normally require toying with complicated back-end services.

Build a Router for Great Good - Preston Pham (15:20-33:53)

Serverless computing is great, but requires routing or some kind of API Continue reading

My Second Cloudflare Company Retreat

Last week, 760 humans from Singapore, London, Beijing, Sydney, Nairobi, Austin, New York, Miami, Washington DC, Warsaw, Munich, Brussels, and Champaign reunited with their San Francisco counterparts for our 9th annual Cloudflare company retreat in the San Francisco Bay Area. The purpose of the company retreat is to bring all global employees together under one roof to bond, build bridges, have fun, and learn – all in support of Cloudflare’s mission to help build a better Internet.

It’s easy to write off corporate retreats as an obligatory series of meetings and tired speeches, but Cloudflare’s retreats are uniquely engaging, personalized, fun, and inspiring. Having grown with Cloudflare over the last year (I started just before our 2017 retreat), I wanted to share some of my experiences to highlight Cloudflare’s incredible culture.

The office was buzzing with different languages and laughter as people hugged and shook hands for the first time after working online together for a year or more. Everyone’s Google calendar looked like a rainbow as we each mined for white space to squeeze in those coveted 1:1s, all-hands presentations, and bowling offsites with our global colleagues.  The buses and Google chats felt like summer camp, with people claiming Continue reading

Every 7.8μs your computer’s memory has a hiccup

Modern DDR3 SDRAM. Source: BY-SA/4.0 by Kjerish

During my recent visit to the Computer History Museum in Mountain View, I found myself staring at some ancient magnetic core memory.

Source: BY-SA/3.0 by Konstantin Lanzet

I promptly concluded I had absolutely no idea on how these things could ever work. I wondered if the rings rotate (they don't), and why each ring has three wires woven through it (and I still don’t understand exactly how these work). More importantly, I realized I have very little understanding on how the modern computer memory - dynamic RAM - works!

Source: Ulrich Drepper's series about memory

I was particularly interested in one of the consequences of how dynamic RAM works. You see, each bit of data is stored by the charge (or lack of it) on a tiny capacitor within the RAM chip. But these capacitors gradually lose their charge over time. To avoid losing the stored data, they must regularly get refreshed to restore the charge (if present) to its original level. This refresh process involves reading the value of every bit and then writing it back. During this "refresh" time, the memory is busy and it can't perform normal operations Continue reading

Serverless Progressive Web Apps using React with Cloudflare Workers

Let me tell you the story of how I learned that you can build Progressive Web Apps on Cloudflare’s network around the globe with one JavaScript bundle that runs both in the browser and on Cloudflare Workers with no modification and no separate bundling for client and server. And when registered as a Service Worker, the same JavaScript bundle will turn your page into a Progressive Web App that doesn’t even make network requests. Here's how that works...

"Any resemblance to actual startups, living or IPO'd, is purely coincidental and unintended" - @sevki

A (possibly apocryphal) Story

I recently met up with some old friends in London who told me they were starting a new business. They did what every coder would do... they quickly hacked something together, bought a domain, and registered the GitHub org and thus Buzzwords was born.

The idea was simple: you could feed the name of your application into a machine learning model and it would generate the configuration files for your deployment for various container orchestrators. They achieved this by going through millions of deployment configurations and training a linear regression model by gamifying quantum computing because blockchain, or something (I told you this Continue reading

Fast Google Fonts with Cloudflare Workers

Google Fonts is one of the most common third-party resources on the web, but carries with it significant user-facing performance issues. Cloudflare Workers running at the edge is a great solution for fixing these performance issues, without having to modify the publishing system for every site using Google Fonts.

This post walks through the implementation details for how to fix the performance of Google Fonts with Cloudflare Workers. More importantly, it also provides code for doing high-performance content modification at the edge using Cloudflare Workers.

Google fonts are SLOW

First, some background. Google Fonts provides a rich selection of royalty-free fonts for sites to use. You select the fonts you want to use, and end up with a simple stylesheet URL to include on your pages, as well as styles to use for applying the fonts to parts of the page:

<link href="https://fonts.googleapis.com/css?family=Open+Sans|Roboto+Slab"
      rel="stylesheet">
<style>
body {
 font-family: 'Open Sans', sans-serif;
}
h1 {
 font-family: 'Roboto Slab', serif;
}

Your visitor’s browser fetches the CSS file as soon as the HTML for the page is available. The browser will request the underlying font files when the browser does layout for the page and discovers that it needs Continue reading

My time as an intern (thus far)

It has been over three months since I started as a marketing intern at Cloudflare. Even before joining the Cloudflare team, I enjoyed reading the technical blog posts about Cloudflare’s use cases and solutions, as well as the inclusive and creative culture. Educating the world about the threats we face on the Internet is something that I found truly valuable. I figured that I would give my own spin on what it’s like to join and work at Cloudflare by writing a blog post too.

Chapter 1: The Path towards the Orange Cloud

Before starting as a freshman in university, I created an online portfolio for my photography. It has been a passion of mine for about 9 years. I tried a multitude of platforms but none afforded me the aesthetic control that I wanted. The only solution was to build and host my own site. I started learning HTML/CSS, a bit of JavaScript and jQuery and so on.

This led to me using the Koken CMS, and hosting it on DigitalOcean in a Docker container. What was left was SSL/TLS encryption and a CDN (my friends in Europe find the loading times unbearable). Continue reading

Cloudflare’s Response to a Privacy Framework

Why We Weighed In on US Privacy Efforts

Cloudflare’s mission is to help build a better internet, and privacy has to be at the heart of that effort. That’s why we submitted comments last week on the National Telecommunications and Information Administration (NTIA)’s request for comment on its proposed approach to advance consumer privacy.

We think it is important for Internet infrastructure companies like us to be a part of the conversation about the future of internet privacy. We want to advocate for an internet that remains accessible to all, while becoming more secure and protective of privacy.

What is NTIA and what is it trying to do?

In 2018, we’ve seen high profile data breaches and data misuse, Europe’s sweeping data protection law – the General Data Protection Regulation (GDPR) – come into effect, and California pass its own comprehensive Consumer Privacy Act (CCPA). All of this has captured the attention of Washington, D.C. lawmakers and regulators.

On September 25, 2018, NTIA began a process to solicit feedback from stakeholders on a proposed approach to consumer data privacy.  NTIA is the Executive Branch agency in the Department of Commerce that is principally Continue reading

Introducing Apps with Workers

Since its inception, Cloudflare Apps has made it possible for website owners to install a tool or a service directly onto a domain they have that lives on the Cloudflare network.  As an App Developer, you’ve had the ability to deliver your product by modifying traffic and content of a user’s website using a combination of Javascript, HTML, CSS injection with the ability to modify Cloudflare features such as DNS.

Today we are announcing a public beta for our Cloudflare Apps with Workers!  If you are a developer who has been using Workers or Cloudflare Apps, this is a significant step forward to bring together our server-less platform allowing you the next wave of innovation in the Cloudflare marketplace.

Wait!  Yet more Workers news?  It’s already been a furious year of releases for Workers, from our GA launch in February to numerous improvements including the beta of Cache API and a flurry of updates during Cloudflare’s Birthday week, where we made available our K/V Storage and WASM support.  With this beta, developers can now package all Workers capabilities, deliver new worker-powered experiences to Cloudflare customers all within a few clicks via Cloudflare Apps.

Now your Apps can be more powerful, Continue reading

Announcing SSH Access through Cloudflare

We held our annual Cloudflare Retreat last week. Over 750 team members from nearly a dozen offices spent three days learning, bonding and some of them got to smash a VPN piñata on stage with a baseball bat. Yes, you read that right.

The latest feature added to Cloudflare Access let us celebrate the replacement of our clunky VPN with a faster, safer way to reach our internal applications. You can now place applications that require SSH connections, like your source control repository, behind Cloudflare Access. We’re excited to release that same feature so that your team can also destroy your own VPN (piñata not included).

How we smashed our VPN

We built Access to replace our corporate VPN. We started with browser-based applications, moved to CLI operations, and then began adding a growing list of single sign-on integrations. Our teammates added single sign-on support to the Cloudflare dashboard by combining Access and our serverless product, Workers. We improved the daily workflow of every team member each time we moved another application behind Access. However, SSH connections held us back. Whenever we needed to push code or review a pull request, we had to fall back to our Continue reading

Bandwidth Alliance Partners – Exciting Choices

We are tremendously excited about the value our Bandwidth Alliance partner ecosystem adds to our customers. We’re on a mission to help make the internet a better place; and ensuring everyone can access cloud resources at zero-egress rates supports that mission in many ways. It’s an easy way for our clients to build modern, cloud-centric applications without the design constraint and financial burden of egress fees.

The cloudflare bandwidth alliance partner landscape continues to grow, and incorporate a diverse group of partners, with today’s second wave announcement.  With over a dozen different partners, the range of choices can quickly become overwhelming. And, while these are all high-quality platforms which we are happy to recommend to our clients - their important differences will help determine the best fit for you, the customer.

In this post, I’ll lay out some of Cloudflare’s approach to this solution design question through the lens of a large client we recently worked with. We apply this approach across our full range of products and services, including many use cases far different from the Storage need we’ll dig into in this post. I hope that this can help all of our clients, or anyone else interested, mirror Continue reading

Expanding the Bandwidth Alliance: sharing the benefits of interconnected networks

At Cloudflare, our mission is to help build a better Internet. That means making the Internet faster, smarter, safer, but also more cost efficient with the help of our partners. We are always on the lookout for ways to help save customers money. With that goal we announced the Bandwidth Alliance with our founding partners during our Birthday week.

The key concept of the Bandwidth Alliance is to help reduce and in many cases waive data transfer charges, sometimes known as "bandwidth” or “egress” charges, for our mutual customers. We achieve this in partnership with the founding partners through strongly interconnected networks over peered connections. These connections typically occur within the same facility with no middleman. So, neither Cloudflare nor the cloud provider bears incremental costs. Further, we will also use our smart routing system (read details in this technical blog post) to ensure that all our customers’ traffic on participating cloud providers, once their systems are set up, qualify for this offer.

Expanding the Alliance: new Partners Committed to Discounting Data Transfer Fees

We are proud to announce the following cloud providers and hosting companies have joined the Bandwidth Alliance in committing to zero data transfer fees for Continue reading

How a Nigerian ISP Accidentally Knocked Google Offline

Last Monday evening — 12 November 2018 — Google and a number of other services experienced a 74 minute outage. It’s not the first time this has happened; and while there might be a temptation to assume that bad actors are at work, incidents like this only serve to demonstrate just how much frailty is involved in how packets get from one point on the Internet to another.

Our logs show that at 21:12 UTC on Monday, a Nigerian ISP, MainOne, accidentally misconfigured part of their network causing a "route leak". This resulted in Google and a number of other networks being routed over unusual network paths. Incidents like this actually happen quite frequently, but in this case, the traffic flows generated by Google users were so great that they overwhelmed the intermediary networks — resulting in numerous services (but predominantly Google) unreachable.

You might be surprised to learn that an error by an ISP somewhere in the world could result in Google and other services going offline. This blog post explains how that can happen and what the Internet community is doing to try to fix this fragility.

What Is A Route Leak, And How Does One Happen?

Continue reading

Real URLs for AMP Cached Content Using Cloudflare Workers

Today, we’re excited to announce our solution for arguably the biggest issue affecting Accelerated Mobile Pages (AMP): the inability to use real origin URLs when serving AMP-cached content. To allow AMP caches to serve content under its origin URL, we implemented HTTP signed exchanges, which extend authenticity and integrity to content cached and served on behalf of a publisher. This logic lives on Cloudflare Workers, meaning that adding HTTP signed exchanges to your content is just a simple Workers application away. Publishers on Cloudflare can now take advantage of AMP performance and have AMP caches serve content with their origin URLs. We're thrilled to use Workers as a core component of this solution.

HTTP signed exchanges are a crucial component of the emerging Web Packaging standard, a set of protocols used to package websites for distribution through optimized delivery systems like Google AMP. This announcement comes just in time for Chrome Dev Summit 2018, where our colleague Rustam Lalkaka spoke about our efforts to advance the Web Packaging standard.

What is Web Packaging and Why Does it Matter?

You may already see the need for Web Packaging on a daily basis. On your smartphone, perhaps you’ve searched for Christmas Continue reading

The rise of multivector DDoS attacks

It's been a while since we last wrote about Layer 3/4 DDoS attacks on this blog. This is a good news - we've been quietly handling the daily onslaught of DDoS attacks. Since our last write-up, a handful of interesting L3/4 attacks have happened. Let's review them.

Gigantic SYN

In April, John tweeted about a gigantic 942Gbps SYN flood:

It was a notable event for a couple of reasons.

First, it was really large. Previously, we've seen only amplification / reflection attacks at terabit scale. In those cases, the attacker doesn't actually have too much capacity. They need to bounce the traffic off other servers to generate a substantial load. This is different from typical "direct" style attacks, like SYN floods. In the SYN flood mentioned by John, all 942Gbps were coming directly from attacker-controlled machines.

Secondly, this attack was truly distributed. Normal SYN floods come from a small number of geographical locations. This one, was all over the globe, hitting all Cloudflare data centers:

Thirdly, the attack seem to be partially spoofed. While our analysis was not conclusive, we saw random, spoofed source IP addresses in the largest internet exchanges. The above Hilbert curve shows the source IP Continue reading

1 Thing You Can Do To Make Your Internet Safer And Faster

On April 1st, 2018, we announced 1.1.1.1, the fastest public DNS resolver in the world ???. Today, we are launching the 1.1.1.1 mobile app to make it incredibly easy to use 1.1.1.1 on your phone.

TL;DR

Any time you are on a public internet connection people can see what sites you visit. Even worse, your Internet Service Provider is very possibly selling all of your browsing history to the highest bidder. We have a tool called 1.1.1.1 which makes it easy to get a faster, more private, Internet experience, but it’s historically been too complex for many people to use, particularly on mobile devices. Today, we’re launching an app you (and everyone you know) can use to use 1.1.1.1 every time your mobile phone connects to the Internet. It’s a free, it’s easy, download it now.

Fastest Public Resolver

We launched 1.1.1.1 on April 1st. Frankly, we’ve been blown away by how many people actually made the switch. Changing your network settings is not easy, but if our traffic amount is any indication, many of you made the effort. Continue reading

Cloud Computing without Containers

Cloudflare has a cloud computing platform called Workers. Unlike essentially every other cloud computing platform I know of, it doesn’t use containers or virtual machines. We believe that is the future of Serverless and cloud computing in general, and I’ll try to convince you why.

Isolates

Two years ago we had a problem. We were limited in how many features and options we could build in-house, we needed a way for customers to be able to build for themselves. We set out to find a way to let people write code on our servers deployed around the world (we had a little over a hundred data centers then, 155 as of this writing). Our system needed to run untrusted code securely, with low overhead. We sit in front of ten million sites and process millions and millions of requests per second, it also had to run very very quickly.

The Lua we had used previously didn’t run in a sandbox; customers couldn’t write their own code without our supervision. Traditional virtualization and container technologies like Kubernetes would have been exceptionally expensive for everyone involved. Running thousands of Kubernetes pods in a single location would be resource intensive, doing it in Continue reading

Improving RubyDocs with Cloudflare Workers and Workers KV

The following is a guest post from Manuel Meurer, Berlin based web developer, entrepreneur, and Ruby on Rails enthusiast. In 2010, he founded Kraut Computing as a one-man web dev shop and launched Uplink, a network for IT experts in Germany, in 2015.

RubyDocs is an open-source service that generates and hosts “fancy docs for any Ruby project”, most notably for the Ruby language itself and for Rails, the most popular Ruby framework. The nifty thing about it is that the docs can be generated for any version of a project — so let’s say you’re working on an old Rails app that still uses version 3.2.22 (released June 16, 2015), then you can really benefit from having access to the docs of that specific version, since a lot of the methods, classes, and concepts of the current Rails version (5.2.1 at the time of writing) don’t exist in that old version.

Scratching an itch

I built RubyDocs back in 2013 to scratch my own itch — a few similar services that I had used over the years had disappeared or hadn’t been regularly updated. After the initial work to get RubyDocs up and running, I continued Continue reading

Lagos, Nigeria – Cloudflare’s 155th city

At just shy of 200 million, Nigeria is the most populous country in Africa (Ethiopia is second and Egypt is third). That’s a lot of people to communicate with the world - and communicate they all do!

According to a published report earlier this year, 84% of the Nigerian population own a mobile device (193 million population and 162 million mobile subscriptions). Again, that’s #1 for any country in Africa. But why so connected? Maybe because Nigeria (and Lagos specifically) is always on the move!

Lagos, as those that know the city say, never sleeps, it’s filled with color from the food to fashion to even the diverse people going about their business. The vibrancy of the city is like a hard slap to the face, no matter what you have been told, your first time here will still knock you out. In Lagos, anything is possible, from the sadness of poverty to the clearly visible upper class, the city sucks you in like a surfers dream wave. Visitor come into Lagos and leave feeling like they’ve been through a unique experience. The traffic is mind blowing and the same goes for the work pace.

Lagos, a city always on Continue reading

Introducing Spectrum with Load Balancing

We’re excited to announce the full integration of Cloudflare Spectrum with Load Balancing. Combining Spectrum with Load Balancing enables traffic management of TCP connections utilising the same battle tested Load Balancer our customers already use for billions of HTTP requests every day.

Customers can configure load balancers with TCP health checks, failover, and steering policies to dictate where traffic should flow. This is live in the Cloudflare dashboard and API — give it a shot!

TCP Health Checks

You can now configure Cloudflare’s Load Balancer health checks to probe any TCP port for an accepted connection. This is in addition to the existing HTTP and HTTPS options.

Health checks are an optional feature within Cloudflare’s Load Balancing product. Without health checks, the Cloudflare Load Balancer will distribute traffic to all origins in the first pool. While this is in itself useful, adding a health check to a Load Balancer provides additional functionality.

With a health check configured for a pool in a Load Balancer, Cloudflare will automatically distribute traffic within a pool to any origins that are marked up by the health check. Unhealthy origins will be dropped automatically. This allows for intelligent failover both within a pool and amongst Continue reading