Stupidly Simple DDoS Protocol (SSDP) generates 100 Gbps DDoS

Last month we shared statistics on some popular reflection attacks. Back then the average SSDP attack size was ~12 Gbps and largest SSDP reflection we recorded was:

• 30 Mpps (millions of packets per second)
• 80 Gbps (billions of bits per second)
• using 940k reflector IPs

This changed a couple of days ago when we noticed an unusually large SSDP amplification. It's worth deeper investigation since it crossed the symbolic threshold of 100 Gbps.

The packets per second chart during the attack looked like this:

The bandwidth usage:

This packet flood lasted 38 minutes. According to our sampled netflow data it utilized 930k reflector servers. We estimate that the during 38 minutes of the attack each reflector sent 112k packets to Cloudflare.

The reflector servers are across the globe, with a large presence in Argentina, Russia and China. Here are the unique IPs per country:

$ cat ips-nf-ct.txt|uniq|cut -f 2|sort|uniq -c|sort -nr|head
 439126 CN
 135783 RU
  74825 AR
  51222 US
  41353 TW
  32850 CA
  19558 MY
  18962 CO
  14234 BR
  10824 KR
  10334 UA
   9103 IT
   ...

The reflector IP distribution across ASNs is typical. It pretty much follows the world’s largest residential ISPs:

$ cat ips-nf-asn.txt |uniq|cut -f 2|sort|uniq  Continue reading

Announcing the Cloudflare Apps Platform and Developer Fund

When we started Cloudflare we had no idea if anyone would validate our core idea. Our idea was what that everyone should have the ability to be as fast and secure as the Internet giants like Google, Facebook, and Microsoft. Six years later, it's incredible how far that core idea has taken us.

CC BY-SA 2.0 image by Mobilus In Mobili

Today, Cloudflare runs one of the largest global networks. We have data centers in 115 cities around the world and continue to expand. We've built a core service that delivers performance, security, availability, and insight to more than 6 million users.

Democratizing the Internet

From the beginning, our goal has been to democratize the Internet. Today we're taking another step toward that goal with the launch of the Cloudflare Apps Platform and the Cloudflare Developer Fund. To understand that, you have to understand where we started.

When we started Cloudflare we needed two things: a collection of users for the service, and finances to help us fund our development. In both cases, people were taking a risk on Cloudflare. Our first users came from Project Honey Pot, which Lee Holloway and I created back in 2004. Members Continue reading

Announcing the New Cloudflare Apps

Today we’re excited to announce the next generation of Cloudflare Apps. Cloudflare Apps is an open platform of tools to build a high quality website. It’s a place where every website owner can select from a vast catalog of Apps which can improve their websites and internet properties in every way imaginable. Selected apps can be previewed and installed instantly with just a few clicks, giving every website owner the power of technical expertise, and every developer the platform only Cloudflare can provide.

Apps can modify content and layout on the page they’re installed on, communicate with external services and dramatically improve websites. Imagine Google Analytics, YouTube videos, in-page chat tools, widgets, themes and every other business which can be built by improving websites. All of these and more can be done with Cloudflare Apps.

Cloudflare Apps makes it possible for a developer in her basement to build the next great new tool and get it on a million websites overnight. With Cloudflare Apps, even the smallest teams can get massive distribution for their apps on the web so that the best products win. With your help we will make it possible for developers like you to build a new Continue reading

Project Jengo: Explaining Challenges to Patent Validity (and a looming threat)

We’ve written a couple times about the problem of patent trolls, and what we are doing in response to the first case a troll filed against Cloudflare. We set a goal to find prior art on all 38 Blackbird Tech patents and applications and then obtain a legal determination that Blackbird Tech’s patents are invalid. Such a determination will end Blackbird’s ability to file or threaten to file abusive patent claims, against us or anyone else.

CC BY-SA 2.0 image by hyku

The patent system exists to reward inventors, so it is no surprise that a patent has to claim something new — an “invention.” Sometimes the United States Patent and Trademark Office (USPTO) — the agency that administers the patent system — mistakenly issues patents that do not claim anything particularly new. The patent examiner may not be aware that the proposed “invention” was already in use in the industry, and the patent applicant (the only party in the process) doesn’t have an incentive to share that information. Often, the USPTO issues patents that are too vague and can later be broadly interpreted by patent owners to cover different and subsequent technologies that could not otherwise Continue reading

When the Internet (Officially) Became the Public Square

Sometimes, well-intended efforts to prevent unacceptable behavior run into the reality of what it means to have an open and free society. That is what happened at the Supreme Court on Monday.

Souvenir Postcard by unknown

The Supreme Court issued an opinion confirming something we at Cloudflare have long believed -- that the First Amendment protects access to the Internet. Using sweeping language, Justice Kennedy compared internet access to access to a street or park, "essential venues for public gatherings to celebrate some views, to protest others, or simply to learn and inquire,” and concluded that "to foreclose access to social media altogether is to prevent the user from engaging in the legitimate exercise of First Amendment rights."

We share this view of the internet as a forum to discuss and debate ideas, and believe that the Court’s opinion is an important reaffirmation of the free speech principles we support.

The Packingham Case

Like many other First Amendment cases, the law at the heart of the Packingham v. North Carolina case presents complex questions about how to protect the community in ways consistent with the right to free speech.

In 2008, North Carolina passed a law making it a Continue reading

Counting things, a lot of different things…

Back in April we announced Rate Limiting of requests for every Cloudflare customer. Being able to rate limit at the edge of the network has many advantages: it’s easier for customers to set up and operate, their origin servers are not bothered by excessive traffic or layer 7 attacks, the performance and memory cost of rate limiting is offloaded to the edge, and more.

In a nutshell, rate limiting works like this:

• Customers can define one or more rate limit rules that match particular HTTP requests (failed login attempts, expensive API calls, etc.)

• Every request that matches the rule is counted per client IP address

• Once that counter exceeds a threshold, further requests are not allowed to reach the origin server and an error page is returned to the client instead

This is a simple yet effective protection against brute force attacks on login pages and other sorts of abusive traffic like L7 DoS attacks.

Doing this with possibly millions of domains and even more millions of rules immediately becomes a bit more complicated. This article is a look at how we implemented a rate limiter able to run quickly and accurately at the edge of the network which Continue reading

Less Is More – Why The IPv6 Switch Is Missing

At Cloudflare we believe in being good to the Internet and good to our customers. By moving on from the legacy world of IPv4-only to the modern-day world where IPv4 and IPv6 are treated equally, we believe we are doing exactly that.

"No matter what happens in life, be good to people. Being good to people is a wonderful legacy to leave behind." - Taylor Swift (whose website has been IPv6 enabled for many many years)

Starting today with free domains, IPv6 is no longer something you can toggle on and off, it’s always just on.

How we got here

Cloudflare has always been a gateway for visitors on IPv6 connections to access sites and applications hosted on legacy IPv4-only infrastructure. Connections to Cloudflare are terminated on either IP version and then proxied to the backend over whichever IP version the backend infrastructure can accept.

That means that a v6-only mobile phone (looking at you, T-Mobile users) can establish a clean path to any site or mobile app behind Cloudflare instead of doing an expensive 464XLAT protocol translation as part of the connection (shaving milliseconds and conserving very precious battery life).

That IPv6 gateway is set by a simple Continue reading

Patent Troll Battle Update: Doubling Down on Project Jengo

Jengo Fett by Brickset (Flickr)

We knew the case against patent trolls was the right one, but we have been overwhelmed by the response to our blog posts on patent trolls and our program for finding prior art on the patents held by Blackbird Tech, which we’ve dubbed Project Jengo. As we discuss in this post, your comments and contributions have allowed us to expand and intensify our efforts to challenge the growing threat that patent trolls pose to innovative tech companies.

We’re SIGNIFICANTLY expanding our program to find prior art on the Blackbird Tech patents

In a little over a week since we started the program, we’ve received 141 separate prior art submissions. But we know there’s an opportunity to find a lot more.

We’ve been impressed with the exceptionally high quality of the submissions. The Cloudflare community of users and readers of our blog are an accomplished bunch, so we have a number of searches that were done by expert engineers and programmers. In one case that stood out to us, someone wrote in about a project they personally had worked on as an engineer back in 1993, which they are convinced is conclusive prior art Continue reading

Reflections on reflection (attacks)

Recently Akamai published an article about CLDAP reflection attacks. This got us thinking. We saw attacks from Conectionless LDAP servers back in November 2016 but totally ignored them because our systems were automatically dropping the attack traffic without any impact.

CC BY 2.0 image by RageZ

We decided to take a second look through our logs and share some statistics about reflection attacks we see regularly. In this blog post, I'll describe popular reflection attacks, explain how to defend against them and why Cloudflare and our customers are immune to most of them.

A recipe for reflection

Let's start with a brief reminder on how reflection attacks (often called "amplification attacks") work.

To bake a reflection attack, the villain needs four ingredients:

• A server capable of performing IP address spoofing.
• A protocol vulnerable to reflection/amplification. Any badly designed UDP-based request-response protocol will do.
• A list of "reflectors": servers that support the vulnerable protocol.
• A victim IP address.

The general idea:

• The villain sends fake UDP requests.
• The source IP address in these packets is spoofed: the attacker sticks the victim's IP address in the source IP address field, not their own IP address as they normally would.
• Each packet Continue reading

Introducing Argo — A faster, more reliable, more secure Internet for everyone

The Internet is inherently unreliable, a collection of networks connected to each other with fiber optics, copper, microwaves and trust. It’s a magical thing, but things on the Internet break all the time; cables get cut, bogus routes get advertised, routers crash. Most of the time, these failures are noticed but inexplicable to the average user — ”The Internet is slow today!” — frustrating user experiences as people go about their lives on the Internet.

Today, to fix all of this, Cloudflare is launching Argo, a “virtual backbone” for the modern Internet. Argo analyzes and optimizes routing decisions across the global Internet in real-time. Think Waze, the automobile route optimization app, but for Internet traffic.

Just as Waze can tell you which route to take when driving by monitoring which roads are congested or blocked, Argo can route connections across the Internet efficiently by avoiding packet loss, congestion, and outages.

Cloudflare’s Argo is able to deliver content across our network with dramatically reduced latency, increased reliability, heightened encryption, and reduced cost vs. an equivalent path across the open Internet. The results are impressive: an average 35% decrease in latency, a 27% decrease in connection errors, and a 60% Continue reading

Introducing Load Balancing & Intelligent Failover with Cloudflare

Cloudflare's Enterprise customers have been using our Load Balancing service since March, and it has been helping them avoid website downtime caused by unreliable hosting providers, Internet outages, or servers. Today, we're bringing Load Balancing to all of our customers.

Even the best caching can't escape the fundamental limitations on performance created by the speed of light. Using Load Balancing, Cloudflare's customers can now route requests between multiple origins, allowing them to serve requests from the closest (and fastest) geographic location.

The Cloudflare Load Balancer automatically sends you notifications when things fail, and when they come back up again, so you can sleep well at night knowing we are keeping your website or API running.

If a DDoS attack can bring down your DNS provider or load balancer, it doesn't matter whether your servers are healthy or not. Our load balancing service runs in Cloudflare's 110+ datacenters, and with experience dealing with some of the largest DDoS attacks, we can withstand traffic volumes that smaller providers, virtual machines or hardware appliances can't. This also allows us to help you avoid business-impacting downtime when major cloud compute providers have issues: when we identify a connectivity reaching your application on AWS, we Continue reading

Detroit and San Diego Data Centers expand Cloudflare network to 26 North American cities

Cloudflare is excited to announce deployments in Detroit (Michigan) and San Diego (California), which are our 114th and 115th data centers respectively. They join Colombo, Sri Lanka and Cape Town, South Africa in the cohort of four new cities added just this week to our growing global network, which spans 57 countries and counting.

For over 6 million Internet properties, we now serve customer traffic from across 26 North American cities, including 22 in the United States alone. We're not going to stop building our network until we're within milliseconds of every Internet user, and to that end, data centers are already in the works in eight additional North American cities (and many others around the world).

Connections

Source: Baja Insider

Detroit and San Diego share something special, as they are immediately adjacent to international borders with Canada and Mexico respectively. Detroit has four border crossings to Windsor, Ontario, including the Ambassador Bridge, which was built in the Roaring Twenties, and accommodates over a quarter of all merchandise trade with Canada.

Founded in 1701, and best known for cars and Motown, Detroit eagerly awaits a 3,000 pound bronze RoboCop statue to watch over Delta City (track progress here). Continue reading

Standing Up to a Dangerous New Breed of Patent Troll

On March 20th, Cloudflare received our first patent infringement claim: Blackbird Tech LLC v. Cloudflare, Inc. Today we’re filing our Answer to that claim in a federal court in Delaware. We have very strong arguments we will present in the litigation, mostly because the patent asserted against us does not have anything to do with our technology.

• The infringement claim is not a close one. The asserted patent, US 6453335 (‘335 patent) was filed in 1998, and describes a system for monitoring an existing data channel and inserting error pages when transmission rates fall below a certain level. Nothing from 1998—including the ’335 patent—comes close to describing our state-of-the-art product that is provisioned via DNS, speeds up internet content delivery, and protects against malicious attackers. Our technology is innovative and different, and Cloudflare’s technology has about 150 patents issued or in process.

• We also expect to show that the patent itself is invalid. For example, if the ’335 patent is read broadly enough to cover our system (which shouldn’t happen), it would also cover any system where electronic communications are examined and redacted or modified. But this is not new. Filtering products performing similar functions were around long before Continue reading

Project Jengo: Cloudflare’s Prior Art Search Bounty

Jengo Fett by Brickset (Flickr)

As readers of this blog likely know, especially if you read this post, Cloudflare has been sued by a dangerous new breed of patent troll, Blackbird Technologies, asserting a very old and very vague patent. And we know we are not alone in being frustrated about the way that such patent trolls inhibit the growth of innovative companies. Cloudflare is asking for your help in this effort, and we’re putting our money where our mouth is.

Patent trolls take advantage of a system they assume is tilted in their favor, where they can take vague technology patents issued years ago and apply them as broadly as imaginable to the latest technology. And they do this without the limitations of having to show the original patent holder would have actually exercised the patent, because most of them don’t, at all. Patent trolls think they can sit back and pick off settlements from companies because their lawsuits are a nuisance and the costs of defending those suits are considerable.

Changing this dynamic and leveling the playing field is going to require an entirely new approach. Fighting such strong, though perverse, economic incentives is going Continue reading

Cape Town (South Africa): Cloudflare Data Center #113

Five fun facts:

• Cape Town is where the Atlantic Ocean and the Indian Ocean meet deep in the Southern Hemisphere.
  
• The city is the start of the Garden Route, a 185 mile (300 km) glorious coastal drive brimming with native flowers and stunning vistas.
  
• Cape Town is the gateway into the Stellenbosch and Franschhoek wine districts that make for very popular weekend excursions.
  
• The imposing Table Mountain can be seen from most of Cape Town. When you take the cable car to the top of the mountain, the view from up there is even more stunning.
  
• Cape Town is where Cloudflare has placed its 113th data center.
  

Second data center in South Africa

Back in December 2014, Cloudflare opened our first data center in Africa and our 30th datacenter globally. That was in Johannesburg, which has since seen over 10x growth in traffic delivered to South Africa and surrounding countries.

Now, we are expanding into our second city in South Africa — Cape Town, bringing us 870 miles (1,400km) closer to millions of Internet users. Only 15% smaller than Johannesburg by population, Cape Town commands a majority of the tourism business for the country.

For Cloudflare, our newest Continue reading

How Cloudflare analyzes 1M DNS queries per second

On Friday, we announced DNS analytics for all Cloudflare customers. Because of our scale –– by the time you’ve finished reading this, Cloudflare DNS will have handled millions of DNS queries –– we had to be creative in our implementation. In this post, we’ll describe the systems that make up DNS Analytics which help us comb through trillions of these logs each month.

How logs come in from the edge

Cloudflare already has a data pipeline for HTTP logs. We wanted to utilize what we could of that system for the new DNS analytics. Every time one of our edge services gets an HTTP request, it generates a structured log message in the Cap’n Proto format and sends it to a local multiplexer service. Given the volume of the data, we chose not to record the full DNS message payload, only telemetry data we are interested in such as response code, size, or query name, which has allowed us to keep only ~150 bytes on average per message. It is then fused with processing metadata such as timing information and exceptions triggered during query processing. The benefit of fusing data and metadata at the edge is that we can spread Continue reading

Colombo, Sri Lanka: Six million Internet properties now faster for six million Internet users

We are excited to add four new data centers this week to Cloudflare's growing network, beginning with Colombo, Sri Lanka. This deployment is our 112th data center globally, and our 38th in Asia.

Faster Performance

CC BY-NC-ND 2.0 image by Pavel Dobrovsky

Six million Internet properties using Cloudflare are now even faster across the island country of Sri Lanka. Previously, local visitors to Cloudflare customers were served out of our Singapore or Dubai data centers.

Latency (ms) decreases 4x to Cloudflare customers. Source: Cedexis

Sri Lanka added over one million Internet users in the past year alone. At ~30% Internet penetration, there is considerable room to grow.

Next Three Cities

Our deployments to be revealed later this week will provide additional redundancy to existing facilities in North America and Africa.

If you enjoy the idea of helping build one of the world's largest networks, come join our team!

Anonymity and Abuse Reports

Last Thursday, ProPublica published an article critiquing our handling of some abuse reports that we receive. Feedback from the article caused us to reevaluate how we handle abuse reports. As a result, we've decided to update our abuse reporting system to allow individuals reporting threats and child sexual abuse material to do so anonymously. We are rolling this change out and expect it to be available by the end of the week.

I appreciate the feedback we received. How we handle abuse reports has evolved over the last six and a half years of Cloudflare's history. I wanted to take this opportunity to walk through some of the rationale that got us to this point and caused us to have a blindspot to the case that was highlighted in the article.

What Is Cloudflare?

Cloudflare is not a hosting provider. We do not store the definitive copy of any of the content that someone may want to file an abuse claim about. If we terminate a customer it doesn’t make the content go away. Instead, we are more akin to a specialized network. One of the functions of the network that we provide is to add security to the content Continue reading

Meet The Brand New DNS Analytics Dashboard

Have you noticed something new in your Cloudflare analytics dashboard this morning? You can now see detailed DNS analytics for your domains on Cloudflare.

If you want to skip to the punch and start exploring, go check it out here. Otherwise, hop on the DNS magic school bus - and let us show you all the neat stats in your now-available DNS analytics.

DNS analytics dashboard: What does it know? Does it know things? Let’s find out.

At the top of the DNS analytics dashboard you can see your DNS traffic health. This “Queries by Response Codes” graph breaks down queries by what response code Cloudflare DNS answered to the visitor. Like HTTP response codes, DNS response codes give an indication of what is happening behind the scenes. Mostly you will just see NOERROR, the HTTP 200 of DNS response codes, and NXDOMAIN, the HTTP 404 of DNS response codes. NXDOMAIN is particularly interesting - what are people querying for that doesn’t exist?

If you are an enterprise customer and you want to know what all the NXDOMAIN queries are, just scroll down a little bit where we show you the top queries for your domain and top queries for Continue reading

Introducing the new Cloudflare Community Forum

Cloudflare’s community of users is vast. With more than 6 million domains registered, our users come in all shapes and sizes and are located all over the world. They can also frequently be found hanging out all around the web, from social media platforms, to Q&A sites, to any number of personal interest forums. Cloudflare users have questions to ask and an awful lot of expertise to share.

It’s with that in mind that we wanted to give Cloudflare users a more centralized location to gather, and to discuss all things Cloudflare. So we have launched a new Cloudflare Community at community.cloudflare.com.

Who is this community for?

It's for anyone and everyone who uses Cloudflare. Whether you are adding your first domain and don’t know what a name server is, or you are managing 1,000s of domains via API, or you are somewhere in between. In the Cloudflare Community you will be able to find tips, tricks, troubleshooting guidance, and recommendations.

We also think this will be a great way to get feedback from users on what’s working for them, what isn’t, and ways that we can make Cloudflare better. There will even be opportunities to Continue reading