High-reliability OCSP stapling and why it matters
At Cloudflare our focus is making the internet faster and more secure. Today we are announcing a new enhancement to our HTTPS service: High-Reliability OCSP stapling. This feature is a step towards enabling an important security feature on the web: certificate revocation checking. Reliable OCSP stapling also improves connection times by up to 30% in some cases. In this post, we’ll explore the importance of certificate revocation checking in HTTPS, the challenges involved in making it reliable, and how we built a robust OCSP stapling service.
Why revocation is hard
Digital certificates are the cornerstone of trust on the web. A digital certificate is like an identification card for a website. It contains identity information including the website’s hostname along with a cryptographic public key. In public key cryptography, each public key has an associated private key. This private key is kept secret by the site owner. For a browser to trust an HTTPS site, the site’s server must provide a certificate that is valid for the site’s hostname and a proof of control of the certificate’s private key. If someone gets access to a certificate’s private key, they can impersonate the site. Private key compromise is a serious risk Continue reading
Participate in the Net Neutrality Day of Action
We at Cloudflare strongly believe in network neutrality, the principle that networks should not discriminate against content that passes through them. We’ve previously posted on our views on net neutrality and the role of the FCC here and here.
In May, the FCC took a first step toward revoking bright-line rules it put in place in 2015 to require ISPs to treat all web content equally. The FCC is seeking public comment on its proposal to eliminate the legal underpinning of the 2015 rules, revoking the FCC's authority to implement and enforce net neutrality protections. Public comments are also requested on whether any rules are needed to prevent ISPs from blocking or throttling web traffic, or creating “fast lanes” for some internet traffic.
To raise awareness about the FCC's efforts, July 12th will be “Internet-Wide Day of Action to save Net Neutrality.” Led by the group Battle for the Net, participating websites will show the world what the web would look like without net neutrality by displaying an alert on their homepage. Website users will be encouraged to contact Congress and the FCC in support of net neutrality.
We wanted to make sure our users had an opportunity to participate in this Continue reading
How to make your site HTTPS-only
The Internet is getting more secure every day as people enable HTTPS, the secure version of HTTP, on their sites and services. Last year, Mozilla reported that the percentage of requests made by Firefox using encrypted HTTPS passed 50% for the first time. HTTPS has numerous benefits that are not available over unencrypted HTTP, including improved performance with HTTP/2, SEO benefits for search engines like Google and the reassuring lock icon in the address bar.
So how do you add HTTPS to your site or service? That’s simple, Cloudflare offers free and automatic HTTPS support for all customers with no configuration. Sign up for any plan and Cloudflare will issue an SSL certificate for you and serve your site over HTTPS.
HTTPS-only
Enabling HTTPS does not mean that all visitors are protected. If a visitor types your website’s name into the address bar of a browser or follows an HTTP link, it will bring them to the insecure HTTP version of your website. In order to make your site HTTPS-only, you need to redirect visitors from the HTTP to the HTTPS version of your site.
Going HTTPS-only should be as easy as a click of a button, so we Continue reading
Three little tools: mmsum, mmwatch, mmhistogram
In a recent blog post, my colleague Marek talked about some SSDP-based DDoS activity we'd been seeing recently. In that blog post he used a tool called mmhistogram to output an ASCII histogram.
That tool is part of a small suite of command-line tools that can be handy when messing with data. Since a reader asked for them to be open sourced... here they are.
mmhistogram
Suppose you have the following CSV of the ages of major Star Wars characters at the time of Episode IV:
Anakin Skywalker (Darth Vader),42
Boba Fett,32
C-3PO,32
Chewbacca,200
Count Dooku,102
Darth Maul,54
Han Solo,29
Jabba the Hutt,600
Jango Fett,66
Jar Jar Binks,52
Lando Calrissian,31
Leia Organa (Princess Leia),19
Luke Skywalker,19
Mace Windu,72
Obi-Wan Kenobi,57
Palpatine,82
Qui-Gon Jinn,92
R2-D2,32
Shmi Skywalker,72
Wedge Antilles,21
Yoda,896
You can get an ASCII histogram of the ages as follows using the mmhistogram tool.
$ cut -d, -f2 epiv | mmhistogram -t "Age"
Age min:19.00 avg:123.90 med=54.00 max:896.00 dev:211.28 count:21
Age:
value |-------------------------------------------------- count
0 | 0
1 | 0
2 | 0
4 | 0
8 | 0
16 |************************************************** 8
32 | ************************* 4
64 | ************************************* 6
128 | ****** 1
256 Continue readingA container identity bootstrapping tool
Everybody has secrets. Software developers have many. Often these secrets—API tokens, TLS private keys, database passwords, SSH keys, and other sensitive data—are needed to make a service run properly and interact securely with other services. Today we’re sharing a tool that we built at Cloudflare to securely distribute secrets to our Dockerized production applications: PAL.
PAL is available on Github: https://github.com/cloudflare/pal.
Although PAL is not currently under active development, we have found it a useful tool and we think the community will benefit from its source being available. We believe that it's better to open source this tool and allow others to use the code than leave it hidden from view and unmaintained.
Secrets in production
CC BY 2.0 image by Personal Creations
How do you get these secrets to your services? If you’re the only developer, or one of a few on a project, you might put the secrets with your source code in your version control system. But if you just store the secrets in plain text with your code, everyone with access to your source repository can read them and use them for nefarious purposes (for example, stealing an API token and pretending to be Continue reading
Stupidly Simple DDoS Protocol (SSDP) generates 100 Gbps DDoS
Last month we shared statistics on some popular reflection attacks. Back then the average SSDP attack size was ~12 Gbps and largest SSDP reflection we recorded was:
- 30 Mpps (millions of packets per second)
- 80 Gbps (billions of bits per second)
- using 940k reflector IPs
This changed a couple of days ago when we noticed an unusually large SSDP amplification. It's worth deeper investigation since it crossed the symbolic threshold of 100 Gbps.
The packets per second chart during the attack looked like this:
The bandwidth usage:
This packet flood lasted 38 minutes. According to our sampled netflow data it utilized 930k reflector servers. We estimate that the during 38 minutes of the attack each reflector sent 112k packets to Cloudflare.
The reflector servers are across the globe, with a large presence in Argentina, Russia and China. Here are the unique IPs per country:
$ cat ips-nf-ct.txt|uniq|cut -f 2|sort|uniq -c|sort -nr|head
439126 CN
135783 RU
74825 AR
51222 US
41353 TW
32850 CA
19558 MY
18962 CO
14234 BR
10824 KR
10334 UA
9103 IT
...
The reflector IP distribution across ASNs is typical. It pretty much follows the world’s largest residential ISPs:
$ cat ips-nf-asn.txt |uniq|cut -f 2|sort|uniq Continue readingAnnouncing the Cloudflare Apps Platform and Developer Fund
When we started Cloudflare we had no idea if anyone would validate our core idea. Our idea was what that everyone should have the ability to be as fast and secure as the Internet giants like Google, Facebook, and Microsoft. Six years later, it's incredible how far that core idea has taken us.
CC BY-SA 2.0 image by Mobilus In Mobili
Today, Cloudflare runs one of the largest global networks. We have data centers in 115 cities around the world and continue to expand. We've built a core service that delivers performance, security, availability, and insight to more than 6 million users.
Democratizing the Internet
From the beginning, our goal has been to democratize the Internet. Today we're taking another step toward that goal with the launch of the Cloudflare Apps Platform and the Cloudflare Developer Fund. To understand that, you have to understand where we started.
When we started Cloudflare we needed two things: a collection of users for the service, and finances to help us fund our development. In both cases, people were taking a risk on Cloudflare. Our first users came from Project Honey Pot, which Lee Holloway and I created back in 2004. Members Continue reading
Announcing the New Cloudflare Apps
Today we’re excited to announce the next generation of Cloudflare Apps. Cloudflare Apps is an open platform of tools to build a high quality website. It’s a place where every website owner can select from a vast catalog of Apps which can improve their websites and internet properties in every way imaginable. Selected apps can be previewed and installed instantly with just a few clicks, giving every website owner the power of technical expertise, and every developer the platform only Cloudflare can provide.
Apps can modify content and layout on the page they’re installed on, communicate with external services and dramatically improve websites. Imagine Google Analytics, YouTube videos, in-page chat tools, widgets, themes and every other business which can be built by improving websites. All of these and more can be done with Cloudflare Apps.
Cloudflare Apps makes it possible for a developer in her basement to build the next great new tool and get it on a million websites overnight. With Cloudflare Apps, even the smallest teams can get massive distribution for their apps on the web so that the best products win. With your help we will make it possible for developers like you to build a new Continue reading
Project Jengo: Explaining Challenges to Patent Validity (and a looming threat)
We’ve written a couple times about the problem of patent trolls, and what we are doing in response to the first case a troll filed against Cloudflare. We set a goal to find prior art on all 38 Blackbird Tech patents and applications and then obtain a legal determination that Blackbird Tech’s patents are invalid. Such a determination will end Blackbird’s ability to file or threaten to file abusive patent claims, against us or anyone else.
The patent system exists to reward inventors, so it is no surprise that a patent has to claim something new — an “invention.” Sometimes the United States Patent and Trademark Office (USPTO) — the agency that administers the patent system — mistakenly issues patents that do not claim anything particularly new. The patent examiner may not be aware that the proposed “invention” was already in use in the industry, and the patent applicant (the only party in the process) doesn’t have an incentive to share that information. Often, the USPTO issues patents that are too vague and can later be broadly interpreted by patent owners to cover different and subsequent technologies that could not otherwise Continue reading
When the Internet (Officially) Became the Public Square
Sometimes, well-intended efforts to prevent unacceptable behavior run into the reality of what it means to have an open and free society. That is what happened at the Supreme Court on Monday.
The Supreme Court issued an opinion confirming something we at Cloudflare have long believed -- that the First Amendment protects access to the Internet. Using sweeping language, Justice Kennedy compared internet access to access to a street or park, "essential venues for public gatherings to celebrate some views, to protest others, or simply to learn and inquire,” and concluded that "to foreclose access to social media altogether is to prevent the user from engaging in the legitimate exercise of First Amendment rights."
We share this view of the internet as a forum to discuss and debate ideas, and believe that the Court’s opinion is an important reaffirmation of the free speech principles we support.
The Packingham Case
Like many other First Amendment cases, the law at the heart of the Packingham v. North Carolina case presents complex questions about how to protect the community in ways consistent with the right to free speech.
In 2008, North Carolina passed a law making it a Continue reading
Counting things, a lot of different things…
Back in April we announced Rate Limiting of requests for every Cloudflare customer. Being able to rate limit at the edge of the network has many advantages: it’s easier for customers to set up and operate, their origin servers are not bothered by excessive traffic or layer 7 attacks, the performance and memory cost of rate limiting is offloaded to the edge, and more.
In a nutshell, rate limiting works like this:
Customers can define one or more rate limit rules that match particular HTTP requests (failed login attempts, expensive API calls, etc.)
Every request that matches the rule is counted per client IP address
Once that counter exceeds a threshold, further requests are not allowed to reach the origin server and an error page is returned to the client instead
This is a simple yet effective protection against brute force attacks on login pages and other sorts of abusive traffic like L7 DoS attacks.
Doing this with possibly millions of domains and even more millions of rules immediately becomes a bit more complicated. This article is a look at how we implemented a rate limiter able to run quickly and accurately at the edge of the network which Continue reading
Less Is More – Why The IPv6 Switch Is Missing
At Cloudflare we believe in being good to the Internet and good to our customers. By moving on from the legacy world of IPv4-only to the modern-day world where IPv4 and IPv6 are treated equally, we believe we are doing exactly that.
"No matter what happens in life, be good to people. Being good to people is a wonderful legacy to leave behind." - Taylor Swift (whose website has been IPv6 enabled for many many years)
Starting today with free domains, IPv6 is no longer something you can toggle on and off, it’s always just on.
How we got here
Cloudflare has always been a gateway for visitors on IPv6 connections to access sites and applications hosted on legacy IPv4-only infrastructure. Connections to Cloudflare are terminated on either IP version and then proxied to the backend over whichever IP version the backend infrastructure can accept.
That means that a v6-only mobile phone (looking at you, T-Mobile users) can establish a clean path to any site or mobile app behind Cloudflare instead of doing an expensive 464XLAT protocol translation as part of the connection (shaving milliseconds and conserving very precious battery life).
That IPv6 gateway is set by a simple Continue reading
Patent Troll Battle Update: Doubling Down on Project Jengo
Jengo Fett by Brickset (Flickr)
We knew the case against patent trolls was the right one, but we have been overwhelmed by the response to our blog posts on patent trolls and our program for finding prior art on the patents held by Blackbird Tech, which we’ve dubbed Project Jengo. As we discuss in this post, your comments and contributions have allowed us to expand and intensify our efforts to challenge the growing threat that patent trolls pose to innovative tech companies.
We’re SIGNIFICANTLY expanding our program to find prior art on the Blackbird Tech patents
In a little over a week since we started the program, we’ve received 141 separate prior art submissions. But we know there’s an opportunity to find a lot more.
We’ve been impressed with the exceptionally high quality of the submissions. The Cloudflare community of users and readers of our blog are an accomplished bunch, so we have a number of searches that were done by expert engineers and programmers. In one case that stood out to us, someone wrote in about a project they personally had worked on as an engineer back in 1993, which they are convinced is conclusive prior art Continue reading
Reflections on reflection (attacks)
Recently Akamai published an article about CLDAP reflection attacks. This got us thinking. We saw attacks from Conectionless LDAP servers back in November 2016 but totally ignored them because our systems were automatically dropping the attack traffic without any impact.
We decided to take a second look through our logs and share some statistics about reflection attacks we see regularly. In this blog post, I'll describe popular reflection attacks, explain how to defend against them and why Cloudflare and our customers are immune to most of them.
A recipe for reflection
Let's start with a brief reminder on how reflection attacks (often called "amplification attacks") work.
To bake a reflection attack, the villain needs four ingredients:
- A server capable of performing IP address spoofing.
- A protocol vulnerable to reflection/amplification. Any badly designed UDP-based request-response protocol will do.
- A list of "reflectors": servers that support the vulnerable protocol.
- A victim IP address.
The general idea:
- The villain sends fake UDP requests.
- The source IP address in these packets is spoofed: the attacker sticks the victim's IP address in the source IP address field, not their own IP address as they normally would.
- Each packet Continue reading
Introducing Argo — A faster, more reliable, more secure Internet for everyone
The Internet is inherently unreliable, a collection of networks connected to each other with fiber optics, copper, microwaves and trust. It’s a magical thing, but things on the Internet break all the time; cables get cut, bogus routes get advertised, routers crash. Most of the time, these failures are noticed but inexplicable to the average user — ”The Internet is slow today!” — frustrating user experiences as people go about their lives on the Internet.
Today, to fix all of this, Cloudflare is launching Argo, a “virtual backbone” for the modern Internet. Argo analyzes and optimizes routing decisions across the global Internet in real-time. Think Waze, the automobile route optimization app, but for Internet traffic.
Just as Waze can tell you which route to take when driving by monitoring which roads are congested or blocked, Argo can route connections across the Internet efficiently by avoiding packet loss, congestion, and outages.
Cloudflare’s Argo is able to deliver content across our network with dramatically reduced latency, increased reliability, heightened encryption, and reduced cost vs. an equivalent path across the open Internet. The results are impressive: an average 35% decrease in latency, a 27% decrease in connection errors, and a 60% Continue reading
Introducing Load Balancing & Intelligent Failover with Cloudflare
Cloudflare's Enterprise customers have been using our Load Balancing service since March, and it has been helping them avoid website downtime caused by unreliable hosting providers, Internet outages, or servers. Today, we're bringing Load Balancing to all of our customers.
Even the best caching can't escape the fundamental limitations on performance created by the speed of light. Using Load Balancing, Cloudflare's customers can now route requests between multiple origins, allowing them to serve requests from the closest (and fastest) geographic location.
The Cloudflare Load Balancer automatically sends you notifications when things fail, and when they come back up again, so you can sleep well at night knowing we are keeping your website or API running.
If a DDoS attack can bring down your DNS provider or load balancer, it doesn't matter whether your servers are healthy or not. Our load balancing service runs in Cloudflare's 110+ datacenters, and with experience dealing with some of the largest DDoS attacks, we can withstand traffic volumes that smaller providers, virtual machines or hardware appliances can't. This also allows us to help you avoid business-impacting downtime when major cloud compute providers have issues: when we identify a connectivity reaching your application on AWS, we Continue reading
Detroit and San Diego Data Centers expand Cloudflare network to 26 North American cities
Cloudflare is excited to announce deployments in Detroit (Michigan) and San Diego (California), which are our 114th and 115th data centers respectively. They join Colombo, Sri Lanka and Cape Town, South Africa in the cohort of four new cities added just this week to our growing global network, which spans 57 countries and counting.
For over 6 million Internet properties, we now serve customer traffic from across 26 North American cities, including 22 in the United States alone. We're not going to stop building our network until we're within milliseconds of every Internet user, and to that end, data centers are already in the works in eight additional North American cities (and many others around the world).
Connections
Source: Baja Insider
Detroit and San Diego share something special, as they are immediately adjacent to international borders with Canada and Mexico respectively. Detroit has four border crossings to Windsor, Ontario, including the Ambassador Bridge, which was built in the Roaring Twenties, and accommodates over a quarter of all merchandise trade with Canada.
Founded in 1701, and best known for cars and Motown, Detroit eagerly awaits a 3,000 pound bronze RoboCop statue to watch over Delta City (track progress here). Continue reading
Standing Up to a Dangerous New Breed of Patent Troll
On March 20th, Cloudflare received our first patent infringement claim: Blackbird Tech LLC v. Cloudflare, Inc. Today we’re filing our Answer to that claim in a federal court in Delaware. We have very strong arguments we will present in the litigation, mostly because the patent asserted against us does not have anything to do with our technology.
The infringement claim is not a close one. The asserted patent, US 6453335 (‘335 patent) was filed in 1998, and describes a system for monitoring an existing data channel and inserting error pages when transmission rates fall below a certain level. Nothing from 1998—including the ’335 patent—comes close to describing our state-of-the-art product that is provisioned via DNS, speeds up internet content delivery, and protects against malicious attackers. Our technology is innovative and different, and Cloudflare’s technology has about 150 patents issued or in process.
We also expect to show that the patent itself is invalid. For example, if the ’335 patent is read broadly enough to cover our system (which shouldn’t happen), it would also cover any system where electronic communications are examined and redacted or modified. But this is not new. Filtering products performing similar functions were around long before Continue reading
Project Jengo: Cloudflare’s Prior Art Search Bounty
Jengo Fett by Brickset (Flickr)
As readers of this blog likely know, especially if you read this post, Cloudflare has been sued by a dangerous new breed of patent troll, Blackbird Technologies, asserting a very old and very vague patent. And we know we are not alone in being frustrated about the way that such patent trolls inhibit the growth of innovative companies. Cloudflare is asking for your help in this effort, and we’re putting our money where our mouth is.
Patent trolls take advantage of a system they assume is tilted in their favor, where they can take vague technology patents issued years ago and apply them as broadly as imaginable to the latest technology. And they do this without the limitations of having to show the original patent holder would have actually exercised the patent, because most of them don’t, at all. Patent trolls think they can sit back and pick off settlements from companies because their lawsuits are a nuisance and the costs of defending those suits are considerable.
Changing this dynamic and leveling the playing field is going to require an entirely new approach. Fighting such strong, though perverse, economic incentives is going Continue reading
Cape Town (South Africa): Cloudflare Data Center #113
Five fun facts:
- Cape Town is where the Atlantic Ocean and the Indian Ocean meet deep in the Southern Hemisphere.
- The city is the start of the Garden Route, a 185 mile (300 km) glorious coastal drive brimming with native flowers and stunning vistas.
- Cape Town is the gateway into the Stellenbosch and Franschhoek wine districts that make for very popular weekend excursions.
- The imposing Table Mountain can be seen from most of Cape Town. When you take the cable car to the top of the mountain, the view from up there is even more stunning.
- Cape Town is where Cloudflare has placed its 113th data center.
Second data center in South Africa
Back in December 2014, Cloudflare opened our first data center in Africa and our 30th datacenter globally. That was in Johannesburg, which has since seen over 10x growth in traffic delivered to South Africa and surrounding countries.
Now, we are expanding into our second city in South Africa — Cape Town, bringing us 870 miles (1,400km) closer to millions of Internet users. Only 15% smaller than Johannesburg by population, Cape Town commands a majority of the tourism business for the country.
For Cloudflare, our newest Continue reading