Mobile Ad Networks as DDoS Vectors: A Case Study
CloudFlare servers are constantly being targeted by DDoS'es. We see everything from attempted DNS reflection attacks to L7 HTTP floods involving large botnets.
Recently an unusual flood caught our attention. A site reliability engineer on call noticed a large number of HTTP requests being issued against one of our customers.
The request
Here is one of the requests:
POST /js/404.js HTTP/1.1
Host: www.victim.com
Connection: keep-alive
Content-Length: 426
Origin: http://attacksite.com
User-Agent: Mozilla/5.0 (Linux; U; Android 4.4.4; zh-cn; MI 4LTE Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/42.0.0.0 Mobile Safari/537.36 XiaoMi/MiuiBrowser/2.1.1
Content-Type: application/x-www-form-urlencoded
Accept: */*
Referer: http://attacksite.com/html/part/86.html
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,en-US;q=0.8
id=datadatadasssssssssssssssssssssssssssssssssssssssssssassssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssadatadata
We received millions of similar requests, clearly suggesting a flood. Let's take a deeper look at this request.
First, let's note that the headers look legitimate. We often see floods issued by Python or Ruby scripts, with weird Accept-Language or User-Agent headers. But this one doesn't look like it. This request is a proper request issued by a real browser.
Next, notice the request is a POST and contains an Origin header — it was issued by an Ajax (XHR) cross Continue reading
iOS 9 — How did the launch really go?
On September 16 2015 at 10:00AM PST, Apple released their latest update to the iPhone: iOS 9. For several days after the announcement, ISPs and customers reported problems downloading iOS 9 due to overloaded servers. Obviously, the demand for iOS 9 was higher than even Apple anticipated, but how much higher? Few organizations outside of Apple have any idea what the actual iOS 9 adoption rates look like.
By analyzing the user agent strings of requests passing through the CloudFlare network, we were able to piece together a pretty good picture of iOS 9 uptake. Here’s an hour-by-hour look at requests from iOS 8 devices (blue) and iOS 9 devices (orange) for the first 24 hours after the announcement.
We started seeing small amounts of iOS 9 usage before it was officially released, followed by a spike immediately after the launch (times are shown in UTC, so the 10:00AM announcement shows up as hour 18). You can also see a second spike at 10:00 UTC when Europe started waking up.
What about iOS 9.1?
Even though the official release was for iOS 9.0, we also found beta iOS 9.1 in the wild. Curious about the comparative traffic Continue reading
Why we raised $110m from Fidelity, Google, Microsoft, Baidu and Qualcomm
The past few years have been marked by tremendous growth for CloudFlare. At the time of our last fundraising in December 2012, CloudFlare was a team of 37 operating a network in 23 cities and 15 countries—today we number over 200 with a presence in 62 cities and 33 countries. We’ve grown from delivering 85 billion page views per month for 500 thousand customers to nearly 1 trillion each month across 4 million Internet properties, all the while protecting our customers from hundreds of billions of cyber threats. The growth and resonance of our service since CloudFlare’s founding 5 years ago is beyond our wildest of expectations, but it is only in the coming years that our scale and efforts to build a better Internet will become visible.
In 2016 alone we will more than double our global presence, increase the size of our network by an order of magnitude, and with that allow millions of new businesses and online publishers to accelerate and secure their online applications and harness the growing power of the Internet economy. Our service is built on the simple premise that any individual or business should be able to quickly and easily ensure the global Continue reading
Announcing CloudFlare’s Internet Summit – And How to Get an Invitation
Five years ago next week, CloudFlare launched its service to the public. We’re celebrating our birthday in a variety of ways, including holding our first-ever Internet Summit on Thursday, September 24th. As part of the Internet Summit, we’re bringing together policymakers, business executives, cybersecurity experts, and academics from all over the world to discuss the threats and opportunities for the Internet over the next five years.
Through a series of fireside chats and panel discussions, featured speakers will discuss the top technology trends shaping business today, including cyber security, mobility, and the Internet of Things. These compelling perspectives will offer insights into the future of the global Internet and its implications on society.
Featured speakers include:
- Toomas Hendrik Ilves, The President of Estonia
- Alex Stamos, chief security officer of Facebook
- Matt Grob, chief technology officer of Qualcomm
- Andrew Ng, chief scientist at Baidu
- Nicole Wong, former US deputy CTO & legal director for products at Twitter
- Andy McAfee, MIT professor & author of "The Second Machine Age"
- Cindy Cohn, executive director of the Electronic Frontier Foundation
- Adam Langley, security engineer at Google
- David Brin, scientist, best-selling author & tech-futurist
- Rajiv Pant, former chief technology officer of The New York Times
- Continue reading
CloudFlare + WHMCS: Faster Websites For Your Customers
We’re at the cPanel Conference in Denver this week, so feel free to drop by our booth and say hello. It’s a great opportunity to connect with our partners and better understand their needs. We’re always trying to streamline our partners’ user experience, and we thought it would be a fitting time to walk through our recently updated WHMCS integration.
CloudFlare’s WHMCS 6.0 plugin lets hosting providers and registrars extend all the benefits of CloudFlare directly to their customers. You can offer your entire user base a global CDN with 62 points of presence, automatic web content optimization, basic DDoS protection, reputation-based threat protection, and much more with virtually no extra work.
These benefits are seamlessly integrated into your WHMCS client. All your customers need to do is click a button, and a new CloudFlare account will be configured for them.
While signing up for an account on www.cloudflare.com only takes a few minutes, users do need to point the relevant DNS records to CloudFlare’s nameservers. Offerring a one-click solution via our WHMCS integration is a great opportunity for hosting providers and registrars to streamline the process for their customers.
Universal SSL with WHMCS
CloudFlare’s Universal Continue reading
How We Extended CloudFlare’s Performance and Security Into Mainland China
CloudFlare launched five years ago. Within a year of our launch, the biggest surprise was the strong global demand for our service. From nearly the beginning, China was the second largest source of traffic by country to our network, behind only the United States.
In retrospect, that shouldn't have been a surprise. In 2010, the year we launched, 34% of China's population, or 450 million people, were online. Today, nearly half the country is online. To put it another way, with 700 million people online, China represents a quarter of all Internet users. If your mission is to help build a better Internet, like CloudFlare's is, then China is a country you cannot ignore.
Consequently, starting in 2011, we began to investigate how CloudFlare could bring our service to the Chinese Internet. Four years later, we're excited to announce the extension of CloudFlare's performance and security platform across mainland China. This is the story of how we did it.
The Challenges
There are three major challenges to extending a service like CloudFlare's across mainland China: technical, economic, and regulatory.
Technical
From a technical perspective, the Chinese Internet, despite its many similarities, is different than the rest of the world. Unlike Continue reading
Kuala Lumpur, Malaysia: CloudFlare’s 45th data center
Kuala Lumpur joins the CloudFlare network as our 45th global point of presence (PoP). While this latest deployment comes only a day after the announcement of our expansion in Berlin (back-to-back!), it's been a long three years since we last crossed a new border in Asia. Kuala Lumpur expands our presence in the Asia-Pacific region to nine PoPs: Kuala Lumpur, Seoul, Tokyo, Osaka, Hong Kong, Singapore, Sydney, Melbourne and Auckland.
No boomerangs allowed
One of the difficulties of delivering content locally in certain parts of Asia (and in many other parts of the world for that matter) is that certain ISPs only connect with other ISPs in the same local Internet ecosystem outside of their national borders. In the absence of domestic interconnection, a request (e.g. an e-mail or web request) from one local ISP to another "boomerangs" outside of the national border before it is ultimately delivered to another local ISP. If you live or travel in certain parts of Asia, this is one of the leading culprits for why the web feels slow. The lack of a domestic and central interconnection point also makes it challenging for networks like CloudFlare, both Continue reading
Berlin, Germany: CloudFlare’s 44th data center
Our data center in Berlin is our 3rd in Germany following Frankfurt and Düsseldorf, 14th in Europe, and 44th globally. Berlin is of considerable importance not just because it is the capital of Europe's most populous country, but also because it is the 2nd largest city in the European Union by population* trailing only London. As of this moment, CloudFlare has a point of presence (PoP) in 7 out of Europe's 10 most populous cities, and we're headed for a perfect 10-for-10.
Ich bin ein Berliner
"I am one with the people of Berlin," best expresses our sentiments following this latest launch, but is more famously a reference to U.S. President John F. Kennedy's June 26th, 1963 speech in West Berlin (and also the source of an amusing urban legend). The story goes that Kennedy should have said "Ich bin Berliner" ("I am a citizen of Berlin"), but instead remarked "Ich bin ein Berliner" which translates as "I am a jelly doughnut."
The Berliner: we treated ourselves to one a few in celebration of the launch
As it turns out, and despite decades of misinformation, Kennedy was linguistically correct. While in proper German an actual Berliner Continue reading
Weird bug of the day: Twitter in-app browser can’t visit site
We keep a close eye on tweets that mention CloudFlare because sometimes we get early warning about odd errors that we are not seeing ourselves through our monitoring systems.
Towards the end of August we saw a small number of tweets like this one:
indicating that trying to browse to a CloudFlare customer web site using the Twitter in-app browser was resulting in an error page. Which was very odd because it was clearly only happening occasionally: very occasionally.
Luckily, the person who tweeted that was in the same timezone as me and able to help debug together (thanks James White!); we discovered that the following sequence of events was necessary to reproduce the bug:
Click on a link in a tweet to a web site that is using an https URL and open in the Twitter in-app browser (not mobile Safari). This site may or may not be a CloudFlare customer.
Then click on a link on that page to a site over an http URL. This site must be on CloudFlare.
BOOM
That explained why this happened very rarely, but the question became... why did it happen at all? After some debugging it appeared to happen in Continue reading
Kernel bypass
In two previous posts we've discussed how to receive 1M UDP packets per second and how to reduce the round trip time. We did the experiments on Linux and the performance was very good considering it's a general purpose operating system.
Unfortunately the speed of vanilla Linux kernel networking is not sufficient for more specialized workloads. For example, here at CloudFlare, we are constantly dealing with large packet floods. Vanilla Linux can do only about 1M pps. This is not enough in our environment, especially since the network cards are capable of handling a much higher throughput. Modern 10Gbps NIC's can usually process at least 10M pps.
CC BY 2.0 image by Tony Webster
It's apparent that the only way to squeeze more packets from our hardware is by working around the Linux kernel networking stack. This is called a "kernel bypass" and in this article we'll dig into various ways of achieving it.
The kernel is insufficient
Let's prepare a small experiment to convince you that working around Linux is indeed necessary. Let's see how many packets can be handled by the kernel under perfect conditions. Passing packets to userspace is costly, so instead let's try to drop Continue reading
Test all the things: IPv6, HTTP/2, SHA-2
CloudFlare constantly tries to stay on the leading edge of Internet technologies so that our customers' web sites use the latest, fastest, most secure protocols. For example, in the past we've enabled IPv6 and SPDY/3.1.
Today we've switched on a test server that is open for people to test compatibility of web clients. It's a mirror of this blog and is served from https://http2.cloudflare.com/. The server uses three technologies that it may be helpful to test with: IPv4/IPv6, HTTP/2 and an SSL certificate that uses SHA-2 for its signature.
The server has both IPv4 and IPv6 addresses.
$ dig +short http2.cloudflare.com A
45.55.83.207
$ dig +short http2.cloudflare.com AAAA
2604:a880:800:10:5ca1:ab1e:f4:e001
The certificate is based on SHA-2 (in this case SHA-256). This is important because SHA-1 is being deprecated by some browsers very soon. On a recent browser the connection will also be secured using ECDHE (for forward secrecy).
And, finally, the server uses HTTP/2 if the browser is capable. For example, in Google Chrome, with the HTTP/2 and SPDY indicator extension the blue lightning bolt indicates that the page was served using HTTP/2:
This server isn't on the normal CloudFlare Continue reading
Simple Helix chooses CloudFlare to ignite white-hot Magento performance
Today’s guest blogger is George Cagle. George is a system administrator at Simple Helix, a CloudFlare partner.
Some months ago, we made a big bet on partnering with CloudFlare for performance improvements and website security for our Magento hosting customers. Customer experience is core to our business and relying on another company is a major deal. CloudFlare is now included in Default–On mode for select Simple Helix hosting plans and can be added to any existing plan. The results have been great and we wanted to share a couple successes with the rest of the CloudFlare community.
Testing the waters
The first thing one notices after melding their site with the worldwide CloudFlare CDN network is just how fast a website becomes. In Simple Helix’s testing, we found that proper CloudFlare implementation can yield 100% speed increases, and an even faster 143% speed increase when paired with the Railgun™ web optimizer for dynamic content.
Adding CloudFlare will certainly improve performance, but it can also significantly improve security through the Web Application Firewall feature. The security benefits of having the CloudFlare service can be seen after just the first few days of adoption as outlined below:
Total number of threats mitigated Continue reading
Railgun v5 has landed: better, faster, lighter
Three years ago we launched Railgun, CloudFlare's origin network optimizer. Railgun allows us to cache the uncacheable to accelerate the connection between CloudFlare and our customers' origin servers. That brings the benefit of a CDN to even dynamic content with no need for 'fast purging' or other tricks. With Railgun even dynamic, ever-changing pages benefit from caching.
CC BY 2.0 image by Nathan E Photography
Over those three years Railgun has been deployed widely by our customers to accelerate the delivery of their web sites and lower their bandwidth costs.
Today we're announcing the availability of Railgun v5 with a number of significant improvements:
We've substantially reduced memory utilization and CPU requirements
Railgun performs delta compression on every request/response requiring CPU (to perform the compression) and memory (to keep a cache of pages to delta against). Version 5 has undergone extensive optimization based on the performance of Railgun on large web sites and at hosting providers. Version 5 requires much less memory and lower CPU.
A new, lighter weight, faster wire protocol
The original Railgun wire protocol that transfer requests and compressed responses between the customer server and CloudFlare's infrastructure has been completely replaced with a new, lighter-weight Continue reading
SXSW Interactive 2016: Vote for CloudFlare’s Submissions
It’s that time of year again, when the end of summer is in sight, students are back in school, football is on TV again, and your social feeds are flooded with “vote for my panel at SXSW” updates. While it feels like our team was just at SXSW, it’s already time to start planning for SXSW ‘16. If these topics interest you, please take a minute to vote for them!
How to vote:
- Create an account here
- Enter your name & email address, then confirm your account
- Log-in with your new account and go to the “PanelPicker”
- Click “search/vote” and search for presentations by title
VOTE!!
*Voting ends on Friday, September 4th!
Just like last year, PanelPicker voting counts for 30% of a panel/presentation’s acceptance to SXSW. Check out the previews of our sessions below. Every vote counts!
CloudFlare's SXSW 2016 Submissions:
1) They’re Coming for our Internet: We can fight back
Join Matthew Prince, CloudFlare’s co-founder and CEO, for a presentation focused on Internet censorship and global security issues. Matthew will share how online censorship varies globally, and how tech giants should collaborate to expand the Internet’s reach, not divide it. He will also cover what your own personal rights are as an online user, and how you can better preserve them. If you're reading this blog post, this is a presentation you won’t want to miss!
Speaker:
Matthew Prince, CloudFlare
2) Innovating Like the “Early Days” 5+ years Later
Innovating is easy in the early days--especially without the legacy systems, prior customer commitments, or formal internal processes that come with time. Fast forward and you have more employees, customers, commitments, internal silos, and business goals than ever before. How do you maintain the agile innovation pace you had early on? This panel of builders and visionaries will share how they stay laser focused on what’s over the horizon, avoiding incrementalism. They’ll share how they keep their teams paving the way for others to follow.
Speakers:
Dane Knecht, CloudFlare
Charise Flynn, Dwolla
Marc Boroditsky, Twilio
3) PR for Startups: Low to No Budget Tips for Today
Learn how to drive PR for your startup--no matter how big/small you are or what your current role is. Join a former tech journalist and PR leaders from growth-stage and unicorn startups--across the enterprise, on-demand, and consumer technology industries--for a candid discussion on navigating the media landscape. Walk away with tips and tools (even free ones!) to drive awareness and take your company to the next level.
Speakers:
Daniella Vallurupalli, CloudFlare
Johnny Brackett, Shyp
Michelle Masek, Imgur
Ryan Lawler, 500 Startups
Please vote and help CloudFlare get to SXSW Interactive 2016! I can already taste the BBQ...
Mombasa, Kenya: CloudFlare’s 43rd data center
Only two weeks after the announcement of our four new points of presence (PoPs) in the Middle East, it is with much hullabaloo that we announce our 43rd PoP, and second in Africa following Johannesburg, in Mombasa, Kenya (a.k.a. “The Castle”). In a challenge that vexed many of our readers, Mombasa is our first PoP to be located in a real life castle-turned-data center (see above). From this castle CloudFlare is already serving networks in every country across East Africa, with reach to many of the region's 30 million+ Internet users.
Building a better Internet in Eastern Africa
While today it feels as if Internet access is ubiquitous, this is most certainly not the case everywhere. The continent of Africa was connected relatively late to the Internet and, in the first years, access was limited to a small segment of the population due to lackluster investment and growth in underlying Infrastructure, and high access costs. Most Africans were also without access to broadband Internet, and were largely limited to viewing content created and hosted half a world away—for the same reason there was little access, there was also no local hosting industry to speak of. By Continue reading
Introducing a Powerful Way to Purge Cache on CloudFlare: Purge by Cache-Tag
Today CloudFlare is introducing a new way to purge the cache using Cache-Tags. Cache-Tags are assigned to cached content via a Cache-Tag response header, and are stored as metadata with cached objects so that global purges take only seconds, targeting a granular, finite set of cached objects.
For example, an e-commerce website can use Cache-Tags to purge all of their catalog images at once, without affecting any of their other assets. A blog can use Cache-Tags to update their JavaScript files in cache, without forcing a cache miss on their CSS. A business can use Cache-Tags to purge cache of all four hundred pages of their blog without purging any of the pages from their core platform.
The CloudFlare Cache
With 42 data centers around the world, web pages served directly from CloudFlare’s cache are guaranteed to be just a few hops away from any visitor, anywhere. With a little bit of fine tuning, many websites succeed in delivering most of their content from cache, saving a majority of bandwidth on their origin servers. One website even managed to reduce their AWS bill by 96% when they started caching assets behind CloudFlare.
CloudFlare’s cache is powerful, but when a Continue reading
Ensuring the web is for everyone
This is the text of an internal email I sent at CloudFlare that we thought worth sharing more widely. I annotated it a bit with links that weren't in the original.
"Tim Berners-Lee- Mosaic by Sue Edkins at Sheen Lane Centre" by Robert Smith - Own work. Licensed under CC BY-SA 4.0 via Commons
Subject: Days of future past
Folks,
One of the exciting things about working at CloudFlare is our continual push to stay on top of what's new for our customers. We've pushed things like IPv6 and SPDY in the past; and we'll soon be giving the world DNSSEC and HTTP/2. In the world of SSL we've stayed on top of changes in recommended cipher suites and offer the latest signature algorithms SHA-2 to our customers.
But as we do this we must not forget the old protocols. Because we serve a truly global audience we serve everyone on the planet. It's easy inside a Silicon Valley bubble to think that everyone is on 1Gbps Internet connection with the latest version of Chrome on a new Mac, but the worldwide reality is far different.
We see every type of machine and browser out there. And Continue reading
Dear Internet, Send Us Your Videos
CloudFlare turns 5 years old this September. It's been an amazing ride since our launch. Before we launched at TechCrunch Disrupt on September 27, 2010, we'd signed up about 1,000 beta customers. It took us nine months to get those first customers. (By comparison, today we typically sign up 1,000 customers every 3 hours.)
Those first beta customers were instrumental. They put up with us when we were had only one data center (in Chicago). They put up with us as we brought traffic online in our next facilities in Ashburn, Virginia and San Jose, California — and had the routing challenges that came along with running a distributed network for the first time. They sent us bug reports, provided us feature requests, and were instrumental to building the foundation that grew into what is CloudFlare today.
Archival Footage
When we launched, we wanted to feature their stories and experience about CloudFlare so we had them submit their stories by video. Here's the video we included as part of our launch presentation.
I'm proud of the fact that more than 80% of those original 1,000 customers are still using CloudFlare five years later.
Send Us Your Stories
As we Continue reading
Now serving the Middle East: 4 new data centers, partnerships
Our last embarkation into a new geography coincided with a significant milestone: our 30th data center (and first in Africa) in Johannesburg, South Africa. And as we march past number 40, we’re proud to announce yet another. Introducing CloudFlare’s latest points of presence (PoPs) in Doha, Qatar; Dubai, United Arab Emirates; Kuwait City, Kuwait; and Muscat, Oman. These data centers are the first wave in our MENA (Middle East/North Africa) expansion, and the 39th, 40th, 41st and 42nd data centers, respectively, to join our global network.
Up to this point all CloudFlare traffic delivered to the MENA region was served from our London, Frankfurt, Marseille, Paris and/or Singapore data centers, with round trip latency of up to 200-350ms. As in Africa, local bandwidth in MENA is notoriously expensive making it cost prohibitive to deliver content locally. That is (once again), until now! We're proud to announce the first of a series of agreements with regional carriers including Etisalat, Omantel, Ooredoo, and Zain to help build a better Internet in the region.
How to build a better Internet
A few of the necessary ingredients to build a better Internet include international connectivity (often in Continue reading
DNS parser, meet Go fuzzer
Here at CloudFlare we are heavy users of the github.com/miekg/dns Go DNS library and we make sure to contribute to its development as much as possible. Therefore when Dmitry Vyukov published go-fuzz and started to uncover tens of bugs in the Go standard library, our task was clear.
Hot Fuzz
Fuzzing is the technique of testing software by continuously feeding it inputs that are automatically mutated. For C/C++, the wildly successful afl-fuzz tool by Michał Zalewski uses instrumented source coverage to judge which mutations pushed the program into new paths, eventually hitting many rarely-tested branches.
go-fuzz applies the same technique to Go programs, instrumenting the source by rewriting it (like godebug does). An interesting difference between afl-fuzz and go-fuzz is that the former normally operates on file inputs to unmodified programs, while the latter asks you to write a Go function and passes inputs to that. The former usually forks a new process for each input, the latter keeps calling the function without restarting often.
There is no strong technical reason for this difference (and indeed afl recently gained the ability to behave like go-fuzz), but it's likely due to the different ecosystems in which they Continue reading