Inside the infamous Mirai IoT Botnet: A Retrospective Analysis
This is a guest post by Elie Bursztein who writes about security and anti-abuse research. It was first published on his blog and has been lightly edited.
This post provides a retrospective analysis of Mirai — the infamous Internet-of-Things botnet that took down major websites via massive distributed denial-of-service using hundreds of thousands of compromised Internet-Of-Things devices. This research was conducted by a team of researchers from Cloudflare, Georgia Tech, Google, Akamai, the University of Illinois, the University of Michigan, and Merit Network and resulted in a paper published at USENIX Security 2017.
At its peak in September 2016, Mirai temporarily crippled several high-profile services such as OVH, Dyn, and Krebs on Security via massive distributed Denial of service attacks (DDoS). OVH reported that these attacks exceeded 1 Tbps—the largest on public record.
What’s remarkable about these record-breaking attacks is they were carried out via small, innocuous Internet-of-Things (IoT) devices like home routers, air-quality monitors, and personal surveillance cameras. At its peak, Mirai infected over 600,000 vulnerable IoT devices, according to our measurements.
This blog post follows the timeline above
- Mirai Genesis: Discusses Mirai’s early days and provides a brief technical overview of how Mirai works and Continue reading
Inside the infamous Mirai IoT Botnet: A Retrospective Analysis
This is a guest post by Elie Bursztein who writes about security and anti-abuse research. It was first published on his blog and has been lightly edited.
This post provides a retrospective analysis of Mirai — the infamous Internet-of-Things botnet that took down major websites via massive distributed denial-of-service using hundreds of thousands of compromised Internet-Of-Things devices. This research was conducted by a team of researchers from Cloudflare (Jaime Cochran, Nick Sullivan), Georgia Tech, Google, Akamai, the University of Illinois, the University of Michigan, and Merit Network and resulted in a paper published at USENIX Security 2017.
At its peak in September 2016, Mirai temporarily crippled several high-profile services such as OVH, Dyn, and Krebs on Security via massive distributed Denial of service attacks (DDoS). OVH reported that these attacks exceeded 1 Tbps—the largest on public record.
What’s remarkable about these record-breaking attacks is they were carried out via small, innocuous Internet-of-Things (IoT) devices like home routers, air-quality monitors, and personal surveillance cameras. At its peak, Mirai infected over 600,000 vulnerable IoT devices, according to our measurements.
This blog post follows the timeline above
- Mirai Genesis: Discusses Mirai’s early days and provides a brief technical overview of Continue reading
There’s Always Cache in the Banana Stand
We’re happy to announce that we now support all HTTP Cache-Control response directives. This puts powerful control in the hands of you, the people running origin servers around the world. We believe we have the strongest support for Internet standard cache-control directives of any large scale cache on the Internet.
Documentation on Cache-Control is available here.
Cloudflare runs a Content Distribution Network (CDN) across our globally distributed network edge. Our CDN works by caching our customers’ web content at over 119 data centers around the world and serving that content to the visitors nearest to each of our network locations. In turn, our customers’ websites and applications are much faster, more
available, and more secure for their end users.
A CDN’s fundamental working principle is simple: storing stuff closer to where it’s needed means it will get to its ultimate destination faster. And, serving something from more places means it’s more reliably available.
To use a simple banana analogy: say you want a banana. You go to your local fruit stand to pick up a bunch to feed your inner monkey. You expect the store to have bananas in stock, which would satisfy your request instantly. But, what if Continue reading
There’s Always Cache in the Banana Stand
We’re happy to announce that we now support all HTTP Cache-Control response directives. This puts powerful control in the hands of you, the people running origin servers around the world. We believe we have the strongest support for Internet standard cache-control directives of any large scale cache on the Internet.
Documentation on Cache-Control is available here.
Cloudflare runs a Content Distribution Network (CDN) across our globally distributed network edge. Our CDN works by caching our customers’ web content at over 119 data centers around the world and serving that content to the visitors nearest to each of our network locations. In turn, our customers’ websites and applications are much faster, more
available, and more secure for their end users.
A CDN’s fundamental working principle is simple: storing stuff closer to where it’s needed means it will get to its ultimate destination faster. And, serving something from more places means it’s more reliably available.
To use a simple banana analogy: say you want a banana. You go to your local fruit stand to pick up a bunch to feed your inner monkey. You expect the store to have bananas in stock, which would satisfy your request instantly. But, what if Continue reading
The Curious Case of Caching CSRF Tokens
It is now commonly accepted as fact that web performance is critical for business. Slower sites can affect conversion rates on e-commerce stores, they can affect your sign-up rate on your SaaS service and lower the readership of your content.
In the run-up to Thanksgiving and Black Friday, e-commerce sites turned to services like Cloudflare to help optimise their performance and withstand the traffic spikes of the shopping season.
In preparation, an e-commerce customer joined Cloudflare on the 9th November, a few weeks before the shopping season. Instead of joining via our Enterprise plan, they were a self-serve customer who signed-up by subscribing to our Business plan online and switching their nameservers over to us.
Their site was running Magento, a notably slow e-commerce platform - filled with lots of interesting PHP, with a considerable amount of soft code in XML. Running version 1.9, the platform was somewhat outdated (Magento was totally rewritten in version 2.0 and subsequent releases).
Despite the somewhat dated technology, the e-commerce site was "good enough" for this customer and had done it's job for many years.
They were the first to notice an interesting technical issue surrounding how performance and security can often Continue reading
The Curious Case of Caching CSRF Tokens
It is now commonly accepted as fact that web performance is critical for business. Slower sites can affect conversion rates on e-commerce stores, they can affect your sign-up rate on your SaaS service and lower the readership of your content.
In the run-up to Thanksgiving and Black Friday, e-commerce sites turned to services like Cloudflare to help optimise their performance and withstand the traffic spikes of the shopping season.
In preparation, an e-commerce customer joined Cloudflare on the 9th November, a few weeks before the shopping season. Instead of joining via our Enterprise plan, they were a self-serve customer who signed-up by subscribing to our Business plan online and switching their nameservers over to us.
Their site was running Magento, a notably slow e-commerce platform - filled with lots of interesting PHP, with a considerable amount of soft code in XML. Running version 1.9, the platform was somewhat outdated (Magento was totally rewritten in version 2.0 and subsequent releases).
Despite the somewhat dated technology, the e-commerce site was "good enough" for this customer and had done it's job for many years.
They were the first to notice an interesting technical issue surrounding how performance and security can often Continue reading
Why Some Phishing Emails Are Mysteriously Disappearing
Phishing is the absolute worst.
Unfortunately, sometimes phishing campaigns use Cloudflare for the very convenient, free DNS. To be clear –– there’s a difference between a compromised server being leveraged to send phishing emails and an intentionally malicious website dedicated to this type of activity. The latter clearly violates our terms of service.
In the past, our Trust and Safety team would kick these intentional phishers off the platform, but now we have a new trick up our sleeve and a way for their malicious emails to mysteriously disappear into the ether.
Background: How Email Works
SMTP - the protocol used for sending email - was finalized in 1982, when it was just a small community online. Many of them knew and trusted each other, and so the protocol was built entirely on trust. In an SMTP message, the MAIL FROM field can be arbitrarily defined. That means you could send an email from any email address, even one you don’t own.
This is great for phishers, and bad for everyone else.
The solution to prevent email spoofing was to create the Sender Policy Framework (SPF). SPF allows the domain owner to specify which servers are allowed to send Continue reading
The FCC Wants to Kill Net Neutrality – Use Battle for the Net on Cloudflare Apps to Fight Back
TL;DR - Net neutrality is under attack. There's an app on Cloudflare Apps that empowers site owners to host a popup on their sites, encouraging users to contact their congresspeople to fight back. Everyone should be doing this right now, before the December 14th FCC vote.
Use Battle for the Net to Call your Congressperson »
Attend Cloudflare's Save the Internet! Net Neutrality Call-A-Thon »
The Federal Communications Commission (FCC) has scheduled a vote to kill its net neutrality rules this Thursday, December 14th. Unfortunately, the expectation is that the FCC will vote to repeal its net neutrality rules. Read about this on Business Insider, Bloomberg, or TechCrunch.
Net neutrality is the principle that networks should not discriminate against content that passes through them. The FCC’s net neutrality rules protect the Internet, users, and companies from abusive behavior by the largest Internet Service Providers (ISPs). Without net neutrality rules in place, ISPs may be able to legally create a "pay to play" system and charge websites to provide content to their customers more quickly. This will create a disadvantage for startups, bloggers, and everyone else who cannot afford to pay fees for their websites to offer faster service.
McAllen, Texas: Cloudflare opens 119th Data Center just north of the Mexico border
Five key facts to know about McAllen, Texas
- McAllen, Texas is on the southern tip of the Rio Grande Valley
- The city is named after John McAllen, who provided land in 1904 to bring the St. Louis, Brownsville & Mexico Railway railway into the area
- McAllen, Texas is named the City of Palms
- The border between Mexico and the USA is less than nine miles away from the data center
- McAllen, Texas is where Cloudflare has placed its 119th data center
Second datacenter in Texas; first on the border with Mexico
While McAllen is close to the Mexican border, its importance goes well beyond that simple fact. The city is halfway between Dallas, Texas (where Cloudflare has an existing datacenter) and Mexico City, the center and capital of Mexico. This means that any Cloudflare traffic delivered into Mexico is better served from McAllen. Removing 500 miles from the latency equation is a good thing. 500 miles equates to around 12 milliseconds of round-trip latency and when a connection operates (as all connections should), as a secure connection, then there can be many round trip communications before the first page starts showing up. Improving latency is key, even if we have Continue reading
The end of the road for Server: cloudflare-nginx
Six years ago when I joined Cloudflare the company had a capital F, about 20 employees, and a software stack that was mostly NGINX, PHP and PowerDNS (there was even a little Apache). Today, things are quite different.
CC BY-SA 2.0 image by Randy Merrill
The F got lowercased, there are now more than 500 people and the software stack has changed radically. PowerDNS is gone and has been replaced with our own DNS server, RRDNS, written in Go. The PHP code that used to handle the business logic of dealing with our customers’ HTTP requests is now Lua code, Apache is long gone and new technologies like Railgun, Warp, Argo and Tiered Cache have been added to our ‘edge’ stack.
And yet our servers still identify themselves in HTTP responses with
Server: cloudflare-nginx
Of course, NGINX is still a part of our stack, but the code that handles HTTP requests goes well beyond the capabilities of NGINX alone. It’s also not hard to imagine a time where the role of NGINX diminishes further. We currently run four instances of NGINX on each edge machine (one for SSL, one for non-SSL, one for caching and one Continue reading
Building a new IMDB: Internet Mince Pie Database
CC-BY-SA 2.0 image by Phil! Gold
Since joining Cloudflare I’ve always known that as we grew, incredible things would be possible. It’s been a long held ambition to work in an organisation with the scale to answer a very controversial and difficult question. To do so would require a collection of individuals with a depth of experience, passion, dedication & above all collaborative spirit.
As Cloudflare’s London office has grown in the last 4 years I believe 2017 is the year we reach the tipping point where this is possible. A paradigm-shift in the type of challenges Cloudflare is able to tackle. We could finally sample every commercially available mince pie in existence before the 1st of December. In doing so, we would know conclusively which mince pie we should all be buying over Christmas to share with our friends & families.
What is a mince pie?
For the uninitiated, a Mince Pie is “a sweet pie of British origin, filled with a mixture of dried fruits and spices called mincemeat, that is traditionally served during the Christmas season in the English world.” - Wikipedia for Mince Pie
The original Mince Pie was typically filled with a mixture Continue reading
On the Leading Edge – Cloudflare named a leader in The Forrester Wave: DDoS Mitigation Solutions
Cloudflare has been recognized as a leader in the “Forrester WaveTM: DDoS Mitigation Solutions, Q4 2017.”
The DDoS landscape continues to evolve. The increase in sophistication, frequency, and range of targets of DDoS attacks has placed greater demands on DDoS providers, many of which were evaluated in the report.
This year, Cloudflare received the highest scores possible in 15 criteria, including:
- Length of Implementation
- Layers 3 and 4 Attacks Mitigation
- DNS Attack Mitigation
- IoT Botnets
- Multi-Vector Attacks
- Filtering Deployment
- Secure Socket Layer Investigation
- Mitigation Capacity
- Pricing Model
We believe that Cloudflare’s position as a leader in the report stems from the following:
An architecture designed to address high-volume attacks. This post written in October 2016 provides some insight into how Cloudflare’s architecture scales to meet the most advanced DDoS attacks differently than legacy scrubbing centers.
In September 2017, due to the size and effectiveness of our network, we announced the elimination of “surge pricing” commonly found in other DDoS vendors by offering unmetered mitigation. Regardless of what Cloudflare plan a customer is on—Free, Pro, Business, or Enterprise—we will never terminate a customer or charge more based on the size of an attack.
Because we protect over 7 Continue reading
CAA of the Wild: Supporting a New Standard
One thing we take pride in at Cloudflare is embracing new protocols and standards that help make the Internet faster and safer. Sometimes this means that we’ll launch support for experimental features or standards still under active development, as we did with TLS 1.3. Due to the not-quite-final nature of some of these features, we limit the availability at the onset to only the most ardent users so we can observe how these cutting-edge features behave in the wild. Some of our observations have helped the community propose revisions to the corresponding RFCs.
We began supporting the DNS Certification Authority Authorization (CAA) Resource Record in June behind a beta flag. Our goal in doing so was to see how the presence of these records would affect SSL certificate issuance by publicly-trusted certification authorities. We also wanted to do so in advance of the 8 September 2017 enforcement date for mandatory CAA checking at certificate issuance time, without introducing a new and externally unproven behavior to millions of Cloudflare customers at once. This beta period has provided invaluable insight as to how CAA records have changed and will continue to change the commercial public-key infrastructure (PKI) ecosystem.
As of today, Continue reading
Make SSL boring again
It may (or may not!) come as surprise, but a few months ago we migrated Cloudflare’s edge SSL connection termination stack to use BoringSSL: Google's crypto and SSL implementation that started as a fork of OpenSSL.
We dedicated several months of work to make this happen without negative impact on customer traffic. We had a few bumps along the way, and had to overcome some challenges, but we ended up in a better place than we were in a few months ago.
TLS 1.3
We have already blogged extensively about TLS 1.3. Our original TLS 1.3 stack required our main SSL termination software (which was based on OpenSSL) to hand off TCP connections to a separate system based on our fork of Go's crypto/tls standard library, which was specifically developed to only handle TLS 1.3 connections. This proved handy as an experiment that we could roll out to our client base in relative safety.
However, over time, this separate system started to make our lives more complicated: most of our SSL-related business logic needed to be duplicated in the new system, which caused a few subtle bugs to pop up, and made it Continue reading
Introducing the Cloudflare Warp Ingress Controller for Kubernetes
It’s ironic that the one thing most programmers would really rather not have to spend time dealing with is... a computer. When you write code it’s written in your head, transferred to a screen with your fingers and then it has to be run. On. A. Computer. Ugh.
Of course, code has to be run and typed on a computer so programmers spend hours configuring and optimizing shells, window managers, editors, build systems, IDEs, compilation times and more so they can minimize the friction all those things introduce. Optimizing your editor’s macros, fonts or colors is a battle to find the most efficient path to go from idea to running code.
CC BY 2.0 image by Yutaka Tsutano
Once the developer is master of their own universe they can write code at the speed of their mind. But when it comes to putting their code into production (which necessarily requires running their programs on machines that they don’t control) things inevitably go wrong. Production machines are never the same as developer machines.
If you’re not a developer, here’s an analogy. Imagine carefully writing an essay on a subject dear to your heart and then publishing it only to be Continue reading
What I learned at my first Cloudflare Retreat
For the last seven years, Cloudflare has taken the entire company off site for a few days at the end of the year for a company retreat. Back in 2010, this meant five people from the San Francisco office. This November, we had 453 employees from our San Francisco, Singapore, London, Champaign (Illinois), New York City, Washington (DC), and Austin (Texas) offices spend time together in Monterey, California.
Knowing that so many teammates would be coming in from all over the world, we used the days leading up to the retreat to hold global team meetings, conduct a session of our home-grown Making Great Managers workshop, and brought in Valerie Aurora from Frame Shift Consulting to lead Ally Skills workshops for the entire company.
On Thursday, buses departed from Cloudflare headquarters and took us all down to Monterey. Our CEO, Matthew Prince, delivered opening remarks over lunch. During his talk, we learned about the imminent acquisition of Neumob, his thoughts about growing pains and how to successfully scale, and were reminded that we are at our best when we are inclusive of everyone. We reflected on how far we’ve come and got an inspiring glimpse of where we are Continue reading
Cloudflare Apps Platform Update: November Edition
Hello Everyone!
My name is Teffen and I’m a developer on the Apps platform. This is the November edition of our newsletter showcasing the latest platform improvements in payment tools, DNS powered apps, customer comments, and more!
Since our last newsletter, dozens of developers like you have reached out with ideas for new kinds of apps that weren’t yet possible. These are some of my favorite conversations because they help us find out which features should be prioritized. With your guidance, we’ve spent this month meticulously converting our supply of Halloween candy into those ideas. Let’s dive in and see what’s new!
? Paid App Product Enhancements
We’ve made it easier to upsell premium features with product specific options. Customers can try out exclusive features before making a purchase, on any site, even without Cloudflare account! Here’s an example of Lead Box using product specific radio buttons:
In this example, a customer can choose to see the newsletter option after choosing the "Pro" plan. Developers can now update the Live Preview in response to this choice. We’ve added new "_product" keyword for this event. Here’s a snippet on how Lead Box handles a customer Continue reading
The New DDoS Landscape
News outlets and blogs will frequently compare DDoS attacks by the volume of traffic that a victim receives. Surely this makes some sense, right? The greater the volume of traffic a victim receives, the harder to mitigate an attack - right?
At least, this is how things used to work. An attacker would gain capacity and then use that capacity to launch an attack. With enough capacity, an attack would overwhelm the victim's network hardware with junk traffic such that they can no longer serve legitimate requests. If your web traffic is served by a server with a 100 Gbps port and someone sends you 200 Gbps, your network will be saturated and the website will be unavailable.
Recently, this dynamic has shifted as attackers have gotten far more sophisticated. The practical realities of the modern Internet have increased the amount of effort required to clog up the network capacity of a DDoS victim - attackers have noticed this and are now choosing to perform attacks higher up the network stack.
In recent months, Cloudflare has seen a dramatic reduction in simple attempts to flood our network with junk traffic. Whilst we continue to see large network level attacks, in Continue reading
Want to try Warp? We just enabled the beta for you
Tomorrow is Thanksgiving in the United States. It’s a holiday for getting together with family characterized by turkey dinner and whatever it is that happens in American football. While celebrating with family is great, if you use a computer for your main line of work, sometimes the conversation turns to how to setup the home wifi or can Russia really use Facebook to hack the US election. Just in case you’re a geek who finds yourself in that position this week, we wanted to give you something to play with. To that end, we’re opening the Warp beta to all Cloudflare users. Feel free to tell your family there’s been an important technical development you need to attend to immediately and enjoy!
Hello Warp! Getting Started
Warp allows you to expose a locally running web server to the internet without having to open up ports in the firewall or even needing a public IP address. Warp connects a web server directly to the Cloudflare network where Cloudflare acts as your web server’s network gateway. Every request reaching your origin must travel to the Cloudflare network where you can apply rate limits, access policies and authentication before the request hits your Continue reading
Releasing AddThis on Cloudflare Apps: Making Disciplined Product Design Decisions
This is a guest post by Emily Schwartz, Product Manager for the AddThis team at Oracle. With a background in digital media that has spanned across NPR, WaPo Labs, Trove, and others, Emily cares deeply about helping publishers leverage data and technology for success.
The Process of Paring Down
When our team learned about the opportunity to build an AddThis app on Cloudflare Apps, I was ready to pounce. Building for distribution platforms is a core part of our business and product strategy, and I knew AddThis could bring a lot to the table for Cloudflare users. With a media background in my pocket, I understand the necessity of making content easily and quickly distributable -- and I wanted to get our tools in front of new users so we could learn more about the critical needs of publishers, merchants, and website owners.
The decision to build was the easy part. What to build was the challenging part.
With time and resources tight, I knew building an app that offered our full suite of website tools wouldn’t be immediately feasible—or even make sense. Share buttons, follow buttons, related posts, list building, link promotion, and tip jar are all Continue reading