The rise of multivector DDoS attacks
It's been a while since we last wrote about Layer 3/4 DDoS attacks on this blog. This is a good news - we've been quietly handling the daily onslaught of DDoS attacks. Since our last write-up, a handful of interesting L3/4 attacks have happened. Let's review them.
Gigantic SYN
In April, John tweeted about a gigantic 942Gbps SYN flood:
It was a notable event for a couple of reasons.
First, it was really large. Previously, we've seen only amplification / reflection attacks at terabit scale. In those cases, the attacker doesn't actually have too much capacity. They need to bounce the traffic off other servers to generate a substantial load. This is different from typical "direct" style attacks, like SYN floods. In the SYN flood mentioned by John, all 942Gbps were coming directly from attacker-controlled machines.
Secondly, this attack was truly distributed. Normal SYN floods come from a small number of geographical locations. This one, was all over the globe, hitting all Cloudflare data centers:
Thirdly, the attack seem to be partially spoofed. While our analysis was not conclusive, we saw random, spoofed source IP addresses in the largest internet exchanges. The above Hilbert curve shows the source IP Continue reading
1 Thing You Can Do To Make Your Internet Safer And Faster
On April 1st, 2018, we announced 1.1.1.1, the fastest public DNS resolver in the world ???. Today, we are launching the 1.1.1.1 mobile app to make it incredibly easy to use 1.1.1.1 on your phone.
TL;DR
Any time you are on a public internet connection people can see what sites you visit. Even worse, your Internet Service Provider is very possibly selling all of your browsing history to the highest bidder. We have a tool called 1.1.1.1 which makes it easy to get a faster, more private, Internet experience, but it’s historically been too complex for many people to use, particularly on mobile devices. Today, we’re launching an app you (and everyone you know) can use to use 1.1.1.1 every time your mobile phone connects to the Internet. It’s a free, it’s easy, download it now.
Fastest Public Resolver
We launched 1.1.1.1 on April 1st. Frankly, we’ve been blown away by how many people actually made the switch. Changing your network settings is not easy, but if our traffic amount is any indication, many of you made the effort. Continue reading
Cloud Computing without Containers
Cloudflare has a cloud computing platform called Workers. Unlike essentially every other cloud computing platform I know of, it doesn’t use containers or virtual machines. We believe that is the future of Serverless and cloud computing in general, and I’ll try to convince you why.
Isolates
Two years ago we had a problem. We were limited in how many features and options we could build in-house, we needed a way for customers to be able to build for themselves. We set out to find a way to let people write code on our servers deployed around the world (we had a little over a hundred data centers then, 155 as of this writing). Our system needed to run untrusted code securely, with low overhead. We sit in front of ten million sites and process millions and millions of requests per second, it also had to run very very quickly.
The Lua we had used previously didn’t run in a sandbox; customers couldn’t write their own code without our supervision. Traditional virtualization and container technologies like Kubernetes would have been exceptionally expensive for everyone involved. Running thousands of Kubernetes pods in a single location would be resource intensive, doing it in Continue reading
Improving RubyDocs with Cloudflare Workers and Workers KV
The following is a guest post from Manuel Meurer, Berlin based web developer, entrepreneur, and Ruby on Rails enthusiast. In 2010, he founded Kraut Computing as a one-man web dev shop and launched Uplink, a network for IT experts in Germany, in 2015.
RubyDocs is an open-source service that generates and hosts “fancy docs for any Ruby project”, most notably for the Ruby language itself and for Rails, the most popular Ruby framework. The nifty thing about it is that the docs can be generated for any version of a project — so let’s say you’re working on an old Rails app that still uses version 3.2.22 (released June 16, 2015), then you can really benefit from having access to the docs of that specific version, since a lot of the methods, classes, and concepts of the current Rails version (5.2.1 at the time of writing) don’t exist in that old version.
Scratching an itch
I built RubyDocs back in 2013 to scratch my own itch — a few similar services that I had used over the years had disappeared or hadn’t been regularly updated. After the initial work to get RubyDocs up and running, I continued Continue reading
Lagos, Nigeria – Cloudflare’s 155th city
At just shy of 200 million, Nigeria is the most populous country in Africa (Ethiopia is second and Egypt is third). That’s a lot of people to communicate with the world - and communicate they all do!
According to a published report earlier this year, 84% of the Nigerian population own a mobile device (193 million population and 162 million mobile subscriptions). Again, that’s #1 for any country in Africa. But why so connected? Maybe because Nigeria (and Lagos specifically) is always on the move!
Lagos, as those that know the city say, never sleeps, it’s filled with color from the food to fashion to even the diverse people going about their business. The vibrancy of the city is like a hard slap to the face, no matter what you have been told, your first time here will still knock you out. In Lagos, anything is possible, from the sadness of poverty to the clearly visible upper class, the city sucks you in like a surfers dream wave. Visitor come into Lagos and leave feeling like they’ve been through a unique experience. The traffic is mind blowing and the same goes for the work pace.
Lagos, a city always on Continue reading
Introducing Spectrum with Load Balancing
We’re excited to announce the full integration of Cloudflare Spectrum with Load Balancing. Combining Spectrum with Load Balancing enables traffic management of TCP connections utilising the same battle tested Load Balancer our customers already use for billions of HTTP requests every day.
Customers can configure load balancers with TCP health checks, failover, and steering policies to dictate where traffic should flow. This is live in the Cloudflare dashboard and API — give it a shot!
TCP Health Checks
You can now configure Cloudflare’s Load Balancer health checks to probe any TCP port for an accepted connection. This is in addition to the existing HTTP and HTTPS options.
Health checks are an optional feature within Cloudflare’s Load Balancing product. Without health checks, the Cloudflare Load Balancer will distribute traffic to all origins in the first pool. While this is in itself useful, adding a health check to a Load Balancer provides additional functionality.
With a health check configured for a pool in a Load Balancer, Cloudflare will automatically distribute traffic within a pool to any origins that are marked up by the health check. Unhealthy origins will be dropped automatically. This allows for intelligent failover both within a pool and amongst Continue reading
Cloudflare’s network boosts security and performance for IBM Cloud customers
Today our partner IBM® announced the general availability of Cloud Internet Services (CIS) Enterprise. It marks a significant step forward in the partnership that we announced at the IBM THINK event in March.
CIS delivers security and performance to IBM Cloud® customers’ internet applications. It brings together Cloudflare’s 150+ points of presence with IBM Cloud’s 60 data centers, stopping attacks before they can even reach the IBM Cloud. CIS Enterprise is integrated into IBM Cloud, allowing IBM Cloud customers to set up and manage Cloudflare’s DDoS mitigation, web application firewall, smart routing and highly customizable load balancer, all from within the IBM Cloud user interface.
Our Network Map (as of 10/18/18). Click here for the latest version
We thought it timely to give a refresher on how Cloudflare’s network supports IBM Cloud customers. The network is designed to meet requirements of the most demanding enterprise customers. It is based on an architecture that differentiates it from legacy CDN, DNS and DDoS-mitigation services to ensure that internet applications stay online, even in the face of extremely high volume attacks or legitimate traffic spikes.
Cloudflare’s network of data centers, distributed across 74 countries (including 22 in China), has a network Continue reading
Cloudflare Peering Portal – Beta
It can be a big deal for Internet users when Cloudflare rolls into town. After our recent Mongolia launch, we received lots of feedback from happy customers that all of a sudden, Internet performance noticeably improved.
As a result, it's not a surprising that we regularly receive requests from all over the world to either peer with our network, or to host a node. However, potential partners are always keen to know just how much traffic will be served over that link. What performance benefits can end-users expect? How much upstream traffic will the ISP save? What new bandwidth will they have available for traffic management?
Starting today, ISPs and hosting providers can request a login to the Cloudflare Peering Portal to find the answers to these questions. After validating ownership of your ASN, the Cloudflare network team will provide a login to the newly launched Peering Portal - Beta. You can find more information at: cloudflare.com/partners/peering-portal/
What problem does peering solve?
If you're new to the core infrastructure of the Internet, the best way to understand peering is to frame the problems it solves:
- Bandwidth costs money
- Internet users don't like slow websites
- Network operators have limited Continue reading
Encrypt that SNI: Firefox edition
A couple of weeks ago we announced support for the encrypted Server Name Indication (SNI) TLS extension (ESNI for short). As promised, our friends at Mozilla landed support for ESNI in Firefox Nightly, so you can now browse Cloudflare websites without leaking the plaintext SNI TLS extension to on-path observers (ISPs, coffee-shop owners, firewalls, …). Today we'll show you how to enable it and how to get full marks on our Browsing Experience Security Check.
Here comes the night
The first step is to download and install the very latest Firefox Nightly build, or, if you have Nightly already installed, make sure it’s up to date.
When we announced our support for ESNI we also created a test page you can point your browser to https://encryptedsni.com which checks whether your browser / DNS configuration is providing a more secure browsing experience by using secure DNS transport, DNSSEC validation, TLS 1.3 & ESNI itself when it connects to our test page. Before you make any changes to your Firefox configuration, you might well see a result something like this:
So, room for improvement! Next, head to the about:config page and look for the network.security.esni.enabled Continue reading
Warum ich Cloudflare helfe in Deutschland, Österreich und der Schweiz zu wachsen
Warum Cloudflare ?
Ich übernehme bei Cloudflare die Verantwortung für die DACH Region (Deutschland, Austria & Switzerland-CH), um das Geschäft von Cloudflare in Deutschland, Österreich und der Schweiz zu erweitern. Die einzigartige Mission von Cloudflare ein besseres Internet zu gestalten, ist der Grund, warum ich diese Verantwortung übernommen habe. Heute kann ich es kaum erwarten dieses Ziel zu erreichen.
Cloudflare trägt dazu bei 10 Millionen Domains schneller und sicherer zu machen indem die Kunden gegen eine große Anzahl von Online-Angriffen geschützt werden und gleichzeitig die Verfügbarkeit garantiert wird. Dies ist die Grundlage für jedes erfolgreiche Geschäft.
Sicherheit, Datenschutz und Geschwindigkeit sind Schlüsselfunktionen für fast jedes Unternehmen: von sehr großen traditionellen bis hin zu reinen Online-Unternehmen oder Einzelpersonen, die ihre eigene persönliche Marke aufbauen. Dazu könnte ich endlos weitere Beispiele nennen und je mehr ich über Cloudflare’s Mission erfahre desto mehr bin ich begeistert.
Eine von Cloudflare’s größten Stärken ist das globale Netzwerk.
Cloudflare verfügt bereits über Rechenzentren in sieben Städten der DACH Region (weitere folgen), um sicherzustellen, dass das Internet für die Nutzer in der Region schnell, sicher und zuverlässig ist.
Ich bin stolz, das erste Büro in Deutschland eröffnen zu dürfen. Gleichzeitig habe ich Continue reading
Why I’m helping Cloudflare grow in Germany, Austria, and Switzerland
Why Cloudflare?
I am incredibly excited to announce that I’m joining Cloudflare as the Head of DACH to help to expand Cloudflare’s demand in Germany, Austria, and Switzerland. Having been in the technology industry for many years, Cloudflare’s mission to help build a better Internet was frankly the reason I joined, and I’m now very eager to start working towards this.
I quickly learned how Cloudflare helps to speed up and secure over 10 million Internet properties by protecting these customers from a wide range of online attacks and providing the reliability needed to run strong businesses. Security, privacy, and performance are key drivers for almost every business: from large traditional enterprises to purely online businesses and even individuals building their own personal brand. I could go on and on. The more I learned, the more excited I became.
One of Cloudflare’s major strengths is its global network. Cloudflare already has data centers in seven cities in the DACH region (with more to come) helping to ensure the Internet is fast, safe, and reliable for users online in the region. So while I get the honor of opening our first office in Germany (in Munich), I loved that Cloudflare had Continue reading
My First Grace Hopper Celebration
Cloudflare #GHC18 team
I am 25+ years into my career in technology, and this was the very first time I attended a conference geared towards women.
A couple of weeks ago I went to Grace Hopper Celebration (#GHC18), and I can still feel the exuberant energy from the 22,000 women over the intensive 3 day conference. I attended with our Cloudflare team; our purpose was to connect with women in the greater tech community and recruit new talent to join our team and mission to help build a better Internet.
Cloudflare prioritizes GHC because we recognize that diversity in our company, and particularly in our technical departments, is crucial to our success. We believe that the best companies are diverse companies. This was Cloudflare’s second time sponsoring GHC, and I was part of the planning committee. This year I headed to the event with 20 of my colleagues to meet all of the incredible attendees, hold on-site interviews, and even host our own Cloudflare panel and luncheon.
Getting to #GHC18
Early Tuesday morning, the day before the conference, as I joined the Southwest Airlines boarding line at Oakland Airport, my fellow passengers were not the usual contingent of Continue reading
A Question of Timing
When considering website performance, the term TTFB - time to first byte - crops up regularly. Often we see measurements from cURL and Chrome, and this article will show what timings those tools can produce, including time to first byte, and discuss whether this is the measurement you are really looking for.
Timing with cURL
cURL is an excellent tool for debugging web requests, and it includes the ability to take timing measurements. Let’s take an example website www.zasag.mn (the Mongolian government), and measure how long a request to its home page takes:
First configure the output format for cURL in ~/.curlrc:
$ cat .curlrc
-w "dnslookup: %{time_namelookup} | connect: %{time_connect} | appconnect: %{time_appconnect} | pretransfer: %{time_pretransfer} | starttransfer: %{time_starttransfer} | total: %{time_total} | size: %{size_download}\n"Now connect to the site dropping the output (-o /dev/null) since we’re only interested in the timing:
$ curl -so /dev/null https://www.zasag.mn
dnslookup: 1.510 | connect: 1.757 | appconnect: 2.256 | pretransfer: 2.259 |
starttransfer: 2.506 | total: 3.001 | size: 53107These timings are in seconds. Depending on your version of cURL, you may Continue reading
Serverless Rust with Cloudflare Workers
The Workers team just announced support for WebAssembly (WASM) within Workers. If you saw my post on Internet Native Apps, you'll know that I believe WebAssembly will play a big part in the apps of the future.
It's exciting times for Rust developers. Cloudflare's Serverless Platform, Cloudflare Workers, allows you to compile your code to WASM, upload to 150+ data centers and invoke those functions just as easily as if they were JavaScript functions. Today I'm going to convert my lipsum generator to use Rust and explore the developer experience (hint: it's already pretty nice).
The Workers teams notes in the documentation:
...WASM is not always the right tool for the job. For lightweight tasks like redirecting a request to a different URL or checking an authorization token, sticking to pure JavaScript is probably both faster and easier than WASM. WASM programs operate in their own separate memory space, which means that it's necessary to copy data in and out of that space in order to operate on it. Code that mostly interacts with external objects without doing any serious "number crunching" likely does not benefit from WASM.
OK, I'm unlikely to gain significant performance improvements on this particular Continue reading
DC CyberWeek Is Here!
Photo by Sarah Ferrante Goodrich / Unsplash
This October is the 15th annual National Cybersecurity Awareness Month in the United States, a collaboration between the US government and industry to raise awareness about the part we can all play in staying more secure online. Here at Cloudflare, where our mission is to help build a better internet, we look forward to this month all year.
As part of this month-long education campaign, Cloudflare is participating in D.C CyberWeek this week, the largest cybersecurity festival in the U.S, taking place in Washington, DC. This year’s event is expected to have over 10,000 attendees, more than 100 events, and feature representatives from over 180 agencies, private companies, and service providers. We will join with other leaders in cybersecurity, to share best practices, find ways to collaborate, and work to achieve common goals.
Along with the United States, the European Union also runs a month-long cyber awareness campaign in October, with the initiative having started back in 2012. The aim of this advocacy campaign is similar: promoting cybersecurity among citizens and organizations, and providing information on available tools and resources. Watch our CTO speak to some of the main considerations around Continue reading
Optimizing HTTP/2 prioritization with BBR and tcp_notsent_lowat
Getting the best end-user performance from HTTP/2 requires good support for resource prioritization. While most web servers support HTTP/2 prioritization, getting it to work well all the way to the browser requires a fair bit of coordination across the networking stack. This article will expose some of the interactions between the web server, Operating System and network and how to tune a server to optimize performance for end users.
tl;dr
On Linux 4.9 kernels and later, enable BBR congestion control and set tcp_notsent_lowat to 16KB for HTTP/2 prioritization to work reliably. This can be done in /etc/sysctl.conf:
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
net.ipv4.tcp_notsent_lowat = 16384
Browsers and Request Prioritization
A single web page is made up of dozens to hundreds of separate pieces of content that a web browser pulls together to create and present to the user. The main content (HTML) for the page you are visiting is a list of instructions on how to construct the page and the browser goes through the instructions from beginning to end to figure out everything it needs to load and how to put it all together. Each piece of content requires a Continue reading
Happy National Coming Out Day: Stories from Proudflare
Today is the 30th Anniversary of National Coming Out Day. We wanted to share some coming out stories from members of Proudflare and draw attention to resources the Human Rights Campaign provides to those who are thinking about coming out or wish to be supportive of those who come out to them.
About National Coming Out Day
On October 11, 1987, about 500,000 people marched on Washington for Lesbian and Gay Rights. This was the second demonstration of this type in the capital and it resulted in the formation of several LGBTQ organizations.
In the late 1980s, the LGBTQ community recognized that they often reacted defensively to anti LGBTQIA+ actions and the community came up with the idea of a national day for celebrating coming out. The anniversary of the 1987 march was chosen as that national day.
Each year on October 11th, National Coming Out Day continues to promote a safe world for LGBTQ individuals to live truthfully and openly.
Source: https://www.hrc.org/resources/the-history-of-coming-out
Coming out stories from Proudflare
Here are seven examples of the coming out stories that surfaced from a company-wide awareness campaign. I hope you’ll enjoy reading these and will find inspiration in them. Let’s Continue reading
Graceful upgrades in Go
The idea behind graceful upgrades is to swap out the configuration and code of a process while it is running, without anyone noticing it. If this sounds error prone, dangerous, undesirable and in general a bad idea – I’m with you. However, sometimes you really need them. Usually this happens in an environment where there is no load balancing layer. We have these at Cloudflare, which led to us investigating and implementing various solutions to this problem.
Coincidentally, implementing graceful upgrades involves some fun low-level systems programming, which is probably why there are already a bajillion options out there. Read on to learn what trade-offs there are, and why you should really really use the Go library we are about to open source. For the impatient, the code is on github and you can read the documentation on godoc.
The basics
So what does it mean for a process to perform a graceful upgrade? Let’s use a web server as an example: we want to be able to fire HTTP requests at it, and never see an error because a graceful upgrade is happening.
We know that HTTP uses TCP under the Continue reading
Introducing Single Sign-On for the Cloudflare Dashboard
The Challenge of Managing User Access to SaaS Applications
As the number of SaaS services people use everyday grows, it has become more challenging to juggle the number of password and multi-factor authentication combinations users have to keep track of to get online.
Adopting identity services have allowed companies to centralize employee authentication. With Cloudflare Access, companies can ensure employees use a company managed identity provider when accessing websites behind Cloudflare. Last week, Sam published a blog on how Cloudflare has made it easier to connect Cloudflare Access to the Atlassian suite of tools.
Since Cloudflare has simplified access control for corporate applications, many enterprise customers have commonly asked for the ability to extend the same ease of access and control to the Cloudflare dashboard itself.
Single Sign-On for the Cloudflare Dashboard
Today, we are announcing support for enterprise customers to use single sign-on (SSO) through their identity provider to access the Cloudflare dashboard.
Cloudflare is a critical piece of infrastructure for customers, and SSO ensures that customers can apply the same authentication policies to access the Cloudflare dashboard as other critical resources.
Once onboarded for SSO, all company user logins to the Cloudflare dashboard redirect to the customer’s identity provider. Once all required authentication checks complete successfully, the user is seamlessly redirected back to dash.cloudflare.com and logged in.
Leveraging Access & Workers to Build SSO
At Cloudflare, we dogfood our own services as both a way to make them better for our customers and to make developing new services more efficient and robust. With SSO, this is no different. Authentication configurations are managed through Access, which allows us to launch with support for the same identity providers available in Access today, including SAML.
Cloudflare is 8 years old and we built our user authentication system way before Cloudflare Access existed. In order to connect Access to our existing authentication system, we built a Cloudflare Worker that converts Access authentication tokens to our own authentication tokens. This greatly simplified the code changes required in our system, and results in faster SSO logins because the Worker runs at the network edge and reduces the number of round trips required to authenticate users.
In addition to leveraging Cloudflare services to build Single Sign-On, we are moving all Cloudflare employees to use SSO through our existing G Suite setup. This ensures Cloudflare can uniformly enforce multi-factor authentication policies for the services we protect with Cloudflare itself.
How to Start using SSO for the Cloudflare Dashboard
Cloudflare Enterprise customers can reach out to their Customer Success Manager to learn how to start using SSO to log-in to the Cloudflare dashboard. If you are interested in using SSO yourself and becoming a Cloudflare Enterprise customer, then please get in touch.
A Tour Inside Cloudflare’s G9 Servers
Cloudflare operates at a significant scale, handling nearly 10% of the Internet HTTP requests that is at peak more than 25 trillion requests through our network every month. To ensure this is as efficient as possible, we own and operate all the equipment in our 154 locations around the world in order to process the volume of traffic that flows through our network. We spend a significant amount of time specing and designing servers that makes up our network to meet our ever changing and growing demands. On regular intervals, we will take everything we've learned about our last generation of hardware and refresh each component with the next generation…
If the above paragraph sounds familiar, it’s a reflecting glance to where we were 5 years ago using today’s numbers. We’ve done so much progress engineering and developing our tools with the latest tech through the years by pushing ourselves at getting smarter in what we do.
Here though we’re going to blog about muscle.
Since the last time we blogged about our G4 servers, we’ve iterated one generation each of the past 5 years. Our latest generation is now the G9 server. From a G4 server comprising 12 Intel Continue reading