Unrest in Gabon leads to Internet shutdown
A second day of rioting in Gabon after the recent election is accompanied by an Internet blackout. Residents of the capital, Libreville, reported that Internet access had been cut and we can confirm that we saw a sudden shutdown of Internet access from Gabon to sites that use CloudFlare.
These three graphs show the major networks inside Gabon shutting off suddenly with a minuscule amount of traffic making it through.
The charts show that Internet access shutdown at different times for different networks. At the time of writing the Internet appears to be almost completely cut off in Gabon.
Panne d'Internet au Gabon après l'élection
Un deuxième jour d'émeutes au Gabon après l'élection récente est accompagnée d'une panne d'Internet. Les résidents de la capitale, Libreville, ont indiqué que l'accès à Internet avait été coupé et CloudFlare peut confirmer que nous avons vu un arrêt brutal de l'accès Internet du Gabon vers nos sites.
Ces trois graphiques montrent que les grands réseaux à l'intérieur du Gabon étaient coupé soudainement.
Les graphiques montrent que l'arrêt de l'accès à Internet à des moments différents pour les différents réseaux. Au moment de la rédaction de l'Internet semble être presque complètement coupé au Gabon.
You Can Finally Get More Page Rules For 5 For $5
Since CloudFlare launched Page Rules in 2012, our Free, Pro and Business users have been asking for a way to get more Page Rules without committing to the next plan up. Starting today, anyone on CloudFlare can add 5 additional Page Rules for just $5/month.
Page Rules allows you to fine tune your site speed and to apply CloudFlare’s wide range of features to specific parts of your site. Page Rules are also accessible over our API, so you can integrate them into your build process or sync them across your domains.
To help you get the most out of Page Rules, we’re also launching a tutorial site that features videos to help you setup Page Rules for specific content management systems like WordPress, Magento and Drupal, and for specific goals like optimizing your website's speed, increasing security, and saving on your bandwidth costs.
How the Consumer Product Safety Commission is (Inadvertently) Behind the Internet’s Largest DDoS Attacks
The mission of the United State's Government's Consumer Product Safety Commission (CPSC) is to protect consumers from injury by products. It's ironic then that the CPSC is playing an unwitting role in most of the largest DDoS attacks seen on the Internet. To understand how, you need to understand a bit about how you launch a high volume DDoS.
Logo of the Consumer Product Safety Commission
Amplification
DDoS attacks are inherently about an attacker sending more traffic to a victim than the victim can handle. The challenge for an attacker is to find a way to generate a large amount of traffic. Launching a DDoS attack is a criminal act, so an attacker can't simply go sign up for large transit contracts. Instead, attackers find ways to leverage other people's resources.
One of the most effective strategies is known as an amplification attack. In these attacks, an attacker can amplify their resources by reflecting them off other resources online that magnify the level of traffic. The most popular amplification vector is known as DNS reflection.
DNS Reflection
We've written about DNS reflection attacks in detail before. The basics are that an attacker generates DNS requests from a network that allows Continue reading
Evenly Distributed Future
Traveling back and forth between the UK and US I often find myself answering the question “What does CloudFlare do?”. That question gets posed by USCIS on arrival and I’ve honed a short and accurate answer: “CloudFlare protects web sites from hackers, makes web sites faster and ensures they work on your computer, phone or tablet.”
If anyone, border agents or others, wants more detail I usually say: “If you run a web site or API for an app and you are Amazon.com, Google, Yahoo or one of a handful of major Internet sites you have the expertise to stay on top of the latest technologies and attacks; you have the staff to accelerate your web site and keep it fully patched. Anyone else, and that’s almost every web site on the Internet, simply will not have the money, people, or knowledge to ‘be a Google’. That’s where CloudFlare comes in: we make sure to stay on top of the latest trends in the Internet so that every web site can ‘be Google’."
The author William Gibson has said many times: “The future is already here Continue reading
The Cuban CDN
On a recent trip to Cuba I brought with me a smartphone and hoped to get Internet access either via WiFi or 3G. I managed that (at a price) but also saw for myself how Cubans get access to an alternate Internet delivered by sneakernet.
Cuba is currently poorly served by the Internet with a small number of public WiFi hotspots. There are currently 175 public WiFi hotspots in the country, many in public parks. In addition, many large hotels also have public WiFi. Since this is the primary way Cubans get Internet access it’s not uncommon to see situations like this:
Getting on the WiFi means buying a card that gives you access for 2 CUC ($2) per hour. These cards have a login number and a password (hidden behind a scratch off panel). The hour can be used in chunks by logging off and on.
There’s also mobile phone access to the Internet (I saw 3G, EDGE and GPRS as I traveled across Cuba), but at 1 CUC ($1) per MB it’s very expensive. The phone company does provide email access (to their own email service) and so some Cubans I met used their phones to get Continue reading
Bandwidth Costs Around the World
CloudFlare protects over 4 million websites using our global network which spans 86 cities across 45 countries. Running this network give us a unique vantage point to track the evolving cost of bandwidth around the world.
CC BY-SA 2.0 image by Quinn Dombrowski
Recap
Two years ago, we previewed the relative cost of bandwidth that we see in different parts of the world. Bandwidth is the largest recurring cost of providing our service. Compared with Europe and North America, there were considerably higher Internet costs in Australia, Asia and Latin America. Even while bandwidth costs tend to trend down over time, driven by competition and decreases in the costs of underlying hardware, we thought it might be interesting to provide an update.
Since August 2014, we have tripled the number of our data centers from 28 to 86, with more to come. CloudFlare hardware is also deployed in new regions such as the Middle East and Africa. Our network spans multiple countries in each continent, and, sometimes, multiple cities in each country.
Traffic across 86 data centers in the CloudFlare network
There are approximately thirteen networks called “Tier 1 networks” (e.g., Telia, GTT, Tata, Cogent) who Continue reading
Accelerating Node.js applications with HTTP/2 Server Push
In April, we announced support for HTTP/2 Server Push via the HTTP Link header. My coworker John has demonstrated how easy it is to add Server Push to an example PHP application.
CC BY 2.0 image by Nicky Fernandes
We wanted to make it easy to improve the performance of contemporary websites built with Node.js. we developed the netjet middleware to parse the generated HTML and automatically add the Link headers. When used with an example Express application you can see the headers being added:
We use Ghost to power this blog, so if your browser supports HTTP/2 you have already benefited from Server Push without realizing it! More on that below.
In netjet, we use the PostHTML project to parse the HTML with a custom plugin. Right now it is looking for images, scripts and external stylesheets. You can implement this same technique in other environments too.
Putting an HTML parser in the response stack has a downside: it will increase the page load latency (or "time to first byte"). In most cases, the added latency will be overshadowed by other parts of your application, such as database access. However, netjet includes an adjustable LRU cache keyed Continue reading
This is strictly a violation of the TCP specification
I was asked to debug another weird issue on our network. Apparently every now and then a connection going through CloudFlare would time out with 522 HTTP error.
CC BY 2.0 image by Chris Combe
522 error on CloudFlare indicates a connection issue between our edge server and the origin server. Most often the blame is on the origin server side - the origin server is slow, offline or encountering high packet loss. Less often the problem is on our side.
In the case I was debugging it was neither. The internet connectivity between CloudFlare and origin was perfect. No packet loss, flat latency. So why did we see a 522 error?
The root cause of this issue was pretty complex. Afterred long debugging we identified an important symptom: sometimes, once in thousands of runs, our test program failed to establish a connection between two daemons on the same machine. To be precise, an NGINX instance was trying to establish a TCP connection to our internal acceleration service on localhost. This failed with a timeout error.
Once we knew what to look for we were able to reproduce this with good old netcat. After a couple of dozen of Continue reading
CloudFlare’s JSON-powered Documentation Generator
Everything that it's possible to do in the CloudFlare Dashboard is also possible through our RESTful API. We use the same API to power the dashboard itself.
In order to keep track of all our endpoints, we use a rich notation called JSON Hyper-Schema. These schemas are used to generate the complete HTML documentation that you can see at https://api.cloudflare.com. Today, we want to share a set of tools that we use in this process.
CC BY 2.0 image by Richard Martin
JSON Schema
JSON Schema is a powerful way to describe your JSON data format. It provides complete structural validation and can be used for things like validation of incoming requests. JSON Hyper-Schema further extends this format with links and gives you a way describe your API.
JSON Schema Example
{
"type": "object",
"properties": {
"name": { "type": "string" },
"age": { "type": "number" },
"address": {
"type": "object",
"properties": {
"street_address": { "type": "string" },
"city": { "type": "string" },
"state": { "type": "string" },
"country": { "type" : "string" }
}
}
}
}
Matching JSON
{
"name": "John Doe",
"age": 45,
"address": {
"street_address": "12433 State St NW",
"city": "Atlanta",
"state": "Georgia",
"country": Continue readingIntroducing the p0f BPF compiler
Two years ago we blogged about our love of BPF (BSD packet filter) bytecode.
CC BY 2.0 image by jim simonson
Then we published a set of utilities we are using to generate the BPF rules for our production iptables: the bpftools.
Today we are very happy to open source another component of the bpftools: our p0f BPF compiler!
Meet the p0f
p0f is a tool written by superhuman Michal Zalewski. The main purpose of p0f is to passively analyze and categorize arbitrary network traffic. You can feed p0f any packet and in return it will derive knowledge about the operating system that sent the packet.
One of the features that caught our attention was the concise yet explanatory signature format used to describe TCP SYN packets.
The p0f SYN signature is a simple string consisting of colon separated values. This string cleanly describes a SYN packet in a human-readable way. The format is pretty smart, skipping the varying TCP fields and keeping focus only on the essence of the SYN packet, extracting the interesting bits from it.
We are using this on daily basis to categorize the packets that we, at CloudFlare, see when we are a target Continue reading
Ask Some HTTP/2 Pros The Hard Questions
We're big fans of HTTP/2 and our customers make up the majority of HTTP/2 enabled domains today. HTTP/2 is a key part of the modern web, and its growth and adoption is changing how websites and applications are built.
On Thursday July 21, 2016, join web performance experts Ilya Grigorik (Web Performance Engineer at Google) and Suzanne Aldrich (Solutions Engineer at CloudFlare), as they discuss the latest in HTTP/2 and Server Push, the tools and resources you can use today to build fast and scalable web apps, and ways to speed up your content on any device.
When: Thursday July 21, 2016 from 1pm-2pm Eastern Time (1500 – 1600 UTC)
Who: CloudFlare’s own Suzanne Aldrich and Ilya Grigorik from Google
Need the basics of HTTP/2 and Server Push? Visit the CloudFlare HTTP/2 website.
CloudFlare sites protected from httpoxy
CC BY 2.0 image by Joe Seggiola
We have rolled out automatic protection for all customers for the the newly announced vulnerability called httpoxy.
This vulnerability affects applications that use “classic” CGI execution models, and could lead to API token disclosure of the services that your application may talk to.
By default httpoxy requests are modified to be harmless and then request is allowed through, however customers who want to outright block those requests can also use the Web Application Firewall rule 100050 in CloudFlare Specials to block requests that could lead to the httpoxy vulnerability.
More data, more data
"multas per gentes et multa per aequora" 1
The life of a request to CloudFlare begins and ends at the edge. But the afterlife! Like Catullus to Bithynia, the log generated by an HTTP request or a DNS query has much, much further to go.
This post comes from CloudFlare's Data Team. It reports the state of processing these sort of edge logs, including what's worked well for us and what remains a challenge in the time since our last post from April 2015.
Numbers, sense
In an edge network, where HTTP and DNS clients connect to thousands of servers distributed across the world, the key is to distribute those servers across many carefully picked points of presence—and with over 80 PoPs, no network has better representation than CloudFlare. The reverse, however, has to happen for that network's logs. After anycast has scattered requests (and queries) to thousands of nodes at the edge, it's the Data Team's job to gather the resulting logs to a small number of central points and consolidate them for easy use by our customers.
The charts above depict (with some artifacts due to counter resets) the total structured logs sent from the edge to one Continue reading
Why we use the Linux kernel’s TCP stack
A recent blog post posed the question Why do we use the Linux kernel's TCP stack?. It triggered a very interesting discussion on Hacker News.
I've also thought about this question while working at CloudFlare. My experience mostly comes from working with thousands of production machines here and I can try to answer the question from that perspective.
CC BY 2.0 image by John Vetterli
Let's start with a broader question - what is the point of running an operating system at all? If you planned on running a single application, having to use a kernel consisting of multiple million lines of code may sound like a burden.
But in fact most of us decide to run some kind of OS and we do that for two reasons. Firstly, the OS layer adds hardware independence and easy to use APIs. With these we can focus on writing the code for any machine - not only the specialized hardware we have at the moment. Secondly, the OS adds a time sharing layer. This allows us to run more than one application at a time. Whether it's a second HTTP server or just a bash session, this ability to share resources Continue reading
HTTP/2 Server Push with multiple assets per Link header
In April we announced that we had added experimental support for HTTP/2 Server Push to all CloudFlare web sites. We did this so that our customers could iterate on this new functionality.
CC BY 2.0 image by https://www.flickr.com/photos/mryipyop/
Our implementation of Server Push made use of the HTTP Link header as detailed in W3C Preload Working Draft.
We also showed how to make Server Push work from within PHP code and many people started testing and using this feature.
However, there was a serious restriction in our initial version: it was not possible to specify more than one asset per Link header for Server Push and many CMS and web development platforms would not allow multiple Link headers.
We have now addressed that problem and it is possible to request that multiple assets be pushed in a single Link header. This change is live and was used to push assets in this blog post to your browser if your browser supports HTTP/2.
When CloudFlare reads a Link header sent by an origin web server it will remove assets that it pushes from the Link header passed on to the web browser. That made it a little difficult Continue reading
The complete guide to Go net/http timeouts
When writing an HTTP server or client in Go, timeouts are amongst the easiest and most subtle things to get wrong: there’s many to choose from, and a mistake can have no consequences for a long time, until the network glitches and the process hangs.
HTTP is a complex multi-stage protocol, so there's no one-size fits all solution to timeouts. Think about a streaming endpoint versus a JSON API versus a Comet endpoint. Indeed, the defaults are often not what you want.
In this post I’ll take apart the various stages you might need to apply a timeout to, and look at the different ways to do it, on both the Server and the Client side.
SetDeadline
First, you need to know about the network primitive that Go exposes to implement timeouts: Deadlines.
Exposed by net.Conn with the Set[Read|Write]Deadline(time.Time) methods, Deadlines are an absolute time which when reached makes all I/O operations fail with a timeout error.
Deadlines are not timeouts. Once set they stay in force forever (or until the next call to SetDeadline), no matter if and how the connection is used in the meantime. So to build a timeout with SetDeadline you'll have to Continue reading
Economical With The Truth: Making DNSSEC Answers Cheap
We launched DNSSEC late last year and are already signing 56.9 billion DNS record sets per day. At this scale, we care a great deal about compute cost. One of the ways we save CPU cycles is our unique implementation of negative answers in DNSSEC.
CC BY-SA 2.0 image by Chris Short
I will briefly explain a few concepts you need to know about DNSSEC and negative answers, and then we will dive into how CloudFlare saves on compute when asked for names that don’t exist.
What You Need To Know: DNSSEC Edition
Here’s a quick summary of DNSSEC:
This is an unsigned DNS answer (unsigned == no DNSSEC):
cloudflare.com. 299 IN A 198.41.214.162
cloudflare.com. 299 IN A 198.41.215.162
This is an answer with DNSSEC:
cloudflare.com. 299 IN A 198.41.214.162
cloudflare.com. 299 IN A 198.41.215.162
cloudflare.com. 299 IN RRSIG A 13 2 300 20160311145051 20160309125051 35273 cloudflare.com. RqRna0qkih8cuki++YbFOkJi0DGeNpCMYDzlBuG88LWqx+Aaq8x3kQZX TzMTpFRs6K0na9NCUg412bOD4LH3EQ==
Answers with DNSSEC contain a signature for every record type that is returned. (In this example, only A records are returned so there is only one signature.) The signatures allow Continue reading
A post-mortem on this morning’s incident
We would like to share more details with our customers and readers on the internet outages that occurred this morning and earlier in the week, and what we are doing to prevent these from happening again.
June 17 incident
On June 17, at 08:32 UTC, our systems detected a significant packet loss between multiple destinations on one of our major transit provider backbone networks, Telia Carrier.
In the timeframe where the incident was being analysed by our engineers, the loss became intermittent and finally disappeared.
Packet loss on Telia Carrier (AS1299)
Today’s incident
Today, Jun 20, at 12:10 UTC, our systems again detected massive packet loss on one of our major transit provider backbone networks: Telia Carrier.
Packet loss on Telia Carrier (AS1299)
Typically, transit providers are very reliable and transport all of our packets from one point of the globe to the other without loss - that’s what we pay them for. In this case, our packets (and that of other Telia customers), were being dropped.
While Internet users usually take it for granted that they can reach any destination in the world from their homes and businesses, the reality is harsher than that. Our planet is big, and Continue reading
Join Us And Paul Vixie On Tuesday To Discuss BIND, Root Servers, And DNS Security
CloudFlare and Gandi have been hosting a speaker series on DNS, previously bringing in the founder of DNS Paul Mockapetris and Dan Kaminsky, who uncovered one of the most critical vulnerabilities in DNS.
Our third and final talk is coming up on June 21st at 6PM PST at the Gandi office in San Francisco (live stream link will be posted on the Meetup page) and you won’t want to miss it, because our speaker is none other than Paul Vixie.
Paul wrote most of BIND Version 8 and hired the team who wrote BIND Version 9, the most widely used DNS software on the Internet. He founded ISC, home of BIND and F-root and later operated C-root.
We'll talk about alternative DNS root servers, email security and spam (Paul founded the first ever anti-spam company), and what needs to be done about DNS and security (Paul added most of BIND’s current security systems).
So come grab a beer and let's geek out about DNS one more time.
Moscow, Russia: CloudFlare’s 83rd data center
Здравствуйте! ))) CloudFlare is excited to announce the newest addition to our network in the largest country in the world (by footprint), increasing both our data center and city count to 83. Moscow is not only the capital and largest city in Russia, it is also home to several Internet exchanges which CloudFlare now participates at: the Moscow Internet Exchange (MSK-IX), Data IX and Global IX (Eurasia Peering coming soon). This raises the number of exchanges that CloudFlare is a participant of to over 120, making us one of the top interconnected networks globally.
Здравствуйте! ))) Мы рады объявить о новом пополнении в нашей сети network в самой большой стране мира (по занимаемой площади), увеличивая как количество наших датацентров, так и количество городов на 83. Москва является не только столицей и самым крупным городом в России, но она также является домом для нескольких точек обмена интернет-трафиком. CloudFlare в настоящее время принимает участие в следующих: Moscow Internet Exchange (MSK-IX), Data IX и Global IX (Eurasia Peering на подходе). Это увеличивает количество точек обмена интернет-трафиком, в которых участвует CloudFlare до 120, тем самым продвигая нас на одну из лидирующих позиций наиболее взаимосвязаных сетей в мире.