Go has a debugger—and it’s awesome!
Something that often, uh... bugs1 Go developers is the lack of a proper debugger. Sure, builds are ridiculously fast and easy, and println(hex.Dump(b)) is your friend, but sometimes it would be nice to just set a breakpoint and step through that endless if chain or print a bunch of values without recompiling ten times.
CC BY 2.0 image by Carl Milner
You could try to use some dirty gdb hacks that will work if you built your binary with a certain linker and ran it on some architectures when the moon was in a waxing crescent phase, but let's be honest, it isn't an enjoyable experience.
Well, worry no more! godebug is here!
godebug is an awesome cross-platform debugger created by the Mailgun team. You can read their introduction for some under-the-hood details, but here's the cool bit: instead of wrestling with half a dozen different ptrace interfaces that would not be portable, godebug rewrites your source code and injects function calls like godebug.Line on every line, godebug.Declare at every variable declaration, and godebug.SetTrace for breakpoints (i.e. wherever you type _ = "breakpoint").
I find this solution brilliant. What you get out Continue reading
How to receive a million packets per second
Last week during a casual conversation I overheard a colleague saying: "The Linux network stack is slow! You can't expect it to do more than 50 thousand packets per second per core!"
That got me thinking. While I agree that 50kpps per core is probably the limit for any practical application, what is the Linux networking stack capable of? Let's rephrase that to make it more fun:
On Linux, how hard is it to write a program that receives 1 million UDP packets per second?
Hopefully, answering this question will be a good lesson about the design of a modern networking stack.
CC BY-SA 2.0 image by Bob McCaffrey
First, let us assume:
Measuring packets per second (pps) is much more interesting than measuring bytes per second (Bps). You can achieve high Bps by better pipelining and sending longer packets. Improving pps is much harder.
Since we're interested in pps, our experiments will use short UDP messages. To be precise: 32 bytes of UDP payload. That means 74 bytes on the Ethernet layer.
For the experiments we will use two physical servers: "receiver" and "sender".
They both have two six core 2GHz Xeon processors. With hyperthreading (HT) enabled Continue reading
iOS Developers — Migrate to iOS 9 with CloudFlare
Thousands of developers use CloudFlare to accelerate and secure the backend of their mobile applications and websites. This week is Apple’s Worldwide Developers Conference (WWDC), where thousands of Apple developers come to San Francisco to talk, learn and share best practices for developing software for Apple platforms. New announcements from Apple this week make CloudFlare an even more obvious choice for application developers.
New operating systems, new application requirements
The flagship announcement of WWDC 2015 was a new version of Apple’s mobile operating system, iOS 9, to be released in September with a developer preview available now. They also announced a new Mac operating system, OS X El Capitan, launching in the fall. Apple has a track record of developing and supporting technologies that enhance user privacy and security with iMessage and Facetime and the trend is continuing with these new operating systems. In both cases, Apple is requiring application developers to make use of two network technologies that CloudFlare is big fan of: HTTPS and IPv6.
For iOS 9 and El Capitan, all applications submitted to the iOS and Mac App Stores must work over IPv6. In previous versions, applications were allowed that only worked with IPv4.
From Continue reading
Four years later and CloudFlare is still doing IPv6 automatically
Over the past four years CloudFlare has helped well over two million websites join the modern web, making us one of the fastest growing providers of IPv6 web connectivity on the Internet. CloudFlare's Automatic IPv6 Gateway allows IPv4-only websites to support IPv6-only clients with zero clicks. No hardware. No software. No code changes. And no need to change your hosting provider.
A Four Year Story
The story of IPv6 support for customers of CloudFlare is about as long as the story of CloudFlare itself. June 6th, 2011 (four years ago) was the original World IPv6 Day, and CloudFlare participated. Each year since, the global Internet community has pushed forward with additional IPv6 deployment. Now, four years later, CloudFlare is celebrating June 6th knowing that our customers are being provided with a solid IPv6 offering that requires zero configuration to enable. CloudFlare is the only global CDN that provides IPv4/IPv6 delivery of content by default and at scale.
IPv6 has been featured in our blog various times over the last four years. We have provided support for legacy logging systems to handle IPv6 addresses, provided DDoS protection on IPv6 alongside classic IPv4 address space, and provided Continue reading
Welcome Acquia!
We’ve had the good fortune to share many great experiences with the Acquia team over the last few years. From breaking bread with founder and CTO Dries Buytaert at SXSW, to skiing the slopes of Park City with the company’s CEO Tom Erickson, to staying up late with their incredible team onboarding a joint customer under a DDoS attack. It’s always a pleasure to spend time with the Acquia team.
Today we are thrilled to welcome Acquia as a CloudFlare Partner. Together we developed Acquia Cloud Edge powered by CloudFlare making it easier for any of their customers to access CloudFlare’s web performance and security solutions. The Acquia Cloud Edge is a family of products that protects websites against security threats, ensures only clean traffic get served, and speeds up site performance no matter where visitors are located.
Acquia Cloud Edge powered by CloudFlare comes as Edge Protect and Edge CDN. Edge Protect defends against DDoS and other network-level attacks. CloudFlare sits on the network edge in front of Acquia web servers, allowing early identification of attack patterns and questionable visitors, and mitigating attacks before they reach a user’s site. Edge CDN accelerates the delivery of digital experiences through CloudFlare’s Continue reading
Logjam: the latest TLS vulnerability explained
Yesterday, a group from INRIA, Microsoft Research, Johns Hopkins, the University of Michigan, and the University of Pennsylvania published a deep analysis of the Diffie-Hellman algorithm as used in TLS and other protocols. This analysis included a novel downgrade attack against the TLS protocol itself called Logjam, which exploits EXPORT cryptography (just like FREAK).
First, let me start by saying that CloudFlare customers are not and were never affected. We don’t support non-EC Diffie-Hellman ciphersuites on either the client or origin side. We also won't touch EXPORT-grade cryptography with a 20ft stick.
But why are CloudFlare customers safe, and how does Logjam work anyway?
The image is "Logjam" as interpreted by @0xabad1dea.
Diffie-Hellman and TLS
This is a detailed technical introduction to how DH works and how it’s used in TLS—if you already know this and want to read about the attack, skip to “Enter export crypto, enter Logjam” below. If, instead, you are not interested in the nuts and bolts and want to know who’s at risk, skip to “So, what’s affected?”
To start a TLS connection, the two sides—client (the browser) and server (CloudFlare)—need to agree securely on a secret key. This process is called Continue reading
CloudFlare Supports the Passage of the USA Freedom Act
Earlier today, the lower house in the U.S. Congress (the House of Representatives) passed the USA FREEDOM Act. The Act, if passed by the Senate and signed by the President, would seek to sunset the National Security Agency’s bulk collection and mass surveillance programs, which may or may not be authorized by Section 215 of the PATRIOT Act. Under this authority the U.S. government has established its broad surveillance programs to indiscriminately collect information. Other governments have followed this lead to create additional surveillance capabilities—most recently, the French Parliament has moved a bill that would allow broad surveillance powers with little judicial oversight.
Restricting routine bulk collection is important: it’s not the government’s job to collect everything that passes over the Internet. The new version of the USA FREEDOM Act keeps useful authorities but ends bulk collection of private data under the PATRIOT Act. It also increases the transparency of the secret FISA court, which reviews surveillance programs—a key start to understanding and fixing broken policies around surveillance. The Act would also allow companies to be more transparent in their reporting related to FISA orders.
To be clear, we continue to be supportive of law enforcement and work Continue reading
CloudFlare “Interview Questions”
For quite some time we've been grilling our candidates about dirty corners of TCP/IP stack. Every engineer here must prove his/her comprehensive understanding of the full network stack. For example: what are the differences in checksumming algorithms between IPv4 and IPv6 stacks?
I'm joking of course, but in the spirit of the old TCP/IP pub game I want to share some of the amusing TCP/IP quirks I've bumped into over the last few months while working on CloudFlare's automatic attack mitigation systems.
CC BY-SA 2.0 image by Daan Berg
Don't worry if you don't know the correct answer: you may always come up with a funny one!
Some of the questions are fairly obvious, some don't have a direct answer and are supposed to provoke a longer discussion. The goal is to encourage our readers to review the dusty RFCs, get interested in the inner workings of the network stack and generally spread the knowledge about the protocols we rely on so much.
Don't forget to add a comment below if you want to share a response!
You think you know all about TCP/IP? Let's find out.
Archaeology
1) What is the lowest TCP port number?
2) The TCP Continue reading
Google PageSpeed Service customers: migrate to CloudFlare for acceleration
This week, Google announced that its hosted PageSpeed Service will be shut down. Everyone using the hosted service needs to move their site elsewhere before August 3 2015 to avoid breaking their website.
We're inviting these hosted customers: don't wait...migrate your site to CloudFlare for global acceleration (and more) right now.
CC BY 2.0 image by Roger
As TechCrunch wrote: "In many ways, PageSpeed Service is similar to what CloudFlare does but without the focus on security."
What is PageSpeed?
PageSpeed started as — and continues — as a Google-created software module for the Apache webserver to rewrite webpages to reduce latency and bandwidth, to help make the web faster.
Google introduced their hosted PageSpeed Service in July 2011, to save webmasters the hassle of installing the software module.
It's the hosted service that is being discontinued.
CloudFlare performance
CloudFlare provides similar capabilities to PageSpeed, such as minification, image compression, and asynchronous loading.
Additionally, CloudFlare offers more performance gains through a global network footprint, Railgun for dynamic content acceleration, built-in SPDY support, and more.
Not just speed
PageSpeed Service customers care about performance, and CloudFlare delivers. CloudFlare also includes security, SSL, Continue reading
Go crypto: bridging the performance gap
It is no secret that we at CloudFlare love Go. We use it, and we use it a LOT. There are many things to love about Go, but what I personally find appealing is the ability to write assembly code!
That is probably not the first thing that pops to your mind when you think of Go, but yes, it does allow you to write code "close to the metal" if you need the performance!
Another thing we do a lot in CloudFlare is... cryptography. To keep your data safe we encrypt everything. And everything in CloudFlare is a LOT.
Unfortunately the built-in cryptography libraries in Go do not perform nearly as well as state-of-the-art implementations such as OpenSSL. That is not acceptable at CloudFlare's scale, therefore we created assembly implementations of Elliptic Curves and AES-GCM for Go on the amd64 architecture, supporting the AES and CLMUL NI to bring performance up to par with the OpenSSL implementation we use for Universal SSL.
We have been using those improved implementations for a while, and attempting to make them part of the official Go build for the good of the community. For now Continue reading
Redesigning CloudFlare
CloudFlare’s original interface grew at an amazing speed. Visually, it hadn't changed much since CloudFlare’s launch in 2010. After several years of new features, settings, and ancillary UIs buried beneath clicks, it became clear that the user experience was lacking and would only get worse as we continued to add features. The question became: How could we make a UI that was versatile, scalable, and consistent?
If you haven’t yet, make sure you read Matthew’s post about the philosophy behind our new interface. This post will go into the details and the thought process behind designing our new dashboard.
Why a redesign?
We needed versatility for a growing variety of users and devices
As CloudFlare has grown, we now have a large variety of customers spanning four very different plan levels. We needed an interface that would work well for both the casual owner of a single blog, an agency managing many client sites, and enterprise customers that demand ultimate control. Also, the rise of responsive design was something we wanted to take seriously — the dashboard should be versatile enough to work just as well on every device.
We needed a platform that we could build upon
We couldn’t Continue reading
An introduction to JavaScript-based DDoS
CloudFlare protects millions of websites from online threats. One of the oldest and most pervasive attacks launched against websites is the Distributed Denial of Service (DDoS) attack. In a typical DDoS attack, an attacker causes a large number of computers to send data to a server, overwhelming its capacity and preventing legitimate users from accessing it.
In recent years, DDoS techniques have become more diversified: attackers are tricking unsuspecting computers into participating in attacks in new and interesting ways. Last year, we saw what was likely the largest attack in history (>400Gbps) performed using NTP reflection. In this attack, the unsuspecting participants were misconfigured NTP servers worldwide. This year, we’re seeing a disturbing new trend: attackers are using malicious JavaScript to trick unsuspecting web users into participating in DDoS attacks.
The total damage that can be caused by a NTP or DNS reflection attack is limited by the number of vulnerable servers. Over time, this number decreases as networks patch their servers, and the maximum size of the attack is capped at the outbound capacity of all the vulnerable servers. For JavaScript-based DDoS, any computer with a browser can be enrolled in the attack, making the potential attack volume nearly Continue reading
Introducing Multi-User Organizations: Share An Account Without Sharing A Login
An enterprise needs security and controls around access.
Your web developer needs to update your website’s logo and make sure it’s live immediately, but doesn’t need access to your SSL keys. Your sysadmin manages your DNS, but doesn’t need to see your visitor traffic. Your marketing team needs to see traffic, but shouldn’t have access to your WAF.
Today CloudFlare is introducing new Multi-User functionality so that many members of a team can work together to manage one CloudFlare account, each with different levels of access.
The Super Admin, and Role-Based Permissions
CloudFlare Multi-User accounts are hierarchical, with the root privileges given to the account’s Super Administrator. The Super Administrator can add or delete users in the organization, change the permissions given to each user, and see and edit all CloudFlare settings. If there is more than one Super Administrator, the Super Administrators can remove each other, which is good practice when an employee leaves the company or switches jobs.
When a user joins a multi-user organization on CloudFlare, they can only see and access the settings that a Super Admin has delegated to them. For example, a user added to the organization as a DNS Administrator would only be Continue reading
CloudFlare’s New Dashboard
When we started CloudFlare, we thought we were building a service to make websites faster and more secure, and we wanted to make the service as easy and accessible as possible. As a result, we built the CloudFlare interface to put basic functions front and center and designed it to look more like a consumer app than the UI for the powerful network it controlled.
Over time, we realized there was a lot more to CloudFlare. In 2011, we added the concept of Apps, and a myriad of additional performance and security features from Rocket Loader to Railgun were added too. All these additional settings got buried under a lowly gear menu next to each site in a customer's account.
While still easier to navigate than the average enterprise app, using our UI could be a frustrating experience. For instance, imagine you wanted to turn on Rocket Loader for multiple sites. You'd have to go to My Websites, click the gear menu next to one of your domains, navigate to CloudFlare Settings, select the Performance Settings tab, scroll to Rocket Loader, then toggle it on. Then you had to go back to My Websites and repeat the process again for Continue reading
New Magento WAF Rule – RCE Vulnerability Protection
Today the Magento Security Team created a new ModSecurity rule and added it to our WAF rules to mitigate an important RCE (remote code execution) vulnerability in the Magento web e-commerce platform. Any customer using the WAF needs to click the ON button next to the “CloudFlare Magento” Group in the WAF Settings to enable protection immediately.
Both Magento version 1.9.1.0 CE and 1.14.1.0 EE are compromised by this vulnerability. CloudFlare WAF protection can help mitigate vulnerabilities like this, but it is vital that Magento users patch Magento immediately. Select and download the patch for SUPEE-5344.
Of Phishing Attacks and WordPress 0days
Proxying around 5% of the Internet’s requests gives us an interesting vantage point from which to observe malicious behavior. It also make us a target. Aside from the many, varied denial of service attacks that break against our defenses we also see huge number of phishing campaigns. In this blog post I will dissect a recent phishing attack that we detected and neutralized with the help of our friends at Bluehost.
An attack that is particularly interesting as it appears to be using a brand new WordPress 0day.
A Day Out Phishing
The first sign we typically see that shows a new phishing campaign is underway are the phishing emails themselves. There's general a constant background noise of a few of these emails targeting individual customers every day. However when a larger campaign starts up, typically that trickle turns into a flood of very similar messages.
Messages like this one:
Note — We will never send you an email like this. If you see one, its fake and should be reported to our abuse team by forwarding it to [email protected].
In terms of the phishing campaign timeline, these emails aren’t the first event. Much like a spider looking to Continue reading
Contributing back to the security community
This Friday at the RSA Conference in San Francisco, along with Marc Rogers, Principal Security Researcher at CloudFlare, I'm speaking about a version of The Grugq's PORTAL, an open source network security device designed to make life easier and safer for anyone traveling, especially internationally, with phones, tablets, laptops, and other network-connected devices.
Portal uses open-source software and services to take inexpensive, commodity travel routers and turn them into powerful security devices. Since this is pretty far from CloudFlare's core business, it warrants a brief digression into why we support projects like this.
Computer security was for a very long time only of interest to hobbyists, academics, and obscure government agencies. Cryptography was an interesting offshoot of number theory, a foundational but very abstract part of mathematics, and many of the early infrastructure components of the Internet didn't include security at all -- there was an assumption that anyone who could gain access would be responsible and well-intentioned, a consequence of the academic origins; after all, why would they want to break or steal things which were freely available.
Before the "cambrian explosion" of commercial computer security, there was still a lot of great security research -- it Continue reading
Oceania Redundancy: Auckland and Melbourne data centers now online
The genesis of our 33rd and 34th data centers in Auckland and Melbourne started a short hop away in nearby Sydney. Prior to these deployments traffic from all of New Zealand and Australia's collective 23 million Internet users was routed through CloudFlare's Sydney data center. Even for those in faraway Perth, the time necessary to reach our Sydney PoP was a mere 55ms of round trip time (RTT). By comparison, the blink of an eye takes 300-400ms. In other words, latency wasn't exactly the pressing concern. The real concern was a failure scenario in our Sydney data center.
Fortunately, our entire architecture starts with an assumption: failure is going to happen. As a result, we plan for failure at every level and have designed a system to gracefully handle it when it occurs. Even though we now maintain multiple layers of redundancy—from power supplies and power circuits to line cards, routing engines and network providers—our ultimate level of redundancy is in the ability to fail out an entire data center in favor of another. In the past we've even written about how this might even play out in the case of a global thermonuclear war. In this instance, the challenge Continue reading
Protection against critical Windows vulnerability (CVE-2015-1635)
A few hours ago, more details surfaced about the MS15-034 vulnerability. Simple PoC code has been widely published that will hang a Windows web server if sent a request with an HTTP Range header containing large byte offsets.
We have rolled out a WAF rule that blocks these requests.
Customers on a paid plan and who have the WAF enabled are automatically protected against this problem. It is highly recommended that you upgrade your IIS and your Windows servers as soon as possible; in the meantime any requests coming into CloudFlare that try and exploit this DoS/RCE will be blocked.
CloudFlare is now a Google Cloud Platform Technology Partner
We’re excited to announce that CloudFlare has just been named a Google Cloud Platform Technology Partner. So what does this mean? Now, Google Cloud Platform customers can experience the best of both worlds—the power and protection of the CloudFlare community along with the flexibility and scalability of Google’s infrastructure.
We share many mutual customers with Google, and this collaboration makes it even easier for Google Cloud Platform customers to get started with CloudFlare.
How does it work?
When CloudFlare is enabled, Google Cloud Platform customers have their infrastructure extended directly to the network edge, giving them faster content delivery as well as heightened optimization and security.
Benefits Include:
2x Web Performance Speed - CloudFlare uses advanced caching and the SPDY protocol to double web content transfer speeds, making web content transfer times significantly faster.
Datacenters at Your Customer’s Doorstep - CloudFlare’s global edge network caches static files close to their destination, meaning that content always loads fast no matter where customers are located. Also, CloudFlare peers with Google in strategic locations globally, improving response times for Google Cloud Platform services.
Protection Against DDoS and SQL Injection Attacks - Because CloudFlare sits on the edge, customers are protected from malicious traffic Continue reading