The Technical Challenges of Building Cloudflare WARP

If you have seen our other post you know that we released WARP to the last members of our waiting list today. With WARP our goal was to secure and improve the connection between your mobile devices and the Internet. Along the way we ran into problems with phone and operating system versions, diverse networks, and our own infrastructure, all while working to meet the pent up demand of a waiting list nearly two million people long.

To understand all these problems and how we solved them we first need to give you some background on how the Cloudflare network works:

How Our Network Works

The Cloudflare network is composed of data centers located in 194 cities and in more than 90 countries. Every Cloudflare data center is composed of many servers that receive a continual flood of requests and has to distribute those requests between the servers that handle them. We use a set of routers to perform that operation:

Our routers listen on Anycast IP addresses which are advertised over the public Internet. If you have a site on Cloudflare, your site is available via two of these addresses. In this case, I am doing a DNS query Continue reading

Inside the Web Browser’s Performance API

Building a beautiful, feature-rich website is easier than ever before. Not long ago, you’d have to fire up a text editor and hand-craft a lot of HTML, CSS, and JavaScript. Today, you can use WYSIWYG tools and third-party libraries that make development much simpler. The flip side of this is that it can be hard to see everything that’s going into your website — and the performance can suffer.

The good news is that modern web browsers expose lots of performance data that can help you understand how your web page performs. With the launch of Browser Insights today, we can analyze the performance from the perspective of the web browser and what the end user actually experiences. In this post, we’ll dive into how we think about performance and utilize the timing APIs in the web browser.

How web browsers measure performance

In the old days, the only way for a developer to profile performance was to intercept requests and measure the time from the beginning of the page load until the end of the load event.

Today, we can use Web APIs that are supported by modern browsers. This is part of the web standard called the Performance Continue reading

Introducing Browser Insights

Speed matters. We know that when your website or app gets faster, users have a better experience and you get more conversions and more revenue. At Cloudflare, we spend our days obsessing about speed and building new features to squeeze out as much performance as possible.

But to improve speed, you first need to measure it. That’s why we’re launching Browser Insights: a new tool that measures the performance of your website from the perspective of your users. Browser Insights lets you dive in to understand where, when, and why web pages are slow. And you can enable it today, for free, with one click.

Why did we build Browser Insights?

Let’s say you run an e-commerce site, and you want to make your conversion rates better. You’ve noticed that there’s a lot of traffic from visitors in Peru, but they have worse conversion than users in North America. Maybe you theorize that it takes a long time to load your checkout page, which causes customers to drop off before checking out. How would you verify that this is happening?

There are a few ways you could do this: you could check your server logs to look at timing Continue reading

Cleaning up bad bots (and the climate)

From the very beginning Cloudflare has been stopping malicious bots from scraping websites, or misusing APIs. Over time we’ve improved our bot detection methods and deployed large machine learning models that are able to distinguish real traffic (be it from humans or apps) from malicious bots. We’ve also built a large catalog of good bots to detect things like helpful indexing by search engines.

But it’s not enough. Malicious bots continue to be a problem on the Internet and we’ve decided to fight back. From today customers have the option of enabling “bot fight mode” in their Cloudflare Dashboard.

Once enabled, when we detect a bad bot, we will do three things: (1) we’re going to disincentivize the bot maker economically by tarpitting them, including requiring them to solve a computationally intensive challenge that will require more of their bot’s CPU; (2) for Bandwidth Alliance partners, we’re going to hand the IP of the bot to the partner and get the bot kicked offline; and (3) we’re going to plant trees to make up for the bot’s carbon cost.

Malicious bots harm legitimate web publishers and applications, hurt hosting providers by misusing resources, and they doubly hurt the planet Continue reading

Welcome to Birthday Week 2019

September has always been a special month for Cloudflare. Nine years ago — on September 27th — we launched Cloudflare. And, each year since, we’ve celebrated our birthday with a week full of new products and innovations that support our mission of helping to build a better Internet.

Our mission guides everything we do. One of the most intentional words in our mission is ‘helping’. Building an Internet that can meet the world’s needs cannot be done by any one company or individual; rather, it takes a global community — from nonprofit organizations and businesses to governments and individuals — collaborating to deliver new standards, technologies, and innovations. We believe Cloudflare should be an active participant in the community and help where we can and should.

Our customers and partners are an active part of the community. I often say that customers are one of my favorite parts of my job (our team is my other favorite part). Our customers give us feedback all the time about what they'd like to see to make their Internet properties more secure, more performant and more reliable. Our partners bring forward standards to help make the Internet run more smoothly. For Birthday Week Continue reading

When TCP sockets refuse to die

While working on our Spectrum server, we noticed something weird: the TCP sockets which we thought should have been closed were lingering around. We realized we don't really understand when TCP sockets are supposed to time out!

In our code, we wanted to make sure we don't hold connections to dead hosts. In our early code we naively thought enabling TCP keepalives would be enough... but it isn't. It turns out a fairly modern TCP_USER_TIMEOUT socket option is equally as important. Furthermore it interacts with TCP keepalives in subtle ways. Many people are confused by this.

In this blog post, we'll try to show how these options work. We'll show how a TCP socket can timeout during various stages of its lifetime, and how TCP keepalives and user timeout influence that. To better illustrate the internals of TCP connections, we'll mix the outputs of the tcpdump and the ss -o commands. This nicely shows the transmitted packets and the changing parameters of the TCP connections.

SYN-SENT

Let's start from the simplest case - what happens when one attempts to establish a connection to a server which discards inbound SYN packets?

$  Continue reading

Weekly Wrap: VMware CEO States IBM Paid Too Much for Red Hat

Cloudflare’s Approach to Research

Cloudflare’s mission is to help build a better Internet. One of the tools used in pursuit of this goal is computer science research. We’ve learned that some of the difficult problems to solve are best approached through research and experimentation to understand the solution before engineering it at scale. This research-focused approach to solving the big problems of the Internet is exemplified by the work of the Cryptography Research team, which leverages research to help build a safer, more secure and more performant Internet. Over the years, the team has worked on more than just cryptography, so we’re taking the model we’ve developed and expanding the scope of the team to include more areas of computer science research. Cryptography Research at Cloudflare is now Cloudflare Research. I am excited to share some of the insights we’ve learned over the years in this blog post.

Cloudflare’s research model

How We Design Features for Wrangler, the Cloudflare Workers CLI

The most recent update to Wrangler, version 1.3.1, introduces important new features for developers building Cloudflare Workers — from built-in deployment environments to first class support for Workers KV. Wrangler is Cloudflare’s first officially supported CLI. Branching into this field of software has been a novel experience for us engineers and product folks on the Cloudflare Workers team.

As part of the 1.3.1 release, the folks on the Workers Developer Experience team dove into the thought process that goes into building out features for a CLI and thinking like users. Because while we wish building a CLI were as easy as our teammate Avery tweeted...

… it brings design challenges that many of us have never encountered. To overcome these challenges successfully requires deep empathy for users across the entire team, as well as the ability to address ambiguous questions related to how developers write Workers.

Wrangler, meet Workers KV

Our new KV functionality introduced a host of new features, from creating KV Continue reading

Cloudflare IPO Scorches Wall Street

How Cloudflare and Wall Street Are Helping Encrypt the Internet Today

Today has been a big day for Cloudflare, as we became a public company on the New York Stock Exchange (NYSE: NET). To mark the occasion, we decided to bring our favorite entropy machines to the floor of the NYSE. Footage of these lava lamps is being used as an additional seed to our entropy-generation system LavaRand — bolstering Internet encryption for over 20 million Internet properties worldwide.

(This is mostly for fun. But when’s the last time you saw a lava lamp on the trading floor of the New York Stock Exchange?)

A little context: generating truly random numbers using computers is impossible, because code is inherently deterministic (i.e. predictable). To compensate for this, engineers draw from pools of randomness created by entropy generators, which is a fancy term for "things that are truly unpredictable".

It turns out that lava lamps are fantastic sources of entropy, as was first shown by Silicon Graphics in the 1990s. It’s a torch we’ve been proud to carry forward: today, Cloudflare uses lava lamps to generate entropy that helps make millions of Internet properties more secure.

Housed in our San Francisco headquarters is a wall filled with dozens of lava lamps, Continue reading

A Letter from Matthew Prince and Michelle Zatlyn

To our potential shareholders:

Cloudflare launched on September 27, 2010. Many great startups pivot over time. We have not. We had a plan and have been purposeful in executing it since our earliest days. While we are still in its early innings, that plan remains clear: we are helping to build a better Internet. Understanding the path we’ve taken to date will help you understand how we plan to operate going forward, and to determine whether Cloudflare is the right investment for you.

Cloudflare was formed to take advantage of a paradigm shift: the world was moving from on-premise hardware and software that you buy to services in the cloud that you rent. Paradigm shifts in technology always create significant opportunities, and we built Cloudflare to take advantage of the opportunities that arose as the world shifted to the cloud.

As we watched packaged software turn into SaaS applications, and physical servers migrate to instances in the public cloud, it was clear that it was only a matter of time before the same happened to network appliances. Firewalls, network optimizers, load balancers, and the myriad of other hardware appliances that Continue reading

How Castle is Building Codeless Customer Account Protection

This is a guest post by Johanna Larsson, of Castle, who designed and built the Castle Cloudflare app and the supporting infrastructure.

Strong security should be easy.

Asking your consumers again and again to take responsibility for their security through robust passwords and other security measures doesn’t work. The responsibility of security needs to shift from end users to the companies who serve them.

Castle is leading the way for companies to better protect their online accounts with millions of consumers being protected every day. Uniquely, Castle extends threat prevention and protection for both pre and post login ensuring you can keep friction low but security high. With realtime responses and automated workflows for account recovery, overwhelmed security teams are given a hand. However, when you’re that busy, sometimes deploying new solutions takes more time than you have. Reducing time to deployment was a priority so Castle turned to Cloudflare Workers.

User security and friction

When security is no longer optional and threats are not black or white, security teams are left with trying to determine how to allow end-user access and transaction completions when there are hints of risk, or when not all of the information is available. Continue reading

Fast WordPress Sites with Bluehost & Cloudflare Workers

WordPress is the most popular CMS (content management system) in the world, powering over a third of the top 10 million websites, according to W3Techs.

WordPress is an open source software project that many website service providers host for end customers to enable them to build WordPress sites and serve that content to visitors over the Internet.  For hosting providers, one of the opportunities and challenges is to host one version of WordPress on their infrastructure that is high performing for all their customers without modifying the WordPress code on a per customer basis.

Hosting providers are increasingly turning to Cloudflare’s Serverless Workers Platform to deliver high performance to their end customers by fixing performance issues at the edge while avoiding modifying code on an individual site basis.

One innovative WordPress hosting provider that Cloudflare has been working with to do this is Bluehost, a recommended web host by WordPress.org. In collaboration with Bluehost, Cloudflare’s Workers have been able to achieve a 40% performance improvement for those sites running Workers. Bluehost started with Cloudflare Workers code for Fast Google Fonts which in-lines the browser-specific font CSS and re-hosts the font files through the page origin. This removes the multiple Continue reading

Cloudflare IPO Targets a $483M Haul, $3.5B Valuation

Announcing the General Availability of API Tokens

APIs at Cloudflare

Today we are announcing the general availability of API Tokens - a scalable and more secure way to interact with the Cloudflare API. As part of making a better internet, Cloudflare strives to simplify manageability of a customer’s presence at the edge. Part of the way we do this is by ensuring that all of our products and services are configurable by API. Customers ranging from partners to enterprises to developers want to automate management of Cloudflare. Sometimes that is done via our API directly, and other times it is done via open source software we help maintain like our Terraform provider or Cloudflare-Go library. It is critical that customers who are automating management of Cloudflare can keep their Cloudflare services as secure as possible.

Least Privilege and Why it Matters

Securing software systems is hard. Limiting what a piece of software can do is a good defense to prevent mistakes or malicious actions from having greater impact than they could. The principle of least privilege helps guide how much access a given system should have to perform actions. Originally formulated by Jerome Saltzer, “Every program and every privileged user of the system should operate using Continue reading

How Customer Success Managers Advocate for and Engage with Customers

“Hi! My name is Cosmin and I am your Customer Success Manager here at Cloudflare” is how I usually introduce myself and almost always I get met by a blank stare. It could be the name (it’s uncommon, to say the least) but in actual fact, many Customer Success professionals go through the same experience. Could it be that the title doesn’t give away much? What does ‘Customer Success Manager’ actually mean? Is that how they call Customer Support nowadays? And in fact, isn’t everyone in a business responsible for ‘customer success’?

Well let me explain..

Customer Success is a relatively new function that is becoming increasingly popular with XaaS businesses (XaaS stands for Anything as a Service).  With any XaaS product, it is no longer the case that you make a significant investment in a perpetual license and are left to figure out how to implement it, how to use it and what else it can do. XaaS businesses operate a subscription model whereby if the product is not actually solving your problem, you can just stop the subscription and move on. Businesses operating a subscription model need to ensure that there is continuous and even increasing Continue reading

Supercharging Firewall Events for Self-Serve

Today, I’m very pleased to announce the release of a completely overhauled version of our Firewall Event log to our Free, Pro and Business customers. This new Firewall Events log is now available in your Dashboard, and you are not required to do anything to receive this new capability.

No more modals!

We have done away with those pesky modals, providing a much smoother user experience. To review more detailed information about an event, you simply click anywhere on the event list row.

In the expanded view, you are provided with all the information you may need to identify or diagnose issues with your Firewall or find more details about a potential threat to your application.

Additional matches per event

Cloudflare has several Firewall features to give customers granular control of their security. With this control comes some complexity when debugging why a request was stopped by the Firewall. To help clarify what happened, we have provided an “Additional matches” count at the bottom for events triggered by multiple services or rules for the same request. Clicking the number expands a list showing each rule and service along with the corresponding action.

Search for any field within a Firewall Event

Continue reading

Cybersecurity Tech Accord Adopts Bug Disclosure Policies

Live Preview: Build and Test Workers Faster with Wrangler CLI 1.2.0

As part of my internship on the Workers Developer Experience team, I set out to polish the Wrangler CLI for Cloudflare Workers. If you're not familiar with Workers, the premise is quite simple: Write a bit of Javascript that takes in an HTTP request, does some processing, and spits out a response. The magic lies in where your Workers scripts run: on Cloudflare's edge network, which spans 193 cities in more than 90 countries. Workers can be used for nearly anything from configuring Cloudflare caching behavior to building entire serverless web applications. And, you don't have to worry about operations at all.

I was excited to focus on Wrangler, because Wrangler aims to make developing and publishing Workers projects a pleasant experience for everyone, whether you're a solo dev working on the next big thing, or an engineer at a Fortune 100 enterprise. The whole point of serverless is about reducing friction, and Wrangler reflects that ethos.

However, when I started at Cloudflare in early June, some parts of the development experience still needed some love. While working on a new WASM tutorial for the Workers documentation, I noticed a storm brewing in my browser…

Wrangler lets you test your Continue reading