Kenton Varda

Author Archives: Kenton Varda

Backwards-compatibility in Cloudflare Workers

Backwards-compatibility in
Cloudflare Workers
Backwards-compatibility in
Cloudflare Workers

Cloudflare Workers is our serverless platform that runs your code in 250+ cities worldwide.

On the Workers team, we have a policy:

A change to the Workers Runtime must never break an application that is live in production.

It seems obvious enough, but this policy has deep consequences. What if our API has a bug, and some deployed Workers accidentally depend on that bug? Then, seemingly, we can't fix the bug! That sounds… bad?

This post will dig deeper into our policy, explaining why Workers is different from traditional server stacks in this respect, and how we're now making backwards-incompatible changes possible by introducing "compatibility dates".

TL;DR: Developers may now opt into backwards-incompatible fixes by setting a compatibility date.

Serverless demands strict compatibility

Workers is a serverless platform, which means we maintain the server stack for you. You do not have to manage the runtime version, you only manage your own code. This means that when we update the Workers Runtime, we update it for everyone. We do this at least once a week, sometimes more.

This means that if a runtime upgrade breaks someone's application, it's really bad. The developer didn't make any change, so won't be watching for Continue reading

Dynamic Process Isolation: Research by Cloudflare and TU Graz

Dynamic Process Isolation: Research by Cloudflare and TU Graz
Dynamic Process Isolation: Research by Cloudflare and TU Graz

Last year, I wrote about the Cloudflare Workers security model, including how we fight Spectre attacks. In that post, I explained that there is no known complete defense against Spectre — regardless of whether you're using isolates, processes, containers, or virtual machines to isolate tenants. What we do have, though, is a huge number of tools to increase the cost of a Spectre attack, to the point where it becomes infeasible. Cloudflare Workers has been designed from the very beginning with protection against side channel attacks in mind, and because of this we have been able to incorporate many defenses that other platforms — such as virtual machines and web browsers — cannot. However, the performance and scalability requirements of edge compute make it infeasible to run every Worker in its own private process, so we cannot rely on the usual defenses provided by the operating system kernel and address space separation.

Given our different approach, we cannot simply rely on others to tell us if we are safe. We had to do our own research. To do this we partnered with researchers at Graz University of Technology (TU Graz) to study the impact of Spectre on our environment. The Continue reading

Durable Objects: Easy, Fast, Correct — Choose three

Durable Objects: Easy, Fast, Correct — Choose three
Durable Objects: Easy, Fast, Correct — Choose three

Storage in distributed systems is surprisingly hard to get right. Distributed databases and consensus are well-known to be extremely hard to build. But, application code isn't necessarily easy either. There are many ways in which apps that use databases can have subtle timing bugs that could result in inconsistent results, or even data loss. Worse, these problems can be very hard to test for, as they'll often manifest only under heavy load, or only after a sudden machine failure.

Up until recently, Durable Objects were no exception. A Durable Object is a special kind of Cloudflare Worker that has access to persistent storage and processes requests in one of Cloudflare’s points of presence. Each Object has its own private storage, accessible through a classical key/value storage API. Like any classical database API, this storage API had to be used carefully to avoid possible race conditions and data loss, especially when performance mattered. And like any classical database API, many apps got it wrong.

However, rather than fix the apps, we decided to fix the model. Last month, we rolled out deep changes to the Durable Objects runtime such that many applications which previously contained subtle race conditions are now correct Continue reading

Containers at the edge: it’s not what you think, or maybe it is

Containers at the edge: it’s not what you think, or maybe it is
Containers at the edge: it’s not what you think, or maybe it is

At Cloudflare, we’re committed to making it as easy as possible for developers to make their ideas come to life. Our announcements this week aim to give developers all the tools they need to build their next application on the edge. These include things like static site hosting, certificate management, and image services, just to name a few.

Today, we’re thrilled to announce that we’re exploring a new type of service at the edge: containers.

This announcement will be exciting to some and surprising to many. On this very blog, we’ve talked about why we believe isolates — rather than containers on the edge — will be the future model for applications on the web.

Containers at the edge: it’s not what you think, or maybe it is

Isolates are best for Distributed Systems

Let us be clear: isolates are the best way to do edge compute, period. The Workers platform is designed to allow developers to treat our global network as one big computer. This has been a long-held dream of generations of engineers, inspiring slogans like "The Network is the Computer" — a trademark which, incidentally, we now own. Isolates and Durable Objects are finally making that vision possible.

In short, isolates excel at distributed systems. They are perfect for Continue reading

Workers Durable Objects Beta: A New Approach to Stateful Serverless

Workers Durable Objects Beta:
A New Approach to Stateful Serverless
Workers Durable Objects Beta:
A New Approach to Stateful Serverless

We launched Cloudflare Workers® in 2017 with a radical vision: code running at the network edge could not only improve performance, but also be easier to deploy and cheaper to run than code running in a single datacenter. That vision means Workers is about more than just edge compute -- we're rethinking how applications are built.

Using a "serverless" approach has allowed us to make deploys dead simple, and using isolate technology has allowed us to deliver serverless more cheaply and without the lengthy cold starts that hold back other providers. We added easy-to-use eventually-consistent edge storage to the platform with Workers KV.

But up until today, it hasn't been possible to manage state with strong consistency, or to coordinate in real time between multiple clients, entirely on the edge. Thus, these parts of your application still had to be hosted elsewhere.

Durable Objects provide a truly serverless approach to storage and state: consistent, low-latency, distributed, yet effortless to maintain and scale. They also provide an easy way to coordinate between clients, whether it be users in a particular chat room, editors of a particular document, or IoT devices in a particular smart home. Durable Objects are the missing piece Continue reading

Workers Security

Workers Security

Workers Security
Hello, I'm an engineer on the Workers team, and today I want to talk to you about security.

Cloudflare is a security company, and the heart of Workers is, in my view, a security project. Running code written by third parties is always a scary proposition, and the primary concern of the Workers team is to make that safe.

For a project like this, it is not enough to pass a security review and say "ok, we're secure" and move on. It's not even enough to consider security at every stage of design and implementation. For Workers, security in and of itself is an ongoing project, and that work is never done. There are always things we can do to reduce the risk and impact of future vulnerabilities.

Today, I want to give you an overview of our security architecture, and then address two specific issues that we are frequently asked about: V8 bugs, and Spectre.

Architectural Overview

Let's start with a quick overview of the Workers Runtime architecture.

Workers Security

There are two fundamental parts of designing a code sandbox: secure isolation and API design.

Isolation

First, we need to create an execution environment where code can't access anything it's not Continue reading

WebAssembly on Cloudflare Workers

WebAssembly on Cloudflare Workers

WebAssembly on Cloudflare Workers

We just announced ten major new products and initiatives over Crypto Week and Birthday Week, but our work is never finished. We're continuously upgrading our existing products with new functionality.

Today, we're extending Cloudflare Workers with support for WebAssembly. All Workers customers can now augment their applications with WASM at no additional cost.

What is WebAssembly?

WebAssembly -- often abbreviated as "WASM" -- is a technology that extends the web platform to support compiled languages like C, C++, Rust, Go, and more. These languages can be compiled to a special WASM binary format and then loaded in a browser.

WASM code is securely sandboxed, just like JavaScript. But, because it is based on compiled lower-level languages, it can be much faster for certain kinds of resource-intensive tasks where JavaScript is not a good fit. In addition to performance benefits, WASM allows you to reuse existing code written in languages other than JavaScript.

What are Workers?

WebAssembly on Cloudflare Workers

For those that don't know: Cloudflare Workers lets you deploy "serverless" JavaScript code directly to our 153-and-growing datacenters. Your Worker handles your site's HTTP traffic directly at the location closest to your end user, allowing you to achieve lower latency and reduce serving costs. Continue reading

Everyone can now run JavaScript on Cloudflare with Workers

Everyone can now run JavaScript on Cloudflare with Workers

Everyone can now run JavaScript on Cloudflare with Workers

Exactly one year ago today, Cloudflare gave me a mission: Make it so people can run code on Cloudflare's edge. At the time, we didn't yet know what that would mean. Would it be container-based? A new Turing-incomplete domain-specific language? Lua? "Functions"? There were lots of ideas.

Eventually, we settled on what now seems the obvious choice: JavaScript, using the standard Service Workers API, running in a new environment built on V8. Five months ago, we gave you a preview of what we were building, and started the beta.

Today, with thousands of scripts deployed and many billions of requests served, Cloudflare Workers is now ready for everyone.

"Moving away from VCL and adopting Cloudflare Workers will allow us to do some creative routing that will let us deliver JavaScript to npm's millions of users even faster than we do now. We will be building our next generation of services on Cloudflare's platform and we get to do it in JavaScript!"

— CJ Silverio, CTO, npm, Inc.

What is the Cloud, really?

Historically, web application code has been split between servers and browsers. Between them lies a vast but fundamentally dumb network which merely ferries data from point to Continue reading

How to Monkey-Patch the Linux Kernel

How to Monkey-Patch the Linux Kernel

I have a weird setup. I type in Dvorak. But, when I hold ctrl or alt, my keyboard reverts to Qwerty.

You see, the classic text-editing hotkeys, ctrl+Z, ctrl+X, ctrl+C, and ctrl+V are all located optimally for a Qwerty layout: next to the control key, easy to reach with your left hand while mousing with your right. In Dvorak, unfortunately, these hotkeys are scattered around mostly on the right half of the keyboard, making them much less convenient. Using Dvorak for typing but Qwerty for hotkeys turns out to be a nice compromise.

But, the only way I could find to make this work on Linux / X was to write a program that uses X "grabs" to intercept key events and rewrite them. That was mostly fine, until recently, when my machine, unannounced, updated to Wayland. Remarkably, I didn't even notice at first! But at some point, I realized my hotkeys weren't working right. You see, Wayland, unlike X, actually has some sensible security rules, and as a result, random programs can't just man-in-the-middle all keyboard events anymore. Which broke my setup.

Yes, that's right, I'm that guy:

How to Monkey-Patch the Linux Kernel

Source: xkcd 1172

So what was I to do? I began Continue reading

Introducing Cloudflare Workers: Run Javascript Service Workers at the Edge

TL;DR: You'll soon be able to deploy Javascript to Cloudflare's edge, written against an API similar to Service Workers.

Try writing a Worker in the playground »

Introduction

Every technology, when sufficiently complicated, becomes programmable.

You see this everywhere, but as a lifelong gamer, my personal favorite example is probably graphics cards. In the '90s, graphics hardware generally provided a fixed set of functionality. The OpenGL standard specified that the geometry pipeline would project points from 3D space onto your viewport, then the raster pipeline would draw triangles between them, with gradient shading and perhaps a texture applied. You could only use one texture at a time. There was only one lighting algorithm, which more or less made every surface look like plastic. If you wanted to do anything else, you often had to give up the hardware entirely and drop back to software.

Of course, new algorithms and techniques were being developed all the time. So, hardware vendors would add the best ideas to their hardware as "extensions". OpenGL ended up with hundreds of vendor-specific extensions to support things like multi-texturing, bump maps, reflections, dynamic shadows, and more.

Then, in 2001, everything changed. The first GPU with a programmable Continue reading