John Graham-Cumming

Author Archives: John Graham-Cumming

Cloudflare em Lisboa

Cloudflare em Lisboa

Eu fui o 24º funcionário da Cloudflare e o primeiro a trabalhar fora de São Francisco. A trabalhar num escritório improvisado em minha casa, e escrevi um pedaço grande do software da Cloudflare antes de ter contratato uma equipa em Londres. Hoje, Cloudflare London, a nossa a sede da EMEA a região da Europa, Médio Oriente e África tem mais de 200 pessoas a trabalhar no edifício histórico County Hall á frente do Parlamento Britânico. O meu escritório improvisado é agora história antiga.

Cloudflare em Lisboa
CC BY-SA 2.0 image by Sridhar Saraf

Cloudflare não parou em Londres. Temos pessoas em Munique, Singapura, Pequim, Austin, Texas, Chicago e Champaign, Illinois, Nova York, Washington,DC, São José, Califórnia, Miami, Florida, Sydney, Austrália e também em Sao Francisco e Londres. Hoje estamos a anunciar o estabelecimento de um novo escritório em Lisboa, Portugal. Como parte da abertura do escritório este Verão irei me deslocar para Lisboa juntamente com um pequeno número de pessoal técnico de outros escritórios da Cloudflare.

Estamos a recrutar em Lisboa neste momento. Pode visitar este link para ver todas as oportunidades actuais. Estamos á procura de candidatos para preencher os cargos de Engenheiro, Segurança, Produto, Produto de Estratégia, Investigação Tecnológica e Continue reading

Cloudflare’s new Lisbon office

Cloudflare's new Lisbon office

I was the 24th employee of Cloudflare and the first outside of San Francisco. Working out of my spare bedroom, I wrote a chunk of Cloudflare’s software before starting to recruit a team in London. Today, Cloudflare London, our EMEA headquarters, has more than 200 people working in the historic County Hall building opposite the Houses of Parliament. My spare bedroom is ancient history.

Cloudflare's new Lisbon office
CC BY-SA 2.0 image by Sridhar Saraf

And Cloudflare didn’t stop at London. We now have people in Munich, Singapore, Beijing, Austin, TX, Chicago and Champaign, IL, New York, Washington, DC, San Jose, CA, Miami, FL, and Sydney, Australia, as well as San Francisco and London. And today we’re announcing the establishment of a new technical hub in Lisbon, Portugal. As part of that office opening I will be relocating to Lisbon this summer along with a small number of technical folks from other Cloudflare offices.

We’re recruiting in Lisbon starting today. Go here to see all the current opportunities. We’re looking for people to fill roles in Engineering, Security, Product, Product Strategy, Technology Research, and Customer Support.

Cloudflare's new Lisbon office
CC BY-SA 2.0 Image by Rustam Aliyev

My first real idea of Lisbon dates to 30 Continue reading

Details of the Cloudflare outage on July 2, 2019

Almost nine years ago, Cloudflare was a tiny company and I was a customer not an employee. Cloudflare had launched a month earlier and one day alerting told me that my little site, jgc.org, didn’t seem to have working DNS any more. Cloudflare had pushed out a change to its use of Protocol Buffers and it had broken DNS.

I wrote to Matthew Prince directly with an email titled “Where’s my dns?” and he replied with a long, detailed, technical response (you can read the full email exchange here) to which I replied:

From: John Graham-Cumming
Date: Thu, Oct 7, 2010 at 9:14 AM
Subject: Re: Where's my dns?
To: Matthew Prince

Awesome report, thanks. I'll make sure to call you if there's a
problem.  At some point it would probably be good to write this up as
a blog post when you have all the technical details because I think
people really appreciate openness and honesty about these things.
Especially if you couple it with charts showing your post launch
traffic increase.

I have pretty robust monitoring of my sites so I get an SMS when
anything fails.  Monitoring shows I was down from 13:03:07  Continue reading

The Network is the Computer: A Conversation with John Gage

The Network is the Computer: A Conversation with John Gage
The Network is the Computer: A Conversation with John Gage

To learn more about the origins of The Network is the Computer®, I spoke with John Gage, the creator of the phrase and the 21st employee of Sun Microsystems. John had a key role in shaping the vision of Sun and had a lot to share about his vision for the future. Listen to our conversation here and read the full transcript below.


[00:00:13]

John Graham-Cumming: I’m talking to John Gage who was what, the 21st employee of Sun Microsystems, which is what Wikipedia claims and it also claims that you created this phrase “The Network is the Computer,” and that's actually one of the things I want to talk about with you a little bit because I remember when I was in Silicon Valley seeing that slogan plastered about the place and not quite understanding what it meant. So do you want to tell me what you meant by it or what Sun meant by it at the time?

[00:00:40]

John Gage: Well, in 2019, recalling what it meant in 1982 or 83’ will be colored by all our experience since then but at the time it seemed so obvious that when we introduced the first scientific workstations, they Continue reading

The Network is the Computer: A Conversation with Ray Rothrock

The Network is the Computer: A Conversation with Ray Rothrock
The Network is the Computer: A Conversation with Ray Rothrock

Last week I spoke with Ray Rothrock, former Director of CAD/CAM Marketing at Sun Microsystems, to discuss his time at Sun and how the Internet has evolved. In this conversation, Ray discusses the importance of trust as a principle, the growth of Sun in sales and marketing, and that time he gave Vice President Bush a Sun demo. Listen to our conversation here and read the full transcript below.

[00:00:07]

John Graham-Cumming: Here I am very lucky to get to talk with Ray Rothrock who was I think one of the first investors in Cloudflare, a Series A investor and got the company a little bit of money to get going, but if we dial back a few earlier years than that, he was also at Sun as the Director of CAD/CAM Marketing. There is a link between Sun and Cloudflare. At least one, but probably more than one, which is that Cloudflare has recently trademarked, “The Network is the Computer”. And that was a Sun trademark, wasn’t it?

[00:00:43]

Ray Rothrock: It was, yes.

[00:00:46]

Graham-Cumming: I talked to John Gage and I asked him about this as well and I asked him to explain to me what it Continue reading

The Network is the Computer: A Conversation with Greg Papadopoulos

The Network is the Computer: A Conversation with Greg Papadopoulos
The Network is the Computer: A Conversation with Greg Papadopoulos

I spoke with Greg Papadopoulos, former CTO of Sun Microsystems, to discuss the origins and meaning of The Network is the Computer®, as well as Cloudflare’s role in the evolution of the phrase. During our conversation, we considered the inevitability of latency, the slowness of the speed of light, and the future of Cloudflare’s newly acquired trademark. Listen to our conversation here and read the full transcript below.


[00:00:08]

John Graham-Cumming: Thank you so much for taking the time to chat with me. I've got Greg Papadopoulos who was CTO of Sun and is currently a venture capitalist. Tell us about “The Network is the Computer.”

[00:00:22]

Greg Papadopoulos: Well, from certainly a Sun perspective, the very first Sun-1 was connected via Internet protocols and at that time there was a big war about what should win from a networking point of view. And there was a dedication there that everything that we made was going to interoperate on the network over open standards, and from day one in the company, it was always that thought. It's really about the collection of these machines and how they interact with one another, and of course that puts the network in Continue reading

The Network is the Computer

The Network is the Computer
The Network is the Computer

We recently registered the trademark for The Network is the Computer®, to encompass how Cloudflare is utilizing its network to pave the way for the future of the Internet.

The phrase was first coined in 1984 by John Gage, the 21st employee of Sun Microsystems, where he was credited with building Sun’s vision around “The Network is the Computer.” When Sun was acquired in 2010, the trademark was not renewed, but the vision remained.

Take it from him:

“When we built Sun Microsystems, every computer we made had the network at its core. But we could only imagine, over thirty years ago, today’s billions of networked devices, from the smallest camera or light bulb to the largest supercomputer, sharing their packets across Cloudflare’s distributed global network.
We based our vision of an interconnected world on open and shared standards. Cloudflare extends this dedication to new levels by openly sharing designs for security and resilience in the post-quantum computer world.
Most importantly, Cloudflare is committed to immediate, open, transparent accountability for network performance. I’m a dedicated reader of their technical blog, as the network becomes central to our security infrastructure and the global economy, demanding even more powerful technical innovation. Continue reading

Cloudflare outage caused by bad software deploy (updated)

This is a short placeholder blog and will be replaced with a full post-mortem and disclosure of what happened today.

For about 30 minutes today, visitors to Cloudflare sites received 502 errors caused by a massive spike in CPU utilization on our network. This CPU spike was caused by a bad software deploy that was rolled back. Once rolled back the service returned to normal operation and all domains using Cloudflare returned to normal traffic levels.

This was not an attack (as some have speculated) and we are incredibly sorry that this incident occurred. Internal teams are meeting as I write performing a full post-mortem to understand how this occurred and how we prevent this from ever occurring again.


Update at 2009 UTC:

Starting at 1342 UTC today we experienced a global outage across our network that resulted in visitors to Cloudflare-proxied domains being shown 502 errors (“Bad Gateway”). The cause of this outage was deployment of a single misconfigured rule within the Cloudflare Web Application Firewall (WAF) during a routine deployment of new Cloudflare WAF Managed rules.

The intent of these new rules was to improve the blocking of inline JavaScript that is used in attacks. These rules were Continue reading

Cloudflare outage caused by bad software deploy (updated)

This is a short placeholder blog and will be replaced with a full post-mortem and disclosure of what happened today.

For about 30 minutes today, visitors to Cloudflare sites received 502 errors caused by a massive spike in CPU utilization on our network. This CPU spike was caused by a bad software deploy that was rolled back. Once rolled back the service returned to normal operation and all domains using Cloudflare returned to normal traffic levels.

This was not an attack (as some have speculated) and we are incredibly sorry that this incident occurred. Internal teams are meeting as I write performing a full post-mortem to understand how this occurred and how we prevent this from ever occurring again.


Update at 2009 UTC:

Starting at 1342 UTC today we experienced a global outage across our network that resulted in visitors to Cloudflare-proxied domains being shown 502 errors (“Bad Gateway”). The cause of this outage was deployment of a single misconfigured rule within the Cloudflare Web Application Firewall (WAF) during a routine deployment of new Cloudflare WAF Managed rules.

The intent of these new rules was to improve the blocking of inline JavaScript that is used in attacks. These rules were Continue reading

Helping To Build Cloudflare, Part 6: What does Cloudflare’s CTO do?

This is the final part of a six part series based on a talk I gave in Trento, Italy. To start from the beginning go here.

If you are still awake there’s really one final question that you might want to know the answer to: What does the CTO do? The reality is that it means different things in different companies. But I can tell you a little about what I do.

The longest temporary job

I didn’t join Cloudflare as CTO. My original job title was Programmer and for the first couple of years I did just that. I wrote a piece of technology called Railgun (a differential compression program used to speed up the connection between Cloudflare and origin web servers) and then I went on to write our WAF. After that I worked on our Go-based DNS server and other parts of the stack.

At some point Lee Holloway decided he didn’t want to manage Cloudflare’s growing staff and Michelle Zatlyn (one of Cloudflare’s founders) asked me if I would ‘temporarily’ manage engineering. This is now the longest temporary job I’ve ever had!

Initially a lot of what I did was manage the team and help interview Continue reading

Helping To Build Cloudflare, Part 5: People: Finding, Nurturing and Learning to Let Go

This is part 5 of a six part series based on a talk I gave in Trento, Italy. To start from the beginning go here.

So, let me talk a bit about people. Software is made by people. Sometimes individuals but more likely by teams. I’ve talked earlier about some aspects of our architecture and our frequent rewrites but it’s people that make all that work.

And, honestly, people can be an utter joy and a total pain. Finding, keeping, nurturing people and teams is the single most important thing you can do in a company. No doubt.

Finding People

Finding people is really hard. Firstly, the technology industry is booming, and so engineers have a lot of choices. Countries create special visas just for them. Politicians line up to create mini-Silicon Valleys in their countries. Life is good!

But the really hard thing is interviewing. How do you find good people from an interview? I don’t know the answer to that. We put people through on average 8 interviews and a pair programming exercise. We look at open source contributions. Sometimes we look at people’s degrees.

We tend to look for potential. An old boss used to say, “Don’t Continue reading

Helping To Build Cloudflare, Part 4: Public Engagement

This is part 4 of a six part series based on a talk I gave in Trento, Italy. To start from the beginning go here.

We don’t believe that any of our software, not a single line of code, provides us with a long-term advantage. We could, today, open source every single line of code at Cloudflare and we don’t believe we’d be hurt by it.

How we think about Open Source

Why don’t we? We actually do open source a lot of code, but we try to be thoughtful about it. Firstly, a lot of our code is so Cloudflare-specific, full of logic about how our service works, that it’s not generic enough for someone else to pick up and use for their service. So, for example, open sourcing the code that runs our web front end would be largely useless.‌‌

But other bits of software are generic. There’s currently a debate going on internally about a piece of software called Quicksilver. I mentioned before that Cloudflare used a distributed key-value store to send configuration to machines across the world. We used to use an open source project called Kyoto Tycoon. It was pretty cool.‌‌

But Continue reading

Helping To Build Cloudflare, Part 2: The Most Difficult Two Weeks

This is part 2 of a six part series based on a talk I gave in Trento, Italy. Part 1 is here.

It’s always best to speak plainly and honestly about the situation you are in. Or as Matthew Prince likes to put it “Panic Early”. Long ago I started a company in Silicon Valley which had the most beautiful code. We could have taught a computer science course from the code base. But we had hardly any customers and we failed to “Panic Early” and not face up to the fact that our market was too small.

Ironically, the CEO of that company used to tell people “Get bad news out fast”. This is a good maxim to live by, if you have bad news then deliver it quickly and clearly. If you don’t the bad news won’t go away, and the situation will likely get worse.

Cloudbleed

Cloudflare had a very, very serious security problem back in 2017. This problem became known as Cloudbleed. We had, without knowing it, been leaking memory from inside our machines into responses returned to web browsers. And because our machines are shared across millions of web sites, that meant that HTTP requests Continue reading

Helping To Build Cloudflare, Part 3: Audacity, Diversity and Change

This is part 3 of a six part series based on a talk I gave in Trento, Italy. To start from the beginning go here.

After Cloudbleed, lots of things changed. We started to move away from memory-unsafe languages like C and C++ (there’s a lot more Go and Rust now). And every SIGABRT or crash on any machine results in an email to me and a message to the team responsible. And I don’t let the team leave those problems to fester.

Making 1.1.1.1

So Cloudbleed was a terrible time. Let’s talk about a great time. The launch of our public DNS resolver 1.1.1.1. That launch is a story of an important Cloudflare quality: audacity. Google had launched 8.8.8.8 years ago and had taken the market for a public DNS resolver by storm. Their address is easy to remember, their service is very fast.‌‌

But we thought we could do better. We thought we could be faster, and we thought we could be more memorable. Matthew asked us to get the address 1.1.1.1 and launch a secure, privacy-preserving, public DNS resolver in a couple of months. Continue reading

Helping To Build Cloudflare, Part 1: How I came to work here

This is the text I prepared for a talk at Speck&Tech in Trento, Italy. I thought it might make a good blog post. Because it is 6,000 words I've split it into six separate posts.

Here's part 1:

I’ve worked at Cloudflare for more than seven years. Cloudflare itself is more than eight years old. So, I’ve been there since it was a very small company. About twenty people in fact. All of those people (except one, me) worked from an office in San Francisco. I was the lone member of the London office.

Today there are 900 people working at Cloudflare spread across offices in San Francisco, Austin, Champaign IL, New York, London, Munich, Singapore and Beijing. In London, my “one-person office” (which was my spare bedroom) is now almost 200 people and in a month, we’ll move into new space opposite Big Ben.

The original Cloudflare London "office"

The numbers tell a story about enormous growth. But it’s growth that’s been very carefully managed. We could have grown much faster (in terms of people); we’ve certainly raised enough money to do so.

I ended up at Cloudflare because I gave a really good talk at a conference. Well, Continue reading

Cloudflare’s network boosts security and performance for IBM Cloud customers

Cloudflare’s network boosts security and performance for IBM Cloud customers

Today our partner IBM® announced the general availability of Cloud Internet Services (CIS) Enterprise. It marks a significant step forward in the partnership that we announced at the IBM THINK event in March.

CIS delivers security and performance to IBM Cloud® customers’ internet applications. It brings together Cloudflare’s 150+ points of presence with IBM Cloud’s 60 data centers, stopping attacks before they can even reach the IBM Cloud. CIS Enterprise is integrated into IBM Cloud, allowing IBM Cloud customers to set up and manage Cloudflare’s DDoS mitigation, web application firewall, smart routing and highly customizable load balancer, all from within the IBM Cloud user interface.  

Cloudflare’s network boosts security and performance for IBM Cloud customers

Our Network Map (as of 10/18/18). Click here for the latest version

We thought it timely to give a refresher on how Cloudflare’s network supports IBM Cloud customers. The network is designed to meet requirements of the most demanding enterprise customers. It is based on an architecture that differentiates it from legacy CDN, DNS and DDoS-mitigation services to ensure that internet applications stay online, even in the face of extremely high volume attacks or legitimate traffic spikes.

Cloudflare’s network of data centers, distributed across 74 countries (including 22 in China), has a network Continue reading

Free to code

This week at the Cloudflare Internet Summit I have the honour of sitting down and talking with Sophie Wilson. She designed the very first ARM processor instruction set in the mid-1980s and was part of the small team that built the foundations for the mobile world we live in: if you are reading this on a mobile device, like a phone or tablet, it almost certainly has an ARM processor in it.

But, despite the amazing success of ARM, it’s not the processor that I think of when I think of Sophie Wilson. It’s the BBC Micro, the first computer I ever owned. And it’s the computer on which Wilson and others created ARM despite it having just an 8-bit 6502 processor and 32k of RAM.

Luckily, I still own that machine and recently plugged it into a TV set and turned it on to make sure it was still working 36 years on (you can read about that one time blue smoke came out of it and my repair). I wanted to experience once more the machine Sophie Wilson helped to design. One vital component of that machine was BBC BASIC, stored in a ROM chip on Continue reading

Statement concerning events at Glowbeam Technologies

All of Cloudflare's staff were shocked at the events depicted in NCIS Season 16 Episode 1 where incorrect use of random numbers for encryption resulted in the insertion of multiple trojan horses that brought a nuclear reactor within seconds of a meltdown.

Although Cloudflare has long been a competitor of the company responsible, Glowbeam Technologies, and uses similar random number generation technology, we would like to emphasize that there are significant differences between the two companies.

Firstly, Cloudflare's Lava Lamps are not an "encryption engine" and thus they are not susceptible to tampering by the janitor.

Secondly, all Cloudflare staff undergo extensive background checks.

Thirdly, we were shocked that Glowbeam Technologies' wall of Lava Lamps was a single point of failure. In contrast, Cloudflare uses multiple sources of randomness.

Lastly, Glowbeam Technologies' CEO confirmed that the company did not use "AES" or "key block ciphers", but instead relied solely on their Lava Lamp "encryption engine". Cloudflare strongly advocates for never writing or inventing encryption algorithms and works closely with groups like the IETF to use standard, well understood encryption.

As a result of these events Cloudflare has acquired the assets of Glowbeam Technologies, please visit glowbeamtechnologies.com for more information.

John Graham-Cumming
Chief Technology Officer
Cloudflare, Inc.

Introducing Workers KV

Introducing Workers KV

In 1864 British computer pioneer Charles Babbage described the first key-value store. It was meant to be part of his Analytical Engine. Sadly, the Analytical Engine, which would have been the first programmable computer, was never built. But Babbage lays out clearly the design for his key-value store in his autobiography. He imagined a read-only store implemented as punched cards. He referred to these as Tables:

I explained that the Tables to be used must, of course, be computed and punched on cards by the machine, in which case they would undoubtedly be correct. I then added that when the machine wanted a tabular number, say the logarithm of a given number, that it would ring a bell and then stop itself. On this, the attendant would look at a certain part of the machine, and find that it wanted the logarithm of a given number, say of 2303. The attendant would then go to the drawer containing the pasteboard cards representing its table of logarithms. From amongst these he would take the required logarithmic card, and place it in the machine.

Introducing Workers KV

Punched card illustration from Babbage’s autobiography showing an integer key (2303) and value representing the decimal part of Continue reading

The QUICening

The QUICening

Six o’clock already, I was just in the middle of a dream, now I’m up, awake, looking at my Twitter stream. As I do that the Twitter app is making multiple API calls over HTTPS to Twitter’s servers somewhere on the Internet.

Those HTTPS connections are running over TCP via my home WiFi and broadband connection. All’s well inside the house, the WiFi connection is interference free thanks to my eero system, the broadband connection is stable and so there’s no packet loss, and my broadband provider’s connection to Twitter’s servers is also loss free.

The QUICening

Those are the perfect conditions for HTTPS running over TCP. Not a packet dropped, not a bit of jitter, no congestion. It’s even the perfect conditions for HTTP/2 where multiple streams of requests and responses are being sent from my phone to websites and APIs as I boot my morning. Unlike HTTP/1.1, HTTP/2 is able to use a single TCP connection for multiple, simultaneously in flight requests. That has a significant speed advantage over the old way (one request after another per TCP connection) when conditions are good.

But I have to catch an early train, got to be to work by nine, so Continue reading

1 2 3 4