Fifty Years Ago

This is a guest post by Steve Crocker of Shinkuro, Inc. and Bill Duvall of Consulair. Fifty years ago they were both present when the first packets flowed on the Arpanet.

On 29 October 2019, Professor Leonard (“Len”) Kleinrock is chairing a celebration at the University of California, Los Angeles (UCLA).  The date is the fiftieth anniversary of the first full system test and remote host-to-host login over the Arpanet.  Following a brief crash caused by a configuration problem, a user at UCLA was able to log in to the SRI SDS 940 time-sharing system.  But let us paint the rest of the picture.

The Arpanet was a bold project to connect sites within the ARPA-funded computer science research community and to use packet-switching as the technology for doing so.  Although there were parallel packet-switching research efforts around the globe, none were at the scale of the Arpanet project. Cooperation among researchers in different laboratories, applying multiple machines to a single problem and sharing of resources were all part of the vision.  And over the fifty years since then, the vision has been fulfilled, albeit with some undesired outcomes mixed in with the enormous benefits.  However, in this blog, we Continue reading

Supporting the latest version of the Privacy Pass Protocol

At Cloudflare, we are committed to supporting and developing new privacy-preserving technologies that benefit all Internet users. In November 2017, we announced server-side support for the Privacy Pass protocol, a piece of work developed in collaboration with the academic community. Privacy Pass, in a nutshell, allows clients to provide proof of trust without revealing where and when the trust was provided. The aim of the protocol is then to allow anyone to prove they are trusted by a server, without that server being able to track the user via the trust that was assigned.

On a technical level, Privacy Pass clients receive attestation tokens from a server, that can then be redeemed in the future. These tokens are provided when a server deems the client to be trusted; for example, after they have logged into a service or if they prove certain characteristics. The redeemed tokens are cryptographically unlinkable to the attestation originally provided by the server, and so they do not reveal anything about the client.

To use Privacy Pass, clients can install an open-source browser extension available in Chrome & Firefox. There have been over 150,000 individual downloads of Privacy Pass worldwide; approximately 130,000 in Chrome and Continue reading

Tales from the Crypt(o team)

Halloween season is upon us. This week we’re sharing a series of blog posts about work being done at Cloudflare involving cryptography, one of the spookiest technologies around. So bookmark this page and come back every day for tricks, treats, and deep technical content.

A long-term mission

Cryptography is one of the most powerful technological tools we have, and Cloudflare has been at the forefront of using cryptography to help build a better Internet. Of course, we haven’t been alone on this journey. Making meaningful changes to the way the Internet works requires time, effort, experimentation, momentum, and willing partners. Cloudflare has been involved with several multi-year efforts to leverage cryptography to help make the Internet better.

Here are some highlights to expect this week:

• We’re renewing Cloudflare’s commitment to privacy-enhancing technologies by sharing some of the recent work being done on Privacy Pass
• We’re helping forge a path to a quantum-safe Internet by sharing some of the results of the Post-quantum Cryptography experiment
• We’re sharing the rust-based software we use to power time.cloudflare.com
• We’re doing a deep dive into the technical details of Encrypted DNS
• We’re announcing support for a new technique we developed with industry partners Continue reading

Public keys are not enough for SSH security

If your organization uses SSH public keys, it’s entirely possible you have already mislaid one. There is a file sitting in a backup or on a former employee’s computer which grants the holder access to your infrastructure. If you share SSH keys between employees it’s likely only a few keys are enough to give an attacker access to your entire system. If you don’t share them, it’s likely your team has generated so many keys you long lost track of at least one.

If an attacker can breach a single one of your client devices it’s likely there is a known_hosts file which lists every target which can be trivially reached with the keys the machine already contains. If someone is able to compromise a team member’s laptop, they could use keys on the device that lack password protection to reach sensitive destinations.

Should that happen, how would you respond and revoke the lost SSH key? Do you have an accounting of the keys which have been generated? Do you rotate SSH keys? How do you manage that across an entire organization so consumed with serving customers that security has to be effortless to be adopted?

Cloudflare Access launched support Continue reading

Cloudflare response to CPDoS exploits

Three vulnerabilities were disclosed as Cache Poisoning Denial of Service attacks in a paper written by Hoai Viet Nguyen, Luigi Lo Iacono, and Hannes Federrath of TH Köln - University of Applied Sciences. These attacks are similar to the cache poisoning attacks presented last year at DEFCON.

Most customers do not have to take any action to protect themselves from the newly disclosed vulnerabilities. Some configuration changes are recommended if you are a Cloudflare customer running unpatched versions of Microsoft IIS and have request filtering enabled on your origin or b) have forced caching of HTTP response code 400 through the use of page rules or Cloudflare Workers.

We have not seen any attempted exploitation of the vulnerabilities described in this paper.

Maintaining the integrity of our content caching infrastructure and ensuring our customers are able to quickly and reliably serve the content they expect to their visitors is of paramount importance to us. In practice, Cloudflare ensures caches serve the content they should in two ways:

1. We build our caching infrastructure to behave in ways compliant with industry standards.
2. We actively add defenses to our caching logic to protect customers from common caching pitfalls. We see our job as Continue reading

Who DDoS’d Austin?

It was a scorching Monday on July 22 as temperatures soared above 37°C (99°F) in Austin, TX, the live music capital of the world. Only hours earlier, the last crowds dispersed from the historic East 6th Street entertainment district. A few blocks away, Cloudflarians were starting to make their way to the office. Little did those early arrivers know that they would soon be unknowingly participating in a Cloudflare time honored tradition of dogfooding new services before releasing them to the wild.

6th East Street, Austin Texas

Dogfooding is when an organization uses its own products. In this case, we dogfed our newest cloud service, Magic Transit, which both protects and accelerates our customers’ entire network infrastructure—not just their web properties or TCP/UDP applications. With Magic Transit, Cloudflare announces your IP prefixes via BGP, attracts (routes) your traffic to our global network edge, blocks bad packets, and delivers good packets to your data centers via Anycast GRE.

We decided to use Austin’s network because we wanted to test the new service on a live network with real traffic from real people and apps. Continue reading

Experiment with HTTP/3 using NGINX and quiche

Just a few weeks ago we announced the availability on our edge network of HTTP/3, the new revision of HTTP intended to improve security and performance on the Internet. Everyone can now enable HTTP/3 on their Cloudflare zone and experiment with it using Chrome Canary as well as curl, among other clients.

We have previously made available an example HTTP/3 server as part of the quiche project to allow people to experiment with the protocol, but it’s quite limited in the functionality that it offers, and was never intended to replace other general-purpose web servers.

We are now happy to announce that our implementation of HTTP/3 and QUIC can be integrated into your own installation of NGINX as well. This is made available as a patch to NGINX, that can be applied and built directly with the upstream NGINX codebase.

It’s important to note that this is not officially supported or endorsed by the NGINX project, it is just something that we, Cloudflare, want to make available to the wider community to help push adoption of QUIC and HTTP/3.

Building

The first step is to download and unpack the NGINX source code. Note that the HTTP/3 and QUIC Continue reading

It’s crowded in here!

We recently gave a presentation on Programming socket lookup with BPF at the Linux Plumbers Conference 2019 in Lisbon, Portugal. This blog post is a recap of the problem statement and proposed solution we presented.

Our edge servers are crowded. We run more than a dozen public facing services, leaving aside the all internal ones that do the work behind the scenes.

Quick Quiz #1: How many can you name? We blogged about them! Jump to answer.

These services are exposed on more than a million Anycast public IPv4 addresses partitioned into 100+ network prefixes.

To keep things uniform every Cloudflare edge server runs all services and responds to every Anycast address. This allows us to make efficient use of the hardware by load-balancing traffic between all machines. We have shared the details of Cloudflare edge architecture on the blog before.

Granted not all services work on all the addresses but rather on a subset of them, covering one or several network prefixes.

So how do you set up your network services to listen on hundreds of IP addresses without driving the network stack over the edge?

Cloudflare engineers have had to ask themselves this question Continue reading

What’s it like to come out as LGBTQIA+ at work?

Today is the 31st Anniversary of National Coming Out Day. I wanted to highlight the importance of this day, share coming out resources, and publish some stories of what it's like to come out in the workplace.

About National Coming Out Day

Thirty-one years ago, on the anniversary of the National March on Washington for Lesbian and Gay Rights, we first observed National Coming Out Day as a reminder that one of our most basic tools is the power of coming out. One out of every two Americans has someone close to them who is gay or lesbian. For transgender people, that number is only one in 10.

Coming out - whether it is as lesbian, gay, bisexual, transgender or queer - STILL MATTERS. When people know someone who is LGBTQ, they are far more likely to support equality under the law. Beyond that, our stories can be powerful to each other.

Each year on October 11th, National Coming Out Day continues to promote a safe world for LGBTQ individuals to live truthfully and openly. Every person who speaks up changes more hearts and minds, and creates new advocates for equality.

For more on coming out, visit HRC's Coming Out Continue reading

Good Morning, Jakarta!

Beneath the veneer of glass and concrete, this is a city of surprises and many faces. On 3rd October 2019, we brought together a group of leaders from across a number of industries to connect in Central Jakarta, Indonesia.

The habit of sharing stories at the lunch table, exchanging ideas, and listening to ideas from the different viewpoints of people from all tiers, paying first-hand attention to all input from customers, and listening to the dreams of some of life’s warriors may sound simple but it is a source of inspiration and encouragement in helping the cyberspace community in this region.

And our new data center in Jakarta extends our Asia Pacific network to 64 cities, and our global network to 194 cities.

Selamat Pagi

Right on time, Kate Fleming extended a warm welcome to our all our Indonesia guests. "We were especially appreciative of the investment of your time that you made coming to join us."

Kate, is the Head of Customer Success for APAC. Australian-born, Kate spent the past 5 years living in Malaysia and Singapore. She leads a team of Customer Success Managers in Singapore. The Customer Success team is dispersed across multiple offices and time Continue reading

Terraforming Cloudflare: in quest of the optimal setup

This is a guest post by Dimitris Koutsourelis and Alexis Dimitriadis, working for the Security Team at Workable, a company that makes software to help companies find and hire great people.

This post is about our introductive journey to the infrastructure-as-code practice; managing Cloudflare configuration in a declarative and version-controlled way. We’d like to share the experience we’ve gained during this process; our pain points, limitations we faced, different approaches we took and provide parts of our solution and experimentations.

Terraform world

Terraform is a great tool that fulfills our requirements, and fortunately, Cloudflare maintains its own provider that allows us to manage its service configuration hasslefree.

On top of that, Terragrunt, is a thin wrapper that provides extra commands and functionality for keeping Terraform configurations DRY, and managing remote state.

The combination of both leads to a more modular and re-usable structure for Cloudflare resources (configuration), by utilizing terraform and terragrunt modules.
We’ve chosen to use the latest version of both tools (Terraform-v0.12 & Terragrunt-v0.19 respectively) and constantly upgrade to take advantage of the valuable new features and functionality, which at this point in time, remove important limitations.

Workable context

Our set up includes Continue reading

Talk Transcript: How Cloudflare Thinks About Security

This is the text I used for a talk at artificial intelligence powered translation platform, Unbabel, in Lisbon on September 25, 2019.

Bom dia. Eu sou John Graham-Cumming o CTO do Cloudflare. E agora eu vou falar em inglês.

Thanks for inviting me to talk about Cloudflare and how we think about security. I’m about to move to Portugal permanently so I hope I’ll be able to do this talk in Portuguese in a few months.

I know that most of you don’t have English as a first language so I’m going to speak a little more deliberately than usual. And I’ll make the text of this talk available for you to read.

But there are no slides today.

I’m going to talk about how Cloudflare thinks about internal security, how we protect ourselves and how we secure our day to day work. This isn’t a talk about Cloudflare’s products.

Culture

Let’s begin with culture.

Many companies have culture statements. I think almost 100% of these are pure nonsense. Culture is how you act every day, not words written in the wall.

One significant piece of company culture is the internal Security Incident mailing list Continue reading

Money Moves: September 2019

SDxCentral’s Top 10 Articles — September 2019

Serverlist Sept. Wrap-up: Static sites, serverless costs, and more

Check out our eighth edition of The Serverlist below. Get the latest scoop on the serverless space, get your hands dirty with new developer tutorials, engage in conversations with other serverless developers, and find upcoming meetups and conferences to attend.

Sign up below to have The Serverlist sent directly to your mailbox.

Learn more about Workers Sites at Austin & San Francisco Meetups

Last Friday, at the end of Cloudflare’s 9th birthday week, we announced Workers Sites.

Now, using the Wrangler CLI, you can deploy entire websites directly to the Cloudflare Network using Cloudflare Workers and Workers KV. If you can statically generate the assets for your site, think create-react-app, Jekyll, or even the WP2Static plugin, you can deploy it to our global network, which spans 194 cities in more than 90 countries.

If you’d like to learn more about how it was built, you can read more about this in the technical blog post. Additionally, I wanted to give you an opportunity to meet with some of the developers who contributed to this product and hear directly from them about their process, potential use cases, and what it took to build.

Check out these events. If you’re based in Austin or San Francisco (more cities coming soon!), join us on-site. If you’re based somewhere else, you can watch the recording of the events afterwards.

Growing Dev Platforms at Scale & Deploying Static Websites

Talk 1: Inspiring with Content: How to Grow Developer Platforms at Scale

Serverless platforms like Cloudflare Workers provide benefits like scalability, high performance, and lower costs. However, Continue reading

Not so static… Introducing the HTMLRewriter API Beta to Cloudflare Workers

Today, we’re excited to announce HTMLRewriter beta — a streaming HTML parser with an easy to use selector based JavaScript API for DOM manipulation, available in the Cloudflare Workers runtime.

For those of you who are unfamiliar, Cloudflare Workers is a lightweight serverless platform that allows developers to leverage Cloudflare’s network to augment existing applications or create entirely new ones without configuring or maintaining infrastructure.

Static Sites to Dynamic Applications

On Friday we announced Workers Sites: a static site deployment workflow built into the Wrangler CLI tool. Now, paired with the HTML Rewriter API, you can perform DOM transformations on top of your static HTML, right on the Cloudflare edge.

You could previously do this by ingesting the entire body of the response into the Worker, however, that method was prone to introducing a few issues. First, parsing a large file was bound to run into memory or CPU limits. Additionally, it would impact your TTFB as the body could no longer be streamed, and the browser would be prevented from doing any speculative parsing to load subsequent assets.

HTMLRewriter was the missing piece to having your application fully live on the edge – soup to nuts. You can Continue reading

¡Bienvenidos a Latinflare!

Our Story

When I first began interviewing with Cloudflare in the Spring of 2019, I came across a Cloudflare blog post announcing Proudflare, the company’s LGBTQIA+ Employee Resource Group (ERG). The post gave me a clear sense of the company’s commitment to diversity and inclusion. I could tell this was a place that values and celebrates diversity, which really appealed to me as I progressed through the interview process with Cloudflare, and ultimately accepted the role.

Fast forward to my Cloudflare new hire orientation, two weeks of training and introductions at our San Francisco HQ. We learned about the various ERGs at Cloudflare including one for Latinx employees. While I had a strong desire to be part of a Latinx ERG, it was clear that the group was actually in need of someone to lead the effort and rally the troops. At Cloudflare, we have offices across the country and around the world. I wasn’t really sure how to launch an ERG that would be global in scope. After meeting with leads from other Cloudflare ERGs, understanding the landscape, and attending an external workshop, everything started to come together.

In early August, we officially gave ourselves Continue reading

Cloudflare’s protection against a new Remote Code Execution vulnerability (CVE-2019-16759) in vBulletin

Cloudflare has released a new rule as part of its Cloudflare Specials Rulesets, to protect our customers against a high-severity vulnerability in vBulletin.  

A new zero-day vulnerability was discovered for vBulletin, a proprietary Internet forum software. By exploiting this vulnerability, bad actors could potentially gain privileged access and control to the host servers on which this software runs, through Remote Code Execution (RCE).

Implications of this vulnerability

At Cloudflare, we use three key indicators to understand the severity of a vulnerability 1) how many customers on Cloudflare are running the affected software 2) the Common Vulnerability Scoring System (CVSS) score, and 3) the OWASP Top 10, an open-source security framework.

We assess this vulnerability to be very significant as it has a CVSS score of 9.8/10 and affects 7 out of the 10 key risk areas of the OWASP 2017 Top 10.

Remote Code Execution is considered a type of injection, which provides the capability to potentially launch a catastrophic attack. Through RCE an attacker can gain privileged access to the host server that might be running the unpatched and vulnerable version of this software. With elevated privileges the attacker could perform malicious activities including discovery of additional Continue reading

Birthday Week 2019 Wrap-up

This week we celebrated Cloudflare’s 9th birthday by launching a variety of new offerings that support our mission: to help build a better Internet.  Below is a summary recap of how we celebrated Birthday Week 2019.

Cleaning up bad bots

Every day Cloudflare protects over 20 million Internet properties from malicious bots, and this week you were invited to join in the fight!  Now you can enable “bot fight mode” in the Firewall settings of the Cloudflare Dashboard and we’ll start deploying CPU intensive code to traffic originating from malicious bots.  This wastes the bots’ CPU resources and makes it more difficult and costly for perpetrators to deploy malicious bots at scale. We’ll also share the IP addresses of malicious bot traffic with our Bandwidth Alliance partners, who can help kick malicious bots offline. Join us in the battle against bad bots – and, as you can read here – you can help the climate too!

Browser Insights

Speed matters, and if you manage a website or app, you want to make sure that you’re delivering a high performing website to all of your global end users. Now you can enable Browser Insights in the Speed section of the Continue reading