Pwned Passwords Padding (ft. Lava Lamps and Workers)

The Pwned Passwords API (part of Troy Hunt’s Have I Been Pwned service) is used tens of millions of times each day, to alert users if their credentials are breached in a variety of online services, browser extensions and applications. Using Cloudflare, the API cached around 99% of requests, making it very efficient to run.

From today, we are offering a new security advancement in the Pwned Passwords API - API clients can receive responses padded with random data. This exists to effectively protect from any potential attack vectors which seek to use passive analysis of the size of API responses to identify which anonymised bucket a user is querying. I am hugely grateful to security researcher Matt Weir who I met at PasswordsCon in Stockholm and has explored proof-of-concept analysis of unpadded API responses in Pwned Passwords and has driven some of the work to consider the addition of padded responses.

Now, by passing a header of “Add-Padding” with a value of “true”, Pwned Passwords API users are able to request padded API responses (to a minimum of 800 entries with additional padding of a further 0-200 entries). The padding consists of randomly generated hash suffixes with the usage Continue reading

Why I’m Excited to Join Cloudflare as its First CIO

I am delighted to share that I have joined Cloudflare as its first Chief Information Officer to help scale the company in this new phase of its business. It’s an incredibly exciting time to be joining Cloudflare, and I am grateful for the opportunity to do my part to help build a better Internet.

At one of my previous companies, I made a bet on Cloudflare to equip us with security and performance solutions across a very decentralized global set of products and services. This is something that would have been very difficult without a cloud solution like Cloudflare’s. Since then I’ve been watching Cloudflare grow, and have always been very impressed by the speed of innovation and transparency, but also how Cloudflare operates: doing the right thing, with integrity, and above all building trust with customers and partners. The “do the right thing, even if it’s hard” mentality that I saw from Cloudflare since I started doing business with them as a customer, was key for me. When I heard that Cloudflare was looking for its first CIO I was excited to have a discussion to see if I could help.

During the interview process I got a sense Continue reading

RPKI and the RTR protocol

Today’s Internet requires stronger protection within its core routing system and as we have already said: it's high time to stop BGP route leaks and hijacks by deploying operationally-excellent RPKI!

Luckily, over the last year plus a lot of good work has happened in this arena. If you’ve been following the growth of RPKI’s validation data, then you’ll know that more and more networks are signing their routes and creating ROA’s or Route Origin Authorizations. These are cryptographically-signed assertions of the validity of an announced IP block and contribute to the further securing of the global routing table that makes for a safer Internet.

The protocol that we have not written much about is RTR. The Resource Public Key Infrastructure (RPKI) to Router Protocol - or RTR Protocol for short. Today we’re fixing that.

RPKI rewind

We have written a few times about RPKI (here and here). We have written about how Cloudflare both signs its announced routes and filters its routing inbound from other networks (both transits and peers) using RPKI data. We also added our efforts in the open-source software space with the release of the Cloudflare RPKI Toolkit.

The primary part of the RPKI (Resource Continue reading

Addressing the Web’s Client-Side Security Challenge

Modern web architecture relies heavily on JavaScript and enabling third-party code to make client-side network requests. These innovations are built on client-heavy frameworks such as Angular, Ember, React, and Backbone that leverage the processing power of the browser to enable the execution of code directly on the client interface/web browser. These third-party integrations provide richness (chat tools, images, fonts) or extract analytics (Google Analytics). Today, up to 70% of the code executing and rendering on your customer’s browser comes from these integrations. All of these software integrations provide avenues for potential vulnerabilities.

Unfortunately, these unmanaged, unmonitored integrations operate without security consideration, providing an expansive attack surface that attackers have routinely exploited to compromise websites. Today, only 2% of the Alexa 1000 global websites were found to deploy client-side security measures to protect websites and web applications against attacks such as Magecart, XSS, credit card skimming, session redirects and website defacement.

Improving website security and ensuring performance with Cloudflare Workers

In this post, we focus on how Cloudflare Workers can be used to improve security and ensure the high performance of web applications. Tala has joined Cloudflare’s marketplace to further our common goals of ensuring website security, preserving data privacy and Continue reading

When Bloom filters don’t bloom

I've known about Bloom filters (named after Burton Bloom) since university, but I haven't had an opportunity to use them in anger. Last month this changed - I became fascinated with the promise of this data structure, but I quickly realized it had some drawbacks. This blog post is the tale of my brief love affair with Bloom filters.

While doing research about IP spoofing, I needed to examine whether the source IP addresses extracted from packets reaching our servers were legitimate, depending on the geographical location of our data centers. For example, source IPs belonging to a legitimate Italian ISP should not arrive in a Brazilian datacenter. This problem might sound simple, but in the ever-evolving landscape of the internet this is far from easy. Suffice it to say I ended up with many large text files with data like this:

This reads as: the IP 192.0.2.1 was recorded reaching Cloudflare data center number 107 with a legitimate request. This data came from many sources, including our active and passive probes, logs of certain domains we own (like cloudflare.com), public sources (like BGP table), etc. The same line would usually be repeated across multiple Continue reading

Technical Details of Why Cloudflare Chose AMD EPYC for Gen X Servers

From the very beginning Cloudflare used Intel CPU-based servers (and, also, Intel components for things like NICs and SSDs). But we're always interested in optimizing the cost of running our service so that we can provide products at a low cost and high gross margin.

We're also mindful of events like the Spectre and Meltdown vulnerabilities and have been working with outside parties on research into mitigation and exploitation which we hope to publish later this year.

We looked very seriously at ARM-based CPUs and continue to keep our software up to date for the ARM architecture so that we can use ARM-based CPUs when the requests per watt is interesting to us.

In the meantime, we've deployed AMD's EPYC processors as part of Gen X server platform and for the first time are not using any Intel components at all. This week, we announced details of this tenth generation of servers. Below is a recap of why we're excited about the design, specifications, and performance of our newest hardware.

Servers for an Accelerated Future

Every server can run every service. This architectural decision has helped us achieve higher efficiency across the Cloudflare network. It has also given us more Continue reading

Going Beyond Black History Month

Around this time of year in the United States, African-Americans are often tasked with explaining why we spend 28 (or in the case of a leap year 29) days celebrating the contributions our ancestors made to this country. It may come in the form of responding to ignorant questions posed in learning environments or expressed in well-crafted articles lauding the relevancy of Black history in our modern time.

Black history is not only relevant, it is how we ensure that our heroes are not forgotten and that we have a viable future in our respective industries. As Carter G. Woodson famously said, “If a race has no history, if it has no worthwhile tradition, it becomes a negligible factor in the thought of the world, and it stands in danger of being exterminated.”

As the US leaders of Afroflare, Cloudflare’s employee resource group (ERG) for employees of African descent, we made a personal commitment this month and beyond to effectively represent, build, and grow at Cloudflare and in the tech industry.

To honor that commitment, we decided to tackle some commonly asked questions about the state of African-Americans in tech.

How many African-Americans work in tech?

The latest Continue reading

Securing Memory at EPYC Scale

Security is a serious business, one that we do not take lightly at Cloudflare. We have invested a lot of effort into ensuring that our services, both external and internal, are protected by meeting or exceeding industry best practices. Encryption is a huge part of our strategy as it is embedded in nearly every process we have. At Cloudflare, we encrypt data both in transit (on the network) and at rest (on the disk). Both practices address some of the most common vectors used to exfiltrate information and these measures serve to protect sensitive data from attackers but,  what about data currently in use?

Can encryption or any technology eliminate all threats? No, but as Infrastructure Security, it’s our job to consider worst-case scenarios. For example, what if someone were to steal a server from one of our data centers? How can we leverage the most reliable, cutting edge, innovative technology to secure all data on that host if it were in the wrong hands? Would it be protected? And, in particular, what about the server’s RAM?

Data in random access memory (RAM) is usually stored in the clear. This can leave data vulnerable to software or hardware probing by Continue reading

Gen X Performance Tuning

We are using AMD 2nd Gen EPYC 7642 for our tenth generation “Gen X” servers. We found many aspects of this processor compelling such as its increase in performance due to its frequency bump and cache-to-core ratio. We have partnered with AMD to get the best performance out of this processor and today, we are highlighting our tuning efforts that led to an additional 6% performance.

Thermal Design Power & Dynamic Power

Thermal design power (TDP) and dynamic power, amongst others, play a critical role when tuning a system. Many share a common belief that thermal design power is the maximum or average power drawn by the processor. The 48-core AMD EPYC 7642 has a TDP rating of 225W which is just as high as the 64-core AMD EPYC 7742. It comes to mind that fewer cores should translate into lower power consumption, so why is the AMD EPYC 7642 expected to draw just as much power as the AMD EPYC 7742?

Let’s take a step back and understand that TDP does not always mean the maximum or average power that the processor will draw. At a glance, Continue reading

Introducing Secrets and Environment Variables to Cloudflare Workers

The Workers team here at Cloudflare has been hard at work shipping a bunch of new features in the last year and we’ve seen some amazing things built with the tools we’ve provided. However, as my uncle once said, with great serverless platform growth comes great responsibility.

One of the ways we can help is by ensuring that deploying and maintaining your Workers scripts is a low risk endeavor. Rotating a set of API keys shouldn’t require risking downtime through code edits and redeployments and in some cases it may not make sense for the developer writing the script to know the actual API key value at all. To help tackle this problem, we’re releasing Secrets and Environment Variables to the Wrangler CLI and Workers Dashboard.

Supporting secrets

As we started to design support for secrets in Workers we had a sense that this was already a big concern for a lot of our users but we wanted to learn about all of the use cases to ensure we were building the right thing. We headed to the community forums, twitter, and the inbox of Louis Grace, business development representative extraordinaire, for some anecdotes about Secrets usage. We also sent Continue reading

Impact of Cache Locality

In the past, we didn't have the opportunity to evaluate as many CPUs as we do today. The hardware ecosystem was simple – Intel had consistently delivered industry leading processors. Other vendors could not compete with them on both performance and cost. Recently it all changed: AMD has been challenging the status quo with their 2nd Gen EPYC processors.

This is not the first time that Intel has been challenged; previously there was Qualcomm, and we worked with AMD and considered their 1st Gen EPYC processors and based on the original Zen architecture, but ultimately, Intel prevailed. AMD did not give up and unveiled their 2nd Gen EPYC processors codenamed Rome based on the latest Zen 2 architecture.

This made many improvements over its predecessors. Improvements include a die shrink from 14nm to 7nm, a doubling of the top end core count from 32 to 64, and a larger L3 cache size. Let’s emphasize again on the size of that L3 cache, which is 32 MiB L3 cache per Core Complex Die (CCD).

This time around, we have taken steps to Continue reading

An EPYC trip to Rome: AMD is Cloudflare’s 10th-generation Edge server CPU

More than 1 billion unique IP addresses pass through the Cloudflare Network each day, serving on average 11 million HTTP requests per second and operating within 100ms of 95% of the Internet-connected population globally. Our network spans 200 cities in more than 90 countries, and our engineering teams have built an extremely fast and reliable infrastructure.

We’re extremely proud of our work and are determined to help make the Internet a better and more secure place. Cloudflare engineers who are involved with hardware get down to servers and their components to understand and select the best hardware to maximize the performance of our stack.

Our software stack is compute intensive and is very much CPU bound, driving our engineers to work continuously at optimizing Cloudflare’s performance and reliability at all layers of our stack. With the server, a straightforward solution for increasing computing power is to have more CPU cores. The more cores we can include in a server, the more output we can expect. This is important for us since the diversity of our products and customers has grown over time with increasing demand that requires our servers to do more. To help us drive compute performance, we needed Continue reading

Seamless remote work with Cloudflare Access

The novel coronavirus is actively changing how organizations work in real-time. According to Fortune, the virus has led to the “world’s largest work-from-home experiment.” As the epidemic crosses borders, employees are staying home and putting new stress on how companies manage remote work.

This is only accelerating an existing trend, however. Remote work has gained real traction in the last decade and Gartner projects that it will only continue. However, teams which are moving to a distributed model tend to do so slowly. When those timelines are accelerated, IT and security administrators need to be able to help their workforce respond without disrupting their team members.

Cloudflare Access can help teams migrate to a model that makes it seamless for users to work from any location, or any device, without the need for lengthy migrations or onboarding sessions. Cloudflare Access can be deployed in less than one hour and bring SaaS-like convenience and speed to the self-hosted applications that previously lived behind a VPN.

Leaving the castle-and-moat

When users share a physical space, working on a private network is easy. Users do not need clunky VPN clients to connect to the resources they need. Team members physically sit close Continue reading

Cloudflare’s Gen X: Servers for an Accelerated Future

“Every server can run every service.”

We designed and built Cloudflare’s network to be able to grow capacity quickly and inexpensively; to allow every server, in every city, to run every service; and to allow us to shift customers and traffic across our network efficiently. We deploy standard, commodity hardware, and our product developers and customers do not need to worry about the underlying servers. Our software automatically manages the deployment and execution of our developers’ code and our customers’ code across our network. Since we manage the execution and prioritization of code running across our network, we are both able to optimize the performance of our highest tier customers and effectively leverage idle capacity across our network.

An alternative approach might have been to run several fragmented networks with specialized servers designed to run specific features, such as the Firewall, DDoS protection or Workers. However, we believe that approach would have resulted in wasted idle resources and given us less flexibility to build new software or adopt the newest available hardware. And a single optimization target means we can provide security and performance at the same time.

We use Anycast to route a web request to the Continue reading

Using your devices as the key to your apps

I keep a very detailed budget. I have for the last 7 years. I manually input every expense into a spreadsheet app and use a combination of sumifs functions to track spending.

Opening the spreadsheet app, and then the specific spreadsheet, every time that I want to submit an expense is a little clunky. I'm working on a new project to make that easier. I'm building a simple web app, with a very basic form, into which I will enter one-off expenses. This form will then append those expenses as rows into the budget workbook.

I want to lock down this project; I prefer that I am the only person with the power to wreck my budget. To do that, I'm going to use Cloudflare Access. With Access, I can require a login to reach the page - no server-side changes required.

Except, I don't want to allow logins from any device. For this project, I want to turn my iPhone into the only device that can reach this app.

To do that, I'll use Cloudflare Access in combination with an open source toolkit from Cloudflare, cfssl. Together, I can convert my device into a secure key for this application Continue reading

Multi-SSO and Cloudflare Access: Adding LinkedIn and GitHub Teams

Cloudflare Access secures internal applications without the hassle, slowness or user headache of a corporate VPN. Access brings the experience we all cherish, of being able to access web sites anywhere, any time from any device, to the sometimes dreary world of corporate applications. Teams can integrate the single sign-on (SSO) option, like Okta or AzureAD, that they’ve chosen to use and in doing so make on-premise or self-managed cloud applications feel like SaaS apps.

However, teams consist of more than just the internal employees that share an identity provider. Organizations work with partners, freelancers, and contractors. Extending access to external users becomes a constant chore for IT and security departments and is a source of security problems.

Cloudflare Access removes that friction by simultaneously integrating with multiple identity providers, including popular services like Gmail or GitHub that do not require corporate subscriptions. External users login with these accounts and still benefit from the same ease-of-use available to internal employees. Meanwhile, administrators avoid the burden in legacy deployments that require onboarding and offboarding new accounts for each project.

We are excited to announce two new integrations that make it even easier for organizations to work securely with third parties. Starting Continue reading

The Serverlist: Globally Distributed Websites, $16M Series A, and more

Check out our twelfth edition of The Serverlist below. Get the latest scoop on the serverless space, get your hands dirty with new developer tutorials, engage in conversations with other serverless developers, and find upcoming meetups and conferences to attend.

Sign up below to have The Serverlist sent directly to your mailbox.

Vetflare, Cloudflare’s Military Veteran Employee Group Launches

“Diversity leads to better outcomes… better decisions, increased innovation, stronger financial returns, and a great place to work for everyone” said Janet Van Huysse, Head of People at Cloudflare during our Q1-2020 kickoff. Veterans, people who have served in the military, are a vital element of a diverse workforce. We come in diverse shapes, sizes, colors, genders, and orientations. We bring diverse skillsets, experiences, and perspectives.  

If you haven’t served in the military and haven’t worked with many veterans, here are some of the things that you can expect from your colleagues or direct reports that are veterans.

Veterans know what it means to SERVE. Indeed, it is a truism that living in service to others is a life well-lived, and that service to others is a foundation of esprit de corps. Though relatively few of us have seen combat, we have all signed a blank check to our nation made payable for any amount, up to and including our lives. This is what it means to become part of something bigger than oneself. This translates to putting our common shared interests ahead of our personal interests even when that means becoming an instrument of a foreign policy we Continue reading

JAMstack at the Edge: How we built Built with Workers… on Workers

I'm extremely stoked to announce Built with Workers today – it's an awesome resource for exploring what you can build with Cloudflare Workers. As Adam explained in our launch post, showcasing developers building incredible projects with tools like Workers KV or our streaming HTML rewriter is a great way to celebrate users of our platform. It also helps encourage developers to try building their dream app on top of Workers. In this post, I’ll explore some of the architectural and implementation designs we made while building the site.

When we first started planning Built with Workers, we wanted to use the site as an opportunity to build a new greenfield application, showcasing the strength of the Workers platform. The Workers Developer Experience team is cross-functional: while we might spend most of our time improving our docs, or developing features for our command-line interface Wrangler, most of us have spent years developing on the web. The prospect of starting a new application is always fun, but in this instance, it was a prime chance to ask (and answer) the question, "If I could build this site on Workers with whatever tools I want, what would I choose?"

A guiding Continue reading

Announcing Built with Workers

Ever since its initial release, Cloudflare Workers has given JavaScript developers a platform to enable building high-performance applications with automatic scaling.

As with any new technology, we know it can be a bit intimidating to get started. For one thing, running code on the edge is a paradigm shift—forcing us to rethink classic web architecture problems, or removing them altogether. For another, since you can build just about anything, it can be challenging to figure out what to build first.

Today we’re launching Built with Workers, a new site designed to help get those creative juices flowing and unblock you, by answering that simple but important question: What can I build with Cloudflare Workers?

Some time in 1999, at age 11, I received my first graphing calculator. It was a TI-82 that my older sister no longer needed. It was on this very calculator that I learned to write code. Looking back, I’m not sure how exactly I had the patience or sanity to figure it all out.

It was a mess. Among the many difficulties were that I had to type the code out on the calculator’s non-QWERTY keyboard, the language I was writing in didn’t have Continue reading