Gen X Performance Tuning

We are using AMD 2nd Gen EPYC 7642 for our tenth generation “Gen X” servers. We found many aspects of this processor compelling such as its increase in performance due to its frequency bump and cache-to-core ratio. We have partnered with AMD to get the best performance out of this processor and today, we are highlighting our tuning efforts that led to an additional 6% performance.

Thermal Design Power & Dynamic Power

Thermal design power (TDP) and dynamic power, amongst others, play a critical role when tuning a system. Many share a common belief that thermal design power is the maximum or average power drawn by the processor. The 48-core AMD EPYC 7642 has a TDP rating of 225W which is just as high as the 64-core AMD EPYC 7742. It comes to mind that fewer cores should translate into lower power consumption, so why is the AMD EPYC 7642 expected to draw just as much power as the AMD EPYC 7742?

Let’s take a step back and understand that TDP does not always mean the maximum or average power that the processor will draw. At a glance, Continue reading

Introducing Secrets and Environment Variables to Cloudflare Workers

The Workers team here at Cloudflare has been hard at work shipping a bunch of new features in the last year and we’ve seen some amazing things built with the tools we’ve provided. However, as my uncle once said, with great serverless platform growth comes great responsibility.

One of the ways we can help is by ensuring that deploying and maintaining your Workers scripts is a low risk endeavor. Rotating a set of API keys shouldn’t require risking downtime through code edits and redeployments and in some cases it may not make sense for the developer writing the script to know the actual API key value at all. To help tackle this problem, we’re releasing Secrets and Environment Variables to the Wrangler CLI and Workers Dashboard.

Supporting secrets

As we started to design support for secrets in Workers we had a sense that this was already a big concern for a lot of our users but we wanted to learn about all of the use cases to ensure we were building the right thing. We headed to the community forums, twitter, and the inbox of Louis Grace, business development representative extraordinaire, for some anecdotes about Secrets usage. We also sent Continue reading

Impact of Cache Locality

In the past, we didn't have the opportunity to evaluate as many CPUs as we do today. The hardware ecosystem was simple – Intel had consistently delivered industry leading processors. Other vendors could not compete with them on both performance and cost. Recently it all changed: AMD has been challenging the status quo with their 2nd Gen EPYC processors.

This is not the first time that Intel has been challenged; previously there was Qualcomm, and we worked with AMD and considered their 1st Gen EPYC processors and based on the original Zen architecture, but ultimately, Intel prevailed. AMD did not give up and unveiled their 2nd Gen EPYC processors codenamed Rome based on the latest Zen 2 architecture.

This made many improvements over its predecessors. Improvements include a die shrink from 14nm to 7nm, a doubling of the top end core count from 32 to 64, and a larger L3 cache size. Let’s emphasize again on the size of that L3 cache, which is 32 MiB L3 cache per Core Complex Die (CCD).

This time around, we have taken steps to Continue reading

An EPYC trip to Rome: AMD is Cloudflare’s 10th-generation Edge server CPU

More than 1 billion unique IP addresses pass through the Cloudflare Network each day, serving on average 11 million HTTP requests per second and operating within 100ms of 95% of the Internet-connected population globally. Our network spans 200 cities in more than 90 countries, and our engineering teams have built an extremely fast and reliable infrastructure.

We’re extremely proud of our work and are determined to help make the Internet a better and more secure place. Cloudflare engineers who are involved with hardware get down to servers and their components to understand and select the best hardware to maximize the performance of our stack.

Our software stack is compute intensive and is very much CPU bound, driving our engineers to work continuously at optimizing Cloudflare’s performance and reliability at all layers of our stack. With the server, a straightforward solution for increasing computing power is to have more CPU cores. The more cores we can include in a server, the more output we can expect. This is important for us since the diversity of our products and customers has grown over time with increasing demand that requires our servers to do more. To help us drive compute performance, we needed Continue reading

Seamless remote work with Cloudflare Access

The novel coronavirus is actively changing how organizations work in real-time. According to Fortune, the virus has led to the “world’s largest work-from-home experiment.” As the epidemic crosses borders, employees are staying home and putting new stress on how companies manage remote work.

This is only accelerating an existing trend, however. Remote work has gained real traction in the last decade and Gartner projects that it will only continue. However, teams which are moving to a distributed model tend to do so slowly. When those timelines are accelerated, IT and security administrators need to be able to help their workforce respond without disrupting their team members.

Cloudflare Access can help teams migrate to a model that makes it seamless for users to work from any location, or any device, without the need for lengthy migrations or onboarding sessions. Cloudflare Access can be deployed in less than one hour and bring SaaS-like convenience and speed to the self-hosted applications that previously lived behind a VPN.

Leaving the castle-and-moat

When users share a physical space, working on a private network is easy. Users do not need clunky VPN clients to connect to the resources they need. Team members physically sit close Continue reading

Cloudflare’s Gen X: Servers for an Accelerated Future

“Every server can run every service.”

We designed and built Cloudflare’s network to be able to grow capacity quickly and inexpensively; to allow every server, in every city, to run every service; and to allow us to shift customers and traffic across our network efficiently. We deploy standard, commodity hardware, and our product developers and customers do not need to worry about the underlying servers. Our software automatically manages the deployment and execution of our developers’ code and our customers’ code across our network. Since we manage the execution and prioritization of code running across our network, we are both able to optimize the performance of our highest tier customers and effectively leverage idle capacity across our network.

An alternative approach might have been to run several fragmented networks with specialized servers designed to run specific features, such as the Firewall, DDoS protection or Workers. However, we believe that approach would have resulted in wasted idle resources and given us less flexibility to build new software or adopt the newest available hardware. And a single optimization target means we can provide security and performance at the same time.

We use Anycast to route a web request to the Continue reading

Using your devices as the key to your apps

I keep a very detailed budget. I have for the last 7 years. I manually input every expense into a spreadsheet app and use a combination of sumifs functions to track spending.

Opening the spreadsheet app, and then the specific spreadsheet, every time that I want to submit an expense is a little clunky. I'm working on a new project to make that easier. I'm building a simple web app, with a very basic form, into which I will enter one-off expenses. This form will then append those expenses as rows into the budget workbook.

I want to lock down this project; I prefer that I am the only person with the power to wreck my budget. To do that, I'm going to use Cloudflare Access. With Access, I can require a login to reach the page - no server-side changes required.

Except, I don't want to allow logins from any device. For this project, I want to turn my iPhone into the only device that can reach this app.

To do that, I'll use Cloudflare Access in combination with an open source toolkit from Cloudflare, cfssl. Together, I can convert my device into a secure key for this application Continue reading

Multi-SSO and Cloudflare Access: Adding LinkedIn and GitHub Teams

Cloudflare Access secures internal applications without the hassle, slowness or user headache of a corporate VPN. Access brings the experience we all cherish, of being able to access web sites anywhere, any time from any device, to the sometimes dreary world of corporate applications. Teams can integrate the single sign-on (SSO) option, like Okta or AzureAD, that they’ve chosen to use and in doing so make on-premise or self-managed cloud applications feel like SaaS apps.

However, teams consist of more than just the internal employees that share an identity provider. Organizations work with partners, freelancers, and contractors. Extending access to external users becomes a constant chore for IT and security departments and is a source of security problems.

Cloudflare Access removes that friction by simultaneously integrating with multiple identity providers, including popular services like Gmail or GitHub that do not require corporate subscriptions. External users login with these accounts and still benefit from the same ease-of-use available to internal employees. Meanwhile, administrators avoid the burden in legacy deployments that require onboarding and offboarding new accounts for each project.

We are excited to announce two new integrations that make it even easier for organizations to work securely with third parties. Starting Continue reading

The Serverlist: Globally Distributed Websites, $16M Series A, and more

Check out our twelfth edition of The Serverlist below. Get the latest scoop on the serverless space, get your hands dirty with new developer tutorials, engage in conversations with other serverless developers, and find upcoming meetups and conferences to attend.

Sign up below to have The Serverlist sent directly to your mailbox.

Vetflare, Cloudflare’s Military Veteran Employee Group Launches

“Diversity leads to better outcomes… better decisions, increased innovation, stronger financial returns, and a great place to work for everyone” said Janet Van Huysse, Head of People at Cloudflare during our Q1-2020 kickoff. Veterans, people who have served in the military, are a vital element of a diverse workforce. We come in diverse shapes, sizes, colors, genders, and orientations. We bring diverse skillsets, experiences, and perspectives.  

If you haven’t served in the military and haven’t worked with many veterans, here are some of the things that you can expect from your colleagues or direct reports that are veterans.

Veterans know what it means to SERVE. Indeed, it is a truism that living in service to others is a life well-lived, and that service to others is a foundation of esprit de corps. Though relatively few of us have seen combat, we have all signed a blank check to our nation made payable for any amount, up to and including our lives. This is what it means to become part of something bigger than oneself. This translates to putting our common shared interests ahead of our personal interests even when that means becoming an instrument of a foreign policy we Continue reading

JAMstack at the Edge: How we built Built with Workers… on Workers

I'm extremely stoked to announce Built with Workers today – it's an awesome resource for exploring what you can build with Cloudflare Workers. As Adam explained in our launch post, showcasing developers building incredible projects with tools like Workers KV or our streaming HTML rewriter is a great way to celebrate users of our platform. It also helps encourage developers to try building their dream app on top of Workers. In this post, I’ll explore some of the architectural and implementation designs we made while building the site.

When we first started planning Built with Workers, we wanted to use the site as an opportunity to build a new greenfield application, showcasing the strength of the Workers platform. The Workers Developer Experience team is cross-functional: while we might spend most of our time improving our docs, or developing features for our command-line interface Wrangler, most of us have spent years developing on the web. The prospect of starting a new application is always fun, but in this instance, it was a prime chance to ask (and answer) the question, "If I could build this site on Workers with whatever tools I want, what would I choose?"

A guiding Continue reading

Announcing Built with Workers

Ever since its initial release, Cloudflare Workers has given JavaScript developers a platform to enable building high-performance applications with automatic scaling.

As with any new technology, we know it can be a bit intimidating to get started. For one thing, running code on the edge is a paradigm shift—forcing us to rethink classic web architecture problems, or removing them altogether. For another, since you can build just about anything, it can be challenging to figure out what to build first.

Today we’re launching Built with Workers, a new site designed to help get those creative juices flowing and unblock you, by answering that simple but important question: What can I build with Cloudflare Workers?

Some time in 1999, at age 11, I received my first graphing calculator. It was a TI-82 that my older sister no longer needed. It was on this very calculator that I learned to write code. Looking back, I’m not sure how exactly I had the patience or sanity to figure it all out.

It was a mess. Among the many difficulties were that I had to type the code out on the calculator’s non-QWERTY keyboard, the language I was writing in didn’t have Continue reading

Empowering Your Privacy

Happy Data Privacy Day! At Cloudflare, our mission is to help build a better Internet, and we believe data privacy is core to that mission. But we know words are cheap — even data brokers who sell your personal information will tell you that “privacy is important” to them. So we wanted to take the opportunity on this Data Privacy Day to show you how our commitment to privacy crosses all levels of the work we do at Cloudflare to help make the Internet more private and secure — and therefore better — for everyone.

Privacy on the Internet means different things to different people. Maybe privacy means you get to control your personal data — who can collect it and how it can be used. Or that you have the right to access and delete your personal information. Or maybe it means your online life is protected from government surveillance or from ad trackers and targeted advertising. Maybe you think you should be able to be completely anonymous online. At Cloudflare, we think all these flavors of privacy are equally important, and as we describe in more detail below, we’ve taken steps to address each of these privacy priorities.

Continue reading

JavaScript Libraries Are Almost Never Updated Once Installed

Cloudflare helps run CDNJS, a very popular way of including JavaScript and other frontend resources on web pages. With the CDNJS team’s permission we collect anonymized and aggregated data from CDNJS requests which we use to understand how people build on the Internet. Our analysis today is focused on one question: once installed on a site, do JavaScript libraries ever get updated?

Let’s consider jQuery, the most popular JavaScript library on Earth. This chart shows the number of requests made for a selected list of jQuery versions over the past 12 months:

Spikes in the CDNJS data as you see with version 3.3.1 are not uncommon as very large sites add and remove CDNJS script tags.

We see a steady rise of version 3.4.1 following its release on May 2nd, 2019. What we don’t see is a substantial decline of old versions. Version 3.2.1 shows an average popularity of 36M requests at the beginning of our sample, and 29M at the end, a decline of approximately 20%. This aligns with a corpus of research which shows the average website lasts somewhere between two and four years. What we don’t see is a decline Continue reading

Announcing the Cloudflare Access App Launch

Every person joining your team has the same question on Day One: how do I find and connect to the applications I need to do my job?

Since launch, Cloudflare Access has helped improve how users connect to those applications. When you protect an application with Access, users never have to connect to a private network and never have to deal with a clunky VPN client. Instead, they reach on-premise apps as if they were SaaS tools. Behind the scenes, Access evaluates and logs every request to those apps for identity, giving administrators more visibility and security than a traditional VPN.

Administrators need about an hour to deploy Access. End user logins take about 20 ms, and that response time is consistent globally. Unlike VPN appliances, Access runs in every data center in Cloudflare’s network in 200 cities around the world. When Access works well, it should be easy for administrators and invisible to the end user.

However, users still need to locate the applications behind Access, and for internally managed applications, traditional dashboards require constant upkeep. As organizations grow, that roster of links keeps expanding. Department leads and IT administrators can create and publish manual lists, but those become Continue reading

Introducing Cloudflare for Campaigns

During the past year, we saw nearly 2 billion global citizens go to the polls to vote in democratic elections. There were major elections in more than 50 countries, including India, Nigeria, and the United Kingdom, as well as elections for the European Parliament. In 2020, we will see a similar number of elections in countries from Peru to Myanmar. In November, U.S citizens will cast their votes for the 46th President, 435 seats in the U.S House of Representatives, 35 of the 100 seats in the U.S. Senate, and many state and local elections.

Recognizing the importance of maintaining public access to election information, Cloudflare launched the Athenian Project in 2017, providing U.S. state and local government entities with the tools needed to secure their election websites for free. As we’ve seen, however, political parties and candidates for office all over the world are also frequent targets for cyberattack. Cybersecurity needs for campaign websites and internal tools are at an all time high.

Although Cloudflare has helped improve the security and performance of political parties and candidates for office all over the world for years, we’ve long felt that we could do more. So today, Continue reading

A cost-effective and extensible testbed for transport protocol development

This was originally published on Perf Planet's 2019 Web Performance Calendar.

At Cloudflare, we develop protocols at multiple layers of the network stack. In the past, we focused on HTTP/1.1, HTTP/2, and TLS 1.3. Now, we are working on QUIC and HTTP/3, which are still in IETF draft, but gaining a lot of interest.

QUIC is a secure and multiplexed transport protocol that aims to perform better than TCP under some network conditions. It is specified in a family of documents: a transport layer which specifies packet format and basic state machine, recovery and congestion control, security based on TLS 1.3, and an HTTP application layer mapping, which is now called HTTP/3.

Let’s focus on the transport and recovery layer first. This layer provides a basis for what is sent on the wire (the packet binary format) and how we send it reliably. It includes how to open the connection, how to handshake a new secure session with the help of TLS, how to send data reliably and how to react when there is packet loss or reordering of packets. Also it includes flow control and congestion control to interact well with other transport protocols in Continue reading

Helping mitigate the Citrix NetScaler CVE with Cloudflare Access

Yesterday, Citrix sent an updated notification to customers warning of a vulnerability in their Application Delivery Controller (ADC) product. If exploited, malicious attackers can bypass the login page of the administrator portal, without authentication, to perform arbitrary code execution.

No patch is available yet. Citrix expects to have a fix for certain versions on January 20 and others at the end of the month.

In the interim, Citrix has asked customers to attempt to mitigate the vulnerability. The recommended steps involve running a number of commands from an administrator command line interface.

The vulnerability relied on by attackers requires that they first be able to reach a login portal hosted by the ADC. Cloudflare can help teams secure that page and the resources protected by the ADC. Teams can place the login page, as well as the administration interface, behind Cloudflare Access’ identity proxy to prevent unauthenticated users from making requests to the portal.

Exploiting URL paths

Citrix ADC, also known as Citrix NetScaler, is an application delivery controller that provides Layer 3 through Layer 7 security for applications and APIs. Once deployed, administrators manage the installation of the ADC through a portal available at a dedicated URL on Continue reading

Accelerating UDP packet transmission for QUIC

This was originally published on Perf Planet's 2019 Web Performance Calendar.

QUIC, the new Internet transport protocol designed to accelerate HTTP traffic, is delivered on top of UDP datagrams, to ease deployment and avoid interference from network appliances that drop packets from unknown protocols. This also allows QUIC implementations to live in user-space, so that, for example, browsers will be able to implement new protocol features and ship them to their users without having to wait for operating systems updates.

But while a lot of work has gone into optimizing TCP implementations as much as possible over the years, including building offloading capabilities in both software (like in operating systems) and hardware (like in network interfaces), UDP hasn't received quite as much attention as TCP, which puts QUIC at a disadvantage. In this post we'll look at a few tricks that help mitigate this disadvantage for UDP, and by association QUIC.

For the purpose of this blog post we will only be concentrating on measuring throughput of QUIC connections, which, while necessary, is not enough to paint an accurate overall picture of the performance of the QUIC protocol (or its implementations) as a whole.

Test Environment

The client used Continue reading

Prototyping optimizations with Cloudflare Workers and WebPageTest

This article was originally published as part of  Perf Planet's 2019 Web Performance Calendar.

Have you ever wanted to quickly test a new performance idea, or see if the latest performance wisdom is beneficial to your site? As web performance appears to be a stochastic process, it is really important to be able to iterate quickly and review the effects of different experiments. The challenge is to be able to arbitrarily change requests and responses without the overhead of setting up another internet facing server. This can be straightforward to implement by combining two of my favourite technologies : WebPageTest and Cloudflare Workers. Pat Meenan sums this up with the following slide from a recent getting the most of WebPageTest presentation:

So what is Cloudflare Workers and why is it ideally suited to easy prototyping of optimizations?

Cloudflare Workers

From the documentation :

Cloudflare Workers provides a lightweight JavaScript execution environment that allows developers to augment existing applications or create entirely new ones without configuring or maintaining infrastructure.A Cloudflare Worker is a programmable proxy which brings the simplicity and flexibility of the Service Workers event-based fetch API from the browser to the edge. This allows a worker to Continue reading