Why I joined Cloudflare — and why I’m excited about Project Pangea

If you are well-prepared to take up the challenge, you will get to experience a moment where you are stepping forward to help build a better world. Personally, I felt exactly that when about a month ago, after a long and (COVID) complicated visa process, I joined Cloudflare as a Systems Engineer in Austin, Texas.

In the early 2000s, I experienced while travelling throughout the Benin Republic (my home country) and West Africa more generally, how challenging accessing the Internet was. I recall that, as students, we were often connecting to the web from cybercafés through limited bandwidth purchased at high cost. It was a luxury to have a broadband connection at home. When access was free (say, from high school premises or at university) we still had bandwidth constraints, and often we could not connect for long. The Internet can efficiently help tackle issues encountered (in areas like education, health, communications, ...) by populations in similar regions, but the lack of easy and affordable access, made it difficult to leverage. It is in such a context that I chose to pursue my studies in telecoms, with the hope of being able to somehow give back to the community by Continue reading

Announcing Project Pangea: Helping Underserved Communities Expand Access to the Internet For Free

Half of the world’s population has no access to the Internet, with many more limited to poor, expensive, and unreliable connectivity. This problem persists despite large levels of public investment, private infrastructure, and effort by local organizers.

Today, Cloudflare is excited to announce Project Pangea: a piece of the puzzle to help solve this problem. We’re launching a program that provides secure, performant, reliable access to the Internet for community networks that support underserved communities, and we’re doing it for free¹ because we want to help build an Internet for everyone.

What is Cloudflare doing to help?

Project Pangea is Cloudflare’s project to help bring underserved communities secure connectivity to the Internet through Cloudflare’s global and interconnected network.

Cloudflare is offering our suite of network services — Cloudflare Network Interconnect, Magic Transit, and Magic Firewall — for free to nonprofit community networks, local networks, or other networks primarily focused on providing Internet access to local underserved or developing areas. This service would dramatically reduce the cost for communities to connect to the Internet, with industry leading security and performance functions built-in:

• Cloudflare Network Interconnect provides access to Cloudflare’s edge in 200+ cities across the globe through Continue reading

Introducing Flarability, Cloudflare’s Accessibility Employee Resource Group

Hello, folks! I’m pleased to introduce myself and Cloudflare’s newest Employee Resource Group (ERG), Flarability, to the world. The 31st anniversary of the signing of the Americans with Disabilities Act (ADA), which happens to fall during Cloudflare’s Impact Week, is an ideal time to raise the subject of accessibility at Cloudflare and around the world.

There are multiple accessibility-related projects and programs at Cloudflare, including office space accessibility and website and product accessibility programs, some of which we will highlight in the stories below. I wanted to share my accessibility story and the story of the birth  and growth of our accessibility community with you.

About Flarability

Flarability began with a conversation between a couple of colleagues, almost two years ago. Some of us had noticed some things about the workspace that weren't as inclusive of people with disabilities as they could have been. For example, the open floor plan in our San Francisco office, as well as the positioning of our interview rooms, made it difficult for some to concentrate in the space. To kick off a community discussion, we formed a chat room, spread the word about our existence, and started hosting some meetings for Continue reading

Welcome to Cloudflare Impact Week

If I'm completely honest, Cloudflare didn't start out as a mission-driven company. When Lee, Michelle, and I first started thinking about starting a company in 2009 we saw an opportunity as the world was shifting from on-premise hardware and software to services in the cloud. It seemed inevitable to us that the same shift would come to security, performance, and reliability services. And, getting ahead of that trend, we could build a great business.

One problem we had was that we knew in order to have a great business we needed to win large organizations with big IT budgets as customers. And, in order to do that, we needed to have the data to build a service that would keep them safe. But we only could get data on security threats once we had customers. So we had a chicken and egg problem.

Our solution was to provide a basic version of Cloudflare's services for free. We reasoned that individual developers and small businesses would sign up for the free service. We'd learn a lot about security threats and performance and reliability opportunities based on their traffic data. And, Continue reading

Cloudflare’s Handling of an RCE Vulnerability in cdnjs

cdnjs provides JavaScript, CSS, images, and fonts assets for websites to reference with more than 4,000 libraries available. By utilizing cdnjs, websites can load faster with less strain on one’s own origin server as files are served directly from Cloudflare’s edge. Recently, a blog post detailed a vulnerability in the way cdnjs’ backend automatically keeps the libraries up to date.

This vulnerability allowed the researcher to execute arbitrary code, granting the ability to modify assets. This blog post details how Cloudflare responded to this report, including the steps we took to block exploitation, investigate potential abuse, and remediate the vulnerability.

This vulnerability is not related to Cloudflare CDN. The cdnjs project is a platform that leverages Cloudflare’s services, but the vulnerability described below relates to cdnjs’ platform only. To be clear, no existing libraries were modified using this exploit. The researcher published a new package which demonstrated the vulnerability and our investigation concluded that the integrity of all assets hosted on cdnjs remained intact.

Disclosure Timeline

As outlined in RyotaK’s blog post, the incident began on 2021-04-06. At around 1100 GMT, RyotaK published a package to npm exploiting the vulnerability. At 1129 GMT, cdnjs processed this package, resulting in Continue reading

AWS’s Egregious Egress

When web hosting services first emerged in the mid-1990s, you paid for everything on a separate meter: bandwidth, storage, CPU, and memory. Over time, customers grew to hate the nickel-and-dime nature of these fees. The market evolved to a fixed-fee model. Then came Amazon Web Services.

AWS was a huge step forward in terms of flexibility and scalability, but a massive step backward in terms of pricing. Nowhere is that more apparent than with their data transfer (bandwidth) pricing. If you look at the (ironically named) AWS Simple Monthly Calculator you can calculate the price they charge for bandwidth for their typical customer. The price varies by region, which shouldn't surprise you because the cost of transit is dramatically different in different parts of the world.

Charging for Stocks, Paying for Flows

AWS charges customers based on the amount of data delivered — 1 terabyte (TB) per month, for example. To visualize that, imagine data is water. AWS fills a bucket full of water and then charges you based on how much water is in the bucket. This is known as charging based on “stocks.”

On the other hand, AWS pays for bandwidth based on the capacity of their Continue reading

Empowering customers with the Bandwidth Alliance

High Egress Fees

Debates over the benefits and drawbacks of walled gardens versus open ecosystems have carried on since the beginnings of the tech industry. As applied to the Internet, we don’t think there’s much to debate. There’s a reason why it’s easier today than ever before to start a company online: open standards. They’ve encouraged a flourishing of technical innovation, made the Internet faster and safer, and easier and less expensive for anyone to have an Internet presence.

Of course, not everyone likes competition. Breaking open standards — with proprietary ones — is a common way to stop competition. In the cloud industry, a more subtle way to gain power over customers and lock them in has emerged. Something that isn’t obvious at the start: high egress fees.

You probably won’t notice them when you embark on your cloud journey. And if you need to bring data into your environment, there’s no data charge. But say you want to get that data out? Or go multi-cloud, and work with another cloud provider who is best-in-class? That’s when the charges start rolling in.

To make matters worse, as the number and diversity of applications in your IT stack increases, the Continue reading

Introducing Workers Usage Notifications

So you’ve built an application on the Workers platform. The first thing you might be wondering after pushing your code out into the world is “what does my production traffic look like?” How many requests is my Worker handling? How long are those requests taking? And as your production traffic evolves overtime it can be a lot to keep up with. The last thing you want is to be surprised by the traffic your serverless application is handling.  But, you have a million things to do in your day job, and having to log in to the Workers dashboard every day to check usage statistics is one extra thing you shouldn’t need to worry about.

Today we’re excited to launch Workers usage notifications that proactively send relevant usage information directly to your inbox. Usage notifications come in two flavors. The first is a weekly summary of your Workers usage with a breakdown of your most popular Workers. The second flavor is an on-demand usage notification, triggered when a worker’s CPU usage is 25% above its average CPU usage over the previous seven days. This on-demand notification helps you proactively catch large changes in Workers usage as soon as those Continue reading

Upgrading the Cloudflare China Network: better performance and security through product innovation and partnership

Core to Cloudflare’s mission of helping build a better Internet is making it easy for our customers to improve the performance, security, and reliability of their digital properties, no matter where in the world they might be. This includes Mainland China. Cloudflare has had customers using our service in China since 2015 and recently, we expanded our China presence through a partnership with JD Cloud, the cloud division of Chinese Internet giant, JD.com. We’ve also had a local office in Beijing for several years, which has given us a deep understanding of the Chinese Internet landscape as well as local customers.

The new Cloudflare China Network built in partnership with JD Cloud has been live for several months, with significant performance and security improvements compared to the previous in-country network. Today, we’re excited to describe the improvements we made to our DNS and DDoS systems, and provide data demonstrating the performance gains customers are seeing. All customers licensed to operate in China can now benefit from these innovations, with the click of a button in the Cloudflare dashboard or via the API.

Serving DNS inside China

With over 14% of all domains on the Internet using Cloudflare’s nameservers we Continue reading

Expanding Cloudflare to 25+ Cities in Brazil

Today, we are excited to announce an expansion we’ve been working on behind the scenes for the last two years: a 25+ city partnership with one of the largest ISPs in Brazil. This is one of the largest simultaneous single-country expansions we’ve done so far.

With this partnership, Brazilians throughout the country will see significant improvement to their Internet experience. Already, the 25th-percentile latency of non-bot traffic (we use that measure as an approximation of physical distance from our servers to end users) has dropped from the mid-20 millisecond range to sub-10 milliseconds. This benefit extends not only to the 25 million Internet properties on our network, but to the entire Internet with Cloudflare services like 1.1.1.1 and WARP. We expect that as we approach 25 cities in Brazil, latency will continue to drop while throughput increases.

This partnership is part of our mission to help create a better Internet and the best development experience for all — not just those in major population centers or in Western markets — and we are excited to take this step on Continue reading

DDoS attack trends for 2021 Q2

Recent weeks have witnessed massive ransomware and ransom DDoS (Distributed Denial of Service) attack campaigns that interrupted aspects of critical infrastructure around the world, including one of the largest petroleum pipeline system operators, and one of the world’s biggest meat processing companies. Earlier this quarter, more than 200 organizations across Belgium, including the government and parliament websites and other services, were also DDoS’d.

And when most of the United States were celebrating Independence Day on July 4, hundreds of US companies were hit by a ransomware attack demanding 70 million USD in Bitcoin. Attackers known to be affiliated with REvil, a Russian ransomware group, exploited multiple previously unknown vulnerabilities in IT management software. The targets included schools, small public-sector bodies, travel and leisure organizations, and credit unions, to name a few. While the threat of ransomware and ransom DDoS is not new (read our posts on ransomware and ransom DDoS from 2021 Q1), the latest attacks on Internet properties ranging from wineries, professional sports teams, ferry services and hospitals has brought them from just being background noise to front page headlines affecting our day-to-day lives. In fact, recent attacks have propelled ransomware and DDoS to the top of US Continue reading

Rich, complex rules for advanced load balancing

Load Balancing — functionality that’s been around for the last 30 years to help businesses leverage their existing infrastructure resources. Load balancing works by proactively steering traffic away from unhealthy origin servers and — for more advanced solutions — intelligently distributing traffic load based on different steering algorithms. This process ensures that errors aren’t served to end users and empowers businesses to tightly couple overall business objectives to their traffic behavior.

What’s important for load balancing today?

We are no longer in the age where setting up a fixed amount of servers in a data center is enough to meet the massive growth of users browsing the Internet. This means that we are well past the time when there is a one size fits all solution to suffice the needs of different businesses. Today, customers look for load balancers that are easy to use, propagate changes quickly, and — especially now — provide the most feature flexibility. Feature flexibility has become so important because different businesses have different paths to success and, consequently, different challenges! Let’s go through a few common use cases:

• You might have an application split into microservices, where specific origins support segments of your application. You Continue reading

Automatic Remediation of Kubernetes Nodes

We use Kubernetes to run many of the diverse services that help us control Cloudflare’s edge. We have five geographically diverse clusters, with hundreds of nodes in our largest cluster. These clusters are self-managed on bare-metal machines which gives us a good amount of power and flexibility in the software and integrations with Kubernetes. However, it also means we don’t have a cloud provider to rely on for virtualizing or managing the nodes. This distinction becomes even more prominent when considering all the different reasons that nodes degrade. With self-managed bare-metal machines, the list of reasons that cause a node to become unhealthy include:

• Hardware failures
• Kernel-level software failures
• Kubernetes cluster-level software failures
• Degraded network communication
• Software updates are required
• Resource exhaustion¹

Unhappy Nodes

We have plenty of examples of failures in the aforementioned categories, but one example has been particularly tedious to deal with. It starts with the following log line from the kernel:

unregister_netdevice: waiting for lo to become free. Usage count = 1

The issue is further observed with the number of network interfaces on the node owned by the Container Network Interface (CNI) plugin getting out of proportion with the number of running pods:

$  Continue reading

The UEFA EURO 2020 final as seen online by Cloudflare Radar

Last night’s Italy-England match was a nail-biter. 1-1 at full time, 1-1 at the end of extra time, and then an amazing penalty shootout with incredible goalkeeping by Pickford and Donnarumma.

Cloudflare has been publishing statistics about all the teams involved in EURO 2020 and traffic to betting websites, sports newspapers, streaming services and sponsors. Here’s a quick look at some specific highlights from England’s and Italy’s EURO 2020.

Two interesting peaks show up in UK visits to sports newspapers: the day after England-Germany and today after England’s defeat. Looks like fans are hungry for analysis and news beyond the goals. You can see all the data on the dedicated England EURO 2020 page on Cloudflare Radar.

But it was a quiet morning for the websites of the England team’s sponsors.

Turning to the winners, we can see that Italian readers are even more interested in knowing more about their team’s success.

And this enthusiasm spills over into visits to the Italian team’s sponsors.

You can follow along on the dedicated Cloudflare Radar page for Italy in EURO 2020.

Visit Cloudflare Radar for information on global Internet trends, trending domains, attacks and usage statistics.

Cloudflare’s Network Doubles CPU Capacity and Expands Into Ten New Cities in Four New Countries

Cloudflare’s global network is always expanding, and 2021 has been no exception. Today, I’m happy to give a mid-year update: we've added ten new Cloudflare cities, with four new countries represented among them. And we've doubled our computational footprint since the start of pandemic-related lockdowns.

No matter what else we do at Cloudflare, constant expansion of our infrastructure to new places is a requirement to help build a better Internet. 2021, like 2020, has been a difficult time to be a global network — from semiconductor shortages to supply-chain disruptions — but regardless, we have continued to expand throughout the entire globe, experimenting with technologies like ARM, ASICs, and Nvidia all the way.

The Cities

Without further ado, here are the new Cloudflare cities: Tbilisi, Georgia; San José, Costa Rica; Tunis, Tunisia; Yangon, Myanmar; Nairobi, Kenya; Jashore, Bangladesh; Canberra, Australia; Palermo, Italy; and Salvador and Campinas, Brazil.

These deployments are spread across every continent except Antarctica.

We’ve solidified our presence in every country of the Caucuses with our first deployment in the country of Georgia in the capital city of Tbilisi. And on the other side of the world, we’ve Continue reading

Browser VNC with Zero Trust Rules

Starting today, we’re excited to share that you can now shift another traditional client-driven use case to a browser. Teams can now provide their users with a Virtual Network Computing (VNC) client fully rendered in the browser with built-in Zero Trust controls.

Like the SSH flow, this allows users to connect from any browser on any device, with no client software needed. The feature runs in every one of our data centers in over 200 cities around the world, bringing the experience closer to your end users. We also built the experience using Cloudflare Workers, to offer nearly instant start times. In the future we will support full auditability of user actions in their VNC and SSH sessions.

A quick refresher on VNC

VNC is a desktop sharing platform built on top of the Remote Frame Buffer protocol that allows for a GUI on any server. It is built to be platform-independent and provides an easy way for administrators to make interfaces available to users that are less comfortable with a command-line to work with a remote machine. Or to complete work better suited for a visual interface.

In my case, the most frequent reason I use VNC is Continue reading

Announcing Rollbacks and API Access for Pages

A couple of months ago, we announced the general availability of Cloudflare Pages: the easiest way to host and collaboratively develop websites on Cloudflare’s global network. It’s been amazing to see over 20,000 incredible sites built by users and hear your feedback. Since then, we’ve released user-requested features like URL redirects, web analytics, and Access integration.

We’ve been listening to your feedback and today we announce two new features: rollbacks and the Pages API. Deployment rollbacks allow you to host production-level code on Pages without needing to stress about broken builds resulting in website downtime. The API empowers you to create custom functionality and better integrate Pages with your development workflows. Now, it’s even easier to use Pages for production hosting.

Rollbacks

You can now rollback your production website to a previous working deployment with just a click of a button. This is especially useful when you want to quickly undo a new deployment for troubleshooting. Before, developers would have to push another deployment and then wait for the build to finish updating production. Now, you can restore a working version within a few moments by rolling back to a previous working build.

To rollback to a previous build, Continue reading

More products, more partners, and a new look for Cloudflare Logs

We are excited to announce a new look and new capabilities for Cloudflare Logs! Customers on our Enterprise plan can now configure Logpush for Firewall Events and Network Error Logs Reports directly from the dashboard. Additionally, it’s easier to send Logs directly to our analytics partners Microsoft Azure Sentinel, Splunk, Sumo Logic, and Datadog. This blog post discusses how customers use Cloudflare Logs, how we’ve made it easier to consume logs, and tours the new user interface.

New data sets for insight into more products

Cloudflare Logs are almost as old as Cloudflare itself, but we have a few big improvements: new datasets and new destinations.

Cloudflare has a large number of products, and nearly all of them can generate Logs in different data sets. We have “HTTP Request” Logs, or one log line for every L7 HTTP request that we handle (whether cached or not). We also provide connection Logs for Spectrum, our proxy for any TCP or UDP based application. Gateway, part of our Cloudflare for Teams suite, can provide Logs for HTTP and DNS traffic.

Today, we are introducing two new data sets:

Firewall Events gives insight into malicious traffic handled by Cloudflare. It provides detailed information Continue reading

Sudan’s exam-related Internet shutdowns

To prevent cheating in exams many countries restrict or even shut down Internet access during critical exam hours. I wrote two weeks ago about Syria having planned Internet shutdowns during June, for exams.

Sudan is doing the same thing and has had four shutdowns so far. Here's the Internet traffic pattern for Sudan over the last seven days. I've circled the shutdowns on Saturday, Sunday, Monday and Tuesday (today, June 22, 2021).

Cloudflare Radar allows anyone to track Internet traffic patterns around the world, and it has country-specific pages. The chart for the last seven days (shown above) came from the dedicated page for Sudan.

The Internet outages start at 0600 UTC (0800 local time) and end three hours later at 0900 UTC (1100 local time). This corresponds to the timings announced by the Sudanese Ministry of Education.

Further shutdowns are likely in Sudan on June 24, 26, 27, 29 and 30 (thanks to Twitter user _adonese for his assistance). Looking deeper into the data, the largest drop in use is for mobile Internet access in Sudan (the message above talks about mobile Internet use being restricted) while some non-mobile access appears to continue.

That can be seen by looking Continue reading

Cloudflare TV: Doing it Live, 1,000 Times and Counting

Last week, Cloudflare TV celebrated its first anniversary the only way it knows how: with a broadcast brimming with live programming spanning everything from the keynotes of Cloudflare Connect, to a day-long virtual career fair, to our flagship game show Silicon Valley Squares.

When our co-founder and CEO Matthew Prince introduced Cloudflare TV to the world last year, he described it as a platform for experimentation. By empowering Cloudflare employees to try whatever they could think up on air — bound only by restraints of common sense — we hoped to unlock aspects of our team’s talent and creativity that otherwise might go untapped in the midst of the pandemic.

The results, as they say, have been extraordinary.

Since launching in June 2020, Cloudflare TV has featured over 1,000 original live episodes covering an incredible array of topics: technical deep dives and tutorials like Hardware at Cloudflare, Leveling up Web Performance with HTTP/3, and Hacker Time. Security expertise from top CISOs and compliance experts. In-depth policy discussions. And of course, updates on Cloudflare’s products with weekly episodes of Latest from Product and Engineering, Estas Semanas en Cloudflare en Español, and launch-day introductions to Magic WAN Continue reading