Production ready eBPF, or how we fixed the BSD socket API

As we develop new products, we often push our operating system - Linux - beyond what is commonly possible. A common theme has been relying on eBPF to build technology that would otherwise have required modifying the kernel. For example, we’ve built DDoS mitigation and a load balancer and use it to monitor our fleet of servers.

This software usually consists of a small-ish eBPF program written in C, executed in the context of the kernel, and a larger user space component that loads the eBPF into the kernel and manages its lifecycle. We’ve found that the ratio of eBPF code to userspace code differs by an order of magnitude or more. We want to shed some light on the issues that a developer has to tackle when dealing with eBPF and present our solutions for building rock-solid production ready applications which contain eBPF.

For this purpose we are open sourcing the production tooling we’ve built for the sk_lookup hook we contributed to the Linux kernel, called tubular. It exists because we’ve outgrown the BSD sockets API. To deliver some products we need features that are just not possible using the standard API.

• Our services are available on millions of Continue reading

Who won Super Bowl LVI? A look at Internet traffic during the big game

“It's ridiculous for a country to get all worked up about a game—except the Super Bowl, of course. Now that's important.”
- Andy Rooney, American radio and television writer

When the Super Bowl is on, there are more winners than just one of the teams playing, especially when we look at Internet trends. By now, everyone knows that the Los Angeles Rams won, but we also want to look at which Super Bowl advertisers were the biggest winners, and how traffic to food delivery services, social media and messaging apps, and sports and betting websites changed throughout the game.

We covered some of these questions during our Super Bowl live-tweeting on our Cloudflare Radar account. (Hint: follow us if you’re interested in Internet trends).

Cloudflare Radar uses a variety of sources to provide aggregate information about Internet traffic and attack trends. In this blog post, as we did last year, we use DNS name resolution data to estimate traffic to websites. We can’t see who visited the websites mentioned, or what anyone did on the websites, but DNS can give us an estimate of the interest generated by the ads or across a set of sites in Continue reading

Cloudflare acquires Vectrix to expand Zero Trust SaaS security

We are excited to share that Vectrix has been acquired by Cloudflare!

Vectrix helps IT and security teams detect security issues across their SaaS applications. We look at both data and users in SaaS apps to alert teams to issues ranging from unauthorized user access and file exposure to misconfigurations and shadow IT.

We built Vectrix to solve a problem that terrified us as security engineers ourselves: how do we know if the SaaS apps we use have the right controls in place? Is our company data protected? SaaS tools make it easy to work with data and collaborate across organizations of any size, but that also makes them vulnerable.

The growing SaaS security problem

The past two years have accelerated SaaS adoption much faster than any of us could have imagined and without much input on how to secure this new business stack.

Google Workspace for collaboration. Microsoft Teams for communication. Workday for HR. Salesforce for customer relationship management. The list goes on.

With this new reliance on SaaS, IT and security teams are faced with a new set of problems like files and folders being made public on the Internet, external users joining private chat channels, or an Continue reading

Adding a CASB to Cloudflare Zero Trust

Earlier today, Cloudflare announced that we have acquired Vectrix, a cloud-access security broker (CASB) company focused on solving the problem of control and visibility in the SaaS applications and public cloud providers that your team uses.

We are excited to welcome the Vectrix team and their technology to the Cloudflare Zero Trust product group. We don’t believe a CASB should be a point solution. Instead, the features of a CASB should be one component of a comprehensive Zero Trust deployment. Each piece of technology, CASB included, should work better together than they would as a standalone product.

We know that this migration is a journey for most customers. That’s true for our own team at Cloudflare, too. We’ve built our own Zero Trust platform to solve problems for customers at any stage of that journey.

Start by defending the resources you control

Several years ago, we protected the internal resources that Cloudflare employees needed by creating a private network with hardware appliances. We deployed applications in a data center and made them available to this network. Users inside the San Francisco office connected to a secure Wi-Fi network that placed them on the network.

For everyone else, we punched a Continue reading

Email Routing is now in open beta, available to everyone

I won’t beat around the bush: we’ve moved Cloudflare Email Routing from closed beta to open beta 🎉

What does this mean? It means that there’s no waitlist anymore; every zone* in every Cloudflare account has Email Routing available to them.

To get started just open one of the zones in your Cloudflare Dashboard and click on Email in the navigation pane.

Our journey so far

Back in September 2021, during Cloudflare’s Birthday Week, we introduced Email Routing as the simplest solution for creating custom email addresses for your domains without the hassle of managing multiple mailboxes.

Many of us at Cloudflare saw a need for this type of product, and we’ve been using it since before it had a UI. After Birthday Week, we started gradually opening it to Cloudflare customers that requested access through the wait list; starting with just a few users per week and gradually ramping up access as we found and fixed edge cases.

Most recently, with users wanting to set up Email Routing for more of their domains and with some of G Suite legacy users looking for an alternative to starting a subscription, we have been onboarding tens of thousands of new zones Continue reading

Slicing and Dicing Instant Logs: Real-time Insights on the Command Line

During Speed Week 2021 we announced a new offering for Enterprise customers, Instant Logs. Since then, the team has not slowed down and has been working on new ways to enable our customers to consume their logs and gain powerful insights into their HTTP traffic in real time.

We recognize that as developers, UIs are useful but sometimes there is the need for a more powerful alternative. Today, I am going to introduce you to Instant Logs in your terminal! In order to get started we need to install a few open-source tools to help us:

• Websocat - to connect to WebSockets.
• Angle Grinder - a utility to slice-and-dice a stream of logs on the command line.

Creating an Instant Log’s Session

For enterprise zones with access to Instant Logs, you can create a new session by sending a POST request to our jobs' endpoint. You will need:

• Your Zone Tag / ID
• An Authentication Key or an API Token with at least the Zone Logs Read permission

curl -X POST 'https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/logpush/edge/jobs' \
-H 'X-Auth-Key: <KEY>' \
-H 'X-Auth-Email: <EMAIL>' \
-H 'Content-Type: application/json' \
--data-raw '{
    "fields": "ClientIP,ClientRequestHost,ClientRequestMethod,ClientRequestPath,EdgeEndTimestamp,EdgeResponseBytes,EdgeResponseStatus,EdgeStartTimestamp,RayID",
    "sample": 1,
    "filter": "",
    "kind": "instant-logs"
}'

Response

The Continue reading

Missing Manuals – io_uring worker pool

Chances are you might have heard of io_uring. It first appeared in Linux 5.1, back in 2019, and was advertised as the new API for asynchronous I/O. Its goal was to be an alternative to the deemed-to-be-broken-beyond-repair AIO, the “old” asynchronous I/O API.

Calling io_uring just an asynchronous I/O API doesn’t do it justice, though. Underneath the API calls, io_uring is a full-blown runtime for processing I/O requests. One that spawns threads, sets up work queues, and dispatches requests for processing. All this happens “in the background” so that the user space process doesn’t have to, but can, block while waiting for its I/O requests to complete.

A runtime that spawns threads and manages the worker pool for the developer makes life easier, but using it in a project begs the questions:

1. How many threads will be created for my workload by default?

2. How can I monitor and control the thread pool size?

I could not find the answers to these questions in either the Efficient I/O with io_uring article, or the Lord of the io_uring guide – two well-known pieces of available documentation.

And while a recent enough io_uring man page touches on the Continue reading

Need to Keep Analytics Data in the EU? Cloudflare Zaraz Can Offer a Solution

A recent decision from the Austrian Data Protection Authority (the Datenschutzbehörde) has network engineers scratching their heads and EU companies that use Google Analytics scrambling. The Datenschutzbehörde found that an Austrian website’s use of Google Analytics violates the EU General Data Protection Regulation (GDPR) as interpreted by the “Schrems II” case because Google Analytics can involve sending full or truncated IP addresses to the United States.

While disabling such trackers might be one (extreme) solution, doing so would leave website operators blind to how users are engaging with their site. A better approach: find a way to use tools like Google Analytics, but do so with an approach that protects the privacy of personal information and keeps it in the EU, avoiding a data transfer altogether. Enter Cloudflare Zaraz.

But before we get into just how Cloudflare Zaraz can help, we need to explain a bit of the background for the Datenschutzbehörde’s ruling, and why it’s a big deal.

What are the privacy and data localization issues?

The GDPR is a comprehensive data privacy law that applies to EU residents’ personal data, regardless of where it is processed. The GDPR itself does not insist that personal data must Continue reading

How to stop running out of ephemeral ports and start to love long-lived connections

Often programmers have assumptions that turn out, to their surprise, to be invalid. From my experience this happens a lot. Every API, technology or system can be abused beyond its limits and break in a miserable way.

It's particularly interesting when basic things used everywhere fail. Recently we've reached such a breaking point in a ubiquitous part of Linux networking: establishing a network connection using the connect() system call.

Since we are not doing anything special, just establishing TCP and UDP connections, how could anything go wrong? Here's one example: we noticed alerts from a misbehaving server, logged in to check it out and saw:

marek@:~# ssh 127.0.0.1
ssh: connect to host 127.0.0.1 port 22: Cannot assign requested address

You can imagine the face of my colleague who saw that. SSH to localhost refuses to work, while she was already using SSH to connect to that server! On another occasion:

marek@:~# dig cloudflare.com @1.1.1.1
dig: isc_socket_bind: address in use

This time a basic DNS query failed with a weird networking error. Failing DNS is a bad sign!

In both cases the problem was Linux running out of ephemeral ports. When Continue reading

Announcing the public launch of Cloudflare’s bug bounty program

Today we are launching Cloudflare’s paid public bug bounty program. We believe bug bounties are a vital part of every security team’s toolbox and have been working hard on improving and expanding our private bug bounty program over the last few years. The first iteration of our bug bounty was a pure vulnerability disclosure program without cash bounties. In 2018, we added a private bounty program and are now taking the next step to a public program.

Starting today, anyone can report vulnerabilities related to any Cloudflare product to our public bug bounty program, hosted on HackerOne’s platform.

Let's walk through our journey so far.

Step 1: starting a vulnerability disclosure program

In 2014, when the company had fewer than 100 employees, we created a responsible disclosure policy to provide a safe place for security researchers to submit potential vulnerabilities to our security team, with some established rules of engagement. A vulnerability disclosure policy is an important first step for a company to take because it is an invitation to researchers to look at company assets without fear of repercussions, provided the researchers follow certain guidelines intended to protect everyone involved. We still stand by that policy and welcome Continue reading

Cloudflare Partner Program Now Supports SASE & Zero Trust Managed Services

The importance of the Cloudflare Partner Network was on full display in 2021, with record level partner growth in 2021 and aiming even higher in 2022. We’ve been listening to our partners and working to constantly strengthen our ability to deliver value for businesses of all types. An area we identified we could do better, is a program to support “service partners” that want to wrap managed and professional services around Cloudflare products. Today, we are excited to announce the next evolution of the Cloudflare Channel and Alliances Partner Program to specifically enable partners that provide services around Cloudflare products with recurring revenue streams as they equip businesses of all sizes and types with Cloudflare’s leading Zero Trust and SASE solutions.

Core to enabling Services Partners are some exciting enhancements:

• New Program Paths
• New Managed Services Partner (MSP) Accreditation.
• New Support & Go-To-Market Motions

New Program Paths

We have seen a 29% increase in ransom DDoS attacks over the past year and a 175% increase just last quarter. Partners continue to be on the front lines helping mitigate and prevent disruption from these events as they extend our services. Our goal for 2022 is to arm our partners with the Continue reading

Happy Data Privacy Day!

Happy Data Privacy Day 2022! Of course, every day is privacy day at Cloudflare, but today gives us a great excuse to talk about one of our favorite topics.

In honor of Privacy Day, we’re highlighting some key topics in data privacy and data protection that helped shape the landscape in 2021, as well as the issues we’ll be thinking about in 2022. The first category that gets our attention is the intersection of data security and data privacy. At Cloudflare, we’ve invested in privacy-focused technologies and security measures that enhance data privacy to help build the third phase of the Internet, the Privacy phase, and we expect to double down on these developments in 2022.

The second category is data localization. While we don’t think you need localization to achieve privacy, the two are inextricably linked in the EU regulatory landscape and elsewhere.

Third, recent regulatory enforcement actions in the EU against websites’ use of cookies have us thinking about how we can help websites run third-party tools, such as analytics, in a faster, more secure, and more privacy-protective way.

Lastly, we’ll continue to focus on the introduction of new or updated data protection regulations around the world, Continue reading

Protecting Holocaust educational websites

Today is the International Holocaust Remembrance Day. On this day, we commemorate the victims that were murdered by the Nazis and their accomplices.

During the Holocaust, and in the events that led to it, the Nazis exterminated one third of the European Jewish population. Six million Jews, along with countless other members of minority and disability groups, were murdered because the Nazis believed they were inferior.

Cloudflare’s Project Galileo provides free protection to at-risk groups across the world including Holocaust educational and remembrance websites. During the past year alone, Cloudflare mitigated over a quarter of a million cyber threats launched against Holocaust-related websites.

Antisemitism and the Final Solution

In the Second World War and the years leading up to it, antisemitism served as the foundation of racist laws and fueled violent Pogroms against Jews. The tipping point was a night of violence known as the Kristallnacht ("Night of Broken Glass"). Jews and other minority groups were outlawed, dehumanized, persecuted and killed. Jewish businesses were boycotted, Jewish books burned and synagogues destroyed. Jews, Roma and other “enemies of the Reich'' were forced into closed ghettos and concentration camps. Finally, as part of the Final Solution for the Jewish Question, Continue reading

Migrating to Cloudflare Email Routing

A few days ago Google announced that the users from the "G Suite legacy free edition" would need to switch to the paid edition before May 1, 2022, to maintain their services and accounts working. Because of this, many people are now considering alternatives.

One use case for G Suite legacy was handling email for custom domains.

In September, during Birthday Week, we announced Cloudflare Email Routing. This service allows you to create any number of custom email addresses you want on top of the domains you already have with Cloudflare and automatically forward the incoming traffic to any destination inboxes you wish.

Email Routing was designed to be privacy-first, secure, powerful, and very simple to use. Also, importantly, it’s available to all our customers for free.

The closed beta allowed us to keep improving the service and make it even more robust, compliant with all the technical nuances of email, and scalable. Today we're pleased to report that we have over two hundred thousand zones testing Email Routing in production, and we started the countdown to open beta and global availability.

With Email Routing, you can effectively start receiving Emails in any of your domains for any number of Continue reading

Incorrect proxying of 24 hostnames on January 24, 2022

On January 24, 2022, as a result of an internal Cloudflare product migration, 24 hostnames (including www.cloudflare.com) that were actively proxied through the Cloudflare global network were mistakenly redirected to the wrong origin. During this incident, traffic destined for these hostnames was passed through to the clickfunnels.com origin and may have resulted in a clickfunnels.com page being displayed instead of the intended website. This was our doing and clickfunnels.com was unaware of our error until traffic started to reach their origin.

API calls or other expected responses to and from these hostnames may not have responded properly, or may have failed completely. For example, if you were making an API call to api.example.com, and api.example.com was an impacted hostname, you likely would not have received the response you would have expected.

Here is what happened:

At 2022-01-24 22:24 UTC we started a migration of hundreds of thousands of custom hostnames to the Cloudflare for SaaS product. Cloudflare for SaaS allows SaaS providers to manage their customers’ websites and SSL certificates at scale - more information is available here. This migration was intended to be completely seamless, with the outcome being enhanced Continue reading

Landscape of API Traffic

In recent years we have witnessed an explosion of Internet-connected applications. Whether it is a new mobile app to find your soulmate, the latest wearable to monitor your vitals, or an industrial solution to detect corrosion, our life is becoming packed with connected systems.

How is the Internet changing because of this shift? This blog provides an overview of how Internet traffic is evolving as Application Programming Interfaces (APIs) have taken the centre stage among the communication technologies. With help from the Cloudflare Radar team, we have harnessed the data from our global network to provide this snapshot of global APIs in 2021.

The huge growth in API traffic comes at a time when Cloudflare has been introducing new technologies that protect applications from nascent threats and vulnerabilities. The release of API Shield with API Discovery, Schema Validation, mTLS and API Abuse Detection has provided customers with a set of tools designed to protect their applications and data based on how APIs work and their challenges.

We are also witnessing increased adoption of new protocols. Among encryption protocols, for example, TLS v1.3 has become the most used protocol for APIs on Cloudflare while, for transport protocols, we Continue reading

Burkina Faso experiencing second major Internet disruption this year

The early hours of Sunday, January 23, 2022, started in Burkina Faso with an Internet outage or shutdown. Heavy gunfire in an army mutiny could be related to the outage according to the New York Times (“mobile Internet services were shut down”). As of today, there are three countries affected by major Internet disruptions — Tonga and Yemen are the others.

Cloudflare Radar shows that Internet traffic dropped significantly in the West African country after ~09:15 UTC (the same in local time) and remains low more than 24 hours later. Burkina Faso also had a mobile Internet shutdown on January 10, 2022, and another we reported in late November 2021.

The main ISPs from Burkina Faso were affected. The two leading Internet Service Providers Orange and FasoNet lost Internet traffic after 09:15 UTC, but also Telecel Faso, as the next chart shows. This morning, at around 10:00 UTC there was some traffic from FasoNet but less than half of what we saw at the same time in preceding days.

It’s not only mobile traffic that is affected. Desktop traffic is also impacted. In Burkina Faso, our data shows that mobile devices normally represent 70% of Internet traffic.

With the Burkina Continue reading

Internet outage in Yemen amid airstrikes

The early hours of Friday, January 21, 2022, started in Yemen with a country-wide Internet outage. According to local and global news reports airstrikes are happening in the country and the outage is likely related, as there are reports that a telecommunications building in Al-Hudaydah where the FALCON undersea cable lands was hit.

Cloudflare Radar shows that Internet traffic dropped close to zero between 21:30 UTC (January 20, 2022) and by 22:00 UTC (01:00 in local time).

The outage affected the main state-owned ISP, Public Telecommunication Corporation (AS30873 in blue in the next chart), which represents almost all the Internet traffic in the country.

Looking at BGP (Border Gateway Protocol) updates from Yemen’s ASNs around the time of the outage, we see a clear spike at the same time the main ASN was affected ~21:55 UTC, January 20, 2022. These update messages are BGP signalling that Yemen’s main ASN was no longer routable, something similar to what we saw happening in The Gambia and Kazakhstan but for very different reasons.

So far, 2022 has started with a few significant Internet disruptions for different reasons:

1. An Internet outage in The Gambia because of a cable problem.
2. An Internet Continue reading

Tonga’s likely lengthy Internet outage

2022 only has 19 days of existence but so far this January, there have already been four significant Internet disruptions:

1. An Internet outage in The Gambia because of a cable problem.
2. An Internet shutdown in Kazakhstan because of unrest.
3. A mobile Internet shutdown in Burkina Faso because of a coup plot.
4. An Internet outage in Tonga because of a volcanic eruption.

The latest Internet outage, in the South Pacific country of Tonga (with 169 islands), is still ongoing. It started with the large eruption of Hunga Tonga–Hunga Haʻapai, an uninhabited volcanic island of the Tongan archipelago on Friday, January 14, 2022. The next day, Cloudflare Radar shows that the Internet outage started at around 03:00 UTC (16:00 local time) — Saturday, January 15, 2022 — and is ongoing for more than four days. Tonga’s 105,000 residents are almost entirely unreachable, according to the BBC.

When we focus on the number of requests by ASN, the country’s main ISPs Digicel and Kalianet started to lose traffic after 03:00 UTC and by 05:30 UTC January 15, 2022, Cloudflare saw close to no traffic at all from them, as shown in the graph below.

Looking at the BGP Continue reading

Tomonari Sato: Why I joined Cloudflare and why I’m helping Cloudflare grow in Japan

This post is also available in 日本語.

I’m excited to announce that I recently joined Cloudflare in Japan as Vice-President and Managing Director, to help build and expand our customer, partner base, and presence in Japan. Cloudflare expanded its network in Japan in 2010, just months after launching. Now, 12 years later, Cloudflare is continuing its mission to help build a better Internet in Japan and across the globe, and I’m looking forward to being able to contribute to that mission!

A little about me

In my 35-year career in the IT industry, I have been fortunate enough to work with some of the biggest technology companies in the world, working in various roles in both sales and technical sides of the business. I consider this one of my biggest strengths. In addition, working in the IT industry has allowed me to acquire industry knowledge across a number of different solutions such as custom development, packaged systems (ERP, CRM), MS Office products, and cloud solutions.

Most recently, I was director of the Enterprise Business Group for Japan at AWS, where I was responsible for all commercial industries such as Manufacturing, Process, Distribution, Retail, Telecommunications, Utility, Media, Service, Pharmaceuticals, among others. Continue reading