Optimizing Magic Firewall’s IP lists
Magic Firewall is Cloudflare’s replacement for network-level firewall hardware. It evaluates gigabits of traffic every second against user-defined rules that can include millions of IP addresses. Writing a firewall rule for each IP address is cumbersome and verbose, so we have been building out support for various IP lists in Magic Firewall—essentially named groups that make the rules easier to read and write. Some users want to reject packets based on our growing threat intelligence of bad actors, while others know the exact set of IPs they want to match, which Magic Firewall supports via the same API as Cloudflare’s WAF.
With all those IPs, the system was using more of our memory budget than we’d like. To understand why, we need to first peek behind the curtain of our magic.
Life inside a network namespace
Magic Transit and Magic WAN enable Cloudflare to route layer 3 traffic, and they are the front door for Magic Firewall. We have previously written about how Magic Transit uses network namespaces to route packets and isolate customer configuration. Magic Firewall operates inside these namespaces, using nftables as the primary implementation of packet filtering.
When a user makes an API request to configure their Continue reading
CVE-2022-1096: How Cloudflare Zero Trust provides protection from zero day browser vulnerabilities
On Friday, March 25, 2022, Google published an emergency security update for all Chromium-based web browsers to patch a high severity vulnerability (CVE-2022-1096). At the time of writing, the specifics of the vulnerability are restricted until the majority of users have patched their local browsers.
It is important everyone takes a moment to update their local web browser. It’s one quick and easy action everyone can contribute to the cybersecurity posture of their team.
Even if everyone updated their browser straight away, this remains a reactive measure to a threat that existed before the update was available. Let’s explore how Cloudflare takes a proactive approach by mitigating the impact of zero day browser threats with our zero trust and remote browser isolation services. Cloudflare’s remote browser isolation service is built from the ground up to protect against zero day threats, and all remote browsers on our global network have already been patched.
How Cloudflare Zero Trust protects against browser zero day threats
Cloudflare Zero Trust applies a layered defense strategy to protect users from zero day threats while browsing the Internet:
- Cloudflare’s roaming client steers Internet traffic over an encrypted tunnel to a nearby Cloudflare data center for inspection and Continue reading
How the Oscars impacted the Internet (at least in the US)
The 94th Academy Awards happened this past Sunday, March 27, 2022. In the global event we got to see several Oscars attributed to winners like CODA, Jane Campion (the director of The Power of the Dog) and also Dune (which won six Oscars), but also moments that had a clear impact in the Internet traffic, like the altercation on stage between Will Smith and Chris Rock.
Cloudflare Radar uses a variety of sources to provide aggregate information about Internet traffic and attack trends. In this blog post, we will use DNS name resolution data as a proxy for traffic to Internet services, as we did for the Super Bowl LVI.
The baseline value for the charts (that are only focused on the US) was calculated by taking the mean DNS traffic level for the associated Internet services between 08:00 - 12:00 PST on Sunday (March 27, 2022) — usually we use UTC, but we chose to use Los Angeles time as that’s where the event took place.
The event started with Beyoncé singing at 17:00 PST and ended at around 20:30. In terms of growth in traffic, the start of the show didn’t show much for social media, although TikTok Continue reading
Ridiculously easy to use Tunnels
A little over a decade ago, Cloudflare launched at TechCrunch Disrupt. At the time, we talked about three core principles that differentiated Cloudflare from traditional security vendors: be more secure, more performant, and ridiculously easy to use. Ease of use is at the heart of every decision we make, and this is no different for Cloudflare Tunnel.
That’s why we’re thrilled to announce today that creating tunnels, which previously required up to 14 commands in the terminal, can now be accomplished in just three simple steps directly from the Zero Trust dashboard.
If you’ve heard enough, jump over to sign-up/teams to unplug your VPN and start building your private network with Cloudflare. If you’re interested in learning more about our motivations for this release and what we’re building next, keep scrolling.
Our connector
Cloudflare Tunnel is the easiest way to connect your infrastructure to Cloudflare, whether that be a local HTTP server, web services served by a Kubernetes cluster, or a private network segment. This connectivity is made possible through our lightweight, open-source connector, cloudflared. Our connector offers high-availability by design, creating four long-lived connections to two distinct data centers within Cloudflare’s network. This means that whether an individual Continue reading
Cloudflare Radar’s new ASN pages
An AS, or Autonomous System, is a group of routable IP prefixes belonging to a single entity, and is one of the key building blocks of the Internet. Internet providers, public clouds, governments, and other organizations have one or more ASes that they use to connect their users or systems to the rest of the Internet by advertising how to reach them.
Per AS traffic statistics and trends help when we need insight into unusual events, like Internet outages, infrastructure anomalies, targeted attacks, or any other changes from service providers.
Today, we are opening more of our data and launching the Cloudflare Radar pages for Autonomous Systems. When navigating to a country or region page on Cloudflare Radar you will see a list of five selected ASes for that country or region. But you shouldn’t feel limited to those, as you can deep dive into any AS by plugging its ASN (Autonomous System Number) into the Radar URL (https://radar.cloudflare.com/asn/<number>). We have excluded some statistical trends from ASes with small amounts of traffic as that data would be difficult to interpret.
The AS page is similar to the country page on Cloudflare Radar. You can find traffic levels, protocol Continue reading
Cloudflare’s investigation of the January 2022 Okta compromise
Today, March 22, 2022 at 03:30 UTC we learnt of a compromise of Okta. We use Okta internally for employee identity as part of our authentication stack. We have investigated this compromise carefully and do not believe we have been compromised as a result. We do not use Okta for customer accounts; customers do not need to take any action unless they themselves use Okta.
Investigation and actions
Our understanding is that during January 2022, hackers outside Okta had access to an Okta support employee’s account and were able to take actions as if they were that employee. In a screenshot shared on social media, a Cloudflare employee’s email address was visible, along with a popup indicating the hacker was posing as an Okta employee and could have initiated a password reset.
We learnt of this incident via Cloudflare’s internal SIRT. SIRT is our Security Incident Response Team and any employee at Cloudflare can alert SIRT to a potential problem. At exactly 03:30 UTC, a Cloudflare employee emailed SIRT with a link to a tweet that had been sent at 03:22 UTC. The tweet indicated that Okta had potentially been breached. Multiple other Cloudflare employees contacted SIRT over the following Continue reading
Get updates on the health of your origin where you need them
We are thrilled to announce the availability of Health Checks in the Cloudflare Dashboard’s Notifications tab, available to all Pro, Business, and Enterprise customers. Now, you can get critical alerts on the health of your origin without checking your inbox! Keep reading to learn more about how this update streamlines notification management and unlocks countless ways to stay informed on the health of your servers.
Keeping your site reliable
We first announced Health Checks when we realized some customers were setting up Load Balancers for their origins to monitor the origins’ availability and responsiveness. The Health Checks product provides a similarly powerful interface to Load Balancing, offering users the ability to ensure their origins meet criteria such as reachability, responsiveness, correct HTTP status codes, and correct HTTP body content. Customers can also receive email alerts when a Health Check finds their origin is unhealthy based on their custom criteria. In building a more focused product, we’ve added a slimmer, monitoring-based configuration, Health Check Analytics, and made it available for all paid customers. Health Checks run in multiple locations within Cloudflare’s edge network, meaning customers can monitor site performance across geographic locations.
What’s new with Health Checks Notifications
Health Checks email Continue reading
Zero Trust for SaaS: Deploying mTLS on custom hostnames
Cloudflare has a large base of Software-as-a-Service (SaaS) customers who manage thousands or millions of their customers’ domains that use their SaaS service. We have helped those SaaS providers grow by extending our infrastructure and services to their customer’s domains through a product called Cloudflare for SaaS. Today, we’re excited to give our SaaS providers a new tool that will help their customers add an extra layer of security: they can now enable mutual TLS authentication on their customer’s domains through our Access product.
Primer on Mutual TLS
When you connect to a website, you should see a lock icon in the address bar — that’s your browser telling you that you’re connecting to a website over a secure connection and that the website has a valid public TLS certificate. TLS certificates keep Internet traffic encrypted using a public/private key pair to encrypt and decrypt traffic. They also provide authentication, proving to clients that they are connecting to the correct server.
To make a secure connection, a TLS handshake needs to take place. During the handshake, the client and the server exchange cryptographic keys, the client authenticates the identity of the server, and both the client and the server generate Continue reading
Network performance update: Security Week
Almost a year ago, we shared extensive benchmarking results of last mile networks all around the world. The results showed that on a range of tests (TCP connection time, time to first byte, time to last byte), and on different measures (p95, mean), Cloudflare was the fastest provider in 49% of networks around the world. Since then, we’ve worked to continuously improve performance towards the ultimate goal of being the fastest everywhere. We set a goal to grow the number of networks where we’re the fastest by 10% every Innovation Week. We met that goal last year, and we’re carrying the work over to 2022.
Today, we’re proud to report we are the fastest provider in 71% of the top 1,000 most reported networks around the world. Of course, we’re not done yet, but we wanted to share the latest results and explain how we did it.
Measuring what matters
To quantify network performance, we have to get enough data from around the world, across all manner of different networks, comparing ourselves with other providers. We used Real User Measurements (RUM) to fetch a 100kb file from several different providers. Users around the world report the performance of different providers. Continue reading
Application security: Cloudflare’s view
Developers, bloggers, business owners, and large corporations all rely on Cloudflare to keep their applications secure, available, and performant.
To meet these goals, over the last twelve years we have built a smart network capable of protecting many millions of Internet properties. As of March 2022, W3Techs reports that:
“Cloudflare is used by 80.6% of all the websites whose reverse proxy service we know. This is 19.7% of all websites”
Netcraft, another provider who crawls the web and monitors adoption puts this figure at more than 20M active sites in their latest Web Server Survey (February 2022):
“Cloudflare continues to make strong gains amongst the million busiest websites, where it saw the only notable increases, with an additional 3,200 sites helping to bring its market share up to 19.4%”
The breadth and diversity of the sites we protect, and the billions of browsers and devices that interact with them, gives us unique insight into the ever-changing application security trends on the Internet. In this post, we share some of those insights we’ve gathered from the 32 million HTTP requests/second that pass through our network.
Definitions
Before we examine the data, it is useful to define Continue reading
New cities on the Cloudflare global network: March 2022 edition
If you follow the Cloudflare blog, you know that we love to add cities to our global map. With each new city we add, we help make the Internet faster, more reliable, and more secure. Today, we are announcing the addition of 18 new cities in Africa, South America, Asia, and the Middle East, bringing our network to over 270 cities globally. We’ll also look closely at how adding new cities improves Internet performance, such as our new locations in Israel, which reduced median response time (latency) from 86ms to 29ms (a 66% improvement) in a matter of weeks for subscribers of one Israeli Internet service provider (ISP).
The Cities
Without further ado, here are the 18 new cities in 10 countries we welcomed to our global network: Accra, Ghana; Almaty, Kazakhstan; Bhubaneshwar, India; Chiang Mai, Thailand; Joinville, Brazil; Erbil, Iraq; Fukuoka, Japan; Goiânia, Brazil; Haifa, Israel; Harare, Zimbabwe; Juazeiro do Norte, Brazil; Kanpur, India; Manaus, Brazil; Naha, Japan; Patna, India; São José do Rio Preto, Brazil; Tashkent, Uzbekistan; Uberlândia, Brazil.
Cloudflare’s ISP Edge Partnership Program
But let’s take a step back Continue reading
Unlocking QUIC’s proxying potential with MASQUE
In the last post, we discussed how HTTP CONNECT can be used to proxy TCP-based applications, including DNS-over-HTTPS and generic HTTPS traffic, between a client and target server. This provides significant benefits for those applications, but it doesn’t lend itself to non-TCP applications. And if you’re wondering whether or not we care about these, the answer is an affirmative yes!
For instance, HTTP/3 is based on QUIC, which runs on top of UDP. What if we wanted to speak HTTP/3 to a target server? That requires two things: (1) the means to encapsulate a UDP payload between client and proxy (which the proxy decapsulates and forward to the target in an actual UDP datagram), and (2) a way to instruct the proxy to open a UDP association to a target so that it knows where to forward the decapsulated payload. In this post, we’ll discuss answers to these two questions, starting with encapsulation.
Encapsulating datagrams
While TCP provides a reliable and ordered byte stream for applications to use, UDP instead provides unreliable messages called datagrams. Datagrams sent or received on a connection are loosely associated, each one is independent from a transport perspective. Applications that are built on top of Continue reading
A Primer on Proxies
Traffic proxying, the act of encapsulating one flow of data inside another, is a valuable privacy tool for establishing boundaries on the Internet. Encapsulation has an overhead, Cloudflare and our Internet peers strive to avoid turning it into a performance cost. MASQUE is the latest collaboration effort to design efficient proxy protocols based on IETF standards. We're already running these at scale in production; see our recent blog post about Cloudflare's role in iCloud Private Relay for an example.
In this blog post series, we’ll dive into proxy protocols.
To begin, let’s start with a simple question: what is proxying? In this case, we are focused on forward proxying — a client establishes an end-to-end tunnel to a target server via a proxy server. This contrasts with the Cloudflare CDN, which operates as a reverse proxy that terminates client connections and then takes responsibility for actions such as caching, security including WAF, load balancing, etc. With forward proxying, the details about the tunnel, such as how it is established and used, whether or not it provides confidentiality via authenticated encryption, and so on, vary by proxy protocol. Before going into specifics, let’s start with one of the most common tunnels Continue reading
Securing Cloudflare Using Cloudflare
When a new security threat arises — a publicly exploited vulnerability (like log4j) or the shift from corporate-controlled environments to remote work or a potential threat actor — it is the Security team’s job to respond to protect Cloudflare’s network, customers, and employees. And as security threats evolve, so should our defense system. Cloudflare is committed to bolstering our security posture with best-in-class solutions — which is why we often turn to our own products as any other Cloudflare customer would.
We’ve written about using Cloudflare Access to replace our VPN, Purpose Justification to create granular access controls, and Magic + Gateway to prevent lateral movement from in-house. We experience the same security needs, wants, and concerns as security teams at enterprises worldwide, so we rely on the same solutions as the Fortune 500 companies that trust Cloudflare for improved security, performance, and speed. Using our own products is embedded in our team’s culture.
Security Challenges, Cloudflare Solutions
We’ve built the muscle to think Cloudflare-first when we encounter a security threat. In fact, many security problems we encounter have a Cloudflare solution.
- Problem: Remote work creates a security blind spot of remote devices and networks.
- Solution: Continue reading
Domain Scoped Roles – Early Access
Today, Cloudflare is making it easier for enterprise account owners to manage their team’s access to Cloudflare by allowing user access to be scoped to sets of domains. Ensuring users have exactly the access they need and no more is critical, and Domain Scoped Roles provide a significant step forward. Additionally, with the introduction of Domain Groups, account owners can grant users access to domains by group instead of individually. Domains can be added or removed from these groups to automatically update the access of those who have access to the group. This reduces toil in managing user access.
One of the most common uses we have seen for Domain Scoped Roles is to limit access to production domains to a small set of team members, while still allowing development and pre-production domains to be open to the rest of the team. That way, someone can’t make changes to a production domain unless they are given access.
How to use Domain Scoped Roles
If you are an enterprise customer please talk with your CSM to get you and your team enrolled. Note that you must have Super Administrator privileges to be able to modify account memberships.
Once the beta has Continue reading
Cloudflare Observability
Whether you’re a software engineer deploying a new feature, network engineer updating routes, or a security engineer configuring a new firewall rule: You need visibility to know if your system is behaving as intended — and if it’s not, to know how to fix it.
Cloudflare is committed to helping our customers get visibility into the services they have protected behind Cloudflare. Being a single pane of glass for all network activity has always been one of Cloudflare’s goals. Today, we’re outlining the future vision for Cloudflare observability.
What is observability?
Observability means gaining visibility into the internal state of a system. It’s used to give users the tools to figure out what’s happening, where it’s happening, and why.
At Cloudflare, we believe that observability has three core components: monitoring, analytics, and forensics. Monitoring measures the health of a system - it tells you when something is going wrong. Analytics give you the tools to visualize data to identify patterns and insights. Forensics helps you answer very specific questions about an event.
Observability becomes particularly important in the context of security to validate that any mitigating actions performed by our security products, such as Firewall or Bot Management, are not Continue reading
Using Cloudflare One to Secure IoT Devices
There is probably an insecure device with an exploitable vulnerability sitting in your house. And your office. And probably even your child’s school. Cameras, printers, speakers, access control readers, thermostats, even heart monitors... all of these devices are, or can be, Internet of Things (IoT) devices. These IoT devices are seamlessly integrated into our modern lives to improve efficiency and control of our environments — yet they are notoriously insecure. This is due to the constrained nature of device hardware and their limited computational capacity, which often lead to minimize access controls, hard-coded passwords, and an inability to patch remotely.
The reality of this threat can play out dramatically. Take, for example, the 2016 Mirai botnet attack, in which hackers exploited millions of IoT devices to become a large-scale botnet network capable of launching DDoS attacks that took down major portions of the Internet, including Twitter, the Guardian, and CNN. These types of attacks are hardly an infrequent occurrence. Cloudflare experienced this reality firsthand in March 2021, when one of our potential vendors for physical security cameras, Verkada, was compromised. The incident allowed a hacker to access Verkada's internal support tools to manage the cameras remotely, enabling them to Continue reading
Commitment to Customer Security
Cloudflare has been hooked on securing customers globally since its inception. Our services protect customer traffic and data as well as our own, and we are continuously improving and expanding those services to respond to the changing threat landscape of the Internet. Proving that commitment is a multi-faceted venture, the Security Team focuses on people, proof, and transparency to ensure every touchpoint with our products and company feels dependable.
People
The breadth of knowledge of the Security Team is wide and bleeding edge. Working as a security team at a security company means being highly technical, diverse, willing to test any and all products on ourselves, and sharing our knowledge with our local and global communities through industry groups and presenting at conferences worldwide. Connecting with our customers and counterparts through meetups and conferences lets us share problems, learn about upcoming industry trends, and share feedback to make improvements to the customer experience. In addition to running a formally documented, risk-based security program for Cloudflare, team members drive continuous improvement efforts across our Product and Infrastructure teams by reviewing and advising on changes, identifying and treating vulnerabilities, controlling authorization and access to systems and data, encrypting data in Continue reading
Zero Trust client sessions
Starting today, you can build Zero Trust rules that require periodic authentication to control network access. We’ve made this feature available for years for web-based applications, but we’re excited to bring this level of granular enforcement to TCP connections and UDP flows.
We’re excited to announce that Zero Trust client-based sessions are now generally available. During CIO Week in 2021, we announced the beta program for this feature. We incorporated feedback from early users into the generally available version. In this post, I will revisit why Zero Trust client-based sessions are important, how the feature works and what we learned during the beta.
Securing traffic with Sessions
We built Zero Trust client-based sessions to enhance the security of Cloudflare’s Zero Trust Network Access (ZTNA). The Zero Trust client is software that runs on a user machine and forwards all traffic from the machine to Cloudflare before it is sent over the Internet. This includes traffic bound for internal IPs and hostnames that typically house sensitive business applications. These sensitive applications were traditionally accessed using a VPN. Unlike VPNs, Cloudflare’s ZTNA allows administrators to set granular policies about who can access a specific resource. The only piece missing was that once Continue reading
Introducing SSH command logging
SSH (Secure Shell Protocol) is an important protocol for managing remote machines. It provides a way for infrastructure teams to remotely and securely manage their fleet of machines. SSH was a step-up in security from other protocols like telnet. It ensures encrypted traffic and enforces per user controls over access to a particular machine. However, it can still introduce a significant security risk. SSH, especially root access, is destructive in the wrong hands (think rm -r *) and can be difficult to track. Logging and securing user actions via SSH typically requires custom development or restrictive software deployments. We’re excited to announce SSH command logging as part of Cloudflare Zero Trust.
Securing SSH access
Security teams put significant effort into securing SSH across their organization because of the negative impact it can have in the wrong hands. Traditional SSH security consists of strong authentication, like certificate based authentication, and tight controls on who has “root” access. Additionally, VPNs and IP allow lists are used to further protect a machine from being publicly accessible to the Internet. The security challenges that remain are visibility and potential for lateral movement.
SSH commands to a remote machine are end-to-end encrypted, which means Continue reading