Using Cloudflare Access with CNI
We are thrilled to introduce an innovative new approach to secure hosted applications via Cloudflare Access without the need for any installed software or custom code on your application server. But before we dive into how this is possible, let's review why Access previously required installed software or custom code on your application server.
Protecting an application with Access
Traditionally, companies used a Virtual Private Network (VPN) to access a hosted application, where all they had to do was configure an IP allowlist rule for the VPN. However, this is a major security threat because anyone on the VPN can access the application, including unauthorized users or attackers.
We built Cloudflare Access to replace VPNs and provide the option to enforce Zero Trust policies in hosted applications. Access allows you to verify a user's identity before they even reach the application. By acting as a proxy in front of your application's hostname (e.g. app.example.com), Cloudflare enables strong verification techniques such as identity, device posture, hardkey MFA, and more. All without having to directly add SSO or Authentication logic directly into your applications.
However, since Access enforces at a hostname level, there is still a potential Continue reading
Cloudflare Aegis: dedicated IPs for Zero Trust migration
Realizing the goals of Zero Trust is a journey: moving from a world of static networking and hardware concepts to organization-based access and continuous validation is not a one-step process. This challenge is never more real than when dealing with IP addresses. For years, companies on the Internet have built hardened systems based on the idea that only users with certain IP addresses can access certain resources. This implies that IP addresses are tied with identity, which is a kluge and can actually open websites up to attack in some cases. For large companies with many origins and applications that need to be protected in a Zero Trust model, it’s important to be able to support their transition to Zero Trust using mTLS, Access, or Tunnel. To make the transition some organizations may need dedicated IP addresses.
Today we’re introducing Cloudflare Aegis: dedicated IPs that we use to send you traffic. This allows you to lock down your services and applications at an IP level and build a protected environment that is application aware, protocol aware, and even IP-aware. Aegis is available today through Early Access for Enterprise customers, and you can talk to your account team if you want Continue reading
Mutual TLS now available for Workers
In today’s digital world, security is a top priority for businesses. Whether you’re a Fortune 500 company or a startup just taking off, it’s essential to implement security measures in order to protect sensitive information. Security starts inside an organization; it starts with having Zero Trust principles that protect access to resources.
Mutual TLS (mTLS) is useful in a Zero Trust world to secure a wide range of network services and applications: APIs, web applications, microservices, databases and IoT devices. Cloudflare has products that enforce mTLS: API Shield uses it to secure API endpoints and Cloudflare Access uses it to secure applications. Now, with mTLS support for Workers you can use Workers to authenticate to services secured by mTLS directly. mTLS for Workers is now generally available for all Workers customers!
A recap on how TLS works
Before diving into mTLS, let’s first understand what TLS (Transport Layer Security) is. Any website that uses HTTPS, like the one you’re reading this blog on, uses TLS encryption. TLS is used to create private communications on the Internet – it gives users assurance that the website you’re connecting to is legitimate and any information passed to it is encrypted.
TLS is enforced Continue reading
How to stay safe from phishing
As you wake up in the morning feeling sleepy and preoccupied, you receive an urgent email from a seemingly familiar source, and without much thought, you click on a link that you shouldn't have. Sometimes it’s that simple, and this more than 30-year-old phishing method means chaos breaks loose – whether it’s your personal bank account or social media, where an attacker also begins to trick your family and friends; or at your company, with what could mean systems and data being compromised, services being disrupted, and all other subsequent consequences. Following up on our “Top 50 Most Impersonated Brands in phishing attacks” post, here are some tips to catch these scams before you fall for them.
We’re all human, and responding to or interacting with a malicious email remains the primary way to breach organizations. According to CISA, 90% of cyber attacks begin with a phishing email, and losses from a similar type of phishing attack, known as business email compromise (BEC), are a $43 billion problem facing organizations. One thing is for sure, phishing attacks are getting more sophisticated every day thanks to emerging tools like AI chatbots and the expanded usage of various communication Continue reading
Welcome to Security Week 2023
Last month I had the chance to attend a dinner with 56 CISOs and CSOs across a range of banking, gaming, ecommerce, and retail companies. We rotated between tables of eight people and talked about the biggest challenges those in the group were facing, and what they were most worried about around the corner. We talk to customers every day at Cloudflare, but this was a unique opportunity to listen to customers (and non-customers) talk to each other. It was a fascinating evening and a few things stood out.
The common thread that dominated the discussions was “how do I convince my business and product teams to do the things I want them to”. Surprisingly little time was spent on specific technical challenges. No one brought up a concern about recent advanced mage cart skimmers, or about protecting their new GraphQL APIs, or how to secure two different cloud vendors at once, or about the size of DDoS attacks consistently getting larger. Over and over again the conversation came back to struggles with getting humans to do the secure thing, or to not do the insecure thing.
This instantly brought to mind a major phishing attack that Cloudflare was Continue reading
Cloudflare’s network expansion in Indonesia
As home to over 200 million Internet users and the fourth-largest population in the world, Indonesians depend on fast and reliable Internet, but this has always been a challenging part of the world for Internet infrastructure. This has real world implications on performance and reliability (IP transit is on average 6x more expensive than our major South East Asian interconnection markets). That said, first we wanted to share what makes things challenging in Indonesia; geography, infrastructure, and market dynamics.
Geography: The Internet backbone for many countries is almost entirely delivered by terrestrial fiber optic cables, where connectivity is more affordable and easier to build when the land mass is contiguous and there is a concentrated population distribution. However, Indonesia is a collection of over 18,000 islands, spanning three time zones, and approximately 3,200 miles (5,100 km) east to west. By comparison, the United States is 2,800 miles (4,500 km) east to west. While parts of Indonesia are geographically close to Singapore (the regional Internet hub with over 60% of the region's data centers) given how large Indonesia is, much of it is far away.
Infrastructure: Indonesia is a large country and to connect it to the rest of the Internet Continue reading
Deploying firmware at Cloudflare-scale: updating thousands of servers in more than 285 cities
As a security company, it’s critical that we have good processes for dealing with security issues. We regularly release software to our servers - on a daily basis even - which includes new features, bug fixes, and as required, security patches. But just as critical is the software which is embedded into the server hardware, known as firmware. Primarily of interest is the BIOS and Baseboard Management Controller (BMC), but many other components also have firmware such as Network Interface Cards (NICs).
As the world becomes more digital, software which needs updating is appearing in more and more devices. As well as my computer, over the last year, I have waited patiently while firmware has updated in my TV, vacuum cleaner, lawn mower and light bulbs. It can be a cumbersome process, including obtaining the firmware, deploying it to the device which needs updating, navigating menus and other commands to initiate the update, and then waiting several minutes for the update to complete.
Firmware updates can be annoying even if you only have a couple of devices. We have more than a few devices at Cloudflare. We have a huge number of servers of varying kinds, from varying vendors, spread Continue reading
Embrace equity on International Women’s Day (and every day)
This post is also available in 简体中文, 日本語, 한국어, Deutsch, Français, Español and 繁體中文.
Happy International Women’s Day! The global theme for 2023 is #EmbraceEquity, which is part of an ongoing effort to raise awareness around “Why equal opportunities are no longer enough.” Today is a time to highlight achievements made by women, but also an opportunity to become better informed, and collaborate and brainstorm about the path forward.
“People start from different places, so true inclusion and belonging require equitable action.” — internationalwomensday.com
Help put an end to gender bias and discrimination
Consider taking a few minutes today to learn about pervasive challenges affecting women, including in the workplace. Since unconscious bias is a major driver of hurdles holding women back, it is beneficial for people of all gender identities to educate ourselves about the varied experiences of others.
Here are some resources to get help get you started:
- Recognize the difference between equity and equality and see why striving for equality can interfere with inclusion-related efforts.
- Read highlights from the Women in the Workplace report from McKinsey and LeanIn.Org to examine factors that are holding women back from Continue reading
Accelerate building resiliency into systems with Cloudflare Workers
In this blog post we’ll discuss how Cloudflare Workers enabled us to quickly improve the resiliency of a legacy system. In particular, we’ll discuss how we prevented the email notification systems within Cloudflare from outages caused by external vendors.
Email notification services
At Cloudflare, we send email notifications to customers such as billing invoices, password resets, OTP logins and certificate status updates. We rely on external Email Service Providers (ESPs) to deliver these emails to customers.
The following diagram shows how the system looks. Multiple services in our control plane dispatch emails through an external email vendor. They use HTTP Transmission APIs and also SMTP to send messages to the vendor. If dispatching an email fails, they are retried with exponential back-off mechanisms. Even when our ESP has outages, the retry mechanisms in place guarantee that we don’t lose any messages.
Why did we need to improve resilience?
In some cases, it isn’t sufficient to just deliver the email to the customer; it must be delivered on time. For example, OTP login emails are extremely time sensitive; their validity is short-lived such that a delay in sending them is as bad as not sending them at all. If the ESP Continue reading
New Zero Trust navigation coming soon (and we need your feedback)
We’re updating the Zero Trust navigation
On March 20, 2023, we will be launching an updated navigation in the Zero Trust dashboard, offering all of our Zero Trust users a more seamless experience across Cloudflare as a whole. This change will allow you to more easily manage your Zero Trust organization alongside your application and network services, developer tools, and more.
As part of this upcoming release, you will see three key changes:
Quicker navigation
Instead of opening another window or typing in a URL, you can go back to the Cloudflare dashboard in one click.
Switch accounts with ease
View and switch accounts at the top of your sidebar.
Resources and support
Find helpful links to our Community, developer documentation, and support team at the top of your navigation bar.
Why we’re updating the Zero Trust navigation
In 2020, Gateway was broadly released as the first Cloudflare product that didn’t require a site hosted on Cloudflare’s infrastructure. In other words, Gateway was unconstrained by the site-specific model most other Cloudflare products relied on at the time, while also used in close conjunction with Access. And so, the Cloudflare for Teams dashboard was built on a new model, designed from Continue reading
The White House’s National Cybersecurity Strategy asks the private sector to step up to fight cyber attacks. Cloudflare is ready
On Thursday, March 2, 2023, the Biden-Harris Administration released the National Cybersecurity Strategy aimed at securing the Internet. Cloudflare welcomes the Strategy, and congratulates the White House on this comprehensive, much-needed policy initiative. The goal of the Strategy is to make the digital ecosystem defensible, resistant, and values-aligned. This is a goal that Cloudflare fully supports. The Strategy recognizes the vital role that the private sector has to play in defending the United States against cyber attacks.
The Strategy aims to make a fundamental shift and transformation of roles, responsibilities, and resources in cyberspace by (1) rebalancing the responsibility to defend cyberspace by shifting the burden away from individuals, small businesses, and local governments, and onto organizations that are most capable and best-positioned to reduce risks, like data holders and technology providers; and (2) realigning incentives to favor long-term investments by balancing defending the United States against urgent threats today and simultaneously investing in a resilient future. The Strategy envisions attaining these goals through five collaborative pillars:
- Pillar One: defending critical infrastructure;
- Pillar Two: disrupting and dismantling threat actors;
- Pillar Three: shaping market forces to drive security and resilience;
- Pillar Four: investing in a resilient future; and
- Pillar Five: forging Continue reading
Keeping the Cloudflare API ‘all green’ using Python-based testing
At Cloudflare, we reuse existing core systems to power multiple products and testing of these core systems is essential. In particular, we require being able to have a wide and thorough visibility of our live APIs’ behaviors. We want to be able to detect regressions, prevent incidents and maintain healthy APIs. That is why we built Scout.
Scout is an automated system periodically running Python tests verifying the end to end behavior of our APIs. Scout allows us to evaluate APIs in production-like environments and thus ensures we can green light a production deployment while also monitoring the behavior of APIs in production.
Why Scout?
Before Scout, we were using an automated test system leveraging the Robot Framework. This older system was limiting our testing capabilities. In fact, we could not easily match json responses against keys we were looking for. We would abandon covering different behaviors of our APIs as it was impossible to decide on which resources a given test suite would run. Two different test suites would create false negatives as they were running on the same account.
Regarding schema validation, only API responses were validated against a json schema and tests would not fail if the Continue reading
How Cloudflare runs Prometheus at scale
We use Prometheus to gain insight into all the different pieces of hardware and software that make up our global network. Prometheus allows us to measure health & performance over time and, if there’s anything wrong with any service, let our team know before it becomes a problem.
At the moment of writing this post we run 916 Prometheus instances with a total of around 4.9 billion time series. Here’s a screenshot that shows exact numbers:
That’s an average of around 5 million time series per instance, but in reality we have a mixture of very tiny and very large instances, with the biggest instances storing around 30 million time series each.
Operating such a large Prometheus deployment doesn’t come without challenges. In this blog post we’ll cover some of the issues one might encounter when trying to collect many millions of time series per Prometheus instance.
Metrics cardinality
One of the first problems you’re likely to hear about when you start running your own Prometheus instances is cardinality, with the most dramatic cases of this problem being referred to as “cardinality explosion”.
So let’s start by looking at what cardinality means from Prometheus' perspective, when it can Continue reading
Oxy is Cloudflare’s Rust-based next generation proxy framework
In this blog post, we are proud to introduce Oxy - our modern proxy framework, developed using the Rust programming language. Oxy is a foundation of several Cloudflare projects, including the Zero Trust Gateway, the iCloud Private Relay second hop proxy, and the internal egress routing service.
Oxy leverages our years of experience building high-load proxies to implement the latest communication protocols, enabling us to effortlessly build sophisticated services that can accommodate massive amounts of daily traffic.
We will be exploring Oxy in greater detail in upcoming technical blog posts, providing a comprehensive and in-depth look at its capabilities and potential applications. For now, let us embark on this journey and discover what Oxy is and how we built it.
What Oxy does
We refer to Oxy as our "next-generation proxy framework". But what do we really mean by “proxy framework”? Picture a server (like NGINX, that reader might be familiar with) that can proxy traffic with an array of protocols, including various predefined common traffic flow scenarios that enable you to route traffic to specific destinations or even egress with a different protocol than the one used for ingress. This server can be configured in many ways Continue reading
How we built an open-source SEO tool using Workers, D1, and Queues
Building applications on Cloudflare Workers has always been fun. Workers applications have low latency response times by default, and easy developer ergonomics thanks to Wrangler. It's no surprise that for years now, developers have been going from idea to production with Workers in just a few minutes.
Internally, we're no different. When a member of our team has a project idea, we often reach for Workers first, and not just for the MVP stage, but in production, too. Workers have been a secret ingredient to Cloudflare’s innovation for some time now, allowing us to build products like Access, Stream and Workers KV. Even better, when we have new ideas and we can use new Cloudflare products to build them, it's a great way to give feedback on those products.
We've discussed this in the past on the Cloudflare blog - in May last year, I wrote how we rebuilt Cloudflare's developer documentation using many of the tools that had recently been released in the Workers ecosystem: Cloudflare Pages for hosting, and Bulk Redirects for the redirect rules. In November, we released a new version of our API documentation, which again used Pages for hosting, and Pages functions for intelligent Continue reading
How Rust and Wasm power Cloudflare’s 1.1.1.1
On April 1, 2018, Cloudflare announced the 1.1.1.1 public DNS resolver. Over the years, we added the debug page for troubleshooting, global cache purge, 0 TTL for zones on Cloudflare, Upstream TLS, and 1.1.1.1 for families to the platform. In this post, we would like to share some behind the scenes details and changes.
When the project started, Knot Resolver was chosen as the DNS resolver. We started building a whole system on top of it, so that it could fit Cloudflare's use case. Having a battle tested DNS recursive resolver, as well as a DNSSEC validator, was fantastic because we could spend our energy elsewhere, instead of worrying about the DNS protocol implementation.
Knot Resolver is quite flexible in terms of its Lua-based plugin system. It allowed us to quickly extend the core functionality to support various product features, like DoH/DoT, logging, BPF-based attack mitigation, cache sharing, and iteration logic override. As the traffic grew, we reached certain limitations.
Lessons we learned
Before going any deeper, let’s first have a bird’s-eye view of a simplified Cloudflare data center setup, which could help us understand what we are going to talk Continue reading
ROFL with a LOL: rewriting an NGINX module in Rust
At Cloudflare, engineers spend a great deal of time refactoring or rewriting existing functionality. When your company doubles the amount of traffic it handles every year, what was once an elegant solution to a problem can quickly become outdated as the engineering constraints change. Not only that, but when you're averaging 40 million requests a second, issues that might affect 0.001% of requests flowing through our network are big incidents which may impact millions of users, and one-in-a-trillion events happen several times a day.
Recently, we've been working on a replacement to one of our oldest and least-well-known components called cf-html, which lives inside the core reverse web proxy of Cloudflare known as FL (Front Line). Cf-html is the framework in charge of parsing and rewriting HTML as it streams back through from the website origin to the website visitor. Since the early days of Cloudflare, we’ve offered features which will rewrite the response body of web requests for you on the fly. The first ever feature we wrote in this way was to replace email addresses with chunks of JavaScript, which would then load the email address when viewed in a web browser. Since bots are often unable Continue reading
One year of war in Ukraine: Internet trends, attacks, and resilience
The Internet has become a significant factor in geopolitical conflicts, such as the ongoing war in Ukraine. Tomorrow marks one year since the Russian invasion of that country. This post reports on Internet insights and discusses how Ukraine's Internet remained resilient in spite of dozens of disruptions in three different stages of the conflict.
Key takeaways:
- Internet traffic shifts in Ukraine are clearly visible from east to west as Ukrainians fled the war, with country-wide traffic dropping as much as 33% after February 24, 2022.
- Air strikes on energy infrastructure starting in October led to widespread Internet disruptions that continue in 2023.
- Application-layer cyber attacks in Ukraine rose 1,300% in early March 2022 compared to pre-war levels.
- Government administration, financial services, and the media saw the most attacks targeting Ukraine.
- Traffic from a number of networks in Kherson was re-routed through Russia between June and October, subjecting traffic to Russia’s restrictions and limitations, including content filtering. Even after traffic ceased to reroute through Russia, those Ukrainian networks saw major outages through at least the end of the year, while two networks remain offline.
- Through efforts on the ground to repair damaged fiber optics and restore electrical power, Ukraine’s networks have Continue reading
The Chief Zero Trust Officer: a new role for a new era of cybersecurity
Setting the stage for Zero Trust
Over the last few years the topic of cyber security has moved from the IT department to the board room. The current climate of geopolitical and economic uncertainty has made the threat of cyber attacks all the more pressing, with businesses of all sizes and across all industries feeling the impact. From the potential for a crippling ransomware attack to a data breach that could compromise sensitive consumer information, the risks are real and potentially catastrophic. Organizations are recognizing the need for better resilience and preparation regarding cybersecurity. It is not enough to simply react to attacks as they happen; companies must proactively prepare for the inevitable in their approach to cybersecurity.
The security approach that has gained the most traction in recent years is the concept of Zero Trust. The basic principle behind Zero Trust is simple: don't trust anything; verify everything. The impetus for a modern Zero Trust architecture is that traditional perimeter-based (castle-and-moat) security models are no longer sufficient in today's digitally distributed landscape. Organizations must adopt a holistic approach to security based on verifying the identity and trustworthiness of all users, devices, and systems that access their networks and data.
So long, and thanks for all the deployments: deprecating Wrangler v1
Cloudflare Workers allow developers to deploy code instantly across the globe. Wrangler is the CLI tool we build (and use!) to create, modify, and upload Workers. We recently announced a new version of Wrangler with a bunch of new features – including offline development, zero-config startup, and developer tools support. Since then, we’ve been working hard to make the developer experience with version 2 as smooth and enjoyable as possible. We’re confident in what we’ve built and are now planning to officially deprecate version 1.
What’s happening?
Version 1 of Wrangler (@cloudflare/wrangler on npm) is now deprecated, which means no new features or bug fixes will be published unless they’re critical. Beginning August 2023, no further updates will be provided and the Wrangler v1 GitHub repo will be archived. We strongly recommend you upgrade to version 2 (wrangler on npm) to receive continued support. We have a migration guide to make this process easy!
Why?
Our goal is to make development on the Cloudflare platform as smooth and enjoyable as possible. Whether that means simplifying common workflows, incorporating powerful tools into the Wrangler codebase, or opening up Wrangler for use as a library Continue reading