Connection coalescing with ORIGIN Frames: fewer DNS queries, fewer connections
This blog reports and summarizes the contents of a Cloudflare research paper which appeared at the ACM Internet Measurement Conference, that measures and prototypes connection coalescing with ORIGIN Frames.
Some readers might be surprised to hear that a single visit to a web page can cause a browser to make tens, sometimes even hundreds, of web connections. Take this very blog as an example. If it is your first visit to the Cloudflare blog, or it has been a while since your last visit, your browser will make multiple connections to render the page. The browser will make DNS queries to find IP addresses corresponding to blog.cloudflare.com and then subsequent requests to retrieve any necessary subresources on the web page needed to successfully render the complete page. How many? Looking below, at the time of writing, there are 32 different hostnames used to load the Cloudflare Blog. That means 32 DNS queries and at least 32 TCP (or QUIC) connections, unless the client is able to reuse (or coalesce) some of those connections.
Each new web connection not only introduces additional load on a server's processing capabilities – potentially leading to scalability challenges during peak usage hours Continue reading
Improving Worker Tail scalability
Being able to get real-time information from applications in production is extremely important. Many times software passes local testing and automation, but then users report that something isn’t working correctly. Being able to quickly see what is happening, and how often, is critical to debugging.
This is why we originally developed the Workers Tail feature - to allow developers the ability to view requests, exceptions, and information for their Workers and to provide a window into what’s happening in real time. When we developed it, we also took the opportunity to build it on top of our own Workers technology using products like Trace Workers and Durable Objects. Over the last couple of years, we’ve continued to iterate on this feature - allowing users to quickly access logs from the Dashboard and via Wrangler CLI.
Today, we’re excited to announce that tail can now be enabled for Workers at any size and scale! In addition to telling you about the new and improved scalability, we wanted to share how we built it, and the changes we made to enable it to scale better.
Why Tail was limited
Tail leverages Durable Objects to handle coordination between the Worker producing messages and Continue reading
Cloudflare Radar’s 2023 overview of new tools and insights
Cloudflare Radar was launched in September 2020, almost three years ago, when the pandemic was affecting Internet traffic usage. It is a free tool to show Internet usage patterns from both human and automated systems, as well as attack trends, top domains, and adoption and usage of browsers and protocols. As Cloudflare has been publishing data-driven insights related to the general Internet for more than 10 years now, Cloudflare Radar is a natural evolution.
This year, we have introduced several new features to Radar, also available through our public API, that enables deeper data exploration. We’ve also launched an Internet Quality section, a Trending Domains section, a URL Scanner tool, and a Routing section to track network interconnection, routing security, and observed routing anomalies.
In this reading list, we want to highlight some of those new additions, as well as some of the Internet disruptions and trends we’ve observed and published posts about during this year, including the war in Ukraine, the impact of Easter, and exam-related shutdowns in Iraq and Algeria.
We also encourage everyone to explore Cloudflare Radar and its new features, and to give you a partial review of the year, in terms of Internet Continue reading
Cloudflare’s tenant platform in action: Meter deploys DNS filtering at scale
In January 2023, we announced support for Managed Service Providers (MSPs) and other businesses to create 'parent-child' and account-level policy configurations when deploying Cloudflare for DNS filtering. Specifically, organizations leverage the integration between our Tenant API and Cloudflare Gateway, our Secure Web Gateway (SWG) to protect their remote or office end users with web filtering and inspection. Already, customers like the US federal government, MalwareBytes, and a large global ISP take advantage of this integration to enable simpler, more flexible policy management across larger deployments across their end customers
Today, we're excited to showcase another similar story: Meter, a provider of Internet infrastructure, is leveraging the Tenant API integration for DNS filtering to help their clients enforce acceptable Internet use policies.
How Meter deploys Cloudflare to secure Internet browsing
Meter, headquartered in San Francisco and founded in 2015, provides Internet infrastructure that includes routing, switching, wireless, and applications. They help deliver faster, more efficient, more secure networking experiences for a diverse range of corporate spaces, including offices, warehouses, retail, manufacturing, biotech, and education institutions.
Meter integrates with the Cloudflare Tenant API to provide DNS filtering to their customers. With the Meter dashboard, Meter customers can set policies to Continue reading
Star your favorite websites in the dashboard
We’re excited to introduce starring, a new dashboard feature built to speed up your workflow. You can now “star” up to 10 of the websites and applications you have on Cloudflare for quicker access.
Star your websites or applications for more efficiency
We have heard from many of our users, particularly ones with tens to hundreds of websites and applications running on Cloudflare, about the need to “favorite” the ones they monitor or configure most often. For example, domains or subdomains that our users designate for development or staging may be accessed in the Cloudflare dashboard daily during a build, migration or a first-time configuration, but then rarely touched for months at a time; yet every time logging in, these users have had to go through multiple steps—searching and paging through results—to navigate to where they need to go. These users seek a more efficient workflow to get to their destination faster. Now, by starring your websites or applications, you can have easier access.
How to get started
Star a website or application
Today, you can star up to 10 items per account. Simply star a website or application you have added to Cloudflare from its Overview page. Once Continue reading
Introducing the 2023 Intern-ets!
This year, Cloudflare welcomed a class of approximately 40 interns, hailing from five different countries for an unforgettable summer. As we joined both remotely and in-person across Cloudflare’s global offices, our experiences spanned a variety of roles from engineering, product management to internal auditing and marketing. Through invaluable mentorship, continuous learning, and the chance to make a real-world impact, our summer was truly enriched at every step. Join us, Anni and Emilie, as we provide an insider's perspective on a summer at Cloudflare, sharing snippets and quotes from our intern cohort.
printf(“Hello Intern-ets!”)
You might have noticed that we have a new name for the interns: the Intern-ets! Our fresh intern nickname was born from a brainstorm between us and our recruiter, Judy. While “Cloudies”, “Cloudterns”, and “Flaries” made the shortlist, a company-wide vote crowned "Intern-ets" as the favorite. And just like that, we've made Cloudflare history!
git commit -m “Innovation!”
We're all incredibly proud to have gotten the opportunity to tackle interesting and highly impactful projects throughout the duration of our internships. To give you a glimpse of our summer, here are a few that showcase the breadth and depth of our experiences.
Mia M., Product Continue reading
Application Security Report: Q2 2023
Cloudflare has a unique vantage point on the Internet. From this position, we are able to see, explore, and identify trends that would otherwise go unnoticed. In this report we are doing just that and sharing our insights into Internet-wide application security trends.
This report is the third edition of our Application Security Report. The first one was published in March 2022, with the second published earlier this year in March, and this is the first to be published on a quarterly basis.
Since the last report, our network is bigger and faster: we are now processing an average of 46 million HTTP requests/second and 63 million at peak. We consistently handle approximately 25 million DNS queries per second. That's around 2.1 trillion DNS queries per day, and 65 trillion queries a month. This is the sum of authoritative and resolver requests served by our infrastructure. Summing up both HTTP and DNS requests, we get to see a lot of malicious traffic. Focusing on HTTP requests only, in Q2 2023 Cloudflare blocked an average of 112 billion cyber threats each day, and this is the data that powers this report.
But as usual, before we dive in, Continue reading
An August reading list about online security and 2023 attacks landscape
In 2023, cybersecurity continues to be in most cases a need-to-have for those who don’t want to take chances on getting caught in a cyberattack and its consequences. Attacks have gotten more sophisticated, while conflicts (online and offline, and at the same time) continue, including in Ukraine. Governments have heightened their cyber warnings and put together strategies, including around critical infrastructure (including health and education). All of this, at a time when there were never so many online risks, but also people online — over five billion in July 2023, 64.5% of the now eight billion that are the world’s total population.
Here we take a look at what we’ve been discussing in 2023, so far, in our Cloudflare blog related to attacks and online security in general, with several August reading list suggestions. From new trends, products, initiatives or partnerships, including AI service safety, to record-breaking blocked cyberattacks. On that note, our AI hub (ai.cloudflare.com) was just launched.
Throughout the year, Cloudflare has continued to onboard customers while they were being attacked, and we have provided protection to many others, including once.net, responsible for the 2023 Eurovision Song Contest online voting system Continue reading
Introducing Cloudflare’s 2023 phishing threats report
After shutting down a ‘phishing-as-a-service’ operation that impacted thousands of victims in 43 countries, INTERPOL recently noted, “Cyberattacks such as phishing may be borderless and virtual in nature, but their impact on victims is real and devastating.” Business email compromise (BEC), a type of malware-less attack that tricks recipients into transferring funds — for example — has cost victims worldwide more than $50 billion, according to the FBI.
It is estimated that 90% of successful cyber attacks start with email phishing, which continues to be very lucrative for attackers. There is not much today that can be done to stop phishing attempts. However, to prevent successful attacks, it is important to understand (and proactively address) evolving phishing trends — including the ways attackers cleverly exploit intended victims’ trust in “known” email senders. To that end, this week Cloudflare published its first Phishing Threats Report.
This report explores key phishing trends and related recommendations, based on email security data from May 2022 to May 2023. During that time, Cloudflare processed approximately 13 billion emails, which included blocking approximately 250 million malicious messages from reaching customers’ inboxes. The report is also informed by a Cloudflare-commissioned survey Continue reading
Wasm core dumps and debugging Rust in Cloudflare Workers
A clear sign of maturing for any new programming language or environment is how easy and efficient debugging them is. Programming, like any other complex task, involves various challenges and potential pitfalls. Logic errors, off-by-ones, null pointer dereferences, and memory leaks are some examples of things that can make software developers desperate if they can't pinpoint and fix these issues quickly as part of their workflows and tools.
WebAssembly (Wasm) is a binary instruction format designed to be a portable and efficient target for the compilation of high-level languages like Rust, C, C++, and others. In recent years, it has gained significant traction for building high-performance applications in web and serverless environments.
Cloudflare Workers has had first-party support for Rust and Wasm for quite some time. We've been using this powerful combination to bootstrap and build some of our most recent services, like D1, Constellation, and Signed Exchanges, to name a few.
Using tools like Wrangler, our command-line tool for building with Cloudflare developer products, makes streaming real-time logs from our applications running remotely easy. Still, to be honest, debugging Rust and Wasm with Cloudflare Workers involves a lot of the good old time-consuming and Continue reading
Debug Queues from the dash: send, list, and ack messages
Today, August 11, 2023, we are excited to announce a new debugging workflow for Cloudflare Queues. Customers using Cloudflare Queues can now send, list, and acknowledge messages directly from the Cloudflare dashboard, enabling a more user-friendly way to interact with Queues. Though it can be difficult to debug asynchronous systems, it’s now easy to examine a queue’s state and test the full flow of information through a queue.
With guaranteed delivery, message batching, consumer concurrency, and more, Cloudflare Queues is a powerful tool to connect services reliably and efficiently. Queues integrate deeply with the existing Cloudflare Workers ecosystem, so developers can also leverage our many other products and services. Queues can be bound to producer Workers, which allow Workers to send messages to a queue, and to consumer Workers, which pull messages from the queue.
We’ve received feedback that while Queues are effective and performant, customers find it hard to debug them. After a message is sent to a queue from a producer worker, there’s no way to inspect the queue’s contents without a consumer worker. The limited transparency was frustrating, and the need to write a skeleton worker just to debug a queue was high-friction.
Now, Continue reading
Introducing scheduled deletion for Cloudflare Stream
Designed with developers in mind, Cloudflare Stream provides a seamless, integrated workflow that simplifies video streaming for creators and platforms alike. With features like Stream Live and creator management, customers have been looking for ways to streamline storage management.
Today, August 11, 2023, Cloudflare Stream is introducing scheduled deletion to easily manage video lifecycles from the Stream dashboard or our API, saving time and reducing storage-related costs. Whether you need to retain recordings from a live stream for only a limited time, or preserve direct creator videos for a set duration, scheduled deletion will simplify storage management and reduce costs.
Stream scheduled deletion
Scheduled deletion allows developers to automatically remove on-demand videos and live recordings from their library at a specified time. Live inputs can be set up with a deletion rule, ensuring that all recordings from the input will have a scheduled deletion date upon completion of the stream.
Let’s see how it works in those two configurations.
Getting started with scheduled deletion for on-demand videos
Whether you run a learning platform where students can upload videos for review, a platform that allows gamers to share clips of their gameplay, or anything in between, scheduled deletion can help Continue reading
Introducing per hostname TLS settings — security fit to your needs
One of the goals of Cloudflare is to give our customers the necessary knobs to enable security in a way that fits their needs. In the realm of SSL/TLS, we offer two key controls: setting the minimum TLS version, and restricting the list of supported cipher suites. Previously, these settings applied to the entire domain, resulting in an “all or nothing” effect. While having uniform settings across the entire domain is ideal for some users, it sometimes lacks the necessary granularity for those with diverse requirements across their subdomains.
It is for that reason that we’re excited to announce that as of today, customers will be able to set their TLS settings on a per-hostname basis.
The trade-off with using modern protocols
In an ideal world, every domain could be updated to use the most secure and modern protocols without any setbacks. Unfortunately, that's not the case. New standards and protocols require adoption in order to be effective. TLS 1.3 was standardized by the IETF in April 2018. It removed the vulnerable cryptographic algorithms that TLS 1.2 supported and provided a performance boost by requiring only one roundtrip, as opposed to two. For a user to benefit from Continue reading
Project Cybersafe Schools: Bringing security tools for free to small K-12 school districts in the US
Like other under-resourced organizations, schools face cyber attacks from malicious actors that can impact their ability to safely perform their basic function: teach children. Schools face email, phishing, and ransomware attacks that slow access and threaten leaks of confidential student data. And these attacks have real effects. In a report issued at the end of 2022, the U.S. Government Accountability Office concluded that schools serving kindergarten through 12th grade (K-12) reported significant educational impact and monetary loss due to cybersecurity incidents, such as ransomware attacks. Recovery time can extend from 2 all the way up to 9 months — that’s almost an entire school year.
Cloudflare’s mission is to help build a better Internet, and we have always believed in helping protect those who might otherwise not have the resources to protect themselves from cyberattack.
It is against this backdrop that we’re very excited to introduce an initiative aimed at small K-12 public school districts: Project Cybersafe Schools. Announced as part of the Back to School Safely: K-12 Cybersecurity Summit at the White House on August 8, 2023, Project Cybersafe Schools will support eligible K-12 public school districts with a package of Zero Trust cybersecurity solutions — for Continue reading
Unmasking the top exploited vulnerabilities of 2022
The Cybersecurity and Infrastructure Security Agency (CISA) just released a report highlighting the most commonly exploited vulnerabilities of 2022. With our role as a reverse proxy to a large portion of the Internet, Cloudflare is in a unique position to observe how the Common Vulnerabilities and Exposures (CVEs) mentioned by CISA are being exploited on the Internet.
We wanted to share a bit of what we’ve learned.
Based on our analysis, two CVEs mentioned in the CISA report are responsible for the vast majority of attack traffic seen in the wild: Log4J and Atlassian Confluence Code Injection. Although CISA/CSA discuss a larger number of vulnerabilities in the same report, our data clearly suggests a major difference in exploit volume between the top two and the rest of the list.
The top CVEs for 2022
Looking at the volume of requests detected by WAF Managed Rules that were created for the specific CVEs listed in the CISA report, we rank the vulnerabilities in order of prevalence:
Integrate Cloudflare Zero Trust with Datadog Cloud SIEM
Cloudflare's Zero Trust platform helps organizations map and adopt a strong security posture. This ranges from Zero Trust Network Access, a Secure Web Gateway to help filter traffic, to Cloud Access Security Broker and Data Loss Prevention to protect data in transit and in the cloud. Customers use Cloudflare to verify, isolate, and inspect all devices managed by IT. Our composable, in-line solutions offer a simplified approach to security and a comprehensive set of logs.
We’ve heard from many of our customers that they aggregate these logs into Datadog’s Cloud SIEM product. Datadog Cloud SIEM provides threat detection, investigation, and automated response for dynamic, cloud-scale environments. Cloud SIEM analyzes operational and security logs in real time – regardless of volume – while utilizing out-of-the-box integrations and rules to detect threats and investigate them. It also automates response and remediation through out-of-the-box workflow blueprints. Developers, security, and operations teams can also leverage detailed observability data and efficiently collaborate to accelerate security investigations in a single, unified platform. We previously had an out-of-the-box dashboard for Cloudflare CDN available on Datadog. These help our customers gain valuable insights into product usage and performance metrics for response times, HTTP status codes, cache hit rate. Continue reading
Hardening Workers KV
Over the last couple of months, Workers KV has suffered from a series of incidents, culminating in three back-to-back incidents during the week of July 17th, 2023. These incidents have directly impacted customers that rely on KV — and this isn’t good enough.
We’re going to share the work we have done to understand why KV has had such a spate of incidents and, more importantly, share in depth what we’re doing to dramatically improve how we deploy changes to KV going forward.
Workers KV?
Workers KV — or just “KV” — is a key-value service for storing data: specifically, data with high read throughput requirements. It’s especially useful for user configuration, service routing, small assets and/or authentication data.
We use KV extensively inside Cloudflare too, with Cloudflare Access (part of our Zero Trust suite) and Cloudflare Pages being some of our highest profile internal customers. Both teams benefit from KV’s ability to keep regularly accessed key-value pairs close to where they’re accessed, as well its ability to scale out horizontally without any need to become an expert in operating KV.
Given Cloudflare’s extensive use of KV, it wasn’t just external customers impacted. Our own internal teams felt the pain Continue reading
Cloudflare Workers database integration with Upstash
During Developer Week we announced Database Integrations on Workers a new and seamless way to connect with some of the most popular databases. You select the provider, authorize through an OAuth2 flow and automatically get the right configuration stored as encrypted environment variables to your Worker.
Today we are thrilled to announce that we have been working with Upstash to expand our integrations catalog. We are now offering three new integrations: Upstash Redis, Upstash Kafka and Upstash QStash. These integrations allow our customers to unlock new capabilities on Workers. Providing them with a broader range of options to meet their specific requirements.
Add the integration
We are going to show the setup process using the Upstash Redis integration.
Select your Worker, go to the Settings tab, select the Integrations tab to see all the available integrations.
After selecting the Upstash Redis integration we will get the following page.
First, you need to review and grant permissions, so the Integration can add secrets to your Worker. Second, we need to connect to Upstash using the OAuth2 flow. Third, select the Redis database we want to use. Then, the Integration will fetch the right information to generate the credentials. Finally, click “Add Continue reading
Cloudflare Radar’s new BGP origin hijack detection system
Border Gateway Protocol (BGP) is the de facto inter-domain routing protocol used on the Internet. It enables networks and organizations to exchange reachability information for blocks of IP addresses (IP prefixes) among each other, thus allowing routers across the Internet to forward traffic to its destination. BGP was designed with the assumption that networks do not intentionally propagate falsified information, but unfortunately that’s not a valid assumption on today’s Internet.
Malicious actors on the Internet who control BGP routers can perform BGP hijacks by falsely announcing ownership of groups of IP addresses that they do not own, control, or route to. By doing so, an attacker is able to redirect traffic destined for the victim network to itself, and monitor and intercept its traffic. A BGP hijack is much like if someone were to change out all the signs on a stretch of freeway and reroute automobile traffic onto incorrect exits.
You can learn more about BGP and BGP hijacking and its consequences in our learning center.
At Cloudflare, we have long been monitoring suspicious BGP anomalies internally. With our recent efforts, we are bringing BGP origin hijack detection to the Cloudflare Radar platform, sharing our detection results with the Continue reading
Routing information now on Cloudflare Radar
Routing is one of the most critical operations of the Internet. Routing decides how and where the Internet traffic should flow from the source to the destination, and can be categorized into two major types: intra-domain routing and inter-domain routing. Intra-domain routing handles making decisions on how individual packets should be routed among the servers and routers within an organization/network. When traffic reaches the edge of a network, the inter-domain routing kicks in to decide what the next hop is and forward the traffic along to the corresponding networks. Border Gateway Protocol (BGP) is the de facto inter-domain routing protocol used on the Internet.
Today, we are introducing another section on Cloudflare Radar: the Routing page, which focuses on monitoring the BGP messages exchanged to extract and present insights on the IP prefixes, individual networks, countries, and the Internet overall. The new routing data allows users to quickly examine routing status of the Internet, examine secure routing protocol deployment for a country, identify routing anomalies, validate IP block reachability and much more from globally distributed vantage points.
It’s a detailed view of how the Internet itself holds together.
Collecting routing statistics
The Internet consists of tens of thousands of interconnected Continue reading