Crawler Hints supports Microsoft’s IndexNow in helping users find new content
The web is constantly changing. Whether it’s news or updates to your social feed, it’s a constant flow of information. As a user, that’s great. But have you ever stopped to think how search engines deal with all the change?
It turns out, they “index” the web on a regular basis — sending bots out, to constantly crawl webpages, looking for changes. Today, bot traffic accounts for about 30% of total traffic on the Internet, and given how foundational search is to using the Internet, it should come as no surprise that search engine bots make up a large proportion of that what might come as a surprise is how inefficient the model is, though: we estimate that over 50% of crawler traffic is wasted effort.
This has a huge impact. There’s all the additional capacity that owners of websites need to bake into their site to absorb the bots crawling all over it. There’s the transmission of the data. There’s the CPU cost of running the bots. And when you’re running at the scale of the Internet, all of this has a pretty big environmental footprint.
Part of the problem, though, is nobody had really stopped to ask: maybe Continue reading
2022 attacks! An August reading list to go “Shields Up”
In 2022, cybersecurity is a must-have for those who don’t want to take chances on getting caught in a cyberattack with difficult to deal consequences. And with a war in Europe (Ukraine) still going on, cyberwar also doesn’t show signs of stopping in a time when there never were so many people online, 4.95 billion in early 2022, 62.5% of the world’s total population (estimates say it grew around 4% during 2021 and 7.3% in 2020).
Throughout the year we, at Cloudflare, have been making new announcements of products, solutions and initiatives that highlight the way we have been preventing, mitigating and constantly learning, over the years, with several thousands of small and big cyberattacks. Right now, we block an average of 124 billion cyber threats per day. The more we deal with attacks, the more we know how to stop them, and the easier it gets to find and deal with new threats — and for customers to forget we’re there, protecting them.
In 2022, we have been onboarding many customers while they’re being attacked, something we know well from the past (Wikimedia/Wikipedia or Eurovision are just two case-studies of many, Continue reading
The mechanics of a sophisticated phishing scam and how we stopped it
Yesterday, August 8, 2022, Twilio shared that they’d been compromised by a targeted phishing attack. Around the same time as Twilio was attacked, we saw an attack with very similar characteristics also targeting Cloudflare’s employees. While individual employees did fall for the phishing messages, we were able to thwart the attack through our own use of Cloudflare One products, and physical security keys issued to every employee that are required to access all our applications.
We have confirmed that no Cloudflare systems were compromised. Our Cloudforce One threat intelligence team was able to perform additional analysis to further dissect the mechanism of the attack and gather critical evidence to assist in tracking down the attacker.
This was a sophisticated attack targeting employees and systems in such a way that we believe most organizations would be likely to be breached. Given that the attacker is targeting multiple organizations, we wanted to share here a rundown of exactly what we saw in order to help other companies recognize and mitigate this attack.
Targeted Text Messages
On July 20, 2022, the Cloudflare Security team received reports of employees receiving legitimate-looking text messages pointing to what appeared to be a Cloudflare Okta login Continue reading
Introducing new Cloudflare for SaaS documentation
As a SaaS provider, you’re juggling many challenges while building your application, whether it’s custom domain support, protection from attacks, or maintaining an origin server. In 2021, we were proud to announce Cloudflare for SaaS for Everyone, which allows anyone to use Cloudflare to cover those challenges, so they can focus on other aspects of their business. This product has a variety of potential implementations; now, we are excited to announce a new section in our Developer Docs specifically devoted to Cloudflare for SaaS documentation to allow you take full advantage of its product suite.
Cloudflare for SaaS solution
You may remember, from our October 2021 blog post, all the ways that Cloudflare provides solutions for SaaS providers:
- Set up an origin server
- Encrypt your customers’ traffic
- Keep your customers online
- Boost the performance of global customers
- Support custom domains
- Protect against attacks and bots
- Scale for growth
- Provide insights and analytics
However, we received feedback from customers indicating confusion around actually using the capabilities of Cloudflare for SaaS because there are so many features! With the existing documentation, it wasn’t 100% clear how to enhance security and performance, or how to support custom domains. Now, we want Continue reading
1.1.1.1 + WARP: More features, still private
It’s a Saturday night. You open your browser, looking for nearby pizza spots that are open. If the search goes as intended, your browser will show you results that are within a few miles, often based on the assumed location of your IP address. At Cloudflare, we affectionately call this type of geolocation accuracy the “pizza test”. When you use a Cloudflare product that sits between you and the Internet (for example, WARP), it’s one of the ways we work to balance user experience and privacy. Too inaccurate and you’re getting pizza places from a neighboring country; too accurate and you’re reducing the privacy benefits of obscuring your location.
With that in mind, we’re excited to announce two major improvements to our 1.1.1.1 + WARP apps: first, an improvement to how we ensure search results and other geographically-aware Internet activity work without compromising your privacy, and second, a larger network with more locations available to WARP+ subscribers, powering even speedier connections to our global network.
A better Internet browsing experience for every WARP user
When we originally built the 1.1.1.1+ WARP mobile app, we wanted to create a consumer-friendly way to connect to Continue reading
Build your next big idea with Cloudflare Pages
Have you ever had a surge of inspiration for a project? That feeling when you have a great idea – a big idea — that you just can’t shake? When all you can think about is putting your hands to your keyboard and hacking away? Building a website takes courage, creativity, passion and drive, and with Cloudflare Pages we believe nothing should stand in the way of that vision.
Especially not a price tag.
Big ideas
We built Pages to be at the center of your developer experience – a way for you to get started right away without worrying about the heavy lift of setting up a fullstack app. A quick commit to your git provider or direct upload to our platform, and your rich and powerful site is deployed to our network of 270+ data centers in seconds. And above all, we built Pages to scale with you as you grow exponentially without getting hit by an unexpected bill.
The limit does not exist
We’re a platform that’s invested in your vision – no matter how wacky and wild (the best ones usually are!). That’s why for many parts of Pages we want your experience to be Continue reading
Building scheduling system with Workers and Durable Objects
We rely on technology to help us on a daily basis – if you are not good at keeping track of time, your calendar can remind you when it's time to prepare for your next meeting. If you made a reservation at a really nice restaurant, you don't want to miss it! You appreciate the app to remind you a day before your plans the next evening.
However, who tells the application when it's the right time to send you a notification? For this, we generally rely on scheduled events. And when you are relying on them, you really want to make sure that they occur. Turns out, this can get difficult. The scheduler and storage backend need to be designed with scale in mind - otherwise you may hit limitations quickly.
Workers, Durable Objects, and Alarms are actually a perfect match for this type of workload. Thanks to the distributed architecture of Durable Objects and their storage, they are a reliable and scalable option. Each Durable Object has access to its own isolated storage and alarm scheduler, both being automatically replicated and failover in case of failures.
There are many use cases where having a reliable scheduler can come Continue reading
Experiment with post-quantum cryptography today
Practically all data sent over the Internet today is at risk in the future if a sufficiently large and stable quantum computer is created. Anyone who captures data now could decrypt it.
Luckily, there is a solution: we can switch to so-called post-quantum (PQ) cryptography, which is designed to be secure against attacks of quantum computers. After a six-year worldwide selection process, in July 2022, NIST announced they will standardize Kyber, a post-quantum key agreement scheme. The standard will be ready in 2024, but we want to help drive the adoption of post-quantum cryptography.
Today we have added support for the X25519Kyber512Draft00 and X25519Kyber768Draft00 hybrid post-quantum key agreements to a number of test domains, including pq.cloudflareresearch.com.
Do you want to experiment with post-quantum on your test website for free? Mail [email protected] to enroll your test website, but read the fine-print below.
What does it mean to enable post-quantum on your website?
If you enroll your website to the post-quantum beta, we will add support for these two extra key agreements alongside the existing classical encryption schemes such as X25519. If your browser doesn’t support these post-quantum key agreements (and none at the time Continue reading
Building and using Managed Components with WebCM
Managed Components are here to shake up the way third-party tools integrate with websites. Two months ago we announced that we’re open sourcing parts of the most innovative technologies behind Cloudflare Zaraz, making them accessible and usable to everyone online. Since then, we’ve been working hard to make sure that the code is well documented and all pieces are fun and easy to use. In this article, I want to show you how Managed Components can be useful for you right now, if you manage a website or if you’re building third-party tools. But before we dive in, let’s talk about the past.
Third-party scripts are a threat to your website
For decades, if you wanted to add an analytics tool to your site, a chat widget, a conversion pixel or any other kind of tool – you needed to include an external script. That usually meant adding some code like this to your website:
<script src=”https://example.com/script.js”></script>
If you think about it – it’s a pretty terrible idea. Not only that you’re now asking the browser to connect to another server, fetch and execute more JavaScript code – you’re also completely giving up the control on your Continue reading
Load Balancing with Weighted Pools
Anyone can take advantage of Cloudflare’s far-reaching network to protect and accelerate their online presence. Our vast number of data centers, and their proximity to Internet users around the world, enables us to secure and accelerate our customers’ Internet applications, APIs and websites. Even a simple service with a single origin server can leverage the massive scale of the Cloudflare network in 270+ cities. Using the Cloudflare cache, you can support more requests and users without purchasing new servers.
Whether it is to guarantee high availability through redundancy, or to support more dynamic content, an increasing number of services require multiple origin servers. The Cloudflare Load Balancer keeps our customer’s services highly available and makes it simple to spread out requests across multiple origin servers. Today, we’re excited to announce a frequently requested feature for our Load Balancer – Weighted Pools!
What’s a Weighted Pool?
Before we can answer that, let’s take a quick look at how our load balancer works and define a few terms:
Origin Servers - Servers which sit behind Cloudflare and are often located in a customer-owned datacenter or at a public cloud provider.
Origin Pool - A logical collection of origin servers. Most pools Continue reading
Running Zig with WASI on Cloudflare Workers
After the recent announcement regarding WASI support in Workers, I decided to see what it would take to get code written in Zig to run as a Worker, and it turned out to be trivial. This post documents the process I followed as a new user of Zig. It’s so exciting to see how Cloudflare Workers is a polyglot platform allowing you to write programs in the language you love, or the language you’re learning!
Hello, World!
I’m not a Zig expert by any means, and to keep things entirely honest I’ve only just started looking into the language, but we all have to start somewhere. So, if my Zig code isn’t perfect please bear with me. My goal was to build a real, small program using Zig and deploy it on Cloudflare Workers. And to see how fast I can go from a blank screen to production code.
My goal for this wasn’t ambitious, just read some text from stdin and print it to stdout with line numbers, like running cat -n. But it does show just how easy the Workers paradigm is. This Zig program works identically on the command-line on my laptop and as an HTTP Continue reading
When the window is not fully open, your TCP stack is doing more than you think
Over the years I've been lurking around the Linux kernel and have investigated the TCP code many times. But when recently we were working on Optimizing TCP for high WAN throughput while preserving low latency, I realized I have gaps in my knowledge about how Linux manages TCP receive buffers and windows. As I dug deeper I found the subject complex and certainly non-obvious.
In this blog post I'll share my journey deep into the Linux networking stack, trying to understand the memory and window management of the receiving side of a TCP connection. Specifically, looking for answers to seemingly trivial questions:
- How much data can be stored in the TCP receive buffer? (it's not what you think)
- How fast can it be filled? (it's not what you think either!)
Our exploration focuses on the receiving side of the TCP connection. We'll try to understand how to tune it for the best speed, without wasting precious memory.
A case of a rapid upload
To best illustrate the receive side buffer management we need pretty charts! But to grasp all the numbers, we need a bit of theory.
We'll draw charts from a receive side of a TCP flow, Continue reading
Cloudflare deployment in Guam
Having fast Internet properties means being as few milliseconds as possible away from our customers and their users, no matter where they are on Earth. And because of the design of Cloudflare's network we don't just make Internet properties faster by being closer, we bring our Zero Trust services closer too. So whether you're connecting to a public API, a website, a SaaS application, or your company's internal applications, we're close by.
This is possible by adding new cities, partners, capacity, and cables. And we have seen over and over again how making the Internet faster in a region also can have a clear impact on traffic: if the experience is quicker, people usually do more online.
Cloudflare’s network keeps increasing, and its global footprint does so accordingly. In April 2022 we announced that the Cloudflare network now spans 275 cities and the number keeps growing.
In this blog post we highlight the deployment of our data center in Hagatna, Guam.
Why a blog about Guam?
Guam is about 2,400 km from both Tokyo in the north and Manila in the west, and about 6,100 km from Honolulu in the east. Honolulu itself is the most remote major city in Continue reading
Using Apache Kafka to process 1 trillion inter-service messages
Cloudflare has been using Kafka in production since 2014. We have come a long way since then, and currently run 14 distinct Kafka clusters, across multiple data centers, with roughly 330 nodes. Between them, over a trillion messages have been processed over the last eight years.
Cloudflare uses Kafka to decouple microservices and communicate the creation, change or deletion of various resources via a common data format in a fault-tolerant manner. This decoupling is one of many factors that enables Cloudflare engineering teams to work on multiple features and products concurrently.
We learnt a lot about Kafka on the way to one trillion messages, and built some interesting internal tools to ease adoption that will be explored in this blog post. The focus in this blog post is on inter-application communication use cases alone and not logging (we have other Kafka clusters that power the dashboards where customers view statistics that handle more than one trillion messages each day). I am an engineer on the Application Services team and our team has a charter to provide tools/services to product teams, so they can focus on their core competency which is delivering value to our customers.
In this blog I’d Continue reading
A story about AF_XDP, network namespaces and a cookie
A crash in a development version of flowtrackd (the daemon that powers our Advanced TCP Protection) highlighted the fact that libxdp (and specifically the AF_XDP part) was not Linux network namespace aware.
This blogpost describes the debugging journey to find the bug, as well as a fix.
flowtrackd is a volumetric denial of service defense mechanism that sits in the Magic Transit customer’s data path and protects the network from complex randomized TCP floods. It does so by challenging TCP connection establishments and by verifying that TCP packets make sense in an ongoing flow.
It uses the Linux kernel AF_XDP feature to transfer packets from a network device in kernel space to a memory buffer in user space without going through the network stack. We use most of the helper functions of the C libbpf with the Rust bindings to interact with AF_XDP.
In our setup, both the ingress and the egress network interfaces are in different network namespaces. When a packet is determined to be valid (after a challenge or under some thresholds), it is forwarded to the second network interface.
For the rest of this post the network setup will be the following:
e.g. eyeball packets Continue reading
Working in public — our docs-as-code approach
Docs-as-code is an approach to writing and publishing documentation with the same tools and processes developers use to create code. This philosophy has become more popular in recent years, especially in tech companies. Automatic link checking is part of this process, which ensures that writer's changes are sound and safe to deploy. By setting the stage with a docs-as-code approach, technical writers can focus on what they do best: ensure that our readers get useful and accurate information that is easy to find, and our documentation speaks a single language.
Besides following a docs-as-code approach, at Cloudflare we handle our documentation changes in public, in our cloudflare-docs GitHub repository. Having our documentation open to external contributions has helped us improve our documentation over time — our community is great at finding issues! While we need to review these contributions and ensure that they fit our style guide and content strategy, the contributions provided by the Cloudflare community have been instrumental in making our documentation better every day. While Cloudflare helps build a better Internet, our community helps build better documentation.
Docs-as-code at Cloudflare
At Cloudflare, we follow a docs-as-code approach to create and publish product documentation in Developer Docs.
Such Continue reading
How the James Webb Telescope’s cosmic pictures impacted the Internet
“Somewhere, something incredible is waiting to be known.” — Carl Sagan
In the past few years, space technology and travel have been trending with increased attention and endeavors (including private ones). In our 2021 Year in Review we showed how NASA and SpaceX flew higher, at least in terms of interest on the Internet.
This week, NASA in collaboration with the European Space Agency (ESA) and the Canadian Space Agency (CSA), released the first images from the James Webb Telescope (JWST) which conducts infrared astronomy to “reveal the unseen universe”.
So, let’s dig into something we really like here at Cloudflare, checking how real life and human interest has an impact on the Internet. In terms of general Continue reading
Mantis – the most powerful botnet to date
In June 2022, we reported on the largest HTTPS DDoS attack that we’ve ever mitigated — a 26 million request per second attack - the largest attack on record. Our systems automatically detected and mitigated this attack and many more. Since then, we have been tracking this botnet, which we’ve called “Mantis”, and the attacks it has launched against almost a thousand Cloudflare customers.
Cloudflare WAF/CDN customers are protected against HTTP DDoS attacks including Mantis attacks. Please refer to the bottom of this blog for additional guidance on how to best protect your Internet properties against DDoS attacks.
Have you met Mantis?
We named the botnet that launched the 26M rps (requests per second) DDoS attack "Mantis" as it is also like the Mantis shrimp, small but very powerful. Mantis shrimps, also known as “thumb-splitters”, are very small; less than 10 cm in length, but their claws are so powerful that they can generate a shock wave with a force of 1,500 Newtons at speeds of 83 km/h from a standing start. Similarly, the Mantis botnet operates a small fleet of approximately 5,000 bots, but with them can generate a massive force — responsible for the largest Continue reading
35,000 new trees in Nova Scotia
Cloudflare is proud to announce the first 35,000 trees from our commitment to help clean up bad bots (and the climate) have been planted.
Working with our partners at One Tree Planted (OTP), Cloudflare was able to support the restoration of 20 hectares of land at Victoria Park in Nova Scotia, Canada. The 130-year-old natural woodland park is located in the heart of Truro, NS, and includes over 3,000 acres of hiking and biking trails through natural gorges, rivers, and waterfalls, as well as an old-growth eastern hemlock forest.
The planting projects added red spruce, black spruce, eastern white pine, eastern larch, northern red oak, sugar maple, yellow birch, and jack pine to two areas of the park. The first area was a section of the park that recently lost a number of old conifers due to insect attacks. The second was an area previously used as a municipal dump, which has since been covered by a clay cap and topsoil.
Our tree commitment began far from the Canadian woodlands. In 2019, we launched an ambitious tool called Bot Fight Mode, which for the first time fought back against bots, targeting scrapers and other automated actors.
Our Continue reading
Waiting Room Event Scheduling protects your site during online events
You've got big plans for your ecommerce strategy in the form of online events — seasonal sales, open registration periods, product drops, ticket sales, and more. With all the hype you've generated, you'll get a lot of site traffic, and that's a good thing! With Waiting Room Event Scheduling, you can protect your servers from being overloaded during your event while delivering a user experience that is unique to the occasion and consistent with your brand. Available now to enterprise customers with an advanced Waiting Room subscription, Event Scheduling allows you to plan changes to your waiting room’s settings and custom queueing page ahead of time, ensuring flawless execution of your online event.
More than always-on protection
We launched Waiting Room to protect our customers' servers during traffic spikes. Waiting Room sends excess visitors to a virtual queue during traffic surges, letting visitors in dynamically as spots become available on your site. By automatically queuing traffic that exceeds your site's capacity, Waiting Room protects your origin servers and your customer experience. Additionally, the Waiting Room's queuing page can be customized to match the look and feel of your site so that your users never feel as though they have left Continue reading