Tiered Cache Smart Topology
A few years ago, we released Argo to help make the Internet faster and more efficient. Argo observes network conditions and finds the optimal route across the Internet for origin server requests, avoiding congestion along the way.
Tiered Cache is an Argo feature that reduces the number of data centers responsible for requesting assets from the origin. With Tiered Cache active, a request in South Africa won’t go directly to an origin in North America, but, instead, look in a large, nearby data center to see if the data requested is cached there first. The number and location of the data centers used by Tiered Cache is controlled by a piece of configuration called the topology. By default, we use a generic topology for every customer that strikes a balance between cache hit ratios and latency that is suitable for most users.
Today we’re introducing Smart Topology, which maximizes cache hit ratios by building on Argo’s internal infrastructure to identify the single best data center for making requests to the origin.
Standard Cache
The standard method for caching assets is to let each data center be a reverse proxy for the origin server. In this scheme, a miss in any Continue reading
2020 U.S. Election: Cybersecurity Analysis
As the election season has ramped down and the new Presidential Administration begins, we think it’s important to assess whether there are lessons we can draw from our experience helping to provide cybersecurity services for those involved in the 2020 U.S. elections.
Cloudflare built the Athenian Project - our project to provide free services to state and local election websites - around the idea that access to the authoritative voting information offered by state and local governments is key to a functioning democracy and that Cloudflare could play an important role in ensuring that election-related websites are protected from cyberattacks intended to disrupt that access. Although the most significant challenges in this election cycle fell outside the realm of cybersecurity, the 2020 election certainly validated the importance of having access to definitive sources of authoritative election information.
We were pleased that the robust cybersecurity preparations we saw for the 2020 U.S. election appeared to be successful. From the Cloudflare perspective, we had the opportunity to witness firsthand the benefits of having access to free cybersecurity services provided to organizations that promote accurate voting information and election results, state and local governments conducting elections, and federal U.S candidates Continue reading
Who won Super Bowl LV? A look at Internet traffic during the game
The obvious answer is the Tampa Bay Buccaneers but the less obvious answer comes from asking “which Super Bowl advertiser got the biggest Internet bump?”. This blog aims to answer that question.
Before, during, and after the game a crack team of three people who work on Cloudflare Radar looked at real time statistics for traffic to advertisers’ websites, social media in the US, US food delivery services, and websites covering (American) football. Luckily, one of us (Kari) is (a) American and (b) a fan of football. Unluckily, one of us (Kari) is a fan of the Kansas City Chiefs.
Cloudflare Radar uses a variety of sources to provide aggregate information about Internet traffic and attack trends. In this blog post we use DNS name resolution data to estimate traffic to websites. We can’t see who visited the websites mentioned below, or what anyone did on the websites, but DNS can give us an estimate of the interest generated by the commercials. This analysis only looked at the top-level names in each domain (so example.com and www.example.com and not any other subdomains).
The Big Picture
To get the ball rolling here’s a look at traffic to Continue reading
Congratulations to Cloudflare’s 2020 Partner Award Winners
We are privileged to share Cloudflare’s inaugural set of Partner Awards. These Awards recognize our partner companies and representatives worldwide who stood out this past year for their investments in acquiring technical expertise in our offerings, for delivering innovative applications and services built on Cloudflare, and for their commitment to customer success.
The unprecedented challenges in 2020 have reinforced how critical it is to have a secure, performant, and reliable Internet. Throughout these turbulent times, our partners have been busy innovating and helping organizations of all sizes and in various industries. By protecting and accelerating websites, applications, and teams with Cloudflare, our partners have helped these organizations adjust, seize new opportunities, and thrive.
Congratulations to each of our award winners. Cloudflare’s mission of helping build a better Internet is more important than ever. And our partners are more critical than ever to achieving our mission. Testifying to Cloudflare’s global reach, our honorees represent companies headquartered in 16 countries.
Cloudflare Partner of the Year Honorees, 2020
Worldwide MSP Partner of the Year: Rackspace Technology
Honors the top performing managed services provider (MSP) partner across Cloudflare's three sales geographies: Americas, APAC, and EMEA.
The effect of switching to TCMalloc on RocksDB memory use
In previous posts we wrote about our configuration distribution system Quicksilver and the story of migrating its storage engine to RocksDB. This solution proved to be fast, resilient and stable. During the migration process, we noticed that Quicksilver memory consumption was unexpectedly high. After our investigation we found out that the root cause was a default memory allocator that we used. Switching memory allocator improved service memory consumption by almost three times.
Unexpected memory growth
After migrating to RocksDB, the memory used by the application increased significantly. Also, the way memory was growing over time looked suspicious. It was around 15GB immediately after start and then was steadily growing for multiple days, until stabilizing at around 30GB. Below, you can see a memory consumption increase after migrating one of our test instances to RocksDB.
We started our investigation with heap profiling with the assumption that we had a memory leak somewhere and found that heap size was almost three times less than the RSS value reported by the operating system. So, if our application does not actually use all this memory, it means that memory is ‘lost’ somewhere between the system and our application, which points to possible problems with Continue reading
Cloudflare Named a ‘Best Place to Work for LGBTQ Equality’
Today we are excited to announce Cloudflare has been named a “Best Place to Work for LGBTQ Equality” by the Human Rights Campaign (HRC). This designation was earned by receiving a perfect score of 100 percent on the HRC’s 2021 Corporate Equality Index. The Corporate Equality Index (CEI) is a nationally recognized benchmarking tool that assesses the inclusivity of corporate policies, practices, and benefits for LGBTQIA+ employees.
Cloudflare’s mission is to “help build a better Internet”. An essential factor that helps us deliver on this mission is our people. When you are solving some of the toughest problems facing the Internet for users worldwide, you need talented individuals that contribute unique outlooks. We strive to build a workplace where our entire team feels comfortable and excited to bring their true authentic selves so they can do their best work.
2021 is the first year Cloudflare has been listed on the Index, but we have been paving this path for quite some time. Back in 2017 a few Cloudflare employees chartered Proudflare, Cloudflare’s first-ever Employee Resource Group (ERG). Proudflare serves as a community space for LGBTQIA+ employees and allies. Check out our Proudflare Launch blog to learn more on that process! Continue reading
Starting a new job in the middle of a pandemic
It has now been more than 90 days since I joined Cloudflare’s EMEA Recruiting Team as a Recruiting Coordinator based in Lisbon. In a year filled with hardships for so many people around the world, I wanted to share my journey. I hope people will relate and feel encouraged to pursue their dreams, even during these challenging times.
When 2020 started, it was not in my plans to change jobs and start working at a new company, completely remote, without ever meeting my colleagues in person or visiting the office. However, that is exactly what happened, and I am so glad I did.
Interviewing with Cloudflare
The number of interviews in the hiring process at Cloudflare may feel overwhelming for some - in my case, I met 11 people during this process. For me, I was glad to have so many chances to get to know the people I would be working with. I believe I got as much out of the conversations as the interviewers did, which is great — a recruitment process should be as much about the company getting to know you, as you getting to know the company.
A great thing about interviewing remotely is that Continue reading
Data Privacy Day 2021 – Looking ahead at the always on, always secure, always private Internet
Welcome to Data Privacy Day 2021! Last year at this time, I was writing about how Cloudflare builds privacy into everything we do, with little idea about how dramatically the world was going to change. The tragedy of the COVID-19 pandemic has reshaped the way we go about our daily lives. Our dependence on the Internet grew exponentially in 2020 as we started working from home, attending school from home, and participating in online weddings, concerts, parties, and more. So as we begin this new year, it’s impossible to think about data privacy in 2021 without thinking about how an always-on, always secure, always private Internet is more important than ever.
The pandemic wasn’t the only thing to dramatically shape data privacy conversations last year. We saw a flurry of new activity on data protection legislation around the globe, and a trend toward data localization in a variety of jurisdictions.
I don’t think I’m taking any risks when I say that 2021 looks to be another busy year in the world of privacy and data protection. Let me tell you a bit about what that looks like for us at Cloudflare. We’ll be spending a lot of time in Continue reading
Automating data center expansions with Airflow
Cloudflare’s network keeps growing, and that growth doesn’t just come from building new data centers in new cities. We’re also upgrading the capacity of existing data centers by adding newer generations of servers — a process that makes our network safer, faster, and more reliable for our users.
Connecting new Cloudflare servers to our network has always been complex, in large part because of the amount of manual effort that used to be required. Members of our Data Center and Infrastructure Operations, Network Operations, and Site Reliability Engineering teams had to carefully follow steps in an extremely detailed standard operating procedure (SOP) document, often copying command-line snippets directly from the document and pasting them into terminal windows.
But such a manual process can only scale so far, and we knew must be a way to automate the installation of new servers.
Here’s how we tackled that challenge by building our own Provisioning-as-a-Service (PraaS) platform and cut by 90% the amount of time our team spent on mundane operational tasks.
Choosing and using an automation framework
When we began our automation efforts, we quickly realized it made sense to replace each of these manual SOP steps with an API-call equivalent and Continue reading
How We Ran a Successful Remote Internship Program in 2020
And how we are planning to do it again in 2021…
On March 5, I sat in a small conference room with a few key contributors in creating and hiring for the Cloudflare summer intern program. With the possibility of office shutdowns looming, the group discussed what an internship would look like without in-person mentorship. How would the managers cope? How would the interns cope? Would it even be worthwhile? After a few minutes of discussions, we settled on ‘absolutely’. A remote summer internship at Cloudflare would be worthwhile for students, mentors, buddies, and managers alike. After all, Cloudflare is an Internet company and we were ready to trust the Internet with a whole lot more than we had anticipated.
The months leading up to the summer were a blur, all I remember is that we did a lot of planning, interviewing and hiring. And I mean, a lot. On April 2, Matthew Prince announced that Cloudflare would be doubling the size of our 2020 intern class in response to other companies cutting their intern programs all together. Due to these cuts, many talented students lost their opportunities for the summer. We knew we couldn’t hire them all so we Continue reading
Introducing Project Fair Shot: Ensuring COVID-19 Vaccine Registration Sites Can Keep Up With Demand
Around the world government and medical organizations are struggling with one of the most difficult logistics challenges in history: equitably and efficiently distributing the COVID-19 vaccine. There are challenges around communicating who is eligible to be vaccinated, registering those who are eligible for appointments, ensuring they show up for their appointments, transporting the vaccine under the required handling conditions, ensuring that there are trained personnel to administer the vaccine, and then doing it all over again as most of the vaccines require two doses.
Cloudflare can't help with most of that problem, but there is one key part that we realized we could help facilitate: ensuring that registration websites don't crash under load when they first begin scheduling vaccine appointments. Project Fair Shot provides Cloudflare's new Waiting Room service for free for any government, municipality, hospital, pharmacy, or other organization responsible for distributing COVID-19 vaccines. It is open to eligible organizations around the world and will remain free until at least July 1, 2021 or longer if there is still more demand for appointments for the vaccine than there is supply.
Crashing Registration Websites
The problem of vaccine scheduling registration websites crashing under load isn't theoretical: it is happening over Continue reading
Cloudflare Waiting Room
Today, we are excited to announce Cloudflare Waiting Room! It will first be available to select customers through a new program called Project Fair Shot which aims to help with the problem of overwhelming demand for COVID-19 vaccinations causing appointment registration websites to fail. General availability in our Business and Enterprise plans will be added in the near future.
Wait, you’re excited about a… Waiting Room?
Most of us are familiar with the concept of a waiting room, and rarely are we excited about the idea of being in one. Usually our first experience of one is at a doctor’s office — yes, you have an appointment, but sometimes the doctor is running late (or one of the patients was). Given the doctor can only see one person at a time… the waiting room was born, as a mechanism to queue up patients.
While servers can handle more concurrent requests than a doctor can, they too can be overwhelmed. If, in a pre-COVID world, you’ve ever tried buying tickets to a popular concert or event, you’ve probably encountered a waiting room online. It limits requests inbound to an application, and places these requests into a virtual queue. Once the number Continue reading
Network-layer DDoS attack trends for Q4 2020
DDoS attack trends in the final quarter of 2020 defied norms in many ways. For the first time in 2020, Cloudflare observed an increase in the number of large DDoS attacks. Specifically, the number of attacks over 500Mbps and 50K pps saw a massive uptick.
In addition, attack vectors continued to evolve, with protocol-based attacks seeing a 3-10x increase compared to the prior quarter. Attackers were also more persistent than ever — nearly 9% of all attacks observed between October and December lasted more than 24 hours.
Below are additional noteworthy observations from the fourth quarter of 2020, which the rest of this blog explores in greater detail.
- Number of attacks: For the first time in 2020, the total number of attacks observed in Q4 decreased compared to the prior quarter.
- Attack duration: 73% of all attacks observed lasted under an hour, a decrease from 88% in Q3.
- Attack vectors: While SYN, ACK, and RST floods continued to be the dominant attack vectors deployed, attacks over NetBIOS saw a whopping 5400% increase, followed by those over ISAKMP and SPSS.
- Global DDoS activity: Our data centers in Mauritius, Romania, and Brunei recorded the highest percentages of DDoS activity relative to non-attack Continue reading
Uganda’s January 13, 2021 Internet Shut Down
Two days ago, through its communications regulator, Uganda's government ordered the "Suspension Of The Operation Of Internet Gateways" the day before the country's general election. This action was confirmed by several users and journalists who got access to the letter sent to Internet providers. In other words, the government effectively cut off Internet access from the population to the rest of the world.
Ahead of tomorrow’s election the Internet has been shutdown in Uganda (confirmed by a few friends in Kampala).
— Samira Sawlani (@samirasawlani) January 13, 2021
Letter from communications commission below: pic.twitter.com/tRpTIXTPcW
On Cloudflare Radar, we want to help anyone understand what happens on the Internet. We are continually monitoring our network and exposing insights, threats, and trends based on the aggregated data that we see.
Uganda's unusual traffic patterns quickly popped up in our charts. Our 7-day change in Internet Traffic chart in Uganda shows a clear drop to near zero starting around 1900 local time, when the providers received the letter.
This is also obvious in the Application-level Attacks chart.
The traffic drop was also confirmed by the Uganda Internet eXchange point, a place where many providers exchange their data traffic, on their Continue reading
KEMTLS: Post-quantum TLS without signatures
The Transport Layer Security protocol (TLS), which secures most Internet connections, has mainly been a protocol consisting of a key exchange authenticated by digital signatures used to encrypt data at transport[1]. Even though it has undergone major changes since 1994, when SSL 1.0 was introduced by Netscape, its main mechanism has remained the same. The key exchange was first based on RSA, and later on traditional Diffie-Hellman (DH) and Elliptic-curve Diffie-Hellman (ECDH). The signatures used for authentication have almost always been RSA-based, though in recent years other kinds of signatures have been adopted, mainly ECDSA and Ed25519. This recent change to elliptic curve cryptography in both at the key exchange and at the signature level has resulted in considerable speed and bandwidth benefits in comparison to traditional Diffie-Hellman and RSA.
TLS is the main protocol that protects the connections we use everyday. It’s everywhere: we use it when we buy products online, when we register for a newsletter — when we access any kind of website, IoT device, API for mobile apps and more, really. But with the imminent threat of the arrival of quantum computers (a threat that seems to be getting closer and closer), we need Continue reading
Holistic web protection: industry recognition for a prolific 2020
I love building products that solve real problems for our customers. These days I don’t get to do so as much directly with our Engineering teams. Instead, about half my time is spent with customers listening to and learning from their security challenges, while the other half of my time is spent with other Cloudflare Product Managers (PMs) helping them solve these customer challenges as simply and elegantly as possible. While I miss the deeply technical engineering discussions, I am proud to have the opportunity to look back every year on all that we’ve shipped across our application security teams.
Taking the time to reflect on what we’ve delivered also helps to reinforce my belief in the Cloudflare approach to shipping product: release early, stay close to customers for feedback, and iterate quickly to deliver incremental value. To borrow a term from the investment world, this approach brings the benefits of compounded returns to our customers: we put new products that solve real-world problems into their hands as quickly as possible, and then reinvest the proceeds of our shared learnings immediately back into the product.
It is these sustained investments that allow us to release a flurry of small improvements Continue reading
Soar: Simulation for Observability, reliAbility, and secuRity
Serving more than approximately 25 million Internet properties is not an easy thing, and neither is serving 20 million requests per second on average. At Cloudflare, we achieve this by running a homogeneous edge environment: almost every Cloudflare server runs all Cloudflare products.
As we offer more and more products and enjoy the benefit of horizontal scalability, our edge stack continues to grow in complexity. Originally, we only operated at the application layer with our CDN service and DoS protection. Then we launched transport layer products, such as Spectrum and Argo. Now we have further expanded our footprint into the IP layer and physical link with Magic Transit. They all run on every machine we have. The work of our engineers enables our products to evolve at a fast pace, and to serve our customers better.
However, such software complexity presents a sheer challenge to operation: the more changes you make, the more likely it is that something is going to break. Continue reading
A Name Resolver for the Distributed Web
The Domain Name System (DNS) matches names to resources. Instead of typing 104.18.26.46 to access the Cloudflare Blog, you type blog.cloudflare.com and, using DNS, the domain name resolves to 104.18.26.46, the Cloudflare Blog IP address.
Similarly, distributed systems such as Ethereum and IPFS rely on a naming system to be usable. DNS could be used, but its resolvers’ attributes run contrary to properties valued in distributed Web (dWeb) systems. Namely, dWeb resolvers ideally provide (i) locally verifiable data, (ii) built-in history, and (iii) have no single trust anchor.
At Cloudflare Research, we have been exploring alternative ways to resolve queries to responses that align with these attributes. We are proud to announce a new resolver for the Distributed Web, where IPFS content indexed by the Ethereum Name Service (ENS) can be accessed.
To discover how it has been built, and how you can use it today, read on.
Welcome to the Distributed Web
IPFS and its addressing system
The InterPlanetary FileSystem (IPFS) is a peer-to-peer network for storing content on a distributed file system. It is composed of a set of computers called nodes that store and relay content using a common Continue reading
Cloudflare Radar’s 2020 Year In Review
Throughout 2020, we tracked changing Internet trends as the SARS-Cov-2 pandemic forced us all to change the way we were living, working, exercising and learning. In early April, we created a dedicated website https://builtforthis.net/ that showed some of the ways in which Internet use had changed, suddenly, because of the crisis.
On that website, we showed how traffic patterns had changed; for example, where people accessed the Internet from, how usage had jumped up dramatically, and how Internet attacks continued unabated and ultimately increased.
Today we are launching a dedicated Year In Review page with interactive maps and charts you can use to explore what changed on the Internet in 2020. Year In Review is part of Cloudflare Radar. We launched Radar in September 2020 to give anyone access to Internet use and abuse trends that Cloudflare normally had reserved only for employees.
Where people accessed the Internet
To get a sense for the Year In Review, let’s zoom in on London (you can do the same with any city from a long list of locations that we’ve analyzed). Here’s a map showing the change in Internet use comparing April (post-lockdown) and February (pre-lockdown). This map compares working hours Continue reading
Untangling Compliance: Working Toward a Global Framework
As part of Cloudflare’s recent Privacy Week we hosted a series of fireside chats on security, privacy, and compliance. Many of these conversations touched on the intricate legal debate being held in Europe around data sovereignty. Here are some of the highlights.
To learn more about the solutions Cloudflare launched to help businesses navigate their compliance needs — including the new data localization suite — see our recent blog post here.
Prof. Dr. Wilfried Bernhardt
Honorary professor -- University of Leipzig,
Attorney, CEO Bernhardt IT Management Consulting GmbH
We have to agree to go down a common road, a common path. And this common path can really only consist of saying: let's sit down together again. I'm talking about the European Commission and, above all, the new administration in the United States. We are all waiting for them expectantly.
And then we look at what our common fundamental values are and see if we don’t simply come together better than we have in the past. After all, our fundamental values are the same: human rights, democracy, the rule of law. You have to concede that there are some differences in understanding when it comes to interpreting what privacy means — Continue reading