Adding the Fallback Pool to the Load Balancing UI and other significant UI enhancements
The Cloudflare Load Balancer was introduced over three years ago to provide our customers with a powerful, easy to use tool to intelligently route traffic to their origins across the world. During the initial design process, one of the questions we had to answer was ‘where do we send traffic if all pools are down?’ We did not think it made sense just to drop the traffic, so we used the concept of a ‘fallback pool’ to send traffic to a ‘pool of last resort’ in the case that no pools were detected as available. While this may still result in an error, it gave an eyeball request a chance at being served successfully in case the pool was still up.
As a brief reminder, a load balancer helps route traffic across your origin servers to ensure your overall infrastructure stays healthy and available. Load Balancers are made up of pools, which can be thought of as collections of servers in a particular location.
Over the past three years, we’ve made many updates to the dashboard. The new designs now support the fallback pool addition to the dashboard UI. The use of a fallback pool is incredibly helpful in Continue reading
The Serverlist: Workers Secrets, Serverless Supremacy, and more!
Check out our thirteenth edition of The Serverlist below. Get the latest scoop on the serverless space, get your hands dirty with new developer tutorials, engage in conversations with other serverless developers, and find upcoming meetups and conferences to attend.
Sign up below to have The Serverlist sent directly to your mailbox.
Remote Work Isn’t Just Video Conferencing: How We Built CloudflareTV
At Cloudflare, we produce all types of video content, ranging from recordings of our Weekly All-Hands to product demos. Being able to stream video on demand has two major advantages when compared to live video:
- It encourages asynchronous communication within the organization
- It extends the life time value of the shared knowledge
Historically, we haven’t had a central, secure repository of all video content that could be easily accessed from the browser. Various teams choose their own platform to share the content. If I wanted to find a recording of a product demo, for example, I’d need to search Google Drive, Gmail and Google Chat with creative keywords. Very often, I would need to reach out to individual teams to finally locate the content.
So we decided we wanted to build CloudflareTV, an internal Netflix-like application that can only be accessed by Cloudflare employees and has all of our videos neatly organized and immediately watchable from the browser.
We wanted to achieve the following when building CloudflareTV:
- Security: make sure the videos are access controlled and not publicly accessible
- Authentication: ensure the application can only be accessed by Cloudflare employees
- Tagging: allow the videos to be categorized so they can Continue reading
Using Cloudflare Gateway to Stay Productive (and turn off distractions) While Working Remotely
This week, like many of you reading this article, I am working from home. I don’t know about you, but I’ve found it hard to stay focused when the Internet is full of news related to the coronavirus.
CNN. Twitter. Fox News. It doesn’t matter where you look, everyone is vying for your attention. It’s totally riveting…
… and it’s really hard not to get distracted.
It got me annoyed enough that I decided to do something about it. Using Cloudflare’s new product, Cloudflare Gateway, I removed all the online distractions I normally get snared by — at least during working hours.
This blog post isn’t very long, but that’s a function of how easy it is to get Gateway up and running!
Getting Started
To get started, you’ll want to set up Gateway under your Cloudflare account. Head to the Cloudflare for Teams dashboard to set it up for free (if you don’t already have a Cloudflare account, hit the ‘Sign up’ button beneath the login form).
If you are using Gateway for the first time, the dashboard will take you through an onboarding experience:
The onboarding flow will help you set up your first location. A location is Continue reading
Keepalives considered harmful
This may sound like a weird title, but hear me out. You’d think keepalives would always be helpful, but turns out reality isn’t always what you expect it to be. It really helps if you read Why does one NGINX worker take all the load? first. This post is an adaptation of a rather old post on Cloudflare’s internal blog, so not all details are exactly as they are in production today but the lessons are still valid.
This is a story about how we were seeing some complaints about sporadic latency spikes, made some unconventional changes, and were able to slash the 99.9th latency percentile by 4x!
Request flow on Cloudflare edge
I'm going to focus only on two parts of our edge stack: FL and SSL.
- FL accepts plain HTTP connections and does the main request logic, including our WAF
- SSL terminates SSL and passes connections to FL over local Unix socket:
Here’s a diagram:
These days we route all traffic through SSL for simplicity, but in the grand scheme of things it’s not going to matter much.
Each of these processes is not itself a single process, but rather a master process and a collection of Continue reading
The problem with thread^W event loops
Back when Cloudflare was created, over 10 years ago now, the dominant HTTP server used to power websites was Apache httpd. However, we decided to build our infrastructure using the then relatively new NGINX server.
There are many differences between the two, but crucially for us, the event loop architecture of NGINX was the key differentiator. In a nutshell, event loops work around the need to have one thread or process per connection by coalescing many of them in a single process, this reduces the need for expensive context switching from the operating system and also keeps the memory usage predictable. This is done by processing each connection until it wants to do some I/O, at that point, the said connection is queued until the I/O task is complete. During that time the event loop is available to process other in-flight connections, accept new clients, and the like. The loop uses a multiplexing system call like epoll (or kqueue) to be notified whenever an I/O task is complete among all the running connections.
In this article we will see that despite its advantages, event loop models also have their limits and falling back to good old threaded architecture is sometimes Continue reading
On the shoulders of giants: recent changes in Internet traffic
As the COVID-19 emergency continues and an increasing number of cities and countries are establishing quarantines or cordons sanitaire, the Internet has become, for many, the primary method to keep in touch with their friends and families. And it's a vital motor of the global economy as many companies have employees who are now working from home.
Traffic towards video conferencing, streaming services and news, e-commerce websites has surged. We've seen growth in traffic from residential broadband networks, and a slowing of traffic from businesses and universities.
The Cloudflare team is fully operational and the Network Operating Center (NOC) is watching the changing traffic patterns in the more than 200 cities in which we operate hardware.
Big changes in Internet traffic aren't unusual. They often occur around large sporting events like the Olympics or World Cup, cultural events like the Eurovision Song Contest and even during Ramadan at the breaking of the fast each day.
The Internet was built to cope with an ever changing environment. In fact, it was literally created, tested, debugged and designed to deal with changing load patterns.
Over the last few weeks, the Cloudflare Network team has noticed some new patterns and we wanted to Continue reading
Announcing Network Analytics
Our Analytics Platform
Back in March 2019, we released Firewall Analytics which provides insights into HTTP security events across all of Cloudflare's protection suite; Firewall rule matches, HTTP DDoS Attacks, Site Security Level which harnesses Cloudflare's threat intelligence, and more. It helps customers tailor their security configurations more effectively. The initial release was for Enterprise customers, however we believe that everyone should have access to powerful tools, not just large enterprises, and so in December 2019 we extended those same enterprise-level analytics to our Business and Pro customers.
Since then, we’ve built on top of our analytics platform; improved the usability, added more functionality and extended it to additional Cloudflare services in the form of Account Analytics, DNS Analytics, Load Balancing Analytics, Monitoring Analytics and more.
Our entire analytics platform harnesses the powerful GraphQL framework which is also available to customers that want to build, export and share their own custom reports and dashboards.
Extending Visibility From L7 To L3
Until recently, all of our dashboards were mostly HTTP-oriented and provided visibility into HTTP attributes such as the user agent, hosts, cached resources, etc. This is valuable to customers that use Cloudflare to protect and accelerate HTTP Continue reading
COVID-19 impacts on Internet traffic: Seattle, Northern Italy and South Korea
The last few weeks have seen unprecedented changes in how people live and work around the world. Over time more and more companies have given their employees the right to work from home, restricted business travel and, in some cases, outright sent their entire workforce home. In some countries, quarantines are in place keeping people restricted to their homes.
These changes in daily life are showing up as changes in patterns of Internet use around the world. In this blog post I take a look at changing patterns in northern Italy, South Korea and the Seattle area of Washington state.
Seattle
To understand how Internet use is changing, it’s first helpful to start with what a normal pattern looks like. Here’s a chart of traffic from our Dallas point of presence in the middle of January 2020.
This is a pretty typical pattern. If you look carefully you can see that Internet use is down a little at the weekend and that Internet usage is diurnal: Internet use drops down during the night and then picks up again in the morning. The peaks occur at around 2100 local time and the troughs in the dead of night at around 0300. Continue reading
Cloudflare’s COVID-19 FAQs
As the status of COVID-19 continues to impact people and businesses around the world, Cloudflare is committed to providing awareness and transparency to our customers, employees, and partners about how we are responding. We do not anticipate any significant disruptions in Cloudflare services.
Our Business Continuity Team is monitoring the situation closely and all company personnel are kept up to date via multiple internal communication channels including a live chat room. Customers and the public are encouraged to visit this blog post for the latest information.
You can check the status of our network at www.cloudflarestatus.com. For COVID-19-related questions that aren’t answered below, please contact our Customer Support Team.
Does Cloudflare have a Business Continuity Team (BCT)?
Yes, Cloudflare’s Business Continuity Team is a cross-functional, geographically diverse group dedicated to navigating through a health crisis like COVID-19 as well as a variety of other scenarios that may impact employee safety and business continuity.
What is Cloudflare’s Business Continuity Plan in the light of COVID-19?
In addition to Cloudflare’s existing Disaster Recovery Plan we have implemented the following strategies:
- Daily Business Continuity Team meetings to determine if updates, changes, or communication need to be provided to customers, partners, and Continue reading
Cloudflare During the Coronavirus Emergency
This email was sent to all Cloudflare customers a short while ago
From: Matthew Prince
Date: Thu, Mar 12, 2020 at 4:20 PM
Subject: Cloudflare During the Coronavirus Emergency
We know that organizations and individuals around the world depend on Cloudflare and our network. I wanted to send you a personal note to let you know how Cloudflare is dealing with the Coronavirus emergency.
First, the health and safety of our employees and customers is our top priority. We have implemented a number of sensible policies to this end, including encouraging many employees to work from home. This, however, hasn't slowed our operations. Our network operations center (NOC), security operations center (SOC), and customer support teams will remain fully operational and can do their jobs entirely remote as needed.
Second, we are tracking Internet usage patterns globally. As more people work from home, peak traffic in impacted regions has increased, on average, approximately 10%. In Italy, which has imposed a nationwide quarantine, peak Internet traffic is up 30%. Traffic patterns have also shifted so peak traffic is occurring earlier in the day in impacted regions. None of these traffic changes raise any concern for us. Cloudflare's network is well provisioned Continue reading
Protect your team with Cloudflare Gateway
On January 7th, we announced Cloudflare for Teams, a new way to protect organizations and their employees globally, without sacrificing performance. Cloudflare for Teams centers around two core products - Cloudflare Access and Cloudflare Gateway. Cloudflare Access is already available and used by thousands of teams around the world to secure internal applications. Cloudflare Gateway solves the other end of the problem by protecting those teams from security threats without sacrificing performance.
Today, we’re excited to announce new secure DNS filtering capabilities in Cloudflare Gateway. Cloudflare Gateway protects teams from threats like malware, phishing, ransomware, crypto-mining and other security threats. You can start using Cloudflare Gateway at dash.teams.cloudflare.com. Getting started takes less than five minutes.
Why Cloudflare Gateway?
We built Cloudflare Gateway to address key challenges our customers experience with managing and securing global networks. The root cause of these challenges is architecture and inability to scale. Legacy network security models solved problems in the 1990s, but teams have continued to attempt to force the Internet of the 2020s through them.
Historically, branch offices sent all of their Internet-bound traffic to one centralized data center at or near corporate headquarters. Administrators configured that to make sure all Continue reading
Open sourcing our Sentry SSO plugin
Cloudflare Access, part of Cloudflare for Teams, replaces legacy corporate VPNs with Cloudflare’s global network. Using your existing identity provider, Access enables your end users to login from anywhere — without a clunky agent or traffic backhaul through a centralized appliance or VPN.
Today, we are open sourcing a plugin that continues to improve that experience by making it easier for teams to use Cloudflare Access with one of the software industry’s most popular engineering tools, Sentry.
What is Sentry?
Sentry is an application that helps software teams find and diagnose errors in their products. We use Sentry here at Cloudflare. When you encounter an error when using a Cloudflare product, like our dashboard, we log that event. We then use Sentry to determine what went wrong.
Sentry can categorize and roll up errors, making it easy to identify new problems before investigating them with the tool’s event logging. Engineering managers here can use the dashboards to monitor the health of a new release. Product managers often use those reports as part of prioritizing what to fix next. Engineers on our team can dig into the individual errors as they release a fix.
Sentry is available in two forms: Continue reading
How Replicated Developers Develop Remotely
This is a guest post by Marc Campbell and Grant Miller, co-founders of Replicated.
Replicated is a 5-year old infrastructure software company working to make it easy for businesses to install and operate third party software. We don’t want you to have to send your data to a multi-tenant SaaS provider just to use their services. Our team is made up of twenty-two people distributed throughout the US. One thing that’s different about Replicated is our developers don’t actually store or execute code on their laptops; all of our development happens on remote instances in the cloud.
Our product, KOTS, runs in Kubernetes and manages the lifecycle of 3rd-party applications in the Kubernetes cluster. Building and validating the product requires a developer to have access to a cluster. But as we started to hire more and more engineers it became ridiculous to ask everyone to run their own local Kubernetes cluster. We needed to both simplify and secure our setup to allow every engineer to run their environment in the cloud, and we needed to do it in a way which was seamless and secure.
Previous Dev Environments with Docker for Mac
We started with each developer building Continue reading
How to Build a Highly Productive Remote Team (or Team of Contractors) with Cloudflare for Teams
Much of IT has been built on two outdated assumptions about how work is done. First, that employees all sit in the same building or branch offices. Second, that those employees will work full-time at the same company for years.
Both of these assumptions are no longer true.
Employees now work from anywhere. In the course of writing this blog post, I opened review tickets in our internal JIRA from my dining table at home. I reviewed internal wiki pages on my phone during my commute on the train. And I spent time reviewing some marketing materials in staging in our CMS.
In a past job, I would have suffered trying to connect to these tools through a VPN. That would have slowed down my work on a laptop and made it nearly impossible to use a phone to catch up on my commute.
The second challenge is ramp-up. I joined Cloudflare a few months ago. As a member of the marketing team, I work closely with our product organization and there are several dozen tools that I need to do that.
I’m hardly alone. The rise of SaaS and custom internal applications means that employees need access to all Continue reading
Cloudflare for Teams Free for Small Businesses During Coronavirus Emergency
There are a lot of people and businesses worldwide that are currently suffering, so I don't want to waste any time in getting to the point.
Beginning today, we are making our Cloudflare for Teams products free to small businesses around the world. Teams enables remote workers to operate securely and easily. We will continue this policy for at least the next 6 months. We're doing this to help ensure that small businesses that implement work from home policies in order to combat the spread of the virus can ensure business continuity. You can learn more and apply at: https://www.cloudflare.com/smallbusiness
We've also helped launch an online hub where small businesses can see technology services available to them for free or a substantial discount from multiple companies, during the Coronavirus Emergency: https://openforbusiness.org
To understand more about why we're doing this, read on.
The IT Strain of WFH
We have a team at Cloudflare carefully monitoring the spread of the SARS-Coronavirus-2, which is responsible for the COVID-19 respiratory disease. Like at many other companies, we have heeded the advice of medical professionals and government agencies and are increasingly allowing employees to work from home in impacted regions in order Continue reading
Why the 100th Anniversary of Women’s Right to Vote in the U.S. is Important to Celebrate on International Women’s Day
Seven months ago, I joined Cloudflare to work on the Public Policy Team focusing on our democracy projects such as Project Galileo, Athenian Project and Cloudflare for Campaigns. Since I joined the team, I have learned a lot about how important cybersecurity protections are for organizations that are the target of sophisticated cyberattacks, while also learning about the complex election security environment in the United States and abroad.
It seems fitting that on International Women’s Day, a day people throughout the world are celebrating the achievements of women, we also celebrate the Centennial Anniversary of the Women’s Suffrage Movement which was the tipping point that gave many women voting rights in the United States.
Since I have been working on Cloudflare’s election security projects, this day means something extra special to me and many of my colleagues who believe that voting is the cornerstone of democracy and that having access to information regarding voting and elections is essential.
Here are five reflections that I want to share on International Women’s Day and the Centennial Anniversary of the Nineteenth Amendment which granted women the right to vote in the United States:
1. The Women’s Suffrage Movement in the United States was Continue reading
How Cloudflare keeps employees productive from any location
Cloudflare employs more than 1,200 people in 13 different offices and maintains a network that operates in 200 cities. To do that, we used to suffer through a traditional corporate VPN that backhauled traffic through a physical VPN appliance. It was, frankly, horrible to work with as a user or IT person.
With today’s mix of on-prem, public cloud and SaaS and a workforce that needs to work from anywhere, be it a coffee shop or home, that model is no longer sustainable. As we grew in headcount, we were spending too much time resolving VPN helpdesk tickets. As offices around the world opened, we could not ask our workforce to sit as every connection had to go back through a central location.
We also had to be ready to scale. Some organizations are currently scrambling to load test their own VPN in the event that their entire workforce needs to work remotely during the COVID-19 outbreak. We could not let a single physical appliance constrain our ability to deliver 26M Internet properties to audiences around the world.
To run a network like Cloudflare, we needed to use Cloudflare’s network to stay fast and secure.
We built Cloudflare Access, part Continue reading
International Women’s Day 2020: Building a Modern Security Team
When we started at Cloudflare in the summer of 2018, we joined a small security team intent on helping it grow quickly. Cloudflare was already a successful “unicorn” startup and its profile was changing fast, providing cyber security protection for millions of Internet-facing properties and moving towards becoming a public company. We were excited to help build the team that would ensure the security of Cloudflare’s systems and the sensitive customer data that flows through them.
Competing for security talent in the tech industry - where every company is investing heavily on security - isn't easy. But, in 18 months, we have grown our team 400% from under 10 people to almost 50 (and still hiring). We are proud that 40% of our team are women and 25% are from an under-represented minority. We believe from experience, and the research shows, that more diverse teams drive better business results and can be a better place to work.
In honor of International Women’s Day this Sunday, we wanted to share some of our lessons learned on how to build a diverse team and inclusive culture on a modern security team.
Lessons Learned Building a Diverse Team
- Our effort to Continue reading
The History of the URL
This issue was occuring now because the ARPANET was on the verge of switching from its original NCP protocol, to the TCP/IP protocol which powers what we now call the Internet. With that switch suddenly there would be a multitude of interconnected networks (an ‘Inter... net’) requiring a more ‘hierarchical’ domain system where ARPANET could resolve its own domains while the other networks resolved theirs.
Other networks at the time had great names like “COMSAT”, “CHAOSNET”, “UCLNET” and “INTELPOSTNET” and were maintained by groups of universities and companies all around the US who wanted to be able to communicate, and could afford to lease 56k lines from the phone company and buy the requisite PDP-11s to handle routing.
In the original ARPANET design, a central Network Information Center Continue reading