Creating serendipity with Python
We've been experimenting with breaking up employees into random groups (of size 4) and setting up video hangouts between them. We're doing this to replace the serendipitous meetings that sometimes occur around coffee machines, in lunch lines or while waiting for the printer. And also, we just want people to get to know each other.
Which lead to me writing some code. The core of which is divide n elements into groups of at least size g minimizing the size of each group. So, suppose an office has 15 employees in it then it would be divided into three groups of sizes 5, 5, 5; if an office had 16 employees it would be 4, 4, 4, 4; if it had 17 employees it would be 4, 4, 4, 5 and so on.
I initially wrote the following code (in Python):
groups = [g] * (n//g)
for e in range(0, n % g):
groups[e % len(groups)] += 1
The first line creates n//g (// is integer division) entries of size g (for example, if g == 4 and n == 17 then groups == [4, 4, 4, 4]). The for loop deals with the 'left over' parts that Continue reading
Through the eyes of a Cloudflare Technical Support Engineer
This post originally appeared on Landing Jobs under the title Mission: Protect the Internet where you can find open positions at Cloudflare Lisbon.
Justina Wong, Technical Support Team Lead in Lisbon, talks about what it’s like working at Cloudflare, and everything you need to know if you want to join us.
Justina joined Cloudflare about three years ago in London as a Technical Support Engineer. Currently, she’s part of their Customer Support team working in Lisbon as a team lead.
I can’t speak for others, but I love the things you can learn from the others. There are so many talented individuals who are willing and ready to teach/share. They are my inspiration and I want to become them!
On a Mission to Protect the Internet
Justina’s favourite Cloudflare products are firewall-related ones. The company’s primary care is for the customers and they want to make attack mitigation as easy as possible. As she puts it, “the fact that these protections are on multiple layers, like L7, L3/4, is very important, and I’m proud to be someone who can help our customers when they face certain attacks.”.
Cloudflare is constantly releasing new products to help build a better Internet, Continue reading
Using HPKE to Encrypt Request Payloads
The Managed Rules team was recently given the task of allowing Enterprise users to debug Firewall Rules by viewing the part of a request that matched the rule. This makes it easier to determine what specific attacks a rule is stopping or why a request was a false positive, and what possible refinements of a rule could improve it.
The fundamental problem, though, was how to securely store this debugging data as it may contain sensitive data such as personally identifiable information from submissions, cookies, and other parts of the request. We needed to store this data in such a way that only the user who is allowed to access it can do so. Even Cloudflare shouldn't be able to see the data, following our philosophy that any personally identifiable information that passes through our network is a toxic asset.
This means we needed to encrypt the data in such a way that we can allow the user to decrypt it, but not Cloudflare. This means public key encryption.
Now we needed to decide on which encryption algorithm to use. We came up with some questions to help us evaluate which one to use:
- What requirements do we have for Continue reading
Introducing: Smarter Tiered Cache Topology Generation
Caching is a magic trick. Instead of a customer’s origin responding to every request, Cloudflare’s 200+ data centers around the world respond with content that is cached geographically close to visitors. This dramatically improves the load performance for web pages while decreasing the bandwidth costs by having Cloudflare respond to a request with cached content.
However, if content is not in cache, Cloudflare data centers must contact the origin server to receive the content. This isn’t as fast as delivering content from cache. It also places load on an origin server, and is more costly compared to serving directly from cache. These issues can be amplified depending on the geographic distribution of a website’s visitors, the number of data centers contacting the origin, and the available origin resources for responding to requests.
To decrease the number of times our network of data centers communicate with an origin, we organize data centers into tiers so that only upper-tier data centers can request content from an origin and then they spread content to lower tiers. This means content that loads faster for visitors, is cheaper to serve, and reduces origin resource consumption.
Today, I’m thrilled to announce a fundamental improvement to Argo Continue reading
Tiered Cache Smart Topology
A few years ago, we released Argo to help make the Internet faster and more efficient. Argo observes network conditions and finds the optimal route across the Internet for origin server requests, avoiding congestion along the way.
Tiered Cache is an Argo feature that reduces the number of data centers responsible for requesting assets from the origin. With Tiered Cache active, a request in South Africa won’t go directly to an origin in North America, but, instead, look in a large, nearby data center to see if the data requested is cached there first. The number and location of the data centers used by Tiered Cache is controlled by a piece of configuration called the topology. By default, we use a generic topology for every customer that strikes a balance between cache hit ratios and latency that is suitable for most users.
Today we’re introducing Smart Topology, which maximizes cache hit ratios by building on Argo’s internal infrastructure to identify the single best data center for making requests to the origin.
Standard Cache
The standard method for caching assets is to let each data center be a reverse proxy for the origin server. In this scheme, a miss in any Continue reading
2020 U.S. Election: Cybersecurity Analysis
As the election season has ramped down and the new Presidential Administration begins, we think it’s important to assess whether there are lessons we can draw from our experience helping to provide cybersecurity services for those involved in the 2020 U.S. elections.
Cloudflare built the Athenian Project - our project to provide free services to state and local election websites - around the idea that access to the authoritative voting information offered by state and local governments is key to a functioning democracy and that Cloudflare could play an important role in ensuring that election-related websites are protected from cyberattacks intended to disrupt that access. Although the most significant challenges in this election cycle fell outside the realm of cybersecurity, the 2020 election certainly validated the importance of having access to definitive sources of authoritative election information.
We were pleased that the robust cybersecurity preparations we saw for the 2020 U.S. election appeared to be successful. From the Cloudflare perspective, we had the opportunity to witness firsthand the benefits of having access to free cybersecurity services provided to organizations that promote accurate voting information and election results, state and local governments conducting elections, and federal U.S candidates Continue reading
Who won Super Bowl LV? A look at Internet traffic during the game
The obvious answer is the Tampa Bay Buccaneers but the less obvious answer comes from asking “which Super Bowl advertiser got the biggest Internet bump?”. This blog aims to answer that question.
Before, during, and after the game a crack team of three people who work on Cloudflare Radar looked at real time statistics for traffic to advertisers’ websites, social media in the US, US food delivery services, and websites covering (American) football. Luckily, one of us (Kari) is (a) American and (b) a fan of football. Unluckily, one of us (Kari) is a fan of the Kansas City Chiefs.
Cloudflare Radar uses a variety of sources to provide aggregate information about Internet traffic and attack trends. In this blog post we use DNS name resolution data to estimate traffic to websites. We can’t see who visited the websites mentioned below, or what anyone did on the websites, but DNS can give us an estimate of the interest generated by the commercials. This analysis only looked at the top-level names in each domain (so example.com and www.example.com and not any other subdomains).
The Big Picture
To get the ball rolling here’s a look at traffic to Continue reading
Congratulations to Cloudflare’s 2020 Partner Award Winners
We are privileged to share Cloudflare’s inaugural set of Partner Awards. These Awards recognize our partner companies and representatives worldwide who stood out this past year for their investments in acquiring technical expertise in our offerings, for delivering innovative applications and services built on Cloudflare, and for their commitment to customer success.
The unprecedented challenges in 2020 have reinforced how critical it is to have a secure, performant, and reliable Internet. Throughout these turbulent times, our partners have been busy innovating and helping organizations of all sizes and in various industries. By protecting and accelerating websites, applications, and teams with Cloudflare, our partners have helped these organizations adjust, seize new opportunities, and thrive.
Congratulations to each of our award winners. Cloudflare’s mission of helping build a better Internet is more important than ever. And our partners are more critical than ever to achieving our mission. Testifying to Cloudflare’s global reach, our honorees represent companies headquartered in 16 countries.
Cloudflare Partner of the Year Honorees, 2020
Worldwide MSP Partner of the Year: Rackspace Technology
Honors the top performing managed services provider (MSP) partner across Cloudflare's three sales geographies: Americas, APAC, and EMEA.
The effect of switching to TCMalloc on RocksDB memory use
In previous posts we wrote about our configuration distribution system Quicksilver and the story of migrating its storage engine to RocksDB. This solution proved to be fast, resilient and stable. During the migration process, we noticed that Quicksilver memory consumption was unexpectedly high. After our investigation we found out that the root cause was a default memory allocator that we used. Switching memory allocator improved service memory consumption by almost three times.
Unexpected memory growth
After migrating to RocksDB, the memory used by the application increased significantly. Also, the way memory was growing over time looked suspicious. It was around 15GB immediately after start and then was steadily growing for multiple days, until stabilizing at around 30GB. Below, you can see a memory consumption increase after migrating one of our test instances to RocksDB.
We started our investigation with heap profiling with the assumption that we had a memory leak somewhere and found that heap size was almost three times less than the RSS value reported by the operating system. So, if our application does not actually use all this memory, it means that memory is ‘lost’ somewhere between the system and our application, which points to possible problems with Continue reading
Cloudflare Named a ‘Best Place to Work for LGBTQ Equality’
Today we are excited to announce Cloudflare has been named a “Best Place to Work for LGBTQ Equality” by the Human Rights Campaign (HRC). This designation was earned by receiving a perfect score of 100 percent on the HRC’s 2021 Corporate Equality Index. The Corporate Equality Index (CEI) is a nationally recognized benchmarking tool that assesses the inclusivity of corporate policies, practices, and benefits for LGBTQIA+ employees.
Cloudflare’s mission is to “help build a better Internet”. An essential factor that helps us deliver on this mission is our people. When you are solving some of the toughest problems facing the Internet for users worldwide, you need talented individuals that contribute unique outlooks. We strive to build a workplace where our entire team feels comfortable and excited to bring their true authentic selves so they can do their best work.
2021 is the first year Cloudflare has been listed on the Index, but we have been paving this path for quite some time. Back in 2017 a few Cloudflare employees chartered Proudflare, Cloudflare’s first-ever Employee Resource Group (ERG). Proudflare serves as a community space for LGBTQIA+ employees and allies. Check out our Proudflare Launch blog to learn more on that process! Continue reading
Starting a new job in the middle of a pandemic
It has now been more than 90 days since I joined Cloudflare’s EMEA Recruiting Team as a Recruiting Coordinator based in Lisbon. In a year filled with hardships for so many people around the world, I wanted to share my journey. I hope people will relate and feel encouraged to pursue their dreams, even during these challenging times.
When 2020 started, it was not in my plans to change jobs and start working at a new company, completely remote, without ever meeting my colleagues in person or visiting the office. However, that is exactly what happened, and I am so glad I did.
Interviewing with Cloudflare
The number of interviews in the hiring process at Cloudflare may feel overwhelming for some - in my case, I met 11 people during this process. For me, I was glad to have so many chances to get to know the people I would be working with. I believe I got as much out of the conversations as the interviewers did, which is great — a recruitment process should be as much about the company getting to know you, as you getting to know the company.
A great thing about interviewing remotely is that Continue reading
Data Privacy Day 2021 – Looking ahead at the always on, always secure, always private Internet
Welcome to Data Privacy Day 2021! Last year at this time, I was writing about how Cloudflare builds privacy into everything we do, with little idea about how dramatically the world was going to change. The tragedy of the COVID-19 pandemic has reshaped the way we go about our daily lives. Our dependence on the Internet grew exponentially in 2020 as we started working from home, attending school from home, and participating in online weddings, concerts, parties, and more. So as we begin this new year, it’s impossible to think about data privacy in 2021 without thinking about how an always-on, always secure, always private Internet is more important than ever.
The pandemic wasn’t the only thing to dramatically shape data privacy conversations last year. We saw a flurry of new activity on data protection legislation around the globe, and a trend toward data localization in a variety of jurisdictions.
I don’t think I’m taking any risks when I say that 2021 looks to be another busy year in the world of privacy and data protection. Let me tell you a bit about what that looks like for us at Cloudflare. We’ll be spending a lot of time in Continue reading
Automating data center expansions with Airflow
Cloudflare’s network keeps growing, and that growth doesn’t just come from building new data centers in new cities. We’re also upgrading the capacity of existing data centers by adding newer generations of servers — a process that makes our network safer, faster, and more reliable for our users.
Connecting new Cloudflare servers to our network has always been complex, in large part because of the amount of manual effort that used to be required. Members of our Data Center and Infrastructure Operations, Network Operations, and Site Reliability Engineering teams had to carefully follow steps in an extremely detailed standard operating procedure (SOP) document, often copying command-line snippets directly from the document and pasting them into terminal windows.
But such a manual process can only scale so far, and we knew must be a way to automate the installation of new servers.
Here’s how we tackled that challenge by building our own Provisioning-as-a-Service (PraaS) platform and cut by 90% the amount of time our team spent on mundane operational tasks.
Choosing and using an automation framework
When we began our automation efforts, we quickly realized it made sense to replace each of these manual SOP steps with an API-call equivalent and Continue reading
How We Ran a Successful Remote Internship Program in 2020
And how we are planning to do it again in 2021…
On March 5, I sat in a small conference room with a few key contributors in creating and hiring for the Cloudflare summer intern program. With the possibility of office shutdowns looming, the group discussed what an internship would look like without in-person mentorship. How would the managers cope? How would the interns cope? Would it even be worthwhile? After a few minutes of discussions, we settled on ‘absolutely’. A remote summer internship at Cloudflare would be worthwhile for students, mentors, buddies, and managers alike. After all, Cloudflare is an Internet company and we were ready to trust the Internet with a whole lot more than we had anticipated.
The months leading up to the summer were a blur, all I remember is that we did a lot of planning, interviewing and hiring. And I mean, a lot. On April 2, Matthew Prince announced that Cloudflare would be doubling the size of our 2020 intern class in response to other companies cutting their intern programs all together. Due to these cuts, many talented students lost their opportunities for the summer. We knew we couldn’t hire them all so we Continue reading
Introducing Project Fair Shot: Ensuring COVID-19 Vaccine Registration Sites Can Keep Up With Demand
Around the world government and medical organizations are struggling with one of the most difficult logistics challenges in history: equitably and efficiently distributing the COVID-19 vaccine. There are challenges around communicating who is eligible to be vaccinated, registering those who are eligible for appointments, ensuring they show up for their appointments, transporting the vaccine under the required handling conditions, ensuring that there are trained personnel to administer the vaccine, and then doing it all over again as most of the vaccines require two doses.
Cloudflare can't help with most of that problem, but there is one key part that we realized we could help facilitate: ensuring that registration websites don't crash under load when they first begin scheduling vaccine appointments. Project Fair Shot provides Cloudflare's new Waiting Room service for free for any government, municipality, hospital, pharmacy, or other organization responsible for distributing COVID-19 vaccines. It is open to eligible organizations around the world and will remain free until at least July 1, 2021 or longer if there is still more demand for appointments for the vaccine than there is supply.
Crashing Registration Websites
The problem of vaccine scheduling registration websites crashing under load isn't theoretical: it is happening over Continue reading
Cloudflare Waiting Room
Today, we are excited to announce Cloudflare Waiting Room! It will first be available to select customers through a new program called Project Fair Shot which aims to help with the problem of overwhelming demand for COVID-19 vaccinations causing appointment registration websites to fail. General availability in our Business and Enterprise plans will be added in the near future.
Wait, you’re excited about a… Waiting Room?
Most of us are familiar with the concept of a waiting room, and rarely are we excited about the idea of being in one. Usually our first experience of one is at a doctor’s office — yes, you have an appointment, but sometimes the doctor is running late (or one of the patients was). Given the doctor can only see one person at a time… the waiting room was born, as a mechanism to queue up patients.
While servers can handle more concurrent requests than a doctor can, they too can be overwhelmed. If, in a pre-COVID world, you’ve ever tried buying tickets to a popular concert or event, you’ve probably encountered a waiting room online. It limits requests inbound to an application, and places these requests into a virtual queue. Once the number Continue reading
Network-layer DDoS attack trends for Q4 2020
DDoS attack trends in the final quarter of 2020 defied norms in many ways. For the first time in 2020, Cloudflare observed an increase in the number of large DDoS attacks. Specifically, the number of attacks over 500Mbps and 50K pps saw a massive uptick.
In addition, attack vectors continued to evolve, with protocol-based attacks seeing a 3-10x increase compared to the prior quarter. Attackers were also more persistent than ever — nearly 9% of all attacks observed between October and December lasted more than 24 hours.
Below are additional noteworthy observations from the fourth quarter of 2020, which the rest of this blog explores in greater detail.
- Number of attacks: For the first time in 2020, the total number of attacks observed in Q4 decreased compared to the prior quarter.
- Attack duration: 73% of all attacks observed lasted under an hour, a decrease from 88% in Q3.
- Attack vectors: While SYN, ACK, and RST floods continued to be the dominant attack vectors deployed, attacks over NetBIOS saw a whopping 5400% increase, followed by those over ISAKMP and SPSS.
- Global DDoS activity: Our data centers in Mauritius, Romania, and Brunei recorded the highest percentages of DDoS activity relative to non-attack Continue reading
Uganda’s January 13, 2021 Internet Shut Down
Two days ago, through its communications regulator, Uganda's government ordered the "Suspension Of The Operation Of Internet Gateways" the day before the country's general election. This action was confirmed by several users and journalists who got access to the letter sent to Internet providers. In other words, the government effectively cut off Internet access from the population to the rest of the world.
Ahead of tomorrow’s election the Internet has been shutdown in Uganda (confirmed by a few friends in Kampala).
— Samira Sawlani (@samirasawlani) January 13, 2021
Letter from communications commission below: pic.twitter.com/tRpTIXTPcW
On Cloudflare Radar, we want to help anyone understand what happens on the Internet. We are continually monitoring our network and exposing insights, threats, and trends based on the aggregated data that we see.
Uganda's unusual traffic patterns quickly popped up in our charts. Our 7-day change in Internet Traffic chart in Uganda shows a clear drop to near zero starting around 1900 local time, when the providers received the letter.
This is also obvious in the Application-level Attacks chart.
The traffic drop was also confirmed by the Uganda Internet eXchange point, a place where many providers exchange their data traffic, on their Continue reading
KEMTLS: Post-quantum TLS without signatures
The Transport Layer Security protocol (TLS), which secures most Internet connections, has mainly been a protocol consisting of a key exchange authenticated by digital signatures used to encrypt data at transport[1]. Even though it has undergone major changes since 1994, when SSL 1.0 was introduced by Netscape, its main mechanism has remained the same. The key exchange was first based on RSA, and later on traditional Diffie-Hellman (DH) and Elliptic-curve Diffie-Hellman (ECDH). The signatures used for authentication have almost always been RSA-based, though in recent years other kinds of signatures have been adopted, mainly ECDSA and Ed25519. This recent change to elliptic curve cryptography in both at the key exchange and at the signature level has resulted in considerable speed and bandwidth benefits in comparison to traditional Diffie-Hellman and RSA.
TLS is the main protocol that protects the connections we use everyday. It’s everywhere: we use it when we buy products online, when we register for a newsletter — when we access any kind of website, IoT device, API for mobile apps and more, really. But with the imminent threat of the arrival of quantum computers (a threat that seems to be getting closer and closer), we need Continue reading
Holistic web protection: industry recognition for a prolific 2020
I love building products that solve real problems for our customers. These days I don’t get to do so as much directly with our Engineering teams. Instead, about half my time is spent with customers listening to and learning from their security challenges, while the other half of my time is spent with other Cloudflare Product Managers (PMs) helping them solve these customer challenges as simply and elegantly as possible. While I miss the deeply technical engineering discussions, I am proud to have the opportunity to look back every year on all that we’ve shipped across our application security teams.
Taking the time to reflect on what we’ve delivered also helps to reinforce my belief in the Cloudflare approach to shipping product: release early, stay close to customers for feedback, and iterate quickly to deliver incremental value. To borrow a term from the investment world, this approach brings the benefits of compounded returns to our customers: we put new products that solve real-world problems into their hands as quickly as possible, and then reinvest the proceeds of our shared learnings immediately back into the product.
It is these sustained investments that allow us to release a flurry of small improvements Continue reading