Reflecting on my first year at Cloudflare as a Field Marketer in APAC
Hey there! I am Els (short form for Elspeth) and I am the Field Marketing and Events Manager for APAC. I am responsible for building brand awareness and supporting our lovely sales team in acquiring new logos across APAC.
I was inspired to write about my first year in Cloudflare, because John, our CTO, encouraged more women to write for our Cloudflare blog after reviewing our blogging statistics and found out that more men than women blog for Cloudflare. I jumped at the chance because I thought this is a great way to share many side stories as people might not know about how it feels to work in Cloudflare.
Why Cloudflare?
Before I continue, I must mention that I really wanted to join Cloudflare after reading our co-founder Michelle’s reply on Quora regarding "What is it like to work in Cloudflare?." Michelle’s answer as follows:
“my answer is 'adult-like.' While we haven’t adopted this as our official company-wide mantra, I like the simplicity of that answer. People work hard, but go home at the end of the day. People care about their work and want to do a great job. When someone does a good job, Continue reading
Diversity Welcome – A Latinx journey into Cloudflare
I came to the United States chasing the love of my life, today my wife, in 2015.
A Spanish native speaker, Portuguese as my second language and born in the Argentine city of Córdoba more than 6,000 miles from San Francisco, there is no doubt that the definition of "Latino" fits me very well and with pride.
Cloudflare was not my first job in this country but it has been the organization in which I have learned many of the things that have allowed me to understand the corporate culture of a society totally alien to the one which I come from.
I was hired in January 2018 as the first Business Development Representative for the Latin America (LATAM) region based in San Francisco. This was long before the company went public in September 2019. The organization was looking for a specialist in Latin American markets with not only good experience and knowledge beyond languages (Spanish/Portuguese), but understanding of the economy, politics, culture, history, go-to-market strategies, etc.—I was lucky enough to be chosen as "that person". Cloudflare invested in me to a great extent and I was amazed at the freedom I had to propose ideas and bring them Continue reading
Internationalizing the Cloudflare Dashboard
Cloudflare’s dashboard now supports four new languages (and multiple locales): Spanish (with country-specific locales: Chile, Ecuador, Mexico, Peru, and Spain), Brazilian Portuguese, Korean, and Traditional Chinese. Our customers are global and diverse, so in helping build a better Internet for everyone, it is imperative that we bring our products and services to customers in their native language.
Since last year Cloudflare has been hard at work internationalizing our dashboard. At the end of 2019, we launched our first language other than US English: German. At the end of March 2020, we released three additional languages: French, Japanese, and Simplified Chinese. If you want to start using the dashboard in any of these languages, you can change your language preference in the top right of the Cloudflare dashboard. The preference selected will be saved and used across all sessions.
In this blog post, I want to help those unfamiliar with internationalization and localization to better understand how it works. I also would like to tell the story of how we made internationalizing and localizing our application a standard and repeatable process along with sharing a few tips that may help you as you do the same.
Beginning the journey
The first Continue reading
Introducing IP Lists
Authentication on the web has been steadily moving to the application layer using services such as Cloudflare Access to establish and enforce software-controlled, zero trust perimeters. However, there are still several important use cases for restricting access at the network-level by source IP address, autonomous system number (ASN), or country. For example, some businesses are prohibited from doing business with customers in certain countries, while others maintain a blocklist of problematic IPs that have previously attacked them.
Enforcing these network restrictions at centralized chokepoints using appliances—hardware or virtualized—adds unacceptable latency and complexity, but doing so performantly for individual IPs at the Cloudflare edge is easy. Today we’re making it just as easy to manage tens of thousands of IPs across all of your zones by grouping them in data structures known as IP Lists. Lists can be stored with metadata at the Cloudflare edge, replicated within seconds to our data centers in 200+ cities, and used as part of our powerful, expressive Firewall Rules engine to take action on incoming requests.
Previously, these sort of network-based security controls have been configured using IP Access or Zone Lockdown rules. Both tools have a number of Continue reading
Why I’m Helping Cloudflare Grow in Japan
If you'd like to read this post in Japanese click here.
I’m excited to say that I’ve recently joined the Cloudflare team as Head of Japan. Cloudflare has had a presence in Japan for a while now, not only with its network spanning the country, but also with many Japanese customers and partners which I’m now looking forward to growing with. In this new role, I’m focused on expanding our capabilities in the Japanese market, building upon our current efforts, and helping more companies in the region address and put an end to the technical pain points they are facing. This is an exciting time for me and an important time for the company. Today, I’m particularly eager to share that we are opening Cloudflare’s first Japan office, in Tokyo! I can’t wait to grow the Cloudflare business and team here.
Why Cloudflare?
The web was built 25 years ago. This invention changed the way people connected—to anyone and anywhere—and the way we work, play, live, learn, and on. We have seen this become more and more complex. With complexities come difficulties, such as ensuring security, performance, and reliability while online. Cloudflare is helping to solve these challenges that businesses Continue reading
Cloudflare outage on July 17, 2020
Today a configuration error in our backbone network caused an outage for Internet properties and Cloudflare services that lasted 27 minutes. We saw traffic drop by about 50% across our network. Because of the architecture of our backbone this outage didn’t affect the entire Cloudflare network and was localized to certain geographies.
The outage occurred because, while working on an unrelated issue with a segment of the backbone from Newark to Chicago, our network engineering team updated the configuration on a router in Atlanta to alleviate congestion. This configuration contained an error that caused all traffic across our backbone to be sent to Atlanta. This quickly overwhelmed the Atlanta router and caused Cloudflare network locations connected to the backbone to fail.
The affected locations were San Jose, Dallas, Seattle, Los Angeles, Chicago, Washington, DC, Richmond, Newark, Atlanta, London, Amsterdam, Frankfurt, Paris, Stockholm, Moscow, St. Petersburg, São Paulo, Curitiba, and Porto Alegre. Other locations continued to operate normally.
For the avoidance of doubt: this was not caused by an attack or breach of any kind.
We are sorry for this outage and have already made a global change to the backbone configuration that will prevent it from being able to occur Continue reading
Serverless Rendering with Cloudflare Workers
Cloudflare’s Workers platform is a powerful tool; a single compute platform for tasks as simple as manipulating requests or complex as bringing application logic to the network edge. Today I want to show you how to do server-side rendering at the network edge using Workers Sites, Wrangler, HTMLRewriter, and tools from the broader Workers platform.
Each page returned to the user will be static HTML, with dynamic content being rendered on our serverless stack upon user request. Cloudflare’s ability to run this across the global network allows pages to be rendered in a distributed fashion, close to the user, with miniscule cold start times for the application logic. Because this is all built into Cloudflare’s edge, we can implement caching logic to significantly reduce load times, support link previews, and maximize SEO rankings, all while allowing the site to feel like a dynamic application.
A Brief History of Web Pages
In the early days of the web pages were almost entirely static - think raw HTML. As Internet connections, browsers, and hardware matured, so did the content on the web. The world went from static sites to more dynamic content, powered by technologies like CGI, PHP, Flash, CSS, JavaScript, and Continue reading
Cloudflare’s first year in Lisbon
A year ago I wrote about the opening of Cloudflare’s office in Lisbon, it’s hard to believe that a year has flown by. At the time I wrote:
Lisbon’s combination of a large and growing existing tech ecosystem, attractive immigration policy, political stability, high standard of living, as well as logistical factors like time zone (the same as the UK) and direct flights to San Francisco made it the clear winner.
We landed in Lisbon with a small team of transplants from other Cloudflare offices. Twelve of us moved from the UK, US and Singapore to bootstrap here. Today we are 35 people with another 10 having accepted offers; we’ve almost quadrupled in a year and we intend to keep growing to around 80 by the end of 2020.
If you read back to my description of why we chose Lisbon only one item hasn’t turned out quite as we expected. Sure enough TAP Portugal does have direct flights to San Francisco but the pandemic put an end to all business flying worldwide for Cloudflare. We all look forward to getting back to being able to visit our colleagues in other locations.
The pandemic also put us in the Continue reading
Cloudflare Network expands to more than 100 Countries
2020 has been a historic year that will forever be associated with the COVID-19 pandemic. Over the past six months, we have seen societies, businesses, and entire industries unsettled. The situation at Cloudflare has been no different. And while this pandemic has affected each and every one of us, we here at Cloudflare have not forgotten what our mission is: to help build a better Internet.
We have expanded our global network to 206 cities across more than 100 countries. This is in addition to completing 40+ datacenter expansion projects and adding over 1Tbps in dedicated “backbone” (transport) capacity connecting our major data centers so far this year.
Pandemic times means new processes
There was zero chance that 2020 would mean business as usual within the Infrastructure department. We were thrown a curve-ball as the pandemic began affecting our supply chains and operations. By April, the vast majority of the world’s passenger flights were grounded. The majority of bulk air freight ships within the lower deck (“belly”) of these flights, which saw an imbalance between supply and demand with the sudden 74% decrease in passenger belly cargo capacity relative to the same period last year.
We were fortunate to have Continue reading
flowtrackd: DDoS Protection with Unidirectional TCP Flow Tracking
Magic Transit is Cloudflare’s L3 DDoS Scrubbing service for protecting network infrastructure. As part of our ongoing investment in Magic Transit and our DDoS protection capabilities, we’re excited to talk about a new piece of software helping to protect Magic Transit customers: flowtrackd. flowrackd is a software-defined DDoS protection system that significantly improves our ability to automatically detect and mitigate even the most complex TCP-based DDoS attacks. If you are a Magic Transit customer, this feature will be enabled by default at no additional cost on July 29, 2020.
TCP-Based DDoS Attacks
In the first quarter of 2020, one out of every two L3/4 DDoS attacks Cloudflare mitigated was an ACK Flood, and over 66% of all L3/4 attacks were TCP based. Most types of DDoS attacks can be mitigated by finding unique characteristics that are present in all attack packets and using that to distinguish ‘good’ packets from the ‘bad’ ones. This is called "stateless" mitigation, because any packet that has these unique characteristics can simply be dropped without remembering any information (or "state") about the other packets that came before it. However, when attack packets have no unique characteristics, then "stateful" mitigation is required, because whether a Continue reading
No Humans Involved: Mitigating a 754 Million PPS DDoS Attack Automatically
On June 21, Cloudflare automatically mitigated a highly volumetric DDoS attack that peaked at 754 million packets per second. The attack was part of an organized four day campaign starting on June 18 and ending on June 21: attack traffic was sent from over 316,000 IP addresses towards a single Cloudflare IP address that was mostly used for websites on our Free plan. No downtime or service degradation was reported during the attack, and no charges accrued to customers due to our unmetered mitigation guarantee.
The attack was detected and handled automatically by Gatebot, our global DDoS detection and mitigation system without any manual intervention by our teams. Notably, because our automated systems were able to mitigate the attack without issue, no alerts or pages were sent to our on-call teams and no humans were involved at all.
During those four days, the attack utilized a combination of three attack vectors over the TCP protocol: SYN floods, ACK floods and SYN-ACK floods. The attack campaign sustained for multiple hours at rates exceeding 400-600 million packets per second Continue reading
Sandboxing in Linux with zero lines of code
Modern Linux operating systems provide many tools to run code more securely. There are namespaces (the basic building blocks for containers), Linux Security Modules, Integrity Measurement Architecture etc.
In this post we will review Linux seccomp and learn how to sandbox any (even a proprietary) application without writing a single line of code.
Tux by Iwan Gabovitch, GPL
Sandbox, Simplified Pixabay License
Linux system calls
System calls (syscalls) is a well-defined interface between userspace applications and the operating system (OS) kernel. On modern operating systems most applications provide only application-specific logic as code. Applications do not, and most of the time cannot, directly access low-level hardware or networking, when they need to store data or send something over the wire. Instead they use system calls to ask the OS kernel to do specific hardware and networking tasks on their behalf:
Apart from providing a generic high level way for applications to interact with the low level hardware, the system call architecture allows the OS kernel to manage available resources between applications as well as enforce policies, like application permissions, networking access control lists etc.
Linux seccomp
Linux seccomp is yet another syscall on Linux, but it is a bit Continue reading
CVE-2020-5902: Helping to protect against the F5 TMUI RCE vulnerability
Cloudflare has deployed a new managed rule protecting customers against a remote code execution vulnerability that has been found in F5 BIG-IP’s web-based Traffic Management User Interface (TMUI). Any customer who has access to the Cloudflare Web Application Firewall (WAF) is automatically protected by the new rule (100315) that has a default action of BLOCK.
Initial testing on our network has shown that attackers started probing and trying to exploit this vulnerability starting on July 3.
F5 has published detailed instructions on how to patch affected devices, how to detect if attempts have been made to exploit the vulnerability on a device and instructions on how to add a custom mitigation. If you have an F5 device, read their detailed mitigations before reading the rest of this blog post.
The most popular probe URL appears to be /tmui/login.jsp/..;/tmui/locallb/workspace/fileRead.jsp followed by /tmui/login.jsp/..;/tmui/util/getTabSet.jsp, /tmui/login.jsp/..;/tmui/system/user/authproperties.jsp and /tmui/login.jsp/..;/tmui/locallb/workspace/tmshCmd.jsp. All contain the critical pattern ..; which is at the heart of the vulnerability.
On July 3 we saw O(1k) probes ramping to O(1m) yesterday. This is because simple test patterns have been added to scanning tools and small test programs made available by Continue reading
Building Cloudflare TV from scratch
Cloudflare TV is inspired by television shows of the 90s that shared the newest, most exciting developments in computing and music videos. We had three basic requirements for Cloudflare TV:
- Guest participation should be as simple as joining a Zoom call
- There should be 24x7 programming. Something interesting should be playing all the time
- Everything should happen in the cloud and we should never have to ask anyone “to leave their computer on” to have the stream running 24 hours a day
We didn’t set out to build Cloudflare TV from scratch
Building a lot of the technology behind Cloudflare TV from scratch was not part of the plan, especially given our aggressive timeline. So why did we decide to pursue it? After evaluating multiple live streaming solutions, we reached the following conclusion:
- 24x7 linear streaming is not something that is a priority for most video streaming platforms. This makes sense: the rise of video-on-demand and event-based live streaming has come at the expense of linear streaming.
- Most broadcasting platforms have their own guest apps which must be downloaded and set up in advance. This introduces unnecessary friction compared to clicking a link in the calendar invite to join a Continue reading
Making the WAF 40% faster
Cloudflare’s Web Application Firewall (WAF) protects against malicious attacks aiming to exploit vulnerabilities in web applications. It is continuously updated to provide comprehensive coverage against the most recent threats while ensuring a low false positive rate.
As with all Cloudflare security products, the WAF is designed to not sacrifice performance for security, but there is always room for improvement.
This blog post provides a brief overview of the latest performance improvements that were rolled out to our customers.
Transitioning from PCRE to RE2
Back in July of 2019, the WAF transitioned from using a regular expression engine based on PCRE to one inspired by RE2, which is based around using a deterministic finite automaton (DFA) instead of backtracking algorithms. This change came as a result of an outage where an update added a regular expression which backtracked enormously on certain HTTP requests, resulting in exponential execution time.
After the migration was finished, we saw no measurable difference in CPU consumption at the edge, but noticed execution time outliers in the 95th and 99th percentiles decreased, something we expected given RE2's guarantees of a linear time execution with the size of the input.
How to test HTTP/3 and QUIC with Firefox Nightly
HTTP/3 is the third major version of the Hypertext Transfer Protocol, which takes the bold step of moving away from TCP to the new transport protocol QUIC in order to provide performance and security improvements.
During Cloudflare's Birthday Week 2019, we were delighted to announce that we had enabled QUIC and HTTP/3 support on the Cloudflare edge network. This was joined by support from Google Chrome and Mozilla Firefox, two of the leading browser vendors and partners in our effort to make the web faster and more reliable for all. A big part of developing new standards is interoperability, which typically means different people analysing, implementing and testing a written specification in order to prove that it is precise, unambiguous, and actually implementable.
At the time of our announcement, Chrome Canary had experimental HTTP/3 support and we were eagerly awaiting a release of Firefox Nightly. Now that Firefox supports HTTP/3 we thought we'd share some instructions to help you enable and test it yourselves.
How do I enable HTTP/3 for my domain?
Simply go to the Cloudflare dashboard and flip the switch from the "Network" tab manually:
Using Firefox Nightly as an HTTP/3 client
Firefox Nightly has experimental support for Continue reading
Catching up with Workers KV
The Workers Distributed Data team has been hard at work since we gave you an update last November. Today, we’d like to share with you some of the stuff that has recently shipped in Workers KV: a new feature and an internal change that should significantly improve latency in some cases. Let’s dig in!
KV Metadata
Workers KV has a fairly straightforward interface: you can put keys and values into KV, and then fetch the value back out by key:
await contents.put(“index.html”, someHtmlContent);
await contents.put(“index.css”, someCssContent);
await contents.put(“index.js”, someJsContent);
// later
let index = await contents.get(“index.html”);
Pretty straightforward. But as you can see from this example, you may store different kinds of content in KV, even if the type is identical. All of the values are strings, but one is HTML, one is CSS, and one is JavaScript. If we were going to serve this content to users, we would have to construct a response. And when we do, we have to let the client know what the content type of that request is: text/html for HTML, text/css for CSS, and text/javascript for JavaScript. If we serve the incorrect content type to Continue reading
Introducing Regional Services
In a world where, increasingly, workloads shift to the cloud, it is often uncertain and unclear how data travels the Internet and in which countries data is processed. Today, Cloudflare is pleased to announce that we're giving our customers control. With Regional Services, we’re providing customers full control over exactly where their traffic is handled.
We operate a global network spanning more than 200 cities. Each data center runs servers with the exact same software stack. This has enabled Cloudflare to quickly and efficiently add capacity where needed. It also allows our engineers to ship features with ease: deploy once and it's available globally.
The same benefit applies to our customers: configure once and that change is applied everywhere in seconds, regardless of whether they’re changing security features, adding a DNS record or deploying a Cloudflare Worker containing code.
Having a homogenous network is great from a routing point of view: whenever a user performs an HTTP request, the closest datacenter is found due to Cloudflare's Anycast network. BGP looks at the hops that would need to be traversed to find the closest data center. This means that someone near the Canadian border (let's say North Dakota) could easily find Continue reading
Lessons from a 2020 intern assignment
This summer, Cloudflare announced that we were doubling the size of our Summer 2020 intern class. Like everyone else at Cloudflare, our interns would be working remotely, and due to COVID-19, many companies had significantly reduced their intern class size, or outright cancelled their programs entirely.
With our announcement came a huge influx of students interested in coming to Cloudflare. For applicants seeking engineering internships, we opted to create an exercise based on our serverless product Cloudflare Workers. I'm not a huge fan of timed coding exercises, which is a pretty traditional way that companies gauge candidate skill, so when I was asked to help contribute an example project that would be used instead, I was excited to jump on the project. In addition, it was a rare chance to have literally thousands of eager pairs of eyes on Workers, and on our documentation, a project that I've been working on daily since I started at Cloudflare over a year ago.
In this blog post, I will explain the details of the full-stack take home exercise that we sent out to our 2020 internship applicants. We asked participants to spend no more than an afternoon working on it, and Continue reading
Cloudflare and Rackspace Technology Expand Partnership with Managed Services
Last year, Cloudflare announced the planned expansion of our partner program to help managed and professional service partners efficiently engage with Cloudflare and join us in our mission to help build a better Internet. We’ve been hard at work growing and expanding our partnerships with some amazing global teams that help us support digital transformation and security needs around the world, and today we’d like to highlight one of our Elite global partners, Rackspace Technology.
Today, we are announcing the expansion of our worldwide reseller partnership with Rackspace Technology to include a series of managed services offerings for Cloudflare. As a result, with Cloudflare Security, Performance, and Reliability with Rackspace Managed Services, customers will not only have access to and the scalability of Cloudflare’s global network and integrated cloud platform of security, performance, and reliability solutions but also benefit from a team of certified, enabled Rackspace experts to configure, onboard, and deploy Cloudflare solutions. Because more than 1 billion unique IP addresses pass through Cloudflare's global network every day, Cloudflare, together with its solutions providers, can build real-world intelligence on the communications occurring over the Internet, and how well they perform. We’ve enjoyed enabling their teams to leverage this Continue reading