Deploy Workers using Terraform
Today we're excited to announce that Cloudflare Workers are now supported in the Cloudflare Terraform Provider.
Terraform is a fantastic tool for configuring your infrastructure. Traditionally if you wanted to spin up, tear down or update some of your infrastructure you would have to click around on a website or make some API calls, which is prone to human error. With Terraform, you define your infrastructure in simple, declarative configuration files and let Terraform figure out how to make the API calls for you. This also lets you treat your infrastructure like your code. You can check your Terraform configuration files into version control and integrate them into your normal software development workflow.
Terraform integrates with many infrastructure providers including Cloudflare. If you'd like to read more about setting up Terraform with Cloudflare, check out Getting started with Terraform and Cloudflare. In this post, I'm going to focus specifically on how to integrate Cloudflare Workers with Terraform.
In this example we're going to create partyparrot.business, and we're going to serve the whole site out of a worker without any origin server. We're starting from scratch here, but if you're already using Cloudflare workers and want to migrate to Continue reading
Norfolk and Richmond, Virginia: Cloudflare’s 152nd and 153rd cities
Virginia has a very important place in Internet history, as well as the history of Cloudflare’s network.
Northern Virginia, in the area around Ashburn VA, has for a long time been core to Internet infrastructure. In the early 1990’s, MAE-East (Metropolitan-Area-Exchange East) , an Internet Exchange Point (IXP) was established. MAE-East and West were some of the earliest IXPs. Internet Exchange Points are crucial interconnection points for ISPs and other Internet Networks to interconnect and exchange traffic. Eco-systems have grown around these through new data center offerings and new Internet platforms. Like many pieces of the Internet, MAE-East had a humble beginning, though not many humble beginnings grew to handle around 50% of Internet traffic exchange.
Cloudflare’s second Data Center, and one that still plays a critical component in our Global Network was Ashburn, Virginia. Similarly across many organizations, the Northern Virginia area has become a Data Center mecca. Many of the largest Clouds have a substantial amount of their footprint in Northern Virginia. Although MAE-East no longer exists, other Internet Exchange Points have come and grown in its place.
Cloudflare’s network has grown beyond what was traditional Interconnection points, like Ashburn/Northern VA, to a new Edge of the Continue reading
Build and Deploy Functions to Cloudflare’s 152+ Data Centers with Serverless
We’re very excited to announce that Cloudflare Workers are now integrated into the Serverless framework as a serverless cloud provider! Serverless’ open source framework has become a must-have for many developers, and we want to make it as simple as possible to harness the power of Cloudflare’s distributed computing network.
Workers have become essential to the way people build applications on the web. The expressiveness of modern JavaScript combined with sub-30 second deploys to Cloudflare’s network of 152+ datacenters means that you can truly build your application into our global compute network. Up until this point, deploying Workers required doing all of your editing through our browser-based IDE or developing one’s own custom tooling on top of our API. But many developers have their own environments and are already familiar with the Serverless framework, so it seemed natural that we build first-class support for Workers into Serverless!
You can now define the entire structure and routing behavior of your Workers scripts in code and deploy them with ease using serverless deploy from your own development environment. Store configuration files in version control alongside your application code. And feel more confident testing your application with serverless invoke, a new way Continue reading
EU Copyright Vote: A Critical Juncture for the Open Internet
Back in June, we blogged about the draft EU copyright proposal which is currently making its way through the legislative process in Brussels. We outlined how under one of the more controversial provisions within the draft Directive, Article 13, certain Internet platforms could be held legally responsible for any copyright content that their users upload and would effectively have to turn to automated filtering solutions to remove infringing content at the point of user upload. Moreover, in order to avoid potential legal liability, it is widely expected that content sharing providers would err on the side of caution and remove excessive amounts of content, resulting in a form of online censorship.
Since that blogpost, the European Parliament Plenary narrowly voted on 5th July to reject the proposal tabled by the Legal Affairs (JURI) Committee and a mandate to negotiate, and now the proposed Directive will undergo a full discussion and rescheduled vote in the next Plenary meeting on 12th September. This was a fantastic outcome, thanks in large part to a groundswell of support from those who value the fundamental right of freedom of expression online. It has presented a window of opportunity to correct the deeply flawed approach to Continue reading
Fixing an old hack – why we are bumping the IPv6 MTU
Back in 2015 we deployed ECMP routing - Equal Cost Multi Path - within our datacenters. This technology allowed us to spread traffic heading to a single IP address across multiple physical servers.
You can think about it as a third layer of load balancing.
- First we split the traffic across multiple IP addresses with DNS.
- Then we split the traffic across multiple datacenters with Anycast.
- Finally, we split the traffic across multiple servers with ECMP.
When deploying ECMP we hit a problem with Path MTU discovery. The ICMP packets destined to our Anycast IP's were being dropped. You can read more about that (and the solution) in the 2015 blog post Path MTU Discovery in practice.
To solve the problem we created a small piece of software, called pmtud (https://github.com/cloudflare/pmtud). Since deploying pmtud, our ECMP setup has been working smoothly.
Hardcoding IPv6 MTU
During that initial ECMP rollout things were broken. To keep services running until pmtud was done, we deployed a quick hack. We reduced the MTU of IPv6 traffic to the minimal possible value: 1280 bytes.
This was done as a tag on a default route. This is Continue reading
Bob Ross, Lorem Ipsum, Heroku and Cloudflare Workers
It may not be immediately obvious how these things are related, but bear with me... It was 4pm Friday and one of the engineers on the Cloudflare Tools team came to me with an emergency. "Steve! The Bob Ross Ipsum generator is down!".
If you've not heard of Lorem Ipsum, it's an extract from a latin poem that designers use as placeholder text when designing the layout of a document. There are generators all over the web that will spit out as much text as you need.
Of course, the web being the web that we all love, there are also endless parodies of Lorem Ipsum. You can generate Hodor Ipsum, Cat Ipsum and Hipster Ipsum. I have a new, undisputed favourite: Bob Ross Ipsum.
Not growing up in the U.S., I hadn't come across the lovable, calm, serene and beautiful human that is Bob Ross. If you haven't spent 30 mins watching him paint a landscape, you should do that now. He built a following as host of the TV show “The Joy of Painting” which ran on the U.S. PBS channel from 1983-1994. He became famous for Continue reading
Introducing Real World Serverless – Practical advice on how to use Cloudflare Workers
We’re getting the best minds on serverless technology from Cloudflare together to lead a series of talks on practical use cases for Cloudflare Workers. Join any of these six global talks for stories of how companies and developers are using serverless in the real world.
San Francisco - London - Austin - Singapore - Sydney - Melbourne
Want a Real World Serverless event in your city? Interested in sharing your stories and experience deploying serverless apps in production? Email [email protected] and let’s put something together.
Check out the event details and register through the Eventbrite links below.
Real World Serverless - San Francisco
Sept 11th, 2018, 6:00pm-9:00pm
In partnership with Serverless Meetup
Location: Heavybit - 325 9th St, San Francisco, CA 94103
View Event Details & Register Here »
Real World Serverless - London
Sept 18th, 2018, 6:00pm-9:00pm
Location: Cloudflare London - 25 Lavington St, Second floor SE1 0NZ London
View Event Details & Register Here »
Real World Serverless - Austin
October 2nd, 2018, 6:00pm-9:00pm
In partnership with ATX Serverless Meetup
Location: Downtown Austin
Website Security Myths
Some conversations are easy; some are difficult. Some are harmonious and some are laborious. But when it comes to website security, the conversation is confusing.
Every organisation agrees, in theory, that their websites need to be secure. But in practice, there is resistance to investing enough time and budget. Reasons for neglecting security include misconceptions surrounding Web Application security.
Below I’ve outlined some of the most common myths and misconceptions that can often put your website at serious security risks.
My website is not the target of an attack because it is small and I run a small business.
An average small business website is attacked 44 times per day. In addition, a low profile website is a nice playground for hackers to try out new tools and techniques. Hackers often use automated tools to find various vulnerable websites and don't discriminate when it comes to the size of the target. Any web application, even if it is not itself a target, may be of interest to attackers. Web applications with lax security are easy pickings for hackers and can be subject to a mass or targeted cyber attack.
The good news is that Continue reading
Welcome, WP Engine!
We’ve had the tremendous pleasure of working with WP Engine for nearly 5 years, starting when both companies employed less than 100 people in total. From the beginning, we noticed striking similarities between our two companies—both were founded in 2010, both are incredibly passionate about their customers’ success, and both strive to make their technology as simple and accessible as possible. Fast forward to 2018: with WP Engine already leveraging Cloudflare for DNS, thousands of mutual WP Engine and Cloudflare customers, and millions of WordPress websites already protected behind Cloudflare, it was a no-brainer to formally partner together.
Today, we are thrilled to announce WP Engine as a Cloudflare partner! The joint offering, Global Edge Security powered by Cloudflare, integrates WP Engine’s platform with Cloudflare’s managed web application firewall (WAF), advanced distributed denial of service mitigation (DDoS), SSL/TLS encryption, and CDN across a global edge network to deliver the world’s most secure and scalable digital experience on WordPress today.
We couldn’t be more excited about our opportunity to collaborate with WP Engine to deploy business-critical security and CDN edge services to Enterprises and SMBs globally.
Protection from Struts Remote Code Execution Vulnerability (S2-057)
On August 22 a new vulnerability in the Apache Struts framework was announced. It allows unauthenticated attackers to perform Remote Code Execution (RCE) on vulnerable hosts.
As security researcher Man Yue Mo explained, the vulnerability has similarities with previous Apache Struts vulnerabilities. The Cloudflare WAF already mitigated these so adjusting our rules to handle the new vulnerability was simple. Within hours of the disclosure we deployed a mitigation with no customer action required.
OGNL, again
Apache Struts RCE payloads often come in the form of Object-Graph Navigation Library (OGNL) expressions. OGNL is a language for interacting with the properties and functions of Java classes and Apache Struts supports it in many contexts.
For example, the snippet below uses OGNL to dynamically insert the value "5" into a webpage by calling a function.
<s:property value="%{getSum(2,3)}" />
OGNL expressions can also be used for more general code execution:
${
#_memberAccess["allowStaticMethodAccess"]=true,
@java.lang.Runtime@getRuntime().exec('calc')
}
Which means if you can find a way to make Apache Struts execute a user supplied OGNL expression, you've found an RCE vulnerability. Security researchers have found a significant number of vulnerabilities where this was the root cause.
What’s different this time?
The major difference between Continue reading
Porting Our Software to ARM64
As we enable more ARM64[1] machines in our network, I want to give some technical insight into the process we went through to reach software parity in our multi-architecture environment.
To give some idea of the scale of this task, it’s necessary to describe the software stack we run on our servers. The foundation is the Linux kernel. Then, we use the Debian distribution as our base operating system. Finally, we install hundreds of packages that we build ourselves. Some packages are based on open-source software, often tailored to better meet our needs. Other packages were written from scratch within Cloudflare.
Industry support for ARM64 is very active, so a lot of open-source software has already been ported. This includes the Linux kernel. Additionally, Debian made ARM64 a first-class release architecture starting with Stretch in 2017. This meant that upon obtaining our ARM64 hardware, a few engineers were able to bring Debian up quickly and smoothly. Our attention then turned to getting all our in-house packages to build and run for ARM64.
Our stack uses a diverse range of programming languages, including C, C++, Go, Lua, Python, and Rust. Different languages have different porting requirements, with some being easier Continue reading
Identifying and alerting on data loss using Cloudflare Workers
Photo by Markus Spiske / Unsplash
You hear about data breaches almost every day in the news these days. New regulations, such as GDPR, require companies to disclose data breaches within 72 hours of becoming aware. Becoming aware of and identifying data breaches as they happen, however, is not an easy task. It is often challenging for companies to become aware of their own data breaches and losses well-before they get picked up by the media.
One symptom of a data breach is data (such as passwords or PII) that should never leave internal systems making its way through an HTTP response into the public Internet. Since Cloudflare Workers sits between your infrastructure and the public for any endpoints exposed to the Internet, Workers can be used as a way of alerting you of canary data leaving.
In the following example, we will be inspecting the content of each response, checking to see if our canary data has leaked out, and if so, returning a static response and calling the PagerDuty API to notify of a potential breach.
Detecting Data Loss
In this example, we’ll be looking for a particular string in the body of the response. This string can Continue reading
Building a Diverse Business Development Team in EMEA
Cloudflare’s mission is to help build a better Internet, and we are also serious about building a diverse workforce where everyone can be themselves, regardless of their gender, sexual orientation, or cultural background.
The Recruiting team sat down with Anil Karavadra, EMEA Head of Business Development, to find out how he has built a diverse team since he joined.
the EMEA team pictured 1st August, some members attending Orientation
Recruiting Team: First question Anil, how did you end up at Cloudflare?
Anil Karavadra: Funny story about this--I was actually approached by my boss! He reached out to me and said “Hey would you be interested in joining Cloudflare?”, and initially I said no because I hadn’t heard of Cloudflare before! Then he suggested “Well how about we share some best practices?”, and I said okay, although I knew he was going to try and pitch Cloudflare to me. After I had a chat with him and he shared his vision to grow the EMEA BDR organisation. This sounded interesting to me so I then went through the interview process and got through. Now that I’m here I love it.
RT: How long have you been Continue reading
Using Edge-Side Includes with Workers for High Availability
Last week, we wrote about implementing ESI with Cloudflare Workers. This is a guest post by Lukas Reider on how to use ESI not only for better performance, but to optimize availability while migrating backends.
In this post, you will learn about how my client Titel Media was able to use Cloudflare Workers to implement simple edge side includes.
The idea is to partially replace the parts of the online magazine highsnobiety.com with a new, and much more refined frontend implementation. In this article, you will get to know the use case, and how I found a powerful application for Cloudflare Workers.
Backstory
My current project, highsnobiety.com is in the process of replacing Wordpress with a dedicated content pipeline and a custom frontend. It is a huge magazine, with tons of contents, hundreds of daily updates and an international team of more than 60 editors, researching and writing exciting stories.
The company behind it, Titel Media GmbH, a publishing house with offices in Berlin, and New York, surely has grown out of Wordpress for hosting their content.
The show must go on
One does not simply rewrite a sophisticated web publishing pipeline like WordPress. Nor does one, simply Continue reading
Using Workers To Make Static Sites Dynamic
The following is a guest post by Paddy Sherry, Lead Developer at Gambling.com Group. They build performance marketing websites and tools, using Cloudflare to serve to their global audience. Paddy is a Web Performance enthusiast with an interest in Serverless Computing.
Choosing technology that is used on a large network of sites is a key architectural decision that must be correct. We build static websites but needed to find a way to make them dynamic to do things like geo targeting, restrict access and A/B testing. This post shares our experiences on what we learned when using Workers to tackle these challenges.
Our Background
At Gambling.com Group, we use Cloudflare on all of our sites so our curiosity level in Workers was higher than most. We are big fans of static websites because nothing is faster than flat HTML. We had been searching for a technology like this for some time and applied to be part of the beta program, so were one of the first to gain access to the functionality.
The reason we were so keen to experiment with Workers is that for anyone running static sites, 99% of the time, the product requirements Continue reading
Introducing ebpf_exporter
This is an adapted transcript of a talk I gave at Promcon 2018. You can find slides with additional information on our Prometheus deployment and presenter notes here. There's also a video.
Tip: you can click on the image to see the original large version.
Here at Cloudflare we use Prometheus to collect operational metrics. We run it on hundreds of servers and ingest millions of metrics per second to get insight into our network and provide the best possible service to our customers.
Prometheus metric format is popular enough, it's now being standardized as OpenMetrics under Cloud Native Computing Foundation. It's exciting to see convergence in long fragmented metrics landscape.
In this blog post we'll talk about how we measure low level metrics and share a tool that can help you to get similar understanding of your systems.
There are two main exporters one can use to get some insight into a Linux system performance.
The first one is node_exporter that gives you information about basics like CPU usage breakdown by type, memory usage, disk IO stats, filesystem and network usage.
The second one is cAdvisor, that gives similar metrics, but drills down to a container level. Instead Continue reading
Refresh Stale DNS Records on 1.1.1.1
You can now refresh 1.1.1.1’s DNS cache for domain names by using the purge cache tool. This is useful for domain owners who have updated their DNS records and want to make sure it is reflected for people who are using 1.1.1.1 as their public DNS resolver.
When a client queries for a domain against 1.1.1.1, the resolver returns the IP address from its cache. The cache TTL for a DNS entry is 3 hours. If the host specifies a cache TTL that is shorter than 3 hours, the resolver respects that. This means, when a domain owner changes the DNS host from one to another, in the worst case, she will have to wait for at least 3 hours before the old IP address expires from 1.1.1.1’s cache. With the help of the purge cache tool, a domain owner can now easily refresh 1.1.1.1’s DNS cache and will not have to wait for the cached entry to expire.
To purge a DNS record, you enter the name of your domain, pick the DNS record type and hit the ‘Purge Cache’ button.
You can Continue reading
Internet Native Applications
I grew up with DOS and Windows 3.1. I remember applications being fast - instant feedback or close to it. Today, native applications like Outlook or Apple Mail still feel fast - click compose and the window is there instantly and it feels snappy. Internet applications do not.
My first Internet experience was paying $30 for a prepaid card with 10 hour access over a 14.4k modem. First, it was bulletin boards and later IRC and the WWW. From my small seaside town in Australia, the Internet was a window into the wider world, but it was slooooooow. In a way, it didn’t matter. The world of opportunities the Internet opened up, from information to music, to socializing and ecommerce, who cared if it was slow? The utility of the Internet and Internet applications meant I would use them regardless of the experience.
Performance improved from the 90s, but in 2008 when I switched from Outlook downloading my Yahoo! email over IMAP to Gmail in the browser, it wasn’t because it was faster - it wasn’t - it was because features like search, backed up mail, and unlimited storage were too good to resist. The cloud computing power Continue reading
Integrating redirection.io with Cloudflare Workers
The following is a guest post by Xavier Lacot, a developer at redirection.io and founder at JoliCode. He works primarily on Web and mobile projects as a consultant, trainer and technical expert.
What is redirection.io
Redirection.io is a Web traffic redirection manager. It provides a collection of tools for website administrators, SEO agencies, and developers, which help analyze HTTP errors, setup HTTP redirections, customize HTTP responses, and monitor the traffic efficiently.
The main part of a traditional redirection.io setup is the proxy, a software component which parses every request to check if a redirection or another response override is required. This "proxy" can be of several types - we provide libraries in several languages - but this setup can be simplified for Cloudflare clients by taking advantage of Cloudflare Workers.
Here come Cloudflare Workers
Earlier this year, Cloudflare unveiled its Workers product, a smart way of running code on the edge of Cloudflare locations. This computing feature is particularly interesting, as it allows performing several traffic operations without requiring any change on your own platform, code, or infrastructure: just enable Workers, write some code, and let Cloudflare handle the magic ✨
In practical terms, Workers Continue reading
How Cloudflare protects customers from cache poisoning
A few days ago, Cloudflare — along with the rest of the world — learned of a "practical" cache poisoning attack. In this post I’ll walk through the attack and explain how Cloudflare mitigated it for our customers. While any web cache is vulnerable to this attack, Cloudflare is uniquely able to take proactive steps to defend millions of customers.
In addition to the steps we’ve taken, we strongly recommend that customers update their origin web servers to mitigate vulnerabilities. Some popular vendors have applied patches that can be installed right away, including Drupal, Symfony, and Zend.
How a shared web cache works
Say a user requests a cacheable file, index.html. We first check if it’s in cache, and if it’s not not, we fetch it from the origin and store it. Subsequent users can request that file from our cache until it expires or gets evicted.
Although contents of a response can vary slightly between requests, customers may want to cache a single version of the file to improve performance:
(See this support page for more info about how to cache HTML with Cloudflare.)
How do we know it’s the same file? We create something Continue reading