EU Copyright Vote: A Critical Juncture for the Open Internet
Back in June, we blogged about the draft EU copyright proposal which is currently making its way through the legislative process in Brussels. We outlined how under one of the more controversial provisions within the draft Directive, Article 13, certain Internet platforms could be held legally responsible for any copyright content that their users upload and would effectively have to turn to automated filtering solutions to remove infringing content at the point of user upload. Moreover, in order to avoid potential legal liability, it is widely expected that content sharing providers would err on the side of caution and remove excessive amounts of content, resulting in a form of online censorship.
Since that blogpost, the European Parliament Plenary narrowly voted on 5th July to reject the proposal tabled by the Legal Affairs (JURI) Committee and a mandate to negotiate, and now the proposed Directive will undergo a full discussion and rescheduled vote in the next Plenary meeting on 12th September. This was a fantastic outcome, thanks in large part to a groundswell of support from those who value the fundamental right of freedom of expression online. It has presented a window of opportunity to correct the deeply flawed approach to Continue reading
Fixing an old hack – why we are bumping the IPv6 MTU
Back in 2015 we deployed ECMP routing - Equal Cost Multi Path - within our datacenters. This technology allowed us to spread traffic heading to a single IP address across multiple physical servers.
You can think about it as a third layer of load balancing.
- First we split the traffic across multiple IP addresses with DNS.
- Then we split the traffic across multiple datacenters with Anycast.
- Finally, we split the traffic across multiple servers with ECMP.
When deploying ECMP we hit a problem with Path MTU discovery. The ICMP packets destined to our Anycast IP's were being dropped. You can read more about that (and the solution) in the 2015 blog post Path MTU Discovery in practice.
To solve the problem we created a small piece of software, called pmtud (https://github.com/cloudflare/pmtud). Since deploying pmtud, our ECMP setup has been working smoothly.
Hardcoding IPv6 MTU
During that initial ECMP rollout things were broken. To keep services running until pmtud was done, we deployed a quick hack. We reduced the MTU of IPv6 traffic to the minimal possible value: 1280 bytes.
This was done as a tag on a default route. This is Continue reading
Bob Ross, Lorem Ipsum, Heroku and Cloudflare Workers
It may not be immediately obvious how these things are related, but bear with me... It was 4pm Friday and one of the engineers on the Cloudflare Tools team came to me with an emergency. "Steve! The Bob Ross Ipsum generator is down!".
If you've not heard of Lorem Ipsum, it's an extract from a latin poem that designers use as placeholder text when designing the layout of a document. There are generators all over the web that will spit out as much text as you need.
Of course, the web being the web that we all love, there are also endless parodies of Lorem Ipsum. You can generate Hodor Ipsum, Cat Ipsum and Hipster Ipsum. I have a new, undisputed favourite: Bob Ross Ipsum.
Not growing up in the U.S., I hadn't come across the lovable, calm, serene and beautiful human that is Bob Ross. If you haven't spent 30 mins watching him paint a landscape, you should do that now. He built a following as host of the TV show “The Joy of Painting” which ran on the U.S. PBS channel from 1983-1994. He became famous for Continue reading
Introducing Real World Serverless – Practical advice on how to use Cloudflare Workers
We’re getting the best minds on serverless technology from Cloudflare together to lead a series of talks on practical use cases for Cloudflare Workers. Join any of these six global talks for stories of how companies and developers are using serverless in the real world.
San Francisco - London - Austin - Singapore - Sydney - Melbourne
Want a Real World Serverless event in your city? Interested in sharing your stories and experience deploying serverless apps in production? Email [email protected] and let’s put something together.
Check out the event details and register through the Eventbrite links below.
Real World Serverless - San Francisco
Sept 11th, 2018, 6:00pm-9:00pm
In partnership with Serverless Meetup
Location: Heavybit - 325 9th St, San Francisco, CA 94103
View Event Details & Register Here »
Real World Serverless - London
Sept 18th, 2018, 6:00pm-9:00pm
Location: Cloudflare London - 25 Lavington St, Second floor SE1 0NZ London
View Event Details & Register Here »
Real World Serverless - Austin
October 2nd, 2018, 6:00pm-9:00pm
In partnership with ATX Serverless Meetup
Location: Downtown Austin
Website Security Myths
Some conversations are easy; some are difficult. Some are harmonious and some are laborious. But when it comes to website security, the conversation is confusing.
Every organisation agrees, in theory, that their websites need to be secure. But in practice, there is resistance to investing enough time and budget. Reasons for neglecting security include misconceptions surrounding Web Application security.
Below I’ve outlined some of the most common myths and misconceptions that can often put your website at serious security risks.
My website is not the target of an attack because it is small and I run a small business.
An average small business website is attacked 44 times per day. In addition, a low profile website is a nice playground for hackers to try out new tools and techniques. Hackers often use automated tools to find various vulnerable websites and don't discriminate when it comes to the size of the target. Any web application, even if it is not itself a target, may be of interest to attackers. Web applications with lax security are easy pickings for hackers and can be subject to a mass or targeted cyber attack.
The good news is that Continue reading
Welcome, WP Engine!
We’ve had the tremendous pleasure of working with WP Engine for nearly 5 years, starting when both companies employed less than 100 people in total. From the beginning, we noticed striking similarities between our two companies—both were founded in 2010, both are incredibly passionate about their customers’ success, and both strive to make their technology as simple and accessible as possible. Fast forward to 2018: with WP Engine already leveraging Cloudflare for DNS, thousands of mutual WP Engine and Cloudflare customers, and millions of WordPress websites already protected behind Cloudflare, it was a no-brainer to formally partner together.
Today, we are thrilled to announce WP Engine as a Cloudflare partner! The joint offering, Global Edge Security powered by Cloudflare, integrates WP Engine’s platform with Cloudflare’s managed web application firewall (WAF), advanced distributed denial of service mitigation (DDoS), SSL/TLS encryption, and CDN across a global edge network to deliver the world’s most secure and scalable digital experience on WordPress today.
We couldn’t be more excited about our opportunity to collaborate with WP Engine to deploy business-critical security and CDN edge services to Enterprises and SMBs globally.
Protection from Struts Remote Code Execution Vulnerability (S2-057)
On August 22 a new vulnerability in the Apache Struts framework was announced. It allows unauthenticated attackers to perform Remote Code Execution (RCE) on vulnerable hosts.
As security researcher Man Yue Mo explained, the vulnerability has similarities with previous Apache Struts vulnerabilities. The Cloudflare WAF already mitigated these so adjusting our rules to handle the new vulnerability was simple. Within hours of the disclosure we deployed a mitigation with no customer action required.
OGNL, again
Apache Struts RCE payloads often come in the form of Object-Graph Navigation Library (OGNL) expressions. OGNL is a language for interacting with the properties and functions of Java classes and Apache Struts supports it in many contexts.
For example, the snippet below uses OGNL to dynamically insert the value "5" into a webpage by calling a function.
<s:property value="%{getSum(2,3)}" />
OGNL expressions can also be used for more general code execution:
${
#_memberAccess["allowStaticMethodAccess"]=true,
@java.lang.Runtime@getRuntime().exec('calc')
}
Which means if you can find a way to make Apache Struts execute a user supplied OGNL expression, you've found an RCE vulnerability. Security researchers have found a significant number of vulnerabilities where this was the root cause.
What’s different this time?
The major difference between Continue reading
Porting Our Software to ARM64
As we enable more ARM64[1] machines in our network, I want to give some technical insight into the process we went through to reach software parity in our multi-architecture environment.
To give some idea of the scale of this task, it’s necessary to describe the software stack we run on our servers. The foundation is the Linux kernel. Then, we use the Debian distribution as our base operating system. Finally, we install hundreds of packages that we build ourselves. Some packages are based on open-source software, often tailored to better meet our needs. Other packages were written from scratch within Cloudflare.
Industry support for ARM64 is very active, so a lot of open-source software has already been ported. This includes the Linux kernel. Additionally, Debian made ARM64 a first-class release architecture starting with Stretch in 2017. This meant that upon obtaining our ARM64 hardware, a few engineers were able to bring Debian up quickly and smoothly. Our attention then turned to getting all our in-house packages to build and run for ARM64.
Our stack uses a diverse range of programming languages, including C, C++, Go, Lua, Python, and Rust. Different languages have different porting requirements, with some being easier Continue reading
Identifying and alerting on data loss using Cloudflare Workers
Photo by Markus Spiske / Unsplash
You hear about data breaches almost every day in the news these days. New regulations, such as GDPR, require companies to disclose data breaches within 72 hours of becoming aware. Becoming aware of and identifying data breaches as they happen, however, is not an easy task. It is often challenging for companies to become aware of their own data breaches and losses well-before they get picked up by the media.
One symptom of a data breach is data (such as passwords or PII) that should never leave internal systems making its way through an HTTP response into the public Internet. Since Cloudflare Workers sits between your infrastructure and the public for any endpoints exposed to the Internet, Workers can be used as a way of alerting you of canary data leaving.
In the following example, we will be inspecting the content of each response, checking to see if our canary data has leaked out, and if so, returning a static response and calling the PagerDuty API to notify of a potential breach.
Detecting Data Loss
In this example, we’ll be looking for a particular string in the body of the response. This string can Continue reading
Building a Diverse Business Development Team in EMEA
Cloudflare’s mission is to help build a better Internet, and we are also serious about building a diverse workforce where everyone can be themselves, regardless of their gender, sexual orientation, or cultural background.
The Recruiting team sat down with Anil Karavadra, EMEA Head of Business Development, to find out how he has built a diverse team since he joined.
the EMEA team pictured 1st August, some members attending Orientation
Recruiting Team: First question Anil, how did you end up at Cloudflare?
Anil Karavadra: Funny story about this--I was actually approached by my boss! He reached out to me and said “Hey would you be interested in joining Cloudflare?”, and initially I said no because I hadn’t heard of Cloudflare before! Then he suggested “Well how about we share some best practices?”, and I said okay, although I knew he was going to try and pitch Cloudflare to me. After I had a chat with him and he shared his vision to grow the EMEA BDR organisation. This sounded interesting to me so I then went through the interview process and got through. Now that I’m here I love it.
RT: How long have you been Continue reading
Using Edge-Side Includes with Workers for High Availability
Last week, we wrote about implementing ESI with Cloudflare Workers. This is a guest post by Lukas Reider on how to use ESI not only for better performance, but to optimize availability while migrating backends.
In this post, you will learn about how my client Titel Media was able to use Cloudflare Workers to implement simple edge side includes.
The idea is to partially replace the parts of the online magazine highsnobiety.com with a new, and much more refined frontend implementation. In this article, you will get to know the use case, and how I found a powerful application for Cloudflare Workers.
Backstory
My current project, highsnobiety.com is in the process of replacing Wordpress with a dedicated content pipeline and a custom frontend. It is a huge magazine, with tons of contents, hundreds of daily updates and an international team of more than 60 editors, researching and writing exciting stories.
The company behind it, Titel Media GmbH, a publishing house with offices in Berlin, and New York, surely has grown out of Wordpress for hosting their content.
The show must go on
One does not simply rewrite a sophisticated web publishing pipeline like WordPress. Nor does one, simply Continue reading
Using Workers To Make Static Sites Dynamic
The following is a guest post by Paddy Sherry, Lead Developer at Gambling.com Group. They build performance marketing websites and tools, using Cloudflare to serve to their global audience. Paddy is a Web Performance enthusiast with an interest in Serverless Computing.
Choosing technology that is used on a large network of sites is a key architectural decision that must be correct. We build static websites but needed to find a way to make them dynamic to do things like geo targeting, restrict access and A/B testing. This post shares our experiences on what we learned when using Workers to tackle these challenges.
Our Background
At Gambling.com Group, we use Cloudflare on all of our sites so our curiosity level in Workers was higher than most. We are big fans of static websites because nothing is faster than flat HTML. We had been searching for a technology like this for some time and applied to be part of the beta program, so were one of the first to gain access to the functionality.
The reason we were so keen to experiment with Workers is that for anyone running static sites, 99% of the time, the product requirements Continue reading
Introducing ebpf_exporter
This is an adapted transcript of a talk I gave at Promcon 2018. You can find slides with additional information on our Prometheus deployment and presenter notes here. There's also a video.
Tip: you can click on the image to see the original large version.
Here at Cloudflare we use Prometheus to collect operational metrics. We run it on hundreds of servers and ingest millions of metrics per second to get insight into our network and provide the best possible service to our customers.
Prometheus metric format is popular enough, it's now being standardized as OpenMetrics under Cloud Native Computing Foundation. It's exciting to see convergence in long fragmented metrics landscape.
In this blog post we'll talk about how we measure low level metrics and share a tool that can help you to get similar understanding of your systems.
There are two main exporters one can use to get some insight into a Linux system performance.
The first one is node_exporter that gives you information about basics like CPU usage breakdown by type, memory usage, disk IO stats, filesystem and network usage.
The second one is cAdvisor, that gives similar metrics, but drills down to a container level. Instead Continue reading
Refresh Stale DNS Records on 1.1.1.1
You can now refresh 1.1.1.1’s DNS cache for domain names by using the purge cache tool. This is useful for domain owners who have updated their DNS records and want to make sure it is reflected for people who are using 1.1.1.1 as their public DNS resolver.
When a client queries for a domain against 1.1.1.1, the resolver returns the IP address from its cache. The cache TTL for a DNS entry is 3 hours. If the host specifies a cache TTL that is shorter than 3 hours, the resolver respects that. This means, when a domain owner changes the DNS host from one to another, in the worst case, she will have to wait for at least 3 hours before the old IP address expires from 1.1.1.1’s cache. With the help of the purge cache tool, a domain owner can now easily refresh 1.1.1.1’s DNS cache and will not have to wait for the cached entry to expire.
To purge a DNS record, you enter the name of your domain, pick the DNS record type and hit the ‘Purge Cache’ button.
You can Continue reading
Internet Native Applications
I grew up with DOS and Windows 3.1. I remember applications being fast - instant feedback or close to it. Today, native applications like Outlook or Apple Mail still feel fast - click compose and the window is there instantly and it feels snappy. Internet applications do not.
My first Internet experience was paying $30 for a prepaid card with 10 hour access over a 14.4k modem. First, it was bulletin boards and later IRC and the WWW. From my small seaside town in Australia, the Internet was a window into the wider world, but it was slooooooow. In a way, it didn’t matter. The world of opportunities the Internet opened up, from information to music, to socializing and ecommerce, who cared if it was slow? The utility of the Internet and Internet applications meant I would use them regardless of the experience.
Performance improved from the 90s, but in 2008 when I switched from Outlook downloading my Yahoo! email over IMAP to Gmail in the browser, it wasn’t because it was faster - it wasn’t - it was because features like search, backed up mail, and unlimited storage were too good to resist. The cloud computing power Continue reading
Integrating redirection.io with Cloudflare Workers
The following is a guest post by Xavier Lacot, a developer at redirection.io and founder at JoliCode. He works primarily on Web and mobile projects as a consultant, trainer and technical expert.
What is redirection.io
Redirection.io is a Web traffic redirection manager. It provides a collection of tools for website administrators, SEO agencies, and developers, which help analyze HTTP errors, setup HTTP redirections, customize HTTP responses, and monitor the traffic efficiently.
The main part of a traditional redirection.io setup is the proxy, a software component which parses every request to check if a redirection or another response override is required. This "proxy" can be of several types - we provide libraries in several languages - but this setup can be simplified for Cloudflare clients by taking advantage of Cloudflare Workers.
Here come Cloudflare Workers
Earlier this year, Cloudflare unveiled its Workers product, a smart way of running code on the edge of Cloudflare locations. This computing feature is particularly interesting, as it allows performing several traffic operations without requiring any change on your own platform, code, or infrastructure: just enable Workers, write some code, and let Cloudflare handle the magic ✨
In practical terms, Workers Continue reading
How Cloudflare protects customers from cache poisoning
A few days ago, Cloudflare — along with the rest of the world — learned of a "practical" cache poisoning attack. In this post I’ll walk through the attack and explain how Cloudflare mitigated it for our customers. While any web cache is vulnerable to this attack, Cloudflare is uniquely able to take proactive steps to defend millions of customers.
In addition to the steps we’ve taken, we strongly recommend that customers update their origin web servers to mitigate vulnerabilities. Some popular vendors have applied patches that can be installed right away, including Drupal, Symfony, and Zend.
How a shared web cache works
Say a user requests a cacheable file, index.html. We first check if it’s in cache, and if it’s not not, we fetch it from the origin and store it. Subsequent users can request that file from our cache until it expires or gets evicted.
Although contents of a response can vary slightly between requests, customers may want to cache a single version of the file to improve performance:
(See this support page for more info about how to cache HTML with Cloudflare.)
How do we know it’s the same file? We create something Continue reading
Edge-Side-Includes with Cloudflare Workers
At Cloudflare we’re accelerating web assets in a number of different ways. Part of this is caching, by storing the response given by the origin server directly within our 151+ global data centers. This will dramatically improve the delivery of the resources as the visitor will directly get them from the data center closest to them, instead of waiting for us to fetch the request from the origin web server.
The issue with dynamic (but not a lot) pages
The subject we’re gonna cover today is the concept of Edge-Side-Includes. And what’s better than a real use-case to introduce what it is used for? Let’s take a website where all pages are including advertisements at the head and bottom. Could we consider these pages static? We couldn’t as at least part of this page is dynamic. Could we consider caching it? That’s a no again as it would mean the first dynamic part rendered will be cached and served for the other visitors trying to get the page. It would be a catastrophe if the advertisements are user-specific.
So the issue here is that we can’t cache the page. That’s quite a shame as it means that we’ll fetch Continue reading
African traffic growth and predictions for the future
Looking back at our historical data, we realized how much the Internet and Cloudflare grew. With more than 150 datacenters, 10 percent of web-based applications, customers everywhere around the world, from the tiny islands in the Pacific to the big metropolises, we have an Internet landscape of almost every country and continent.
Cloudflare’s mission is to help build a better Internet. To do that we operate datacenters across the globe. By having datacenters close to end user we provide a fast, secure experience for everyone. Today I’d like to talk about our datacenters in Africa and our plans to serve a population of 1.2 billion people over 58 countries.
Internet penetration in developed countries skyrocketed since the 2000s, Internet usage is growing rapidly across Africa. We are seeing a 4% to 7% increase in traffic month on month. As of July 2018, we have 8 datacenters on the African Continent:
Enable Private DNS with 1.1.1.1 on Android 9 Pie
Recently, Google officially launched Android 9 Pie, which includes a slew of new features around digital well-being, security, and privacy. If you’ve poked around the network settings on your phone while on the beta or after updating, you may have noticed a new Private DNS Mode now supported by Android.
This new feature simplifies the process of configuring a custom secure DNS resolver on Android, meaning parties between your device and the websites you visit won’t be able to snoop on your DNS queries because they’ll be encrypted. The protocol behind this, TLS, is also responsible for the green lock icon you see in your address bar when visiting websites over HTTPS. The same technology is useful for encrypting DNS queries, ensuring they cannot be tampered with and are unintelligible to ISPs, mobile carriers, and any others in the network path between you and your DNS resolver. These new security protocols are called DNS over HTTPS, and DNS over TLS.
Configuring 1.1.1.1
Android Pie only supports DNS over TLS. To enable this on your device:
- Go to Settings → Network & internet → Advanced → Private DNS.
- Select the Private DNS provider hostname option.
- Enter
Continue reading