Introducing the Cloudflare Onion Service
- When: a cold San Francisco summer afternoon
- Where: Room 305, Cloudflare
- Who: 2 from Cloudflare + 9 from the Tor Project
What could go wrong?
Bit of Background
Two years ago this week Cloudflare introduced Opportunistic Encryption, a feature that provided additional security and performance benefits to websites that had not yet moved to HTTPS. Indeed, back in the old days some websites only used HTTP --- weird, right? “Opportunistic” here meant that the server advertised support for HTTP/2 via an HTTP Alternative Service header in the hopes that any browser that recognized the protocol could take advantage of those benefits in subsequent requests to that domain.
Around the same time, CEO Matthew Prince wrote about the importance and challenges of privacy on the Internet and tasked us to find a solution that provides convenience, security, and anonymity.
From neutralizing fingerprinting vectors and everyday browser trackers that Privacy Badger feeds on, all the way to mitigating correlation attacks that only big actors are capable of, guaranteeing privacy is a complicated challenge. Fortunately, the Tor Project addresses this extensive adversary model in Tor Browser.
However, the Internet is full of bad actors, and distinguishing Continue reading
RPKI and BGP: our path to securing Internet Routing
This article will talk about our approach to network security using technologies like RPKI to sign Internet routes and protect our users and customers from route hijacks and misconfigurations. We are proud to announce we have started deploying active filtering by using RPKI for routing decisions and signing our routes.
Back in April, articles including our blog post on BGP and route-leaks were reported in the news, highlighting how IP addresses can be redirected maliciously or by mistake. While enormous, the underlying routing infrastructure, the bedrock of the Internet, has remained mostly unsecured.
At Cloudflare, we decided to secure our part of the Internet by protecting our customers and everyone using our services including our recursive resolver 1.1.1.1.
From BGP to RPKI, how do we Internet ?
A prefix is a range of IP addresses, for instance, 10.0.0.0/24, whose first address is 10.0.0.0 and the last one is 10.0.0.255. A computer or a server usually have one. A router creates a list of reachable prefixes called a routing table and uses this routing table to transport packets from a source to a destination.
On the Internet, network Continue reading
RPKI – The required cryptographic upgrade to BGP routing
We have talked about the BGP Internet routing protocol before. We have talked about how we build a more resilient network and how we can see outages at a country-level via BGP. We have even talked about the network community that is vital to the operation of the global Internet.
Today we need to talk about why existing operational practices for BGP routing and filtering have to significantly improve in order to finally stop route leaks and hijacks; which are sadly pervasive in today’s Internet routing world. In fact, the subtle art of running a BGP network and the various tools (both online and within your a networks subsystems) that are vital to making the Internet routing world a safe and reliable place to operate need to improve.
Internet routing and BGP and security along with its operational expertise must improve globally.
Nothing specific triggered today’s writing except the fact that Cloudflare has decided that it's high-time we took a leadership role to finally secure BGP routing. We believe that each and every network needs to change its mindset towards BGP security both on a day-by-day and a long-term basis.
It's time to stop Continue reading
Expanding DNSSEC Adoption
Cloudflare first started talking about DNSSEC in 2014 and at the time, Nick Sullivan wrote: “DNSSEC is a valuable tool for improving the trust and integrity of DNS, the backbone of the modern Internet.”
Over the past four years, it has become an even more critical part of securing the internet. While HTTPS has gone a long way in preventing user sessions from being hijacked and maliciously (or innocuously) redirected, not all internet traffic is HTTPS. A safer Internet should secure every possible layer between a user and the origin they are intending to visit.
As a quick refresher, DNSSEC allows a user, application, or recursive resolver to trust that the answer to their DNS query is what the domain owner intends it to be. Put another way: DNSSEC proves authenticity and integrity (though not confidentiality) of a response from the authoritative nameserver. Doing so makes it much harder for a bad actor to inject malicious DNS records into the resolution path through BGP Leaks and cache poisoning. Trust in DNS matters even more when a domain is publishing record types that are used to declare trust for other systems. As a specific example, DNSSEC is helpful for preventing Continue reading
End-to-End Integrity with IPFS
This post describes how to use Cloudflare's IPFS gateway to set up a website which is end-to-end secure, while maintaining the performance and reliability benefits of being served from Cloudflare’s edge network. If you'd rather read an introduction to the concepts behind IPFS first, you can find that in our announcement. Alternatively, you could skip straight to the developer docs to learn how to set up your own website.
By 'end-to-end security', I mean that neither the site owner nor users have to trust Cloudflare to serve the correct documents, like they do now. This is similar to how using HTTPS means you don't have to trust your ISP to not modify or inspect traffic.
CNAME Setup with Universal SSL
The first step is to choose a domain name for your website. Websites should be given their own domain name, rather than served directly from the gateway by root hash, so that they are considered a distinct origin by the browser. This is primarily to prevent cache poisoning, but there are several functional advantages as well. It gives websites their own instance of localStorage and their own cookie jar which are sandboxed from inspection and manipulation by malicious third-party documents. Continue reading
Cloudflare goes InterPlanetary – Introducing Cloudflare’s IPFS Gateway
Today we’re excited to introduce Cloudflare’s IPFS Gateway, an easy way to access content from the InterPlanetary File System (IPFS) that doesn’t require installing and running any special software on your computer. We hope that our gateway, hosted at cloudflare-ipfs.com, will serve as the platform for many new highly-reliable and security-enhanced web applications. The IPFS Gateway is the first product to be released as part of our Distributed Web Gateway project, which will eventually encompass all of our efforts to support new distributed web technologies.
This post will provide a brief introduction to IPFS. We’ve also written an accompanying blog post describing what we’ve built on top of our gateway, as well as documentation on how to serve your own content through our gateway with your own custom hostname.
Quick Primer on IPFS
Usually, when you access a website from your browser, your browser tracks down the origin server (or servers) that are the ultimate, centralized repository for the website’s content. It then sends a request from your computer to that origin server, wherever it is in the world, and that server sends the content back to your computer. This system has served the Internet well for decades, Continue reading
Welcome to Crypto Week
The Internet is an amazing invention. We marvel at how it connects people, connects ideas, and makes the world smaller. But the Internet isn’t perfect. It was put together piecemeal through publicly funded research, private investment, and organic growth that has left us with an imperfect tapestry. It’s also evolving. People are constantly developing creative applications and finding new uses for existing Internet technology. Issues like privacy and security that were afterthoughts in the early days of the Internet are now supremely important. People are being tracked and monetized, websites and web services are being attacked in interesting new ways, and the fundamental system of trust the Internet is built on is showing signs of age. The Internet needs an upgrade, and one of the tools that can make things better, is cryptography.
Every day this week, Cloudflare will be announcing support for a new technology that uses cryptography to make the Internet better. Everything we are announcing this week is free to use and provides a meaningful step towards supporting a new capability or structural reinforcement. So why are we doing this? Because it’s good for the users and good for the Internet. Welcome to Crypto Week!
A more Continue reading
JAMstack podcast episode: Listen to Cloudflare’s Kenton Varda speak about originless code
JAMstack Radio is a show all about the JAMstack, a new way to build fast & secure apps or websites. In the most recent episode, the host, Brian Douglas, met with Kenton Varda, tech lead for Cloudflare Workers and author of Sandstorm.io to discuss some of the infinite uses for running code at the edge.
Listen to what Kenton had to say about serverless technology in this twenty two minute podcast here:
Here's the transcript of the podcast as well:
Brian Douglas: Welcome to another installment of JAMstack Radio. In the room I've got Kenton Varda from Cloudflare.
Kenton Varda: Thanks for having me.
Brian: Thanks for coming all the way across San Francisco to chat with me in person. I'm curious who Kenton is, but I'm also curious what Cloudflare is. Can you answer both questions? Let's start with, "Who is Kenton?"
Kenton: I'm an engineer. I'm the architect of Cloudflare Workers. In a past life I worked for Google for several years. I was once known as the "protocol buffers guy," I was the one who open sourced that. And I founded a company called Sandstorm that was later acquired by Cloudflare.
Brian: I'm Continue reading
Cache API for Cloudflare Workers is now in Beta!
In October of last year we announced the launch of Cloudflare Workers. Workers allows you to run JavaScript from 150+ of Cloudflare’s data centers. This means that from the moment a request hits the Cloudflare network, you have full control over its destiny. One of the benefits of using Workers in combination with Cloudflare’s cache is that Workers allow you to have programmatic, and thus very granular control over the Cloudflare cache.
You can choose what to cache, how long to cache it for, the source it should be cached from, and you can even modify the cached result after it is retrieved from the cache.
We have seen many of our existing customers use Workers to enhance their usage of the Cloudflare cache, and we have seen many new customers join Cloudflare to take advantage of these unique benefits.
(Re-)Introducing the Cache API
You can always have more control, so today we are announcing support for the Cache API! As some of you may know, Cloudflare Workers are built against the existing Service Worker APIs. One of the reasons we originally chose to model Cloudflare Workers after Service Workers was due to the existing familiarity and audience of Service Continue reading
Deploy Workers using Terraform
Today we're excited to announce that Cloudflare Workers are now supported in the Cloudflare Terraform Provider.
Terraform is a fantastic tool for configuring your infrastructure. Traditionally if you wanted to spin up, tear down or update some of your infrastructure you would have to click around on a website or make some API calls, which is prone to human error. With Terraform, you define your infrastructure in simple, declarative configuration files and let Terraform figure out how to make the API calls for you. This also lets you treat your infrastructure like your code. You can check your Terraform configuration files into version control and integrate them into your normal software development workflow.
Terraform integrates with many infrastructure providers including Cloudflare. If you'd like to read more about setting up Terraform with Cloudflare, check out Getting started with Terraform and Cloudflare. In this post, I'm going to focus specifically on how to integrate Cloudflare Workers with Terraform.
In this example we're going to create partyparrot.business, and we're going to serve the whole site out of a worker without any origin server. We're starting from scratch here, but if you're already using Cloudflare workers and want to migrate to Continue reading
Norfolk and Richmond, Virginia: Cloudflare’s 152nd and 153rd cities
Virginia has a very important place in Internet history, as well as the history of Cloudflare’s network.
Northern Virginia, in the area around Ashburn VA, has for a long time been core to Internet infrastructure. In the early 1990’s, MAE-East (Metropolitan-Area-Exchange East) , an Internet Exchange Point (IXP) was established. MAE-East and West were some of the earliest IXPs. Internet Exchange Points are crucial interconnection points for ISPs and other Internet Networks to interconnect and exchange traffic. Eco-systems have grown around these through new data center offerings and new Internet platforms. Like many pieces of the Internet, MAE-East had a humble beginning, though not many humble beginnings grew to handle around 50% of Internet traffic exchange.
Cloudflare’s second Data Center, and one that still plays a critical component in our Global Network was Ashburn, Virginia. Similarly across many organizations, the Northern Virginia area has become a Data Center mecca. Many of the largest Clouds have a substantial amount of their footprint in Northern Virginia. Although MAE-East no longer exists, other Internet Exchange Points have come and grown in its place.
Cloudflare’s network has grown beyond what was traditional Interconnection points, like Ashburn/Northern VA, to a new Edge of the Continue reading
Build and Deploy Functions to Cloudflare’s 152+ Data Centers with Serverless
We’re very excited to announce that Cloudflare Workers are now integrated into the Serverless framework as a serverless cloud provider! Serverless’ open source framework has become a must-have for many developers, and we want to make it as simple as possible to harness the power of Cloudflare’s distributed computing network.
Workers have become essential to the way people build applications on the web. The expressiveness of modern JavaScript combined with sub-30 second deploys to Cloudflare’s network of 152+ datacenters means that you can truly build your application into our global compute network. Up until this point, deploying Workers required doing all of your editing through our browser-based IDE or developing one’s own custom tooling on top of our API. But many developers have their own environments and are already familiar with the Serverless framework, so it seemed natural that we build first-class support for Workers into Serverless!
You can now define the entire structure and routing behavior of your Workers scripts in code and deploy them with ease using serverless deploy from your own development environment. Store configuration files in version control alongside your application code. And feel more confident testing your application with serverless invoke, a new way Continue reading
EU Copyright Vote: A Critical Juncture for the Open Internet
Back in June, we blogged about the draft EU copyright proposal which is currently making its way through the legislative process in Brussels. We outlined how under one of the more controversial provisions within the draft Directive, Article 13, certain Internet platforms could be held legally responsible for any copyright content that their users upload and would effectively have to turn to automated filtering solutions to remove infringing content at the point of user upload. Moreover, in order to avoid potential legal liability, it is widely expected that content sharing providers would err on the side of caution and remove excessive amounts of content, resulting in a form of online censorship.
Since that blogpost, the European Parliament Plenary narrowly voted on 5th July to reject the proposal tabled by the Legal Affairs (JURI) Committee and a mandate to negotiate, and now the proposed Directive will undergo a full discussion and rescheduled vote in the next Plenary meeting on 12th September. This was a fantastic outcome, thanks in large part to a groundswell of support from those who value the fundamental right of freedom of expression online. It has presented a window of opportunity to correct the deeply flawed approach to Continue reading
Fixing an old hack – why we are bumping the IPv6 MTU
Back in 2015 we deployed ECMP routing - Equal Cost Multi Path - within our datacenters. This technology allowed us to spread traffic heading to a single IP address across multiple physical servers.
You can think about it as a third layer of load balancing.
- First we split the traffic across multiple IP addresses with DNS.
- Then we split the traffic across multiple datacenters with Anycast.
- Finally, we split the traffic across multiple servers with ECMP.
When deploying ECMP we hit a problem with Path MTU discovery. The ICMP packets destined to our Anycast IP's were being dropped. You can read more about that (and the solution) in the 2015 blog post Path MTU Discovery in practice.
To solve the problem we created a small piece of software, called pmtud (https://github.com/cloudflare/pmtud). Since deploying pmtud, our ECMP setup has been working smoothly.
Hardcoding IPv6 MTU
During that initial ECMP rollout things were broken. To keep services running until pmtud was done, we deployed a quick hack. We reduced the MTU of IPv6 traffic to the minimal possible value: 1280 bytes.
This was done as a tag on a default route. This is Continue reading
Bob Ross, Lorem Ipsum, Heroku and Cloudflare Workers
It may not be immediately obvious how these things are related, but bear with me... It was 4pm Friday and one of the engineers on the Cloudflare Tools team came to me with an emergency. "Steve! The Bob Ross Ipsum generator is down!".
If you've not heard of Lorem Ipsum, it's an extract from a latin poem that designers use as placeholder text when designing the layout of a document. There are generators all over the web that will spit out as much text as you need.
Of course, the web being the web that we all love, there are also endless parodies of Lorem Ipsum. You can generate Hodor Ipsum, Cat Ipsum and Hipster Ipsum. I have a new, undisputed favourite: Bob Ross Ipsum.
Not growing up in the U.S., I hadn't come across the lovable, calm, serene and beautiful human that is Bob Ross. If you haven't spent 30 mins watching him paint a landscape, you should do that now. He built a following as host of the TV show “The Joy of Painting” which ran on the U.S. PBS channel from 1983-1994. He became famous for Continue reading
Introducing Real World Serverless – Practical advice on how to use Cloudflare Workers
We’re getting the best minds on serverless technology from Cloudflare together to lead a series of talks on practical use cases for Cloudflare Workers. Join any of these six global talks for stories of how companies and developers are using serverless in the real world.
San Francisco - London - Austin - Singapore - Sydney - Melbourne
Want a Real World Serverless event in your city? Interested in sharing your stories and experience deploying serverless apps in production? Email [email protected] and let’s put something together.
Check out the event details and register through the Eventbrite links below.
Real World Serverless - San Francisco
Sept 11th, 2018, 6:00pm-9:00pm
In partnership with Serverless Meetup
Location: Heavybit - 325 9th St, San Francisco, CA 94103
View Event Details & Register Here »
Real World Serverless - London
Sept 18th, 2018, 6:00pm-9:00pm
Location: Cloudflare London - 25 Lavington St, Second floor SE1 0NZ London
View Event Details & Register Here »
Real World Serverless - Austin
October 2nd, 2018, 6:00pm-9:00pm
In partnership with ATX Serverless Meetup
Location: Downtown Austin
Website Security Myths
Some conversations are easy; some are difficult. Some are harmonious and some are laborious. But when it comes to website security, the conversation is confusing.
Every organisation agrees, in theory, that their websites need to be secure. But in practice, there is resistance to investing enough time and budget. Reasons for neglecting security include misconceptions surrounding Web Application security.
Below I’ve outlined some of the most common myths and misconceptions that can often put your website at serious security risks.
My website is not the target of an attack because it is small and I run a small business.
An average small business website is attacked 44 times per day. In addition, a low profile website is a nice playground for hackers to try out new tools and techniques. Hackers often use automated tools to find various vulnerable websites and don't discriminate when it comes to the size of the target. Any web application, even if it is not itself a target, may be of interest to attackers. Web applications with lax security are easy pickings for hackers and can be subject to a mass or targeted cyber attack.
The good news is that Continue reading
Welcome, WP Engine!
We’ve had the tremendous pleasure of working with WP Engine for nearly 5 years, starting when both companies employed less than 100 people in total. From the beginning, we noticed striking similarities between our two companies—both were founded in 2010, both are incredibly passionate about their customers’ success, and both strive to make their technology as simple and accessible as possible. Fast forward to 2018: with WP Engine already leveraging Cloudflare for DNS, thousands of mutual WP Engine and Cloudflare customers, and millions of WordPress websites already protected behind Cloudflare, it was a no-brainer to formally partner together.
Today, we are thrilled to announce WP Engine as a Cloudflare partner! The joint offering, Global Edge Security powered by Cloudflare, integrates WP Engine’s platform with Cloudflare’s managed web application firewall (WAF), advanced distributed denial of service mitigation (DDoS), SSL/TLS encryption, and CDN across a global edge network to deliver the world’s most secure and scalable digital experience on WordPress today.
We couldn’t be more excited about our opportunity to collaborate with WP Engine to deploy business-critical security and CDN edge services to Enterprises and SMBs globally.
Protection from Struts Remote Code Execution Vulnerability (S2-057)
On August 22 a new vulnerability in the Apache Struts framework was announced. It allows unauthenticated attackers to perform Remote Code Execution (RCE) on vulnerable hosts.
As security researcher Man Yue Mo explained, the vulnerability has similarities with previous Apache Struts vulnerabilities. The Cloudflare WAF already mitigated these so adjusting our rules to handle the new vulnerability was simple. Within hours of the disclosure we deployed a mitigation with no customer action required.
OGNL, again
Apache Struts RCE payloads often come in the form of Object-Graph Navigation Library (OGNL) expressions. OGNL is a language for interacting with the properties and functions of Java classes and Apache Struts supports it in many contexts.
For example, the snippet below uses OGNL to dynamically insert the value "5" into a webpage by calling a function.
<s:property value="%{getSum(2,3)}" />
OGNL expressions can also be used for more general code execution:
${
#_memberAccess["allowStaticMethodAccess"]=true,
@java.lang.Runtime@getRuntime().exec('calc')
}
Which means if you can find a way to make Apache Struts execute a user supplied OGNL expression, you've found an RCE vulnerability. Security researchers have found a significant number of vulnerabilities where this was the root cause.
What’s different this time?
The major difference between Continue reading
Porting Our Software to ARM64
As we enable more ARM64[1] machines in our network, I want to give some technical insight into the process we went through to reach software parity in our multi-architecture environment.
To give some idea of the scale of this task, it’s necessary to describe the software stack we run on our servers. The foundation is the Linux kernel. Then, we use the Debian distribution as our base operating system. Finally, we install hundreds of packages that we build ourselves. Some packages are based on open-source software, often tailored to better meet our needs. Other packages were written from scratch within Cloudflare.
Industry support for ARM64 is very active, so a lot of open-source software has already been ported. This includes the Linux kernel. Additionally, Debian made ARM64 a first-class release architecture starting with Stretch in 2017. This meant that upon obtaining our ARM64 hardware, a few engineers were able to bring Debian up quickly and smoothly. Our attention then turned to getting all our in-house packages to build and run for ARM64.
Our stack uses a diverse range of programming languages, including C, C++, Go, Lua, Python, and Rust. Different languages have different porting requirements, with some being easier Continue reading