Diving into Technical SEO using Cloudflare Workers

This is a guest post by Igor Krestov and Dan Taylor. Igor is a lead software developer at SALT.agency, and Dan a lead technical SEO consultant, and has also been credited with coining the term “edge SEO”. SALT.agency is a technical SEO agency with offices in London, Leeds, and Boston, offering bespoke consultancy to brands around the world. You can reach them both via Twitter.

With this post we illustrate the potential applications of Cloudflare Workers in relation to search engine optimization, which is more commonly referred to as ‘SEO’ using our research and testing over the past year making Sloth.

This post is aimed at readers who are both proficient in writing performant JavaScript, as well as complete newcomers, and less technical stakeholders, who haven’t really written many lines of code before.

Endless practical applications to overcome obstacles

Working with various clients and projects over the years we’ve continuously encountered the same problems and obstacles in getting their websites to a point of “technical SEO excellence”. A lot of these problems come from platform restriction at an enterprise level, legacy tech stacks, incorrect builds, and years of patching together various services and infrastructures.

As a team of Continue reading

Stopping Drupal’s SA-CORE-2019-003 Vulnerability

On the 20th February 2019, Drupal announced that they had discovered a severe vulnerability and that they would be releasing a patch for it the next day. Drupal is a Content Management System used by many of our customers, which made it important that our WAF protect against the vulnerability as quickly as possible.

As soon as Drupal released their patch, we analysed it to establish what kind of payloads could be used against it and created rules to mitigate these. By analysing the patch we were able to put together WAF rules to protect cloudflare customers running Drupal.

We identified the type of vulnerability we were dealing within 15 minutes. From here, we were able to deploy rules to block the exploit well before any real attacks were seen.

The exploit

As Drupal's release announcement explains, a site is affected if:

• It has the Drupal 8 RESTful API enabled                                      
• Or it uses one of the 8 modules found to be affected

From looking at the patch we very quickly realised the exploit would be based on deserialization. The option ['allowed_classes' Continue reading

Building fast interpreters in Rust

In the previous post we described the Firewall Rules architecture and how the different components are integrated together. We also mentioned that we created a configurable Rust library for writing and executing Wireshark®-like filters in different parts of our stack written in Go, Lua, C, C++ and JavaScript Workers.

With a mixed set of requirements of performance, memory safety, low memory use, and the capability to be part of other products that we’re working on like Spectrum, Rust stood out as the strongest option.

We have now open-sourced this library under our Github account: https://github.com/cloudflare/wirefilter. This post will dive into its design, explain why we didn’t use a parser generator and how our execution engine balances security, runtime performance and compilation cost for the generated filters.

Parsing Wireshark syntax

When building a custom Domain Specific Language (DSL), the first thing we need to be able to do is parse it. This should result in an intermediate representation (usually called an Abstract Syntax Tree) that can be inspected, traversed, analysed and, potentially, serialised.

There are different ways to perform such conversion, such as:

1. Manual char-by-char parsing using state machines, regular expression and/or native string APIs.
2. Parser combinators, which use Continue reading

How we made Firewall Rules

Recently we launched Firewall Rules, a new feature that allows you to construct expressions that perform complex matching against HTTP requests and then choose how that traffic is handled. As a Firewall feature you can, of course, block traffic. The expressions we support within Firewall Rules along with powerful control over the order in which they are applied allows complex new behaviour.

In this blog post I tell the story of Cloudflare’s Page Rules mechanism and how Firewall Rules came to be. Along the way I’ll look at the technical choices that led to us building the new matching engine in Rust.

The evolution of the Cloudflare Firewall

Cloudflare offers two types of firewall for web applications, a managed firewall in the form of a WAF where we write and maintain the rules for you, and a configurable firewall where you write and maintain rules. In this article, we will focus on the configurable firewall.

One of the earliest Cloudflare firewall features was the IP Access Rule. It dates backs to the earliest versions of the Cloudflare Firewall and simply allows you to block traffic from specific IP addresses:

if request IP equals 203.0.113.1 then block  Continue reading

Deploying Workers with GitHub Actions + Serverless

If you weren’t aware, Cloudflare Workers, our serverless programming platform, allows you to deploy code onto our 165 data centers around the world.

Want to automatically deploy Workers directly from a GitHub repository? Now you can with our official GitHub Action. This Action is an extension of our existing integration with the Serverless Framework. It runs in a containerized GitHub environment and automatically deploys your Worker to Cloudflare. We chose to utilize the Serverless Framework within our GitHub Action to raise awareness of their awesome work and to enable even more serverless applications to be built with Cloudflare Workers. This Action can be used to deploy individual Worker scripts as well; the Serverless Framework is being used in the background as the deployment mechanism.

Before going into the details, we’ll quickly go over what GitHub Actions are.

GitHub Actions

GitHub Actions allow you to trigger commands in reaction to GitHub events. These commands run in containers and can receive environment variables. Actions could trigger build, test, or deployment commands across a variety of providers. They can also be linked and run sequentially (i.e. ‘if the build passes, deploy the app’). Similar to many CI/CD tools, these commands run Continue reading

New Firewall Tab and Analytics

At Cloudflare, one of our top priorities is to make our products and services intuitive so that we can enable customers to accelerate and protect their Internet properties. We're excited to launch two improvements designed to make our Firewall easier to use and more accessible, and helping our customers better manage and visualize their threat-related data.

New Firewall Tabs for ease of access

We have re-organised our features into meaningful pages: Events, Firewall Rules, Managed Rules, Tools, and Settings. Our customers will see an Overview tab, which contains our new Firewall Analytics, detailed below.

All the features you know and love are still available, and can be found in one of the four new tabs. Here is a breakdown of their new locations.

If the new sub navigation has not appeared, you may need to re-login to the dashboard or clear your browser’s cookies.

New Firewall Analytics for analysing events and Continue reading

Digital Evidence Across Borders and Engagement with Non-U.S. Authorities

Since we first started reporting in 2013, our transparency report has focused on requests from U.S. law enforcement. Previous versions of the report noted that, as a U.S. company, we ask non-U.S. law enforcement agencies to obtain formal U.S. legal process before providing customer data.

As more countries pass laws that seek to extend beyond their national borders and as we expand into new markets, the question of how to handle requests from non-U.S. law enforcement has become more complicated. It seems timely to talk about our engagement with non-U.S. law enforcement and how our practice is changing. But first, some background on the changes that we’ve seen over the last year.

Law enforcement access to data across borders

The explosion of cloud services -- and the fact that data may be stored outside the countries of residence of those who generated it -- has been a challenge for governments conducting law enforcement investigations. A number of U.S. laws, like the Stored Communications Act or the Electronic Communications Privacy Act restrict companies from providing particular types of data, such as the content of communications, to any person or entity, including foreign law enforcement Continue reading

Out of the Clouds and into the weeds: Cloudflare’s approach to abuse in new products

In a blogpost yesterday, we addressed the principles we rely upon when faced with numerous and various requests to address the content of websites that use our services. We believe the building blocks that we provide for other people to share and access content online should be provided in a content-neutral way. We also believe that our users should understand the policies we have in place to address complaints and law enforcement requests, the type of requests we receive, and the way we respond to those requests. In this post, we do the dirty work of addressing how those principles are put into action, specifically with regard to Cloudflare’s expanding set of features and products.

Abuse reports and new products

Currently, we receive abuse reports and law enforcement requests on fewer than one percent of the more than thirteen million domains that use Cloudflare’s network. Although the reports we receive run the gamut -- from phishing, malware or other technical abuses of our network to complaints about content -- the overwhelming majority are allegations of copyright violations copyright or violations of other intellectual property rights. Most of the complaints that we receive do not identify concerns with particular Cloudflare services Continue reading

The Serverlist Newsletter 2nd Edition: Available Now

Check out our second edition of The Serverlist below. Get the latest scoop on the serverless space, get your hands dirty with new developer tutorials, engage in conversations with other serverless developers, and find upcoming meetups and conferences to attend.

Sign up below to have The Serverlist sent directly to your mailbox.

Unpacking the Stack and Addressing Complaints about Content

Although we are focused on protecting and optimizing the operation of the Internet, Cloudflare is sometimes the target of complaints or criticism about the content of a very small percentage of the more than thirteen million websites that use our service. Our termination of services to the Daily Stormer website a year and a half ago drew significant attention to our approach to these issues and prompted a lot of thinking on our part.  

At the time, Matthew wrote that calls for service providers to reject some online content should start with a consideration of how the Internet works and how the services at issue up and down the stack interact with that content. He tasked Cloudflare’s policy team with engaging broadly to try and find an answer. With some time having passed, we want to take stock of what we’ve learned and where we stand in addressing problematic content online.  

The aftermath of the Daily Stormer decision

The weeks immediately following the decision in August 2017 were filled with conversations. Matthew made sure the Cloudflare team accepted every single invitation to talk about these issues; we didn’t simply put out a press release or “no comment” anyone. Continue reading

Cloudflare Signs European Commission Declaration on Gender Balanced Company Culture

Last week Cloudflare attended a roundtable meeting in Brussels convened by the European Commissioner for Digital Economy and Society, Mariya Gabriel, with all signatories of the Tech Leaders’ Declaration on Gender Balanced Company Culture. Cloudflare joined this European Commission initiative late last year and, along with other companies, we are committed to taking a hands-on approach to close the digital gender divide in skills, inception of technologies, access and career opportunities.

In particular, we have all committed to implementing, promoting and spreading five specific actions to achieve equality of opportunities for women in our companies and in the digital sector at large:

1. Instil an inclusive, open, female-friendly company culture
2. Recruit and invest in diversity
3. Give women in tech their voice and visibility
4. Create the leaders of the future
5. Become an advocate for change

The project, spearheaded by the Digital Commissioner as part of a range of actions to promote gender balance in the digital industry, allows for the exchange of ideas and best practices among companies, with opportunities to chart progress and also to discuss the challenges we face. Many companies around the table shared their inspiring stories of steps taken at company level to encourage diversity, push back against Continue reading

Logpush: the Easy Way to Get Your Logs to Your Cloud Storage

Introducing Logpush

Today, we’re excited to announce a new way to get your logs: Logpush, a tool for uploading your logs to your cloud storage provider, such as Amazon S3 or Google Cloud Storage. It’s now available in Early Access for Enterprise domains.

We first explained Cloudflare’s logging functionality almost six years ago. Since then, the number of domains on our network has grown by ten times. We’ve continued giving our Enterprise customers the ability to download logs using a REST API, which has gotten a large number of functional and technical updates. We’ve also been paying attention to how our customers’ needs have evolved, especially as we protect and accelerate increasingly larger domains. This led to the development of Logpush.

The Value of Logs

Cloudflare works by being an intermediary between our customers’ websites, applications, and devices, and their end-users or potential attackers. As part of providing our service, we create a record of each request that goes through our network. These records (or request logs) have detailed information regarding the connecting client, our actions—including whether the request was served by the cache or blocked by our firewall—and the response from the origin web server. For Enterprise customers Continue reading

Cloudflare Transparency Update: Joining Cloudflare’s Flock of (Warrant) Canaries

Today, Cloudflare is releasing its transparency report for the second half of 2018. We have been publishing biannual Transparency Reports since 2013.

We believe an essential part of earning the trust of our customers is being transparent about our features and services, what we do – and do not do – with our users’ data, and generally how we conduct ourselves in our engagement with third parties such as law enforcement authorities.  We also think that an important part of being fully transparent is being rigorously consistent and anticipating future circumstances, so our users not only know how we have behaved in the past, but are able to anticipate with reasonable certainty how we will act in the future, even in difficult cases.

As part of that effort, we have set forth certain ‘warrant canaries’ – statements of things we have never done as a company. As described in greater detail below, the report published today adds three new ‘warrant canaries’, which is the first time we’ve added to that list since 2013. The transparency report is also distinguished because it adds new reporting on requests for user information from foreign law enforcement, and requests for user information that we Continue reading

Cloudflare’s RPKI Toolkit

A few months ago, we made a first then a second announcement about Cloudflare’s involvement in Resource Public Key Infrastructure (RPKI), and our desire to make BGP Internet routing more secure. Our mission is to build a safer Internet. We want to make it easier for network operators to deploy RPKI.

Today’s article is going to cover our experience and the tools we are using. As a brief reminder, RPKI is a framework that allows networks to deploy route filtering using cryptography-validated information. Picture TLS certificates for IP addresses and Autonomous System Numbers (ASNs)

What it means for you:

We validate our IP routes. This means, as a 1.1.1.1 DNS resolver user, you are less likely to be victim of cache poisoning. We signed our IP routes. This means a user browsing the websites on Cloudflare’s network are unlikely to experience route hijacks.

All our Points of Presence which have a router compatible with The Resource Public Key Infrastructure (RPKI) to Router Protocol (RTR protocol) are connected to our custom software called GoRTR and are now filtering invalid routes. The deployment amounts to around 70% of our network.

We received many questions regarding the amount of invalid Continue reading

My Cloudflaraversery: Things I’ve Learned Along the Way

A year ago, I joined the marketing team at Cloudflare.

I was first attracted to Cloudflare by its audacious mission: to help build a better Internet. As someone who’s spent most of my professional life working on programs — in marketing, policy, communications, and advocacy — that build trust and confidence in the Internet, Cloudflare’s mission resonated with me.

But it wasn’t just the mission — it was the product too. Over its eight years, the company has developed a growing platform of products and solutions that help millions of online properties — from nonprofits and hobbyists to small businesses and large enterprises — protect and accelerate anything connected to the Internet. For me, joining the Cloudflare team was an opportunity to help advance a mission and a product that is doing good in the world.

It’s been an exciting year and I want to take the opportunity to reflect on a few things I’ve learned along the way.

First, trust is everything

During my first few months at Cloudflare, I spoke with dozens and dozens of customers. I wanted to understand Cloudflare from their perspective. What challenges do they face? What progress are Continue reading

Cloudflare Registrar at three months

We announced Cloudflare Registrar in September. We launched the product by making it available in waves to our existing customers. During that time we gathered feedback and continued making improvements to the product while also adding more TLDs.

Staring today, we’re excited to make Cloudflare Registrar available to all of our customers. Cloudflare Registrar only charges you what we pay to the registry for your domain and any user can now rely on that at-cost pricing to manage their domain. As part of this announcement, we’d like to share some insights and data about domain registration that we learned during the early access period.

One-click DNS security makes a difference

When you launch your domain to the world, you rely on the Domain Name System (DNS) to direct your users to the address for your site. However, DNS cannot guarantee that your visitors reach your content because DNS, in its basic form, lacks authentication. If someone was able to poison the DNS responses for your site, they could hijack your visitors' DNS requests.

The Domain Name System Security Extensions (DNSSEC) can help prevent that type of attack by adding a chain of trust to DNS queries. When you enable DNSSEC Continue reading

Cloudflare Access now supports RDP

Last fall, the United States FBI warned organizations of an increase in attacks that exploit vulnerabilities in the Remote Desktop Protocol (RDP). Attackers stole sensitive data and compromised networks by taking advantage of desktops left unprotected. Like legacy VPNs, RDP configurations made work outside of the office corporate network possible by opening a hole in it.

Starting today, you can use Cloudflare Access to connect over RDP without sacrificing security or performance. Access enables your team to lock down remote desktops like you do physical ones while using your SSO credentials to authenticate each connection request.

Stronger passwords with identity provider integration

The FBI cited weak passwords and unrestricted port access to RDP ports as serious risks that led to the rise in RDP-based attacks. Cloudflare Access addresses those vulnerabilities by removing them altogether.

When users connect over RDP, they enter a local password to login to the target machine. However, organizations rarely manage these credentials. Instead, users set and save these passwords on an ad-hoc basis outside of the single sign-on credentials used for other services. That oversight leads to outdated, reused, and ultimately weak passwords.

Cloudflare Access integrates with the identity credentials your team already uses. Whether your Continue reading

Join us for 5 serverless events in SF Bay Area this week

Developer Week Bay Area is happening this week and Cloudflare engineers and developer relations team members are delivering several talks around the Bay. Join us in San Francisco and Oakland for the following talks. We’ll hope to see you soon.

WebAssembly on the Server, npm & genomics tools @ Cloudflare

We've partnered with the WebAssembly SF meetup group to curate three talks from Zack Bloom of Cloudflare, Laurie Voss of npm, and Robert Aboukhalil of Invitae.
Event Type: Meetup
Location: Cloudflare HQ, San Francisco, CA
Date: February 20, 2019

View Event Details & Register Here »

Serverless: An Inside Look

Cloudflare engineers are delivering three serverless talks in downtown Oakland: How Workers Work, Security: the Serverless Future, and Building a Serverless World (Map) with IoT and Workers.
Event Type: Meetup
Location: At My Sphere, Oakland, CA
Date: February 21, 2019

View Event Details & Register Here »

Developer Week Bay Area

Cloudflare will be at Developer Week Bay Area. Be sure to check out Single-Process Serverless, Building an Iot World (Map) with Serverless, and Make Your Existing Application Serverless talks.
Event Type: Conference
Location: Oakland Convention Center, Oakland, CA
Date: February 20-24, 2019

Register Here for a free Open conference ticket »

Stop the Bots: Practical Lessons in Machine Learning

Bot-powered credential stuffing is a scourge on the modern Internet. These attacks attempt to log into and take over a user’s account by assaulting password forms with a barrage of dictionary words and previously stolen account credentials, with the aim of performing fraudulent transactions, stealing sensitive data, and compromising personal information.

At Cloudflare we’ve built a suite of technologies to combat bots, many of them grounded in Machine Learning. ML is a hot topic these days, but the literature tends to focus on improving the core technology — and not how these learning machines are incorporated into real-world organizations.

Given how much experience we have with ML (which we employ for many security and performance products, in addition to bot management), we wanted to share some lessons learned with regard to how this technology manifests in actual products.

There tend to be three stages every company goes through in the life cycle of infusing machine learning into their DNA. They are:

• Business Intelligence
• Standalone Machine Learning
• Machine Learning Productization

These concepts are a little abstract — so let’s walk through how they might apply to a tangible field we all know and love: dental insurance.

Business Intelligence

Many companies already Continue reading

Announcing workers.dev

We are working really hard to allow you to deploy Workers without having a Cloudflare domain. You will soon be able to deploy your Cloudflare Workers to a subdomain-of-your-choice.workers.dev, which you can go claim now on workers.dev!

Why are we doing this?

You may have read the announcement blog post for Workers (or one of the many tutorials and guest posts), and thought “let me give this a try!”. If you’re an existing Cloudflare customer, you logged into the dashboard, and found a new icon called “Workers”, paid $5 and were on your way. If you’re not, you clicked “Sign Up”, but instead of getting to create and deploy a Worker, we asked you for your domain (if you didn’t have one, we had you register one), and move your nameservers.

Since launch, we have had tons of people who wanted to build a new serverless project from scratch or just try Workers out, but found it difficult to get started. We want to make it easier for anyone to get started building and deploying serverless applications.

How did we get here?

The way you get started on Workers today reflects our journey as a company. Continue reading