Cloudflare’s team of security analysts monitor for upcoming threats and vulnerabilities and where possible put protection in place for upcoming threats before they compromise our customers. This post examines how we protected people against a new major vulnerability in the Drupal CMS, nicknamed Drupalgeddon 2.

Two weeks after adding protection with WAF rule ID D0003 which mitigates the critical remote code execution Drupal exploit (SA-CORE-2018-002/CVE-2018-7600), we have seen significant spikes of attack attempts. Since the 13th of April the Drupal security team has been aware of automated attack attempts and it significantly increased the security risk score of the vulnerability. It makes sense to go back and analyse what happened in the last seven days in Cloudflare’s WAF environment.

What is Drupalgeddon 2

The vulnerability potentially allows attackers to exploit multiple attack vectors on a Drupal site, which could make a site completely compromised.

Drupal introduced renderable arrays, which are a key-value structure, with keys starting with a ‘#’ symbol, that allows you to alter data during form rendering. These arrays however, did not have enough input validation. This means that an attacker could inject a custom renderable array on one of these keys in the form structure.

Continue reading

This post is inspired by a very good blog post from one of my colleague in the US, which I really appreciated as I was a newcomer to the company. It was great to see what it is like working for Cloudflare after one year and to learn from the lessons she had learnt.

I'll try to do the same in three parts. Beginning with how my on-boarding went, my first customer experiences and finally what is my day-to-day life at Cloudflare. These writings only reflect my personal feelings and thoughts. The experience is different for each and every newcomer to Cloudflare.

Chapter 1 - On-boarding, being impressed and filling the (big) knowledge gaps.

Before I joined Cloudflare, I was working as a Security Consultant in Paris, France. I never had the opportunity to move abroad to speak English (me.englishLevel = 0), I never had any reason to live outside of France and was at the same time looking for another Job. Perfect then!

When I saw the job posting, I immediately applied as I knew the company well, the mindset and the products Cloudflare provided. It took me 6 months to get the offer probably because Continue reading

In previous blog post we discussed how we use the TPROXY iptables module to power Cloudflare Spectrum. With TPROXY we solved a major technical issue on the server side, and we thought we might find another use for it on the client side of our product.

This is Addressograph. Source Wikipedia

When building an application level proxy, the first consideration is always about retaining real client source IP addresses. Some protocols make it easy, e.g. HTTP has a defined X-Forwarded-For header^[1], but there isn't a similar thing for generic TCP tunnels.

Others have faced this problem before us, and have devised three general solutions:

(1) Ignore the client IP

For certain applications it may be okay to ignore the real client IP address. For example, sometimes the client needs to identify itself with a username and password anyway, so the source IP doesn't really matter. In general, it's not a good practice because...

(2) Nonstandard TCP header

A second method was developed by Akamai: the client IP is saved inside a custom option in the TCP header in the SYN packet. Early implementations of this method weren't conforming to any standards, e.g. using option field 28 Continue reading

As engineers at Cloudflare quickly adapt our software stack to run on ARM, a few parts of our software stack have not been performing as well on ARM processors as they currently do on our Xeon® Silver 4116 CPUs. For the most part this is a matter of Intel specific optimizations some of which utilize SIMD or other special instructions.

One such example is the venerable jpegtran, one of the workhorses behind our Polish image optimization service.

A while ago I optimized our version of jpegtran for Intel processors. So when I ran a comparison on my test image, I was expecting that the Xeon would outperform ARM:

vlad@xeon:~$ time  ./jpegtran -outfile /dev/null -progressive -optimise -copy none test.jpg

real    0m2.305s
user    0m2.059s
sys     0m0.252s

vlad@arm:~$ time ./jpegtran -outfile /dev/null -progressive -optimise -copy none test.jpg

real    0m8.654s
user    0m8.433s
sys     0m0.225s

Ideally we want to have the ARM performing at or above 50% of the Xeon performance per core. This would make sure we have no performance regressions, and net performance gain, since the ARM CPUs have double the core count as our current 2 socket setup.

In this case, however, I Continue reading

Today we are introducing Spectrum, which brings Cloudflare’s security and acceleration to the whole spectrum of TCP ports and protocols for our Enterprise customers. It’s DDoS protection for any box, container or VM that connects to the internet; whether it runs email, file transfer or a custom protocol, it can now get the full benefits of Cloudflare. If you want to skip ahead and see it in action, you can scroll to the video demo at the bottom.

DDoS Protection

The core functionality of Spectrum is its ability to block large DDoS attacks. Spectrum benefits from Cloudflare’s existing DDoS mitigation (which this week blocked a 900 Gbps flood). Spectrum’s DDoS protection has already been battle tested. Just soon as we opened up Spectrum for beta, Spectrum received its first SYN flood.

One of Spectrum's earliest deployments was in front of Hypixel’s infrastructure. Hypixel runs the largest minecraft server, and because gamers can be - uh, passionate - they were one of the earliest targets of the terabit-per-second Mirai botnet. “Hypixel was one of the first subjects of the Mirai botnet DDoS attacks and frequently receives large attacks. Before Spectrum, we had to rely on unstable services & techniques Continue reading

Today we are introducing Spectrum: a new Cloudflare feature that brings DDoS protection, load balancing, and content acceleration to any TCP-based protocol.

CC BY-SA 2.0 image by Staffan Vilcans

Soon after we started building Spectrum, we hit a major technical obstacle: Spectrum requires us to accept connections on any valid TCP port, from 1 to 65535. On our Linux edge servers it's impossible to "accept inbound connections on any port number". This is not a Linux-specific limitation: it's a characteristic of the BSD sockets API, the basis for network applications on most operating systems. Under the hood there are two overlapping problems that we needed to solve in order to deliver Spectrum:

• how to accept TCP connections on all port numbers from 1 to 65535
• how to configure a single Linux server to accept connections on a very large number of IP addresses (we have many thousands of IP addresses in our anycast ranges)

Assigning millions of IPs to a server

Cloudflare’s edge servers have an almost identical configuration. In our early days, we used to assign specific /32 (and /128) IP addresses to the loopback network interface^[1]. This worked well when we had dozens of IP Continue reading

This is a Korean translation of a prior post by Marek Majkowski.

얼마전 우리는 Spectrum을 발표하였습니다: 어떤 TCP 기반의 프로토콜이라도 DDoS 방어, 로드밸런싱 그리고 컨텐츠 가속을 할 수 있는 새로운 Cloudflare의 기능입니다.

CC BY-SA 2.0 image by Staffan Vilcans

Spectrum을 만들기 시작하고 얼마 되지 않아서 중요한 기술적 난관에 부딛히게 되었습니다: Spectrum은 1부터 65535 사이의 어떤 유효한 TCP 포트라도 접속을 허용해야 합니다. 우리의 리눅스 엣지 서버에서는 "임의의 포트 번호에 인바운드 연결을 허용"은 불가능합니다. 이것은 리눅스만의 제한은 아닙니다: 이것은 대부분 운영 체제의 네트워크 어플리케이션의 기반인 BSD 소켓 API의 특성입니다. 내부적으로 Spectrum을 완성하기 위해서 풀어야 하는 서로 겹치는 문제가 둘 있었습니다:

• 1에서 65535 사이의 모든 포트 번호에 TCP 연결을 어떻게 받아들일 것인가
• 매우 많은 수의 IP 주소로 오는 연결을 받아들이도록 단일 리눅스 서버를 어떻게 설정할 것인가 (우리는 애니캐스트 대역에 수많은 IP주소를 갖고 있습니다)

서버에 수백만의 IP를 할당

Cloudflare의 엣지 서버는 거의 동일한 구성을 갖고 있습니다. 초창기에는 루프백 네트워크 인터페이스에 특정한 /32 (그리고 /128) IP 주소를 할당하였습니다^[1]. 이것은 수십개의 IP주소만 갖고 있었을 때에는 잘 동작 하였지만 더 성장함에 따라 확대 적용하는 것에는 실패하였습니다.

그때 "AnyIP" 트릭이 등장하였습니다. AnyIP는 단일 주소가 아니라 전체 IP 프리픽스 (서브넷)을 루프백 인터페이스에 할당하도록 해 줍니다. 사실 AnyIP를 많이 사용하고 있습니다: 여러분 컴퓨터에는 루브백 인터페이스에 Continue reading

Recently we announced our fast, privacy-centric DNS resolver 1.1.1.1, supported by our global network. As you can see 1.1.1.1 is very easy to remember, which is both a blessing and a curse. In the time leading up to the announcement of the resolver service we began testing reachability to 1.1.1.1, primarily using the RIPE Atlas probes. The RIPE Atlas project is an extensive collection of small monitoring devices hosted by the public around the world. Currently there are over 10,000 active probes hosted in over 3,000 networks, giving great vantage points for testing. We found large numbers of probes unable to query 1.1.1.1, but successfully able to query 1.0.0.1 in almost all cases. 1.0.0.1 is the secondary address we have assigned for the resolver, to allow clients who are unable to reach 1.1.1.1 to be able to make DNS queries.

This blog focuses on IPv4. We provide four IPs (two for each IP address family) in order to provide a path toward the DNS resolver independent of IPv4 or IPv6 reachability.

1.0.0.0/8 was assigned in Continue reading

If you want to skip ahead to instructions, scroll to the next section. But I, like a TLS handshake, am very verbose so please enjoy this opener.

Imagine this scenario - I'm at a restaurant and need to have a private phone conversation but unfortunately my phone's battery is drained. To get around this problem, I borrow my friend's phone and dial the number - to protect my privacy I walk outside. When I'm done with the call, I come back inside and return the phone.

Whilst the phone itself doesn't store the coversation I've had, it does have a log of the recently dialed number, if the friend from whom I borrowed the phone wanted to, they could easily see who I actually called - even if they don't specifically know the topic of conversation.

Sometimes, the data about who you've spoken to can tell an aweful lot about the conversation - if someone was to call an emotional support hotline or a debt collector, you could probably infer a lot about the conversation from the caller ID.

When we browse the internet, we use encryption to try and protect the conversations we have. When you connect to a Continue reading

Yesterday Cloudflare launched Argo Tunnel. In the words of the product team:

Argo Tunnel exposes applications running on your local web server, on any network with an Internet connection, without adding DNS records or configuring a firewall or router. It just works.

Once I grokked this, the first thing that came to mind was that I could actually use one of my Raspberry Pi's sitting around to serve a website, without:

• A flaky DDNS running on my router
• Exposing my home network to the world
• A cloud VM

Ooooh... so exciting.

The Rig

I'll assume you already have a Raspberry Pi with Raspbian on it.

Plug the Pi into your router. It should now have an IP address. Look that up in your router’s admin UI:

OK, that's promising. Let's connect to that IP using the default pi/raspberry credentials:

$ ssh 192.168.8.26 -l pi
[email protected]'s password: 

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sun Mar 18 23:24:11 2018 from  Continue reading

Photo from Wikimedia Commons

Today we’re introducing Argo Tunnel, a private connection between your web server and Cloudflare. Tunnel makes it so that only traffic that routes through Cloudflare can reach your server.

You can think of Argo Tunnel as a virtual P.O. box. It lets someone send you packets without knowing your real address. In other words, it’s a private link. Only Cloudflare can see the server and communicate with it, and for the rest of the internet, it’s unroutable, as if the server is not even there.

How this used to be done

This type of private deployment used to be accomplished with GRE tunnels. But GRE tunnels are expensive and slow, they don’t really make sense in a 2018 internet.

GRE is a tunneling protocol for sending data between two servers by simulating a physical link. Configuring a GRE tunnel requires coordination between network administrators from both sides of the connection. It is an expensive service that is usually only available for large corporations with dedicated budgets. The GRE protocol encapsulates packets inside other packets, which means that you will have to either lower the MTU of your origin servers, or have your router do Continue reading

How great would it be to have a dashboard with a holistic view of threats, malicious server activity, vulnerabilities, sensitive data access levels and a daily scan of resources across all of your applications and services? Now you can.

Cloudflare is thrilled to announce its integration with Cloud Security Command Center (Cloud SCC) for Google Cloud Platform: A security and data risk platform helping enterprises gather data, identify threats, and act on them before they result in business damage or loss.

The advantage of the Cloud SCC solution is that it surfaces insights from both the Google Cloud Platform, as well as Cloudflare’s edge, in a unified dashboard.

What Cloudflare data is visible within the Cloud SCC dashboard?

Through Cloudflare’s API endpoints, data is pushed to Google’s Cloud SCC dashboard and domain name information mapped to the appropriate Google Cloud asset. Cloudflare’s branded card in the Cloud SCC dashboard is automatically populated with a summary of top theat origins, top types of threats, and latest Web Application Firewall (WAF) events.

To view a full list of Cloudflare events, click on the Cloudflare card in Cloud SCC and it will take you to a “Cloudflare Findings” page. From there, you can Continue reading

Cloudflare's mission is to help build a better Internet. We're excited today to take another step toward that mission with the launch of 1.1.1.1 — the Internet's fastest, privacy-first consumer DNS service. This post will talk a little about what that is and a lot about why we decided to do it. (If you're interested in the technical details on how we built the service, check out Ólafur Guðmundsson's accompanying post.)

Quick Primer On DNS

DNS is the directory of the Internet. Whenever you click on a link, send an email, open a mobile app, often one of the first things that has to happen is your device needs to look up the address of a domain. There are two sides of the DNS network: Authoritative (the content side) and Resolver (the consumer side).

Every domain needs to have an Authoritative DNS provider. Cloudflare, since our launch in September 2010, has run an extremely fast and widely-used Authoritative DNS service. 1.1.1.1 doesn't (directly) change anything about Cloudflare's Authoritative DNS service.

On the other side of the DNS system are resolvers. Every device that connects to the Internet needs a DNS resolver. By default, Continue reading

Cloudflare’s mission is to help build a better Internet and today we are releasing our DNS resolver, 1.1.1.1 - a recursive DNS service. With this offering, we’re fixing the foundation of the Internet by building a faster, more secure and privacy-centric public DNS resolver. The DNS resolver, 1.1.1.1, is available publicly for everyone to use - it is the first consumer-focused service Cloudflare has ever released.

We’re using the following IPv4 addresses for our resolver: 1.1.1.1 and 1.0.0.1. Easy to remember. These addresses have been provided to Cloudflare by APNIC for both joint research and this service. You can read more about their work via the APNIC blog.

DNS resolver, 1.1.1.1, is served by Cloudflare’s Global Anycast Network.

Background: A quick refresher on the role of the resolver in DNS

Our friends at DNSimple have made this amazing DNS Tutorial for anyone to fill in their gaps on how DNS works. They explain all about resolvers, root name servers, and much more in a very informative way.

When resolving a domain name, a query travels from your end system (i.e. a web browser) to Continue reading

Hot off the presses! Cloudflare just completed provisioning our Luxembourg City and Chișinău data centers, expanding our Europe network to 41 cities, and our global network to 151 cities across 74 countries. In the coming days, we'll ramp up traffic from across millions of websites using Cloudflare, and get routes optimized across all networks. Cloudflare is a participant at the Chișinău Internet Exchange (KIVIX), Luxembourg Commercial Internet eXchange (LU-CIX), and Moldova Internet Exchange (MD-IX), amongst ~180 other interconnection points.

This has been an exciting month, with 31 cities added just in March, for an average of one per day! Collectively, they provide additional resilience and performance across countries spanning a population of over one billion people. To recap, here's the list of our newest data centers: Beirut, Phnom Penh, Kathmandu, Istanbul, Reykjavík, Riyadh, Macau, Baghdad, Houston, Indianapolis, Montgomery, Pittsburgh, Sacramento, Mexico City, Tel Aviv, Durban, Port Louis, Cebu City, Edinburgh, Riga, Tallinn, Vilnius, Calgary, Saskatoon, Winnipeg, Jacksonville, Memphis, Tallahassee, Bogotá, Luxembourg and Chișinău!

We are very excited to surpass a milestone of 150 cities, or our sixth cohort of Continue reading

At 2625 meters (8612 feet) above sea level, Bogotá (Colombia) is one of the four highest capital cities in the world. Now, it is also home to Cloudflare's 149th data center.

This is the 29th city to be added just in March, and joins our existing Colombia datacenter in Medellín, launched four years ago.

CC BY-SA 2.0 image by nigel_sb

Bogotá is the third largest city in South America after São Paulo (Brazil) and Lima (Peru). Bogotanos affectionately known as Rolos are proud of their city with its rich cultural heritage, and its modern transportation systems (Ciclovias, Transmilenio) despite the heavy traffic. Whether you are visiting the world famous gold museum or savoring the mouthwatering Ajiaco soup, Bogotá has something for everyone, and visitors are always warmly received by the locals.

CC BY-SA 2.0 image by krossbow

Bogotá is our 11th deployment in the Latin America and Caribbean Region, and is located at a Tier III facility in the Bogota Free Trade Zone specially developed to attract ICT Investments. We'll continue our expansions in the Latin America and Caribbean region (and around the world!).

Come meet the Cloudflare team at the LACNIC29 Meeting in end April Continue reading

Good things come in threes! Following the launch of three data centers each in the Baltics (Riga, Tallinn, Vilnius) and in the Canadian Prairies (Calgary, Saskatoon, Winnipeg), we're thrilled to announce three new data centers in the Southern United States!

Located in Jacksonville (Florida), Memphis (Tennessee), and Tallahassee (Florida), they represent the 146th, 147th and 148th cities across our growing global network, and our 40th, 41st and 42nd cities just in North America. They join existing Cloudflare facilities in the US, including other Florida / Tennessee deployments such as Miami, Tampa and Nashville. Just in March, we've added deployments in 28 new cities worldwide, which help reduce latency to millions of Internet properties using Cloudflare, while expanding our capacity to withstand new and familiar attacks.

Photo of Jacksonville Beach by Lance Asper / Unsplash

Whether you're doing the Memphis Main Street Crawl, experiencing history through a visit to Tallahassee's Mission San Luis de Apalachee, or just relaxing by the stunning beaches of Jacksonville, you'll be close to the nearest Cloudflare data center.

The Cloudflare Global Anycast Network

This map reflects the network as of the publish date of this blog post. For the most up to date directory Continue reading

We just turned up Calgary, Saskatoon and Winnipeg - Cloudflare’s 143rd, 144th, and 145th data centers. This brings our Canadian presence to six cities, joining Toronto, Montreal and Vancouver. I grew up just outside of Saskatoon, so I couldn’t be happier that Cloudflare’s network has expanded to the Canadian Prairies. My parents still live there and I just booked flights to go and visit them this summer. When I tell people that I grew up in Saskatchewan, most people don’t know a lot about the region, so I wanted to share some of my favorite things about the region, starting from west to east:

• Calgary was home to the 1988 Winter Olympics and is a 90 minute drive from Banff, an incredible National Park that is absolutely worth visiting. Calgary has grown quickly over the last twenty years because of all the natural resources, including oil and gas. They host a famous rodeo, Calgary Stampede, for 10 days every summer. Definitely something to add to your bucket list. With Cloudflare’s new deployment in Calgary, we’ll make the Internet even faster for visitors. Hello Calgary!

• About 1 million people live in the province of Saskatchewan. Saskatoon is the Continue reading