We are very excited to announce Cloudflare’s 126th data center in Riyadh, Saudi Arabia (only hours after launching in Reykjavík!). This joins our existing Middle East facilities to provide even stronger coverage and resilience for over 7 million Internet properties across the region.

Our newest deployment was made possible in partnership with Zain, which now experiences reduced latency for every Internet user accessing every Internet facing application using Cloudflare. At least four additional Middle East deployments are already in the works.

Saudi Arabia

Photo by Mohammed Alamri / Unsplash

Over 30 million people live in Saudi Arabia, which is also the 13th largest country by area at over 830,000 square miles. In 2020, alongside the launch of entirely new “economic cities”, we might witness the opening of the world’s tallest skyscraper at a staggering 1,000m height, located in Jeddah. More modestly, but in much less than two years from now, we also expect to place a Cloudflare data center there.

Saudi Arabia has an incredibly young demographic, as over half of the population is less than 25. Additional 4G LTE deployments, while also paving the way for 5G, should drive increased Internet usage.

Stay tuned as Continue reading

Iceland is a small country in Northern Europe, a land of active volcanoes and boiling hot geysers. The geology and climate creates unique conditions for running compute power. With an abundance of green electricity and natural cooling, many companies are placing high power machines in Iceland to run power intensive, heat generating operations. Reykjavík is our 125th location globally.

Connecting to the rest of the world

A unique aspect about Iceland relates to how it connects to the Internet, being situated on the Mid-Atlantic Ridge means submarine cables are necessary to reach networks in other countries. Iceland has three active fibre optic submarine cables that land on its shores: FARICE-1, DANICE and Greenland Connect. Due to the distance, latency to the nearest Cloudflare locations in London and Copenhagen starts at 35 milliseconds. By deploying in Reykjavík, we're able to drive down latency even further to a minimum of under 1 millisecond.

Iceland is unique in many ways, but is no different from other countries when it comes to exchanging internet traffic. ISNIC, Iceland's Network Information Centre runs RIX, the Reykjavík Internet Exchange. Cloudflare is the only CDN network connected to RIX, allowing traffic to flow directly to Continue reading

One of our large scale data infrastructure challenges here at Cloudflare is around providing HTTP traffic analytics to our customers. HTTP Analytics is available to all our customers via two options:

• Analytics tab in Cloudflare dashboard
• Zone Analytics API with 2 endpoints
  • Dashboard endpoint
  • Co-locations endpoint (Enterprise plan only)

In this blog post I'm going to talk about the exciting evolution of the Cloudflare analytics pipeline over the last year. I'll start with a description of the old pipeline and the challenges that we experienced with it. Then, I'll describe how we leveraged ClickHouse to form the basis of a new and improved pipeline. In the process, I'll share details about how we went about schema design and performance tuning for ClickHouse. Finally, I'll look forward to what the Data team is thinking of providing in the future.

Let's start with the old data pipeline.

Old data pipeline

The previous pipeline was built in 2014. It has been mentioned previously in Scaling out PostgreSQL for CloudFlare Analytics using CitusDB and More data, more data blog posts from the Data team.

It had following components:

• Log forwarder - collected Cap'n Proto formatted logs from the edge, notably DNS and Nginx logs, Continue reading

We’re at the EDGE of our seats, about to LANd in Austin, Texas in route for SXSW. (TKIP, hip, hooray!)

ARP you going to be there? We R going to have three epoch sessions by Cloudflare speakers. Ifdown, seems apt you could SELECT to JOIN. Cat make it? Not a bg deal, wget it (though it mega hertz we won’t C you). All the audio from the three sessions will be recorded, you can listen to the cd.

WPS! I almost forgot to tel(net) you whois going to be there, and WAN and where to go.

On Friday, March 9, I’m moderating a panel with Emily Schechter from Google, Aaron DeVera from Deloitte and Gabe Kassel from eero about how Wi-Fi networks work and WEP happens when attackers coax people into joining insecure networks. It’s at Salon K in the Hilton at 3:30PM.

On Sunday the 11th, Nitin Rao is on a panel with Heather West from Mozilla, Stefan Lederer from Bitmovin and Fred Benenson from Unlimited Liability Corporation LLC about the impact of the recent revocation of Net Neutrality rules on online video streaming. It’s at 11AM at Salon J in the Continue reading

A week ago we published a story about new amplification attacks using memcached protocol on UDP port 11211. A few things happened since then:

• Github announced it was a target of 1.3Tbps memcached attack.
• OVH and Arbor reported similar large attacks with the peak reported at 1.7Tbps.

Let's take a deep breath and discuss why such large DDoS attacks are even possible on the modern internet.

Large attacks use IP spoofing

CC BY-SA 2.0 image by DaPuglet

All the gigantic headline-grabbing attacks are what we call "L3" (Layer 3 OSI^[1]). This kind of attack has a common trait - the malicious software sends as many packets as possible onto the network. For greater speed these packets are hand crafted by attackers - they are not generated using high-level, well-behaved libraries. Packets are mashed together as a series of bytes and fired onto the network to inflict the greatest damage.

L3 attacks can be divided into two categories, depending on where the attacker directs their traffic:

• Direct: where the traffic is sent directly against a victim IP. A SYN flood is a common attack of this type.

• Amplification: the traffic is sent to vulnerable Continue reading

Cloudflare is excited to turn up our newest data center in Istanbul, Turkey. This is our 124th data center globally (and 62nd country), and it is throwing a curveball in our data center by continent tracking. Istanbul is one of the only cities in the world to span two continents: Europe and Asia. Technically, we’ll specify this is our 34th data center in Europe. In the coming weeks, we’ll continue to attract more traffic to this deployment as more networks interconnect with us locally.

March 2018 is a big month for us, as we’ll be announcing (on average) nearly one new Cloudflare data center per day. Stay tuned as we continue to meaningfully expand our geographic coverage and capacity.

Turkish Internet

The Hagia Sophia - Photo by Blaque X / Unsplash

Istanbul itself is home to more than 16 million people, and Turkey is home to over 80 million people. For reference, Turkey’s population is comparable to Germany’s, where Cloudflare turned up its 11th, 31st, 44th, 72nd and 110th data centers in Frankfurt, Düsseldorf, Berlin, Hamburg and Munich. Internet usage in Turkey is approaching 70%, while the rate of Turkish households with access to Internet now exceeds Continue reading

We at Cloudflare are long time Kafka users, first mentions of it date back to beginning of 2014 when the most recent version was 0.8.0. We use Kafka as a log to power analytics (both HTTP and DNS), DDOS mitigation, logging and metrics.

Firehose CC BY 2.0 image by RSLab

While the idea of unifying abstraction of the log remained the same since then (read this classic blog post from Jay Kreps if you haven't), Kafka evolved in other areas since then. One of these improved areas was compression support. Back in the old days we've tried enabling it a few times and ultimately gave up on the idea because of unresolved issues in the protocol.

Kafka compression overview

Just last year Kafka 0.11.0 came out with the new improved protocol and log format.

The naive approach to compression would be to compress messages in the log individually:

Edit: originally we said this is how Kafka worked before 0.11.0, but that appears to be false.

Compression algorithms work best if they have more data, so in the new log format messages (now called records) are packed back to back and compressed in Continue reading

CC BY-SA 2.0 image by David Trawin

Over last couple of days we've seen a big increase in an obscure amplification attack vector - using the memcached protocol, coming from UDP port 11211.

In the past, we have talked a lot about amplification attacks happening on the internet. Our most recent two blog posts on this subject were:

• SSDP amplifications crossing 100Gbps. Funnily enough, since then we were a target of an 196Gbps SSDP attack.
• General statistics about various amplification attacks we see.

The general idea behind all amplification attacks is the same. An IP-spoofing capable attacker sends forged requests to a vulnerable UDP server. The UDP server, not knowing the request is forged, politely prepares the response. The problem happens when thousands of responses are delivered to an unsuspecting target host, overwhelming its resources - most typically the network itself.

Amplification attacks are effective, because often the response packets are much larger than the request packets. A carefully prepared technique allows an attacker with limited IP spoofing capacity (such as 1Gbps) to launch very large attacks (reaching 100s Gbps) "amplifying" the attacker's bandwidth.

Memcrashed

Obscure amplification attacks happen all the time. We often see "chargen" or "call Continue reading

I joined Cloudflare last week as an Engineering Manager, having previously spent 4 years working as the head of the software engineering community in the UK Government’s Digital Service (GDS). You only get one chance to be a new starter at each new place, so it’s important to make the most of the experience. Also, the job of Engineering Manager is different in every organisation, so it’s important to understand what the expectations and need for the role is in Cloudflare.

To help with this, I started by sketching out some objectives for my first week.

• Meet all my team members in a 1 to 1 meeting
• Know the skills and motivations of each team member
• Write down the expectations of my manager and team
• Write down the scope of the job
• Write a list of technologies to get an understanding of
• Be mentored by one of the team in one of these technologies
• Review what I can do to help with diversity and inclusion
• Understand the management structure - draw a diagram of it
• Spend 30 minutes with the most senior person I can to understand their aims
• Play with the product myself

Some of these are a bit Continue reading

Last week Troy Hunt launched his Pwned Password v2 service which has an API handled and cached by Cloudflare using a clever anonymity scheme.

The following simple code can check if a password exists in Troy's database without sending the password to Troy. The details of how it works are found in the blog post above.

use strict;
use warnings;

use LWP::Simple qw/$ua get/;
$ua->agent('Cloudflare Test/0.1');
use Digest::SHA1 qw/sha1_hex/;

uc(sha1_hex($ARGV[0]))=~/^(.{5})(.+)/;
print get("https://api.pwnedpasswords.com/range/$1")=~/$2/?'Pwned':'Ok', "\n";

It's just as easy to implement the same check in other languages, such as JavaScript, which made me realize that I could incorporate the check into a Cloudflare Worker. With a little help from people who know JavaScript far better than me, I wrote the following Worker:

addEventListener('fetch', event => {
  event.respondWith(fetchAndCheckPassword(event.request))
})

async function fetchAndCheckPassword(req) {
  if (req.method == "POST") {
    try {
      const post = await req.formData()
      const pwd = post.get('password')
      const enc = new TextEncoder("utf-8").encode(pwd)

      let hash = await crypto.subtle.digest("SHA-1", enc)
      let hashStr = hex(hash).toUpperCase()
  
      const prefix = hashStr.substring(0, 5)
      const suffix = hashStr.substring(5)

      const pwndpwds = await fetch('https://api.pwnedpasswords.com/range/' + prefix)
      const t =  Continue reading

(This is a crosspost of a blog post originally published on Google Cloud blog)

One of the great things about container technology is that it delivers the same experience and functionality across different platforms. This frees you as a developer from having to rewrite or update your application to deploy it on a new cloud provider—or lets you run it across multiple cloud providers. With a containerized application running on multiple clouds, you can avoid lock-in, run your application on the cloud for which it’s best suited, and lower your overall costs.

If you’re using Kubernetes, you probably manage traffic to clusters and services across multiple nodes using internal load-balancing services, which is the most common and practical approach. But if you’re running an application on multiple clouds, it can be hard to distribute traffic intelligently among them. In this blog post, we show you how to use Cloudflare Load Balancer in conjunction with Kubernetes so you can start to achieve the benefits of a multi-cloud configuration.

To continue reading follow the Google Cloud blog here or if you are ready to get started we created a guide on how to deploy an application using Kubernetes on GCP and AWS Continue reading

We said that we would head to the mountains for Cloudflare’s 123rd data center, and mountains feature prominently as we talk about Kathmandu, Nepal, home of our newest deployment and our 42nd data center in Asia!

Five and three quarter key facts to get started:

• Nepal is home to the highest mountain in the world.
• Kathmandu has more UNESCO heritage sites in its immediate area than any other capital!
• The Nepalese flag isn’t a rectangle. It’s not even close!
• Nepal has never been conquered or ruled by another country.
• Kathmandu, Nepal is where Cloudflare has placed its 123rd data center.
• Nepal’s timezone is 5 hours 45 minutes ahead of GMT.

Mountains

The mountainous nation of Nepal is home to Mount Everest, the highest mountain in the world, known in Nepali as Sagarmāthā. Most of us learn that at school; however there’s plenty of other mountains located in Nepal. Here’s the ones above 8,000 meters (extracted from the full list) to get you started:

• Mount Everest at 8,848 meters
• Kanchenjunga at 8,586 meters
• Lhotse at 8,516 meters
• Makalu at 8,463 meters
• Cho Oyu at 8,201 meters
• Dhaulagiri I at 8,167 meters
• Manaslu at 8,156 meters
• Annapurna I at 8,091 meters

Continue reading

Almost a year ago, I began my journey in the tech industry at a growing company called Cloudflare. I’m a 30-something paralegal and although I didn’t know how to write code (yet), I was highly motivated and ready to crush. I had worked hard for the previous two years, focused on joining a thriving company where I could grow my intelligence, further develop my skill set and work alongside successful professionals. And finally, my hard work paid off; I landed the job at Cloudflare and booked a seat on the rocket ship.

After the initial whirlwind that accompanies this fast-paced field subsided, motivation, inspiration, success, momentum and endurance began to flood my neurons. I loved the inner workings of a successful startup, felt the good and bad of the tech industry, related to and admired the female executives and most importantly, wanted to give something back to the community that adopted me.

Venus Approaching the Sun Source: Flickr

During a routine chat with my dad, I pitched what I thought was a crazy idea. Crazy because I was so used to being told “no” at previous jobs, used to not having my ideas taken seriously, and also used to not Continue reading

Today, v2 of Pwned Passwords was released as part of the Have I Been Pwned service offered by Troy Hunt. Containing over half a billion real world leaked passwords, this database provides a vital tool for correcting the course of how the industry combats modern threats against password security.

I have written about how we need to rethink password security and Pwned Passwords v2 in the following post: How Developers Got Password Security So Wrong. Instead, in this post I want to discuss one of the technical contributions Cloudflare has made towards protecting user information when using this tool.

Cloudflare continues to support Pwned Passwords by providing CDN and security functionality such that the data can easily be made available for download in raw form to organisations to protect their customers. Further; as part of the second iteration of this project, I have also worked with Troy on designing and implementing API endpoints that support anonymised range queries to function as an additional layer of security for those consuming the API, that is visible to the client.

This contribution allows for Pwned Passwords clients to use range queries to search for breached passwords, without having to disclose a complete unsalted Continue reading

Both in our real lives, and online, there are times where we need to authenticate ourselves - where we need to confirm we are who we say we are. This can be done using three things:

• Something you know
• Something you have
• Something you are

Passwords are an example of something you know; they were introduced in 1961 for computer authentication for a time-share computer in MIT. Shortly afterwards, a PhD researcher breached this system (by being able to simply download a list of unencrypted passwords) and used the time allocated to others on the computer.

As time has gone on; developers have continued to store passwords insecurely, and users have continued to set them weakly. Despite this, no viable alternative has been created for password security. To date, no system has been created that retains all the benefits that passwords offer as researchers have rarely considered real world constraints^[1]. For example; when using fingerprints for authentication, engineers often forget that there is a sizable percentage of the population that do not have usable fingerprints or hardware upgrade costs.

Cracking Passwords

In the 1970s, people started thinking about how to better store passwords and cryptographic hashing started to Continue reading

Cloudflare is excited to turn up our newest data center in Phnom Penh, Cambodia, making over 7 million Internet properties even faster. This is our 122nd data center globally, and our 41st data center in Asia. By the end of 2018, we expect that 95% of the world's population will live in a country with a Cloudflare data center, as we grow our global network to span 200 cities.

Cambodian Internet

Home to over 16 million people, Cambodia has a relatively low base of Internet penetration (~25%) today, but is seeing an increasing number of Internet users coming online. For perspective, Cambodia has approximately the same number of Internet users as Lebanon (where we just turned up our 121st data center!) or Singapore (from where we used to serve a portion of Cambodian visitors).

In the coming weeks, we’ll further optimize our routing for Cloudflare customers and expect to see a growing number of ISPs pick up our customers’ traffic on a low latency path.

Latency from a Cambodian ISP (SINET) to Cloudflare customers decreases 10x

Coming up next

Next up, in fact, thousands of feet further up, we head to the mountains for Cloudflare’s 123rd data center. Following Continue reading

At Cloudflare we like Go. We use it in many in-house software projects as well as parts of bigger pipeline systems. But can we take Go to the next level and use it as a scripting language for our favourite operating system, Linux?

gopher image CC BY 3.0 Renee French
Tux image CC0 BY OpenClipart-Vectors

Why consider Go as a scripting language

Short answer: why not? Go is relatively easy to learn, not too verbose and there is a huge ecosystem of libraries which can be reused to avoid writing all the code from scratch. Some other potential advantages it might bring:

• Go-based build system for your Go project: go build command is mostly suitable for small, self-contained projects. More complex projects usually adopt some build system/set of scripts. Why not have these scripts written in Go then as well?
• Easy non-privileged package management out of the box: if you want to use a third-party library in your script, you can simply go get it. And because the code will be installed in your GOPATH, getting a third-party library does not require administrative privileges on the system (unlike some other scripting languages). This is especially useful in large Continue reading

This is a Korean translation of a prior post by Ignat Korchagin.

Cloudflare에서는 Go를 좋아합니다. Go는 많은 내부 소프트웨어 프로젝트와 거대한 파이프라인 시스템의 일부로도 사용되고 있습니다. 하지만 Go를 한단계 더 끌어 올려서 우리가 선호하는 운영체제인 리눅스의 스크립트 언어로 사용할 수 있을까요?

gopher image CC BY 3.0 Renee French | Tux image CC0 BY OpenClipart-Vectors

Go를 왜 스크립트 언어로 고려하는가

간단한 답은: 왜 안되나요? Go는 비교적 쉽게 배울 수 있고 아주 복잡하지도 않고, 코드를 처음부터 작성해야 하는 일을 피하기 위해 재사용 가능한 라이브러리의 거대한 에코시스템이 있습니다. 추가로 다음과 같은 잠재적인 장점이 있습니다:

• 여러분의 Go 프로젝트를 위한 Go 기반 빌드 시스템: go build 명령은 대부분의 소규모이며 독립적인 프로젝트에 적합합니다. 더 복잡한 프로젝트는 대부분 별도의 빌드 시스템/스크립트 세트를 채용하고 있습니다. 이런 스크립트도 Go로 작성 가능하지 않을까요?
• 바로 이용 가능한 별도 권한 없는 패키지 관리: 여러분의 프로그램에서 서드 파티 라이브러리를 사용하고 싶다면 단순히 go get 을 사용하면 됩니다. 그리고 이 코드가 여러분의 GOPATH에 설치되므로, 서드파티 라이브러리를 받는 것은 시스템의 별도 운영 권한을 필요로 하지 않습니다(다른 일부 스크립트 언어와 달리). 이것은 대규모의 기업 환경에서 특히 유용합니다.
• 초기 단계 프로젝트를 위한 빠른 코드 프로토타이핑: 최초로 돌아가는 코드를 작성할 때 컴파일 되기 위해서 많은 편집을 해야 하고 "편집->빌드->체크" 사이클을 위해 많은 키보드 입력을 Continue reading

To everyone in Cloudflare, account security is one of our most important tasks. We recognize that to every customer on our platform, we are critical infrastructure. We also know that the simplest attacks often lead to the most devastating of outcomes. Most people think that if they are going to get hacked it will be by some clever ”zero day”. The reality couldn’t be farther from the truth.

Attackers are smart and they have realized that even in 2018, the human is still the weakest link in the chain. The 2017 Verizon breach report identified that 81% of hacking related breaches occurred as a result of weak credentials or credential theft, an increase from the 63% reported in 2016’s breach report.

Source: Verizon 2017 data breach report

Your credentials are as important as your house or car keys. If someone copies or steals them, the repercussions can be catastrophic. If you suspect someone has access to your house keys you change your locks. If you aren’t fast enough, someone might break in.

Likewise if you realize that someone might have access to your password, the remedy is to change it. Too often, as with house keys, we are slow to Continue reading