When Cloudflare launched, three of the original five cities in our network - Chicago, Ashburn and San Jose - were located in the United States. Since then, we have grown the breadth of the global network considerably to span 66 countries, and even added expanded the US footprint to twenty five locations. Even as a highly international business, the United States continues to be home to a number of our customers and the majority of Cloudflare employees.

Today, we expand our network in the United States even further by adding five new locations: Houston (Texas), Indianapolis (Indiana), Montgomery (Alabama), Pittsburgh (Pennsylvania) and Sacramento (California) as our 129th, 130th, 131st, 132nd and 133rd data centers respectively. They represent states that collectively span nearly 100 million people. In North America alone, the Cloudflare network now spans 37 cities, including thirty in the US.

In each of these new locations, we connect with at least one major local Internet service provider and also openly peer using at least one major Internet exchange. We are participants at CyrusOne IX Houston, Midwest IX Indianapolis, Montgomery IX, Pittsburgh IX, and the upcoming Sacramento IX.

These deployments improves performance, security and reliability Continue reading

Cloudflare's newest data center is located in Baghdad, Iraq, in the region often known as the cradle of civilization. This expands our growing Middle East presence, while serving as our 45th data center in Asia, and 128th data center globally.

Even while accelerating over 7 million Internet properties, this deployment helps our effort to be closer to every Internet user. Previous, ISPs such as Earthlink were served from our Frankfurt data center. Nearly 40 million people live in Iraq.

Rich Cuisine

One of the world's largest producers of the sweet date palm, Iraq's cuisine dates back over 10,000 years and includes favorites such as,

• Kleicha: Date-filled cookies flavored with cardamom, saffron and rose water
• Mezza: a selection of appetizers to begin the meal
• Iraqi Dolma: stuffed vegetables with a tangy sauce
• Iraqi Biryani: cooked rice with spices, beans, grilled nuts and meat / vegetables
• Masgouf: whole baked fish marinated in oil, salt, pepper, turmeric and tamarind

New data centers

Baghdad is the first of eight deployments joining the Cloudflare global network just this week. Stay tuned!

The Cloudflare Global Anycast Network

This map reflects the network as of the publish date of this blog Continue reading

A Tale of Two New Relationships

Late last spring, we were seeking to expand our connections inside of IBM. IBM had first become a direct Cloudflare customer in 2016, when its X-force Exchange business selected Cloudflare, instead of traditional scrubbing center solutions, for DDoS protection, WAF, and Load Balancing. We had friendly relationships with several people inside of IBM’s Softlayer business. We learned that the IBM “Networking Tribe” was evaluating various solutions to fill product gaps that their cloud customers were experiencing for DDoS, DNS, WAF, and load balancing.

In trying to engage with the people leading the effort, I made a casual phone call late on a Friday afternoon to one of the IBMers based in Raleigh, NC. When he understood that I was from Cloudflare, he replied, “Oh, I know Cloudflare. You guys do DDoS protection, right?” I replied, “Well, yes, we do offer DDoS protection, but we also offer a number of other security and performance services.” He indicated that he would be in the Bay Area two weeks later, and that he would bring his team to our office if we could make the time.

Also late last spring, my wife delivered our baby Continue reading

Exactly one year ago today, Cloudflare gave me a mission: Make it so people can run code on Cloudflare's edge. At the time, we didn't yet know what that would mean. Would it be container-based? A new Turing-incomplete domain-specific language? Lua? "Functions"? There were lots of ideas.

Eventually, we settled on what now seems the obvious choice: JavaScript, using the standard Service Workers API, running in a new environment built on V8. Five months ago, we gave you a preview of what we were building, and started the beta.

Today, with thousands of scripts deployed and many billions of requests served, Cloudflare Workers is now ready for everyone.

What is the Cloud, really?

Historically, web application code has been split between servers and browsers. Between them lies a vast but fundamentally dumb network which merely ferries data from point to Continue reading

On June 4, Cloudflare will be dropping support for TLS 1.0 and 1.1 on api.cloudflare.com. Additionally, the dashboard will be moved from www.cloudflare.com/a to dash.cloudflare.com and will require a browser that supports TLS 1.2 or higher.

No changes will be made to customer traffic that is proxied through our network, though you may decide to enforce a minimum version for your own traffic. We will soon expose TLS analytics that indicate the percent of connections to your sites using TLS 1.0-1.3, and controls to set a specific minimum version. Currently, you may enforce version 1.2 or higher using the Require Modern TLS setting.

Prior to June 4, API calls made with TLS 1.0 or 1.1 will have warning messages inserted into responses and dashboard users will see a banner encouraging you to upgrade your browser. Additional details on these changes, and a complete schedule of planned events can be found in the timeline below.

Background

Transport Layer Security (TLS) is the protocol used on the web today to encrypt HTTPS connections. Version 1.0 was standardized almost 20 years ago as the successor to SSL Continue reading

Cloudflare's 127th data center is now live in Macau, helping make over 7 million Internet facing applications safer and faster. This is our 44th data center in Asia.

Cloudflare 將在澳門啟用全球第127個數據中心, 幫助超過 7,000,000 客戶的互聯網資產運行得更快、更安全。澳門也是我們在亞洲的第44個數據中心。

O 127º centro de dados da Cloudflare agora está em funcionamento em Macau, ajudando a tornar mais de 7 milhões de aplicações voltadas para a Internet de forma mais segura e rápida. Estamos felizes em compartilhar que este é o nosso 44º centro de dados na Ásia.

CC BY-NC-ND 2.0 image by kidchen915

Blending Chinese and Portuguese culture, just last year, Macau welcomed over 30 million visitors. Visit Macau to experience its unique and extravagant entertainment scene, see scenic spots such as the Ruins of St Paul, Senado Square, attempt the world's highest bungy jump from Macau Tower, or enjoy the foodie paradise Macau delivers!

有著與眾不同的中國及葡萄牙文化融合景觀，澳門至去年為止已經吸引了三千萬遊客來一睹她的風采。你可以拜訪著名的娛樂景觀，像是聖保祿大教堂遺址，議事亭廣場，挑戰澳門的美食，亦可以選擇從全世界最高的高空彈跳地點ㄧ澳門旅遊塔上一躍而下。

Combinando cultura chinesa e portuguesa, o ano passado, Macau recebeu mais de 30 milhões de visitantes. Recomendamos que visite Macau para experimentar a cena de entretenimento única e extravagante, explore os pontos cénica, como ruínas de São Paulo, Praça do Senado, e ainda, tente o "bungee jumping" mais alto do mundo deste da Torre de Macau, ou aproveite Continue reading

We are very excited to announce Cloudflare’s 126th data center in Riyadh, Saudi Arabia (only hours after launching in Reykjavík!). This joins our existing Middle East facilities to provide even stronger coverage and resilience for over 7 million Internet properties across the region.

Our newest deployment was made possible in partnership with Zain, which now experiences reduced latency for every Internet user accessing every Internet facing application using Cloudflare. At least four additional Middle East deployments are already in the works.

Saudi Arabia

Photo by Mohammed Alamri / Unsplash

Over 30 million people live in Saudi Arabia, which is also the 13th largest country by area at over 830,000 square miles. In 2020, alongside the launch of entirely new “economic cities”, we might witness the opening of the world’s tallest skyscraper at a staggering 1,000m height, located in Jeddah. More modestly, but in much less than two years from now, we also expect to place a Cloudflare data center there.

Saudi Arabia has an incredibly young demographic, as over half of the population is less than 25. Additional 4G LTE deployments, while also paving the way for 5G, should drive increased Internet usage.

Stay tuned as Continue reading

Iceland is a small country in Northern Europe, a land of active volcanoes and boiling hot geysers. The geology and climate creates unique conditions for running compute power. With an abundance of green electricity and natural cooling, many companies are placing high power machines in Iceland to run power intensive, heat generating operations. Reykjavík is our 125th location globally.

Connecting to the rest of the world

A unique aspect about Iceland relates to how it connects to the Internet, being situated on the Mid-Atlantic Ridge means submarine cables are necessary to reach networks in other countries. Iceland has three active fibre optic submarine cables that land on its shores: FARICE-1, DANICE and Greenland Connect. Due to the distance, latency to the nearest Cloudflare locations in London and Copenhagen starts at 35 milliseconds. By deploying in Reykjavík, we're able to drive down latency even further to a minimum of under 1 millisecond.

Iceland is unique in many ways, but is no different from other countries when it comes to exchanging internet traffic. ISNIC, Iceland's Network Information Centre runs RIX, the Reykjavík Internet Exchange. Cloudflare is the only CDN network connected to RIX, allowing traffic to flow directly to Continue reading

One of our large scale data infrastructure challenges here at Cloudflare is around providing HTTP traffic analytics to our customers. HTTP Analytics is available to all our customers via two options:

• Analytics tab in Cloudflare dashboard
• Zone Analytics API with 2 endpoints
  • Dashboard endpoint
  • Co-locations endpoint (Enterprise plan only)

In this blog post I'm going to talk about the exciting evolution of the Cloudflare analytics pipeline over the last year. I'll start with a description of the old pipeline and the challenges that we experienced with it. Then, I'll describe how we leveraged ClickHouse to form the basis of a new and improved pipeline. In the process, I'll share details about how we went about schema design and performance tuning for ClickHouse. Finally, I'll look forward to what the Data team is thinking of providing in the future.

Let's start with the old data pipeline.

Old data pipeline

The previous pipeline was built in 2014. It has been mentioned previously in Scaling out PostgreSQL for CloudFlare Analytics using CitusDB and More data, more data blog posts from the Data team.

It had following components:

• Log forwarder - collected Cap'n Proto formatted logs from the edge, notably DNS and Nginx logs, Continue reading

We’re at the EDGE of our seats, about to LANd in Austin, Texas in route for SXSW. (TKIP, hip, hooray!)

ARP you going to be there? We R going to have three epoch sessions by Cloudflare speakers. Ifdown, seems apt you could SELECT to JOIN. Cat make it? Not a bg deal, wget it (though it mega hertz we won’t C you). All the audio from the three sessions will be recorded, you can listen to the cd.

WPS! I almost forgot to tel(net) you whois going to be there, and WAN and where to go.

On Friday, March 9, I’m moderating a panel with Emily Schechter from Google, Aaron DeVera from Deloitte and Gabe Kassel from eero about how Wi-Fi networks work and WEP happens when attackers coax people into joining insecure networks. It’s at Salon K in the Hilton at 3:30PM.

On Sunday the 11th, Nitin Rao is on a panel with Heather West from Mozilla, Stefan Lederer from Bitmovin and Fred Benenson from Unlimited Liability Corporation LLC about the impact of the recent revocation of Net Neutrality rules on online video streaming. It’s at 11AM at Salon J in the Continue reading

A week ago we published a story about new amplification attacks using memcached protocol on UDP port 11211. A few things happened since then:

• Github announced it was a target of 1.3Tbps memcached attack.
• OVH and Arbor reported similar large attacks with the peak reported at 1.7Tbps.

Let's take a deep breath and discuss why such large DDoS attacks are even possible on the modern internet.

Large attacks use IP spoofing

CC BY-SA 2.0 image by DaPuglet

All the gigantic headline-grabbing attacks are what we call "L3" (Layer 3 OSI^[1]). This kind of attack has a common trait - the malicious software sends as many packets as possible onto the network. For greater speed these packets are hand crafted by attackers - they are not generated using high-level, well-behaved libraries. Packets are mashed together as a series of bytes and fired onto the network to inflict the greatest damage.

L3 attacks can be divided into two categories, depending on where the attacker directs their traffic:

• Direct: where the traffic is sent directly against a victim IP. A SYN flood is a common attack of this type.

• Amplification: the traffic is sent to vulnerable Continue reading

Cloudflare is excited to turn up our newest data center in Istanbul, Turkey. This is our 124th data center globally (and 62nd country), and it is throwing a curveball in our data center by continent tracking. Istanbul is one of the only cities in the world to span two continents: Europe and Asia. Technically, we’ll specify this is our 34th data center in Europe. In the coming weeks, we’ll continue to attract more traffic to this deployment as more networks interconnect with us locally.

March 2018 is a big month for us, as we’ll be announcing (on average) nearly one new Cloudflare data center per day. Stay tuned as we continue to meaningfully expand our geographic coverage and capacity.

Turkish Internet

The Hagia Sophia - Photo by Blaque X / Unsplash

Istanbul itself is home to more than 16 million people, and Turkey is home to over 80 million people. For reference, Turkey’s population is comparable to Germany’s, where Cloudflare turned up its 11th, 31st, 44th, 72nd and 110th data centers in Frankfurt, Düsseldorf, Berlin, Hamburg and Munich. Internet usage in Turkey is approaching 70%, while the rate of Turkish households with access to Internet now exceeds Continue reading

We at Cloudflare are long time Kafka users, first mentions of it date back to beginning of 2014 when the most recent version was 0.8.0. We use Kafka as a log to power analytics (both HTTP and DNS), DDOS mitigation, logging and metrics.

Firehose CC BY 2.0 image by RSLab

While the idea of unifying abstraction of the log remained the same since then (read this classic blog post from Jay Kreps if you haven't), Kafka evolved in other areas since then. One of these improved areas was compression support. Back in the old days we've tried enabling it a few times and ultimately gave up on the idea because of unresolved issues in the protocol.

Kafka compression overview

Just last year Kafka 0.11.0 came out with the new improved protocol and log format.

The naive approach to compression would be to compress messages in the log individually:

Edit: originally we said this is how Kafka worked before 0.11.0, but that appears to be false.

Compression algorithms work best if they have more data, so in the new log format messages (now called records) are packed back to back and compressed in Continue reading

CC BY-SA 2.0 image by David Trawin

Over last couple of days we've seen a big increase in an obscure amplification attack vector - using the memcached protocol, coming from UDP port 11211.

In the past, we have talked a lot about amplification attacks happening on the internet. Our most recent two blog posts on this subject were:

• SSDP amplifications crossing 100Gbps. Funnily enough, since then we were a target of an 196Gbps SSDP attack.
• General statistics about various amplification attacks we see.

The general idea behind all amplification attacks is the same. An IP-spoofing capable attacker sends forged requests to a vulnerable UDP server. The UDP server, not knowing the request is forged, politely prepares the response. The problem happens when thousands of responses are delivered to an unsuspecting target host, overwhelming its resources - most typically the network itself.

Amplification attacks are effective, because often the response packets are much larger than the request packets. A carefully prepared technique allows an attacker with limited IP spoofing capacity (such as 1Gbps) to launch very large attacks (reaching 100s Gbps) "amplifying" the attacker's bandwidth.

Memcrashed

Obscure amplification attacks happen all the time. We often see "chargen" or "call Continue reading

I joined Cloudflare last week as an Engineering Manager, having previously spent 4 years working as the head of the software engineering community in the UK Government’s Digital Service (GDS). You only get one chance to be a new starter at each new place, so it’s important to make the most of the experience. Also, the job of Engineering Manager is different in every organisation, so it’s important to understand what the expectations and need for the role is in Cloudflare.

To help with this, I started by sketching out some objectives for my first week.

• Meet all my team members in a 1 to 1 meeting
• Know the skills and motivations of each team member
• Write down the expectations of my manager and team
• Write down the scope of the job
• Write a list of technologies to get an understanding of
• Be mentored by one of the team in one of these technologies
• Review what I can do to help with diversity and inclusion
• Understand the management structure - draw a diagram of it
• Spend 30 minutes with the most senior person I can to understand their aims
• Play with the product myself

Some of these are a bit Continue reading

Last week Troy Hunt launched his Pwned Password v2 service which has an API handled and cached by Cloudflare using a clever anonymity scheme.

The following simple code can check if a password exists in Troy's database without sending the password to Troy. The details of how it works are found in the blog post above.

use strict;
use warnings;

use LWP::Simple qw/$ua get/;
$ua->agent('Cloudflare Test/0.1');
use Digest::SHA1 qw/sha1_hex/;

uc(sha1_hex($ARGV[0]))=~/^(.{5})(.+)/;
print get("https://api.pwnedpasswords.com/range/$1")=~/$2/?'Pwned':'Ok', "\n";

It's just as easy to implement the same check in other languages, such as JavaScript, which made me realize that I could incorporate the check into a Cloudflare Worker. With a little help from people who know JavaScript far better than me, I wrote the following Worker:

addEventListener('fetch', event => {
  event.respondWith(fetchAndCheckPassword(event.request))
})

async function fetchAndCheckPassword(req) {
  if (req.method == "POST") {
    try {
      const post = await req.formData()
      const pwd = post.get('password')
      const enc = new TextEncoder("utf-8").encode(pwd)

      let hash = await crypto.subtle.digest("SHA-1", enc)
      let hashStr = hex(hash).toUpperCase()
  
      const prefix = hashStr.substring(0, 5)
      const suffix = hashStr.substring(5)

      const pwndpwds = await fetch('https://api.pwnedpasswords.com/range/' + prefix)
      const t =  Continue reading

(This is a crosspost of a blog post originally published on Google Cloud blog)

One of the great things about container technology is that it delivers the same experience and functionality across different platforms. This frees you as a developer from having to rewrite or update your application to deploy it on a new cloud provider—or lets you run it across multiple cloud providers. With a containerized application running on multiple clouds, you can avoid lock-in, run your application on the cloud for which it’s best suited, and lower your overall costs.

If you’re using Kubernetes, you probably manage traffic to clusters and services across multiple nodes using internal load-balancing services, which is the most common and practical approach. But if you’re running an application on multiple clouds, it can be hard to distribute traffic intelligently among them. In this blog post, we show you how to use Cloudflare Load Balancer in conjunction with Kubernetes so you can start to achieve the benefits of a multi-cloud configuration.

To continue reading follow the Google Cloud blog here or if you are ready to get started we created a guide on how to deploy an application using Kubernetes on GCP and AWS Continue reading

We said that we would head to the mountains for Cloudflare’s 123rd data center, and mountains feature prominently as we talk about Kathmandu, Nepal, home of our newest deployment and our 42nd data center in Asia!

Five and three quarter key facts to get started:

• Nepal is home to the highest mountain in the world.
• Kathmandu has more UNESCO heritage sites in its immediate area than any other capital!
• The Nepalese flag isn’t a rectangle. It’s not even close!
• Nepal has never been conquered or ruled by another country.
• Kathmandu, Nepal is where Cloudflare has placed its 123rd data center.
• Nepal’s timezone is 5 hours 45 minutes ahead of GMT.

Mountains

The mountainous nation of Nepal is home to Mount Everest, the highest mountain in the world, known in Nepali as Sagarmāthā. Most of us learn that at school; however there’s plenty of other mountains located in Nepal. Here’s the ones above 8,000 meters (extracted from the full list) to get you started:

• Mount Everest at 8,848 meters
• Kanchenjunga at 8,586 meters
• Lhotse at 8,516 meters
• Makalu at 8,463 meters
• Cho Oyu at 8,201 meters
• Dhaulagiri I at 8,167 meters
• Manaslu at 8,156 meters
• Annapurna I at 8,091 meters

Continue reading

Almost a year ago, I began my journey in the tech industry at a growing company called Cloudflare. I’m a 30-something paralegal and although I didn’t know how to write code (yet), I was highly motivated and ready to crush. I had worked hard for the previous two years, focused on joining a thriving company where I could grow my intelligence, further develop my skill set and work alongside successful professionals. And finally, my hard work paid off; I landed the job at Cloudflare and booked a seat on the rocket ship.

After the initial whirlwind that accompanies this fast-paced field subsided, motivation, inspiration, success, momentum and endurance began to flood my neurons. I loved the inner workings of a successful startup, felt the good and bad of the tech industry, related to and admired the female executives and most importantly, wanted to give something back to the community that adopted me.

Venus Approaching the Sun Source: Flickr

During a routine chat with my dad, I pitched what I thought was a crazy idea. Crazy because I was so used to being told “no” at previous jobs, used to not having my ideas taken seriously, and also used to not Continue reading