Cloudflare is excited to turn up our newest data center in Istanbul, Turkey. This is our 124th data center globally (and 62nd country), and it is throwing a curveball in our data center by continent tracking. Istanbul is one of the only cities in the world to span two continents: Europe and Asia. Technically, we’ll specify this is our 34th data center in Europe. In the coming weeks, we’ll continue to attract more traffic to this deployment as more networks interconnect with us locally.

March 2018 is a big month for us, as we’ll be announcing (on average) nearly one new Cloudflare data center per day. Stay tuned as we continue to meaningfully expand our geographic coverage and capacity.

Turkish Internet

The Hagia Sophia - Photo by Blaque X / Unsplash

Istanbul itself is home to more than 16 million people, and Turkey is home to over 80 million people. For reference, Turkey’s population is comparable to Germany’s, where Cloudflare turned up its 11th, 31st, 44th, 72nd and 110th data centers in Frankfurt, Düsseldorf, Berlin, Hamburg and Munich. Internet usage in Turkey is approaching 70%, while the rate of Turkish households with access to Internet now exceeds Continue reading

We at Cloudflare are long time Kafka users, first mentions of it date back to beginning of 2014 when the most recent version was 0.8.0. We use Kafka as a log to power analytics (both HTTP and DNS), DDOS mitigation, logging and metrics.

Firehose CC BY 2.0 image by RSLab

While the idea of unifying abstraction of the log remained the same since then (read this classic blog post from Jay Kreps if you haven't), Kafka evolved in other areas since then. One of these improved areas was compression support. Back in the old days we've tried enabling it a few times and ultimately gave up on the idea because of unresolved issues in the protocol.

Kafka compression overview

Just last year Kafka 0.11.0 came out with the new improved protocol and log format.

The naive approach to compression would be to compress messages in the log individually:

Edit: originally we said this is how Kafka worked before 0.11.0, but that appears to be false.

Compression algorithms work best if they have more data, so in the new log format messages (now called records) are packed back to back and compressed in Continue reading

CC BY-SA 2.0 image by David Trawin

Over last couple of days we've seen a big increase in an obscure amplification attack vector - using the memcached protocol, coming from UDP port 11211.

In the past, we have talked a lot about amplification attacks happening on the internet. Our most recent two blog posts on this subject were:

• SSDP amplifications crossing 100Gbps. Funnily enough, since then we were a target of an 196Gbps SSDP attack.
• General statistics about various amplification attacks we see.

The general idea behind all amplification attacks is the same. An IP-spoofing capable attacker sends forged requests to a vulnerable UDP server. The UDP server, not knowing the request is forged, politely prepares the response. The problem happens when thousands of responses are delivered to an unsuspecting target host, overwhelming its resources - most typically the network itself.

Amplification attacks are effective, because often the response packets are much larger than the request packets. A carefully prepared technique allows an attacker with limited IP spoofing capacity (such as 1Gbps) to launch very large attacks (reaching 100s Gbps) "amplifying" the attacker's bandwidth.

Memcrashed

Obscure amplification attacks happen all the time. We often see "chargen" or "call Continue reading

I joined Cloudflare last week as an Engineering Manager, having previously spent 4 years working as the head of the software engineering community in the UK Government’s Digital Service (GDS). You only get one chance to be a new starter at each new place, so it’s important to make the most of the experience. Also, the job of Engineering Manager is different in every organisation, so it’s important to understand what the expectations and need for the role is in Cloudflare.

To help with this, I started by sketching out some objectives for my first week.

• Meet all my team members in a 1 to 1 meeting
• Know the skills and motivations of each team member
• Write down the expectations of my manager and team
• Write down the scope of the job
• Write a list of technologies to get an understanding of
• Be mentored by one of the team in one of these technologies
• Review what I can do to help with diversity and inclusion
• Understand the management structure - draw a diagram of it
• Spend 30 minutes with the most senior person I can to understand their aims
• Play with the product myself

Some of these are a bit Continue reading

Last week Troy Hunt launched his Pwned Password v2 service which has an API handled and cached by Cloudflare using a clever anonymity scheme.

The following simple code can check if a password exists in Troy's database without sending the password to Troy. The details of how it works are found in the blog post above.

use strict;
use warnings;

use LWP::Simple qw/$ua get/;
$ua->agent('Cloudflare Test/0.1');
use Digest::SHA1 qw/sha1_hex/;

uc(sha1_hex($ARGV[0]))=~/^(.{5})(.+)/;
print get("https://api.pwnedpasswords.com/range/$1")=~/$2/?'Pwned':'Ok', "\n";

It's just as easy to implement the same check in other languages, such as JavaScript, which made me realize that I could incorporate the check into a Cloudflare Worker. With a little help from people who know JavaScript far better than me, I wrote the following Worker:

addEventListener('fetch', event => {
  event.respondWith(fetchAndCheckPassword(event.request))
})

async function fetchAndCheckPassword(req) {
  if (req.method == "POST") {
    try {
      const post = await req.formData()
      const pwd = post.get('password')
      const enc = new TextEncoder("utf-8").encode(pwd)

      let hash = await crypto.subtle.digest("SHA-1", enc)
      let hashStr = hex(hash).toUpperCase()
  
      const prefix = hashStr.substring(0, 5)
      const suffix = hashStr.substring(5)

      const pwndpwds = await fetch('https://api.pwnedpasswords.com/range/' + prefix)
      const t =  Continue reading

(This is a crosspost of a blog post originally published on Google Cloud blog)

One of the great things about container technology is that it delivers the same experience and functionality across different platforms. This frees you as a developer from having to rewrite or update your application to deploy it on a new cloud provider—or lets you run it across multiple cloud providers. With a containerized application running on multiple clouds, you can avoid lock-in, run your application on the cloud for which it’s best suited, and lower your overall costs.

If you’re using Kubernetes, you probably manage traffic to clusters and services across multiple nodes using internal load-balancing services, which is the most common and practical approach. But if you’re running an application on multiple clouds, it can be hard to distribute traffic intelligently among them. In this blog post, we show you how to use Cloudflare Load Balancer in conjunction with Kubernetes so you can start to achieve the benefits of a multi-cloud configuration.

To continue reading follow the Google Cloud blog here or if you are ready to get started we created a guide on how to deploy an application using Kubernetes on GCP and AWS Continue reading

We said that we would head to the mountains for Cloudflare’s 123rd data center, and mountains feature prominently as we talk about Kathmandu, Nepal, home of our newest deployment and our 42nd data center in Asia!

Five and three quarter key facts to get started:

• Nepal is home to the highest mountain in the world.
• Kathmandu has more UNESCO heritage sites in its immediate area than any other capital!
• The Nepalese flag isn’t a rectangle. It’s not even close!
• Nepal has never been conquered or ruled by another country.
• Kathmandu, Nepal is where Cloudflare has placed its 123rd data center.
• Nepal’s timezone is 5 hours 45 minutes ahead of GMT.

Mountains

The mountainous nation of Nepal is home to Mount Everest, the highest mountain in the world, known in Nepali as Sagarmāthā. Most of us learn that at school; however there’s plenty of other mountains located in Nepal. Here’s the ones above 8,000 meters (extracted from the full list) to get you started:

• Mount Everest at 8,848 meters
• Kanchenjunga at 8,586 meters
• Lhotse at 8,516 meters
• Makalu at 8,463 meters
• Cho Oyu at 8,201 meters
• Dhaulagiri I at 8,167 meters
• Manaslu at 8,156 meters
• Annapurna I at 8,091 meters

Continue reading

Almost a year ago, I began my journey in the tech industry at a growing company called Cloudflare. I’m a 30-something paralegal and although I didn’t know how to write code (yet), I was highly motivated and ready to crush. I had worked hard for the previous two years, focused on joining a thriving company where I could grow my intelligence, further develop my skill set and work alongside successful professionals. And finally, my hard work paid off; I landed the job at Cloudflare and booked a seat on the rocket ship.

After the initial whirlwind that accompanies this fast-paced field subsided, motivation, inspiration, success, momentum and endurance began to flood my neurons. I loved the inner workings of a successful startup, felt the good and bad of the tech industry, related to and admired the female executives and most importantly, wanted to give something back to the community that adopted me.

Venus Approaching the Sun Source: Flickr

During a routine chat with my dad, I pitched what I thought was a crazy idea. Crazy because I was so used to being told “no” at previous jobs, used to not having my ideas taken seriously, and also used to not Continue reading

Today, v2 of Pwned Passwords was released as part of the Have I Been Pwned service offered by Troy Hunt. Containing over half a billion real world leaked passwords, this database provides a vital tool for correcting the course of how the industry combats modern threats against password security.

I have written about how we need to rethink password security and Pwned Passwords v2 in the following post: How Developers Got Password Security So Wrong. Instead, in this post I want to discuss one of the technical contributions Cloudflare has made towards protecting user information when using this tool.

Cloudflare continues to support Pwned Passwords by providing CDN and security functionality such that the data can easily be made available for download in raw form to organisations to protect their customers. Further; as part of the second iteration of this project, I have also worked with Troy on designing and implementing API endpoints that support anonymised range queries to function as an additional layer of security for those consuming the API, that is visible to the client.

This contribution allows for Pwned Passwords clients to use range queries to search for breached passwords, without having to disclose a complete unsalted Continue reading

Both in our real lives, and online, there are times where we need to authenticate ourselves - where we need to confirm we are who we say we are. This can be done using three things:

• Something you know
• Something you have
• Something you are

Passwords are an example of something you know; they were introduced in 1961 for computer authentication for a time-share computer in MIT. Shortly afterwards, a PhD researcher breached this system (by being able to simply download a list of unencrypted passwords) and used the time allocated to others on the computer.

As time has gone on; developers have continued to store passwords insecurely, and users have continued to set them weakly. Despite this, no viable alternative has been created for password security. To date, no system has been created that retains all the benefits that passwords offer as researchers have rarely considered real world constraints^[1]. For example; when using fingerprints for authentication, engineers often forget that there is a sizable percentage of the population that do not have usable fingerprints or hardware upgrade costs.

Cracking Passwords

In the 1970s, people started thinking about how to better store passwords and cryptographic hashing started to Continue reading

Cloudflare is excited to turn up our newest data center in Phnom Penh, Cambodia, making over 7 million Internet properties even faster. This is our 122nd data center globally, and our 41st data center in Asia. By the end of 2018, we expect that 95% of the world's population will live in a country with a Cloudflare data center, as we grow our global network to span 200 cities.

Cambodian Internet

Home to over 16 million people, Cambodia has a relatively low base of Internet penetration (~25%) today, but is seeing an increasing number of Internet users coming online. For perspective, Cambodia has approximately the same number of Internet users as Lebanon (where we just turned up our 121st data center!) or Singapore (from where we used to serve a portion of Cambodian visitors).

In the coming weeks, we’ll further optimize our routing for Cloudflare customers and expect to see a growing number of ISPs pick up our customers’ traffic on a low latency path.

Latency from a Cambodian ISP (SINET) to Cloudflare customers decreases 10x

Coming up next

Next up, in fact, thousands of feet further up, we head to the mountains for Cloudflare’s 123rd data center. Following Continue reading

At Cloudflare we like Go. We use it in many in-house software projects as well as parts of bigger pipeline systems. But can we take Go to the next level and use it as a scripting language for our favourite operating system, Linux?

gopher image CC BY 3.0 Renee French
Tux image CC0 BY OpenClipart-Vectors

Why consider Go as a scripting language

Short answer: why not? Go is relatively easy to learn, not too verbose and there is a huge ecosystem of libraries which can be reused to avoid writing all the code from scratch. Some other potential advantages it might bring:

• Go-based build system for your Go project: go build command is mostly suitable for small, self-contained projects. More complex projects usually adopt some build system/set of scripts. Why not have these scripts written in Go then as well?
• Easy non-privileged package management out of the box: if you want to use a third-party library in your script, you can simply go get it. And because the code will be installed in your GOPATH, getting a third-party library does not require administrative privileges on the system (unlike some other scripting languages). This is especially useful in large Continue reading

This is a Korean translation of a prior post by Ignat Korchagin.

Cloudflare에서는 Go를 좋아합니다. Go는 많은 내부 소프트웨어 프로젝트와 거대한 파이프라인 시스템의 일부로도 사용되고 있습니다. 하지만 Go를 한단계 더 끌어 올려서 우리가 선호하는 운영체제인 리눅스의 스크립트 언어로 사용할 수 있을까요?

gopher image CC BY 3.0 Renee French | Tux image CC0 BY OpenClipart-Vectors

Go를 왜 스크립트 언어로 고려하는가

간단한 답은: 왜 안되나요? Go는 비교적 쉽게 배울 수 있고 아주 복잡하지도 않고, 코드를 처음부터 작성해야 하는 일을 피하기 위해 재사용 가능한 라이브러리의 거대한 에코시스템이 있습니다. 추가로 다음과 같은 잠재적인 장점이 있습니다:

• 여러분의 Go 프로젝트를 위한 Go 기반 빌드 시스템: go build 명령은 대부분의 소규모이며 독립적인 프로젝트에 적합합니다. 더 복잡한 프로젝트는 대부분 별도의 빌드 시스템/스크립트 세트를 채용하고 있습니다. 이런 스크립트도 Go로 작성 가능하지 않을까요?
• 바로 이용 가능한 별도 권한 없는 패키지 관리: 여러분의 프로그램에서 서드 파티 라이브러리를 사용하고 싶다면 단순히 go get 을 사용하면 됩니다. 그리고 이 코드가 여러분의 GOPATH에 설치되므로, 서드파티 라이브러리를 받는 것은 시스템의 별도 운영 권한을 필요로 하지 않습니다(다른 일부 스크립트 언어와 달리). 이것은 대규모의 기업 환경에서 특히 유용합니다.
• 초기 단계 프로젝트를 위한 빠른 코드 프로토타이핑: 최초로 돌아가는 코드를 작성할 때 컴파일 되기 위해서 많은 편집을 해야 하고 "편집->빌드->체크" 사이클을 위해 많은 키보드 입력을 Continue reading

To everyone in Cloudflare, account security is one of our most important tasks. We recognize that to every customer on our platform, we are critical infrastructure. We also know that the simplest attacks often lead to the most devastating of outcomes. Most people think that if they are going to get hacked it will be by some clever ”zero day”. The reality couldn’t be farther from the truth.

Attackers are smart and they have realized that even in 2018, the human is still the weakest link in the chain. The 2017 Verizon breach report identified that 81% of hacking related breaches occurred as a result of weak credentials or credential theft, an increase from the 63% reported in 2016’s breach report.

Source: Verizon 2017 data breach report

Your credentials are as important as your house or car keys. If someone copies or steals them, the repercussions can be catastrophic. If you suspect someone has access to your house keys you change your locks. If you aren’t fast enough, someone might break in.

Likewise if you realize that someone might have access to your password, the remedy is to change it. Too often, as with house keys, we are slow to Continue reading

Happy Valentine's Day, Internet!

There’s a special place in our heart for all the open source projects that support the Internet and improve the lives of everyone in the developer community, and today seems like an appropriate time to express the gratitude we have for the non-profit / volunteer-run projects that hold everything together.

Cloudflare uses a lot of open source software and also contributes to open source. Informally, Cloudflare has already been upgrading the plans of certain eligible open source projects that have reached out to us or that we have interfaced with. Here are some of the projects whose landing pages are already protected by Cloudflare.

A subset of open source projects on Cloudflare. See more >>

To really pay the goodwill forward, we want to make this opportunity common knowledge in the developer community. In 2018, we intend to provide free Cloudflare Pro Plan upgrades to eligible open source projects (subject to a case-by-case evaluation) that:

1. provide engineering tools or resources to the developer community; and
2. are volunteer-run or working on a non-profit basis.

Are you an open source project using @Cloudflare? We want to give you a free Pro Plan to thank you for your work Continue reading

I’m excited to announce that I’ve joined Cloudflare as Head of Asia. This is an important time for the company as we continue to grow our presence in the region and build on the successes we’ve already had in our Singapore office. In this new role, I’m eager to grow our brand recognition in Asia and optimize our reach to clients by building up teams and channel partners.

A little about me

I’m a Californian with more than 20 years of experience growing businesses across Asia. I initially came to Asia with the Boston Consulting Group and since then I’ve helped Google and Twitter start and grow their businesses in Singapore and Asia. In many cases throughout my career, I’ve been one of the very first employees (sometimes the first) on the ground in this part of the world. To me, the Asian market presents an often untapped opportunity for companies looking to expand, and it’s a challenge that has appealed to me throughout my career.

This year's Chinese New Year celebration

Why Cloudflare?

I’m driven by opportunities to work with global businesses that drive change and are full of ambitious and passionate people. Cloudflare’s mission is to help build Continue reading

Google just announced that beginning in July 2018, with the release of Chrome 68, web pages loaded without HTTPS will be marked as “not secure”.

More than half of web visitors will soon see this warning when browsing unencrypted HTTP sites, according to data from Cloudflare’s edge that shows 56.62% of desktop requests originate from Chrome. Users presented with this warning will be less likely to interact with these sites or trust their content, so it’s imperative that site operators not yet using HTTPS have a plan to do so by July.

How did we get here (and why)?

To those who have followed the Chrome team’s public statements, this announcement comes as no surprise. Google has been gearing up for this change since 2014, as Chrome boss Parisa Tabriz tweeted and Chris Palmer memorialized in a widely distributed email. While this step is an important and potentially jarring one for users, it’s by no means the last step that Google will take to influence website administrator behavior for the better.

But why are they making this change (now)? Google’s primary motivation for driving HTTPS adoption is simple: a safe browsing experience is good for business. Users that feel Continue reading

Lebanon is a historic country, home to two cities among the oldest in the world. There’s a vast mix of influences from the East and West. It’s also the smallest country in continental Asia.

CC-BY-SA Gregor Rom

Lebanon’s connection to the Internet

Lebanon is a little different to most other countries when it comes to the internet, with all connectivity to the outside world flowing via a single network, Ogero. Traffic to Lebanon was previously served from our existing deployments in Marseille and Paris, due to where Ogero connects to the rest of the internet. By deploying locally in Beirut, round-trip latency is cut by around 50 milliseconds. This might seem like almost nothing, but it adds up when you factor in a DNS lookup and 3-way handshake required to open a TCP connection. Internet penetration in Lebanon according to different sources is around 75%, which is quite high. However, the speed available to end users is low, typically in single digit megabits per second.

The Ministry of Telecommunications has an ambitious plan to Continue reading

Courtesty of PublicDomainPictures.net

As we have talked about repeatedly in this blog, we at Cloudflare are not fans of the behavior of patent trolls. They prey upon innovative companies using overly-broad patents in an attempt to bleed settlements out of their targets. When we were first sued by a patent troll called Blackbird Technologies last spring, we decided that we weren’t going along with their game by agreeing to a modest settlement in lieu of going through the considerable effort and expense of litigation. We decided to fight.

We’re happy to report that earlier today, the United States District Court for the Northern District of California dismissed the case that Blackbird brought against Cloudflare. In a two-page order (copied below) Judge Vince Chhabria noted that “[a]bstract ideas are not patentable” and then held that Blackbird’s attempted assertion of the patent “attempts to monopolize the abstract idea of monitoring a preexisting data stream between a server” and is invalid as a matter of law. That means that Blackbird loses no matter what the facts of the case would have been.

The court’s ruling comes in response to a preliminary motion filed by Cloudflare under Section 101 of the U. Continue reading