Archive

Category Archives for "CloudFlare"

SYN packet handling in the wild

SYN packet handling in the wild

Here at Cloudflare, we have a lot of experience of operating servers on the wild Internet. But we are always improving our mastery of this black art. On this very blog we have touched on multiple dark corners of the Internet protocols: like understanding FIN-WAIT-2 or receive buffer tuning.

SYN packet handling in the wild
CC BY 2.0 image by Isaí Moreno

One subject hasn't had enough attention though - SYN floods. We use Linux and it turns out that SYN packet handling in Linux is truly complex. In this post we'll shine some light on this subject.

The tale of two queues

SYN packet handling in the wild

First we must understand that each bound socket, in the "LISTENING" TCP state has two separate queues:

  • The SYN Queue
  • The Accept Queue

In the literature these queues are often given other names such as "reqsk_queue", "ACK backlog", "listen backlog" or even "TCP backlog", but I'll stick to the names above to avoid confusion.

SYN Queue

The SYN Queue stores inbound SYN packets[1] (specifically: struct inet_request_sock). It's responsible for sending out SYN+ACK packets and retrying them on timeout. On Linux the number of retries is configured with:

$ sysctl net.ipv4.tcp_synack_retries
net.ipv4.tcp_synack_retries = 5

The docs describe Continue reading

Welcome Salt Lake City and Get Ready for a Massive Expansion

Welcome Salt Lake City and Get Ready for a Massive Expansion

Welcome Salt Lake City and Get Ready for a Massive Expansion

We just turned up Salt Lake City, Utah — Cloudflare's 120th data center. Salt Lake holds a special place in Cloudflare's history. I grew up in the region and still have family there. Back in 2004, Lee Holloway and I lived just up into the mountains in Park City when we built Project Honey Pot, the open source project that inspired the original idea for Cloudflare.

Welcome Salt Lake City and Get Ready for a Massive Expansion

Salt Lake also holds a special place in the history of the Internet. The University of Utah, based there, was one of the original four Arpanet locations (along with UCLA, UC Santa Barbara, and the Stanford Research Institute). The school also educated the founders of great technology companies like Silicon Graphics, Adobe, Atari, Netscape, and Pixar. Many were graduates of the computer graphics department lead by Professors Ivan Sutherland and David Evans.

Welcome Salt Lake City and Get Ready for a Massive Expansion

In 1980, when I was seven years old, my grandmother, who lived a few blocks from the University, gave me an Apple II+ for Christmas. I took to it like a duck to water. My mom enrolled in a continuing education computer course at the University of Utah teaching BASIC programming. I went with her to the classes. Unbeknownst to the Continue reading

An Explanation of the Meltdown/Spectre Bugs for a Non-Technical Audience

Last week the news of two significant computer bugs was announced. They've been dubbed Meltdown and Spectre. These bugs take advantage of very technical systems that modern CPUs have implemented to make computers extremely fast. Even highly technical people can find it difficult to wrap their heads around how these bugs work. But, using some analogies, it's possible to understand exactly what's going on with these bugs. If you've found yourself puzzled by exactly what's going on with these bugs, read on — this blog is for you.

When you come to a fork in the road, take it.” — Yogi Berra

Late one afternoon walking through a forest near your home and navigating with the GPS you come to a fork in the path which you’ve taken many times before. Unfortunately, for some mysterious reason your GPS is not working and being a methodical person you like to follow it very carefully.

Cooling your heels waiting for GPS to start working again is annoying because you are losing time when you could be getting home. Instead of waiting, you decide to make an intelligent guess about which path is most likely based on past experience and set Continue reading

An Explanation of the Meltdown/Spectre Bugs for a Non-Technical Audience

Last week the news of two significant computer bugs was announced. They've been dubbed Meltdown and Spectre. These bugs take advantage of very technical systems that modern CPUs have implemented to make computers extremely fast. Even highly technical people can find it difficult to wrap their heads around how these bugs work. But, using some analogies, it's possible to understand exactly what's going on with these bugs. If you've found yourself puzzled by exactly what's going on with these bugs, read on — this blog is for you.

When you come to a fork in the road, take it.” — Yogi Berra

Late one afternoon walking through a forest near your home and navigating with the GPS you come to a fork in the path which you’ve taken many times before. Unfortunately, for some mysterious reason your GPS is not working and being a methodical person you like to follow it very carefully.

Cooling your heels waiting for GPS to start working again is annoying because you are losing time when you could be getting home. Instead of waiting, you decide to make an intelligent guess about which path is most likely based on past experience and set Continue reading

Keeping your GDPR Resolutions

Keeping your GDPR Resolutions

For many of us, a New Year brings a renewed commitment to eat better, exercise regularly, and read more (especially the Cloudflare blog). But as we enter 2018, there is a unique and significant new commitment approaching -- protecting personal data and complying with the European Union’s (EU) General Data Protection Regulation (GDPR).

As many of you know by now, the GDPR is a sweeping new EU law that comes into effect on May 25, 2018. The GDPR harmonizes data privacy laws across the EU and mandates how companies collect, store, delete, modify and otherwise process personal data of EU citizens.

Since our founding, Cloudflare has believed that the protection of our customers’ and their end users’ data is essential to our mission to help build a better internet.

Keeping your GDPR ResolutionsImage by GregMontani via Wikimedia Commons

Need a Data Processing Agreement?

As we explained in a previous blog post last August, Cloudflare has been working hard to achieve GDPR compliance in advance of the effective date, and is committed to help our customers and their partners prepare for GDPR compliance on their side. We understand that compliance with a new set of privacy laws can be challenging, and we are here Continue reading

Keeping your GDPR Resolutions

Keeping your GDPR Resolutions

For many of us, a New Year brings a renewed commitment to eat better, exercise regularly, and read more (especially the Cloudflare blog). But as we enter 2018, there is a unique and significant new commitment approaching -- protecting personal data and complying with the European Union’s (EU) General Data Protection Regulation (GDPR).

As many of you know by now, the GDPR is a sweeping new EU law that comes into effect on May 25, 2018. The GDPR harmonizes data privacy laws across the EU and mandates how companies collect, store, delete, modify and otherwise process personal data of EU citizens.

Since our founding, Cloudflare has believed that the protection of our customers’ and their end users’ data is essential to our mission to help build a better internet.

Keeping your GDPR ResolutionsImage by GregMontani via Wikimedia Commons

Need a Data Processing Agreement?

As we explained in a previous blog post last August, Cloudflare has been working hard to achieve GDPR compliance in advance of the effective date, and is committed to help our customers and their partners prepare for GDPR compliance on their side. We understand that compliance with a new set of privacy laws can be challenging, and we are here Continue reading

How “expensive” is crypto anyway?

I wouldn’t be surprised if the title of this post attracts some Bitcoin aficionados, but if you are such, I want to disappoint you. For me crypto means cryptography, not cybermoney, and the price we pay for it is measured in CPU cycles, not USD.

If you got to this second paragraph you probably heard that TLS today is very cheap to deploy. Considerable effort was put to optimize the cryptography stacks of OpenSSL and BoringSSL, as well as the hardware that runs them. However, aside for the occasional benchmark, that can tell us how many GB/s a given algorithm can encrypt, or how many signatures a certain elliptic curve can generate, I did not find much information about the cost of crypto in real world TLS deployments.

CC BY-SA 2.0 image by Michele M. F.

As Cloudflare is the largest provider of TLS on the planet, one would think we perform a lot of cryptography related tasks, and one would be absolutely correct. More than half of our external traffic is now TLS, as well as all of our internal traffic. Being in that position means that crypto performance is critical to our success, and as it Continue reading

How “expensive” is crypto anyway?

I wouldn’t be surprised if the title of this post attracts some Bitcoin aficionados, but if you are such, I want to disappoint you. For me crypto means cryptography, not cybermoney, and the price we pay for it is measured in CPU cycles, not USD.

If you got to this second paragraph you probably heard that TLS today is very cheap to deploy. Considerable effort was put to optimize the cryptography stacks of OpenSSL and BoringSSL, as well as the hardware that runs them. However, aside for the occasional benchmark, that can tell us how many GB/s a given algorithm can encrypt, or how many signatures a certain elliptic curve can generate, I did not find much information about the cost of crypto in real world TLS deployments.

CC BY-SA 2.0 image by Michele M. F.

As Cloudflare is the largest provider of TLS on the planet, one would think we perform a lot of cryptography related tasks, and one would be absolutely correct. More than half of our external traffic is now TLS, as well as all of our internal traffic. Being in that position means that crypto performance is critical to our success, and as it Continue reading

Why TLS 1.3 isn’t in browsers yet

Why TLS 1.3 isn't in browsers yet

Why TLS 1.3 isn't in browsers yet

Upgrading a security protocol in an ecosystem as complex as the Internet is difficult. You need to update clients and servers and make sure everything in between continues to work correctly. The Internet is in the middle of such an upgrade right now. Transport Layer Security (TLS), the protocol that keeps web browsing confidential (and many people persist in calling SSL), is getting its first major overhaul with the introduction of TLS 1.3. Last year, Cloudflare was the first major provider to support TLS 1.3 by default on the server side. We expected the client side would follow suit and be enabled in all major browsers soon thereafter. It has been over a year since Cloudflare’s TLS 1.3 launch and still, none of the major browsers have enabled TLS 1.3 by default.

The reductive answer to why TLS 1.3 hasn’t been deployed yet is middleboxes: network appliances designed to monitor and sometimes intercept HTTPS traffic inside corporate environments and mobile networks. Some of these middleboxes implemented TLS 1.2 incorrectly and now that’s blocking browsers from releasing TLS 1.3. However, simply blaming network appliance vendors would be disingenuous. The deeper truth of the Continue reading

Why TLS 1.3 isn’t in browsers yet

Why TLS 1.3 isn't in browsers yet

Why TLS 1.3 isn't in browsers yet

Upgrading a security protocol in an ecosystem as complex as the Internet is difficult. You need to update clients and servers and make sure everything in between continues to work correctly. The Internet is in the middle of such an upgrade right now. Transport Layer Security (TLS), the protocol that keeps web browsing confidential (and many people persist in calling SSL), is getting its first major overhaul with the introduction of TLS 1.3. Last year, Cloudflare was the first major provider to support TLS 1.3 by default on the server side. We expected the client side would follow suit and be enabled in all major browsers soon thereafter. It has been over a year since Cloudflare’s TLS 1.3 launch and still, none of the major browsers have enabled TLS 1.3 by default.

The reductive answer to why TLS 1.3 hasn’t been deployed yet is middleboxes: network appliances designed to monitor and sometimes intercept HTTPS traffic inside corporate environments and mobile networks. Some of these middleboxes implemented TLS 1.2 incorrectly and now that’s blocking browsers from releasing TLS 1.3. However, simply blaming network appliance vendors would be disingenuous. The deeper truth of the Continue reading

Concise (Post-Christmas) Cryptography Challenges

Concise (Post-Christmas) Cryptography Challenges

It's the day after Christmas; or, depending on your geography, Boxing Day. With the festivities over, you may still find yourself stuck at home and somewhat bored.

Either way; here are three relatively short cryptography challenges, you can use to keep you momentarily occupied. Other than the hints (and some internet searching), you shouldn't require a particularly deep cryptography knowledge to start diving into these challenges. For hints and spoilers, scroll down below the challenges!

Concise (Post-Christmas) Cryptography Challenges

Challenges

Password Hashing

The first one is simple enough to explain; here are 5 hashes (from user passwords), crack them:

$2y$10$TYau45etgP4173/zx1usm.uO34TXAld/8e0/jKC5b0jHCqs/MZGBi
$2y$10$qQVWugep3jGmh4ZHuHqw8exczy4t8BZ/Jy6H4vnbRiXw.BGwQUrHu
$2y$10$DuZ0T/Qieif009SdR5HD5OOiFl/WJaDyCDB/ztWIM.1koiDJrN5eu
$2y$10$0ClJ1I7LQxMNva/NwRa5L.4ly3EHB8eFR5CckXpgRRKAQHXvEL5oS
$2y$10$LIWMJJgX.Ti9DYrYiaotHuqi34eZ2axl8/i1Cd68GYsYAG02Icwve

HTTP Strict Transport Security

A website works by redirecting its www. subdomain to a regional subdomain (i.e. uk.), the site uses HSTS to prevent SSLStrip attacks. You can see cURL requests of the headers from the redirects below, how would you practically go about stripping HTTPS in this example?

$ curl -i http://www.example.com
HTTP/1.1 302 Moved Temporarily  
Server: nginx  
Date: Tue, 26 Dec 2017 12:26:51 GMT  
Content-Type: text/html  
Transfer-Encoding: chunked  
Connection: keep-alive  
location: https://uk.example.com/  
$ curl -i http://uk.example.com
HTTP/1.1 200 OK  
Server: nginx  
Content-Type: text/html;  Continue reading

Concise (Post-Christmas) Cryptography Challenges

Concise (Post-Christmas) Cryptography Challenges

It's the day after Christmas; or, depending on your geography, Boxing Day. With the festivities over, you may still find yourself stuck at home and somewhat bored.

Either way; here are three relatively short cryptography challenges, you can use to keep you momentarily occupied. Other than the hints (and some internet searching), you shouldn't require a particularly deep cryptography knowledge to start diving into these challenges. For hints and spoilers, scroll down below the challenges!

Concise (Post-Christmas) Cryptography Challenges

Challenges

Password Hashing

The first one is simple enough to explain; here are 5 hashes (from user passwords), crack them:

$2y$10$TYau45etgP4173/zx1usm.uO34TXAld/8e0/jKC5b0jHCqs/MZGBi
$2y$10$qQVWugep3jGmh4ZHuHqw8exczy4t8BZ/Jy6H4vnbRiXw.BGwQUrHu
$2y$10$DuZ0T/Qieif009SdR5HD5OOiFl/WJaDyCDB/ztWIM.1koiDJrN5eu
$2y$10$0ClJ1I7LQxMNva/NwRa5L.4ly3EHB8eFR5CckXpgRRKAQHXvEL5oS
$2y$10$LIWMJJgX.Ti9DYrYiaotHuqi34eZ2axl8/i1Cd68GYsYAG02Icwve

HTTP Strict Transport Security

A website works by redirecting its www. subdomain to a regional subdomain (i.e. uk.), the site uses HSTS to prevent SSLStrip attacks. You can see cURL requests of the headers from the redirects below, how would you practically go about stripping HTTPS in this example?

$ curl -i http://www.example.com
HTTP/1.1 302 Moved Temporarily
Server: nginx
Date: Tue, 26 Dec 2017 12:26:51 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
location: https://uk.example.com/
$ curl -i http://uk.example.com
HTTP/1.1 200 OK
Server: nginx
Content-Type: text/html;  Continue reading

The History of Stock Quotes

The History of Stock Quotes

In honor of all the fervor around Bitcoin, we thought it would be fun to revisit the role finance has had in the history of technology even before the Internet came around. This was adapted from a post which originally appeared on the Eager blog.

On 10th of April 1814, almost one hundred thousand troops fought the battle of Toulouse in Southern France. The war had ended on April 6th. Messengers delivering news of Napoleon Is abdication and the end of the war wouldn’t reach Toulouse until April 12th.

The issue was not the lack of a rapid communication system in France, it just hadn’t expanded far enough yet. France had an elaborate semaphore system. Arranged all around the French countryside were buildings with mechanical flags which could be rotated to transmit specific characters to the next station in line. When the following station showed the same flag positions as this one, you knew the letter was acknowledged, and you could show the next character. This system allowed roughly one character to be transmitted per minute, with the start of a message moving down the line at almost 900 miles per hour. It wouldn’t expand to Toulouse until 1834 however, Continue reading

The History of Stock Quotes

The History of Stock Quotes

In honor of all the fervor around Bitcoin, we thought it would be fun to revisit the role finance has had in the history of technology even before the Internet came around. This was adapted from a post which originally appeared on the Eager blog.

On 10th of April 1814, almost one hundred thousand troops fought the battle of Toulouse in Southern France. The war had ended on April 6th. Messengers delivering news of Napoleon Is abdication and the end of the war wouldn’t reach Toulouse until April 12th.

The issue was not the lack of a rapid communication system in France, it just hadn’t expanded far enough yet. France had an elaborate semaphore system. Arranged all around the French countryside were buildings with mechanical flags which could be rotated to transmit specific characters to the next station in line. When the following station showed the same flag positions as this one, you knew the letter was acknowledged, and you could show the next character. This system allowed roughly one character to be transmitted per minute, with the start of a message moving down the line at almost 900 miles per hour. It wouldn’t expand to Toulouse until 1834 however, Continue reading

Simple Cyber Security Tips (for your Parents)

Simple Cyber Security Tips (for your Parents)

Today, December 25th, Cloudflare offices around the world are taking a break. From San Francisco to London and Singapore; engineers have retreated home for the holidays (albeit with those engineers on-call closely monitoring their mobile phones).

Whilst our Support and SRE teams operated on a schedule to ensure fingers were on keyboards; on Saturday, I headed out of the London bound for the Warwickshire countryside. Away from the barracks of the London tech scene, it didn't take long for the following conversation to happen:

  • Family member: "So what do you do nowadays?"
  • Me: "I work in Cyber Security."
  • Family member: "There seems to be a new cyber attack every day on the news! What can I possibly do to keep myself safe?"

If you work in the tech industry, you may find a family member asking you for advice on cybersecurity. This blog post will hopefully save you Continue reading

Simple Cyber Security Tips (for your Parents)

Simple Cyber Security Tips (for your Parents)

Today, December 25th, Cloudflare offices around the world are taking a break. From San Francisco to London and Singapore; engineers have retreated home for the holidays (albeit with those engineers on-call closely monitoring their mobile phones).

Whilst our Support and SRE teams operated on a schedule to ensure fingers were on keyboards; on Saturday, I headed out of the London bound for the Warwickshire countryside. Away from the barracks of the London tech scene, it didn't take long for the following conversation to happen:

  • Family member: "So what do you do nowadays?"
  • Me: "I work in Cyber Security."
  • Family member: "There seems to be a new cyber attack every day on the news! What can I possibly do to keep myself safe?"

If you work in the tech industry, you may find a family member asking you for advice on cybersecurity. This blog post will hopefully save you Continue reading

TLS 1.3 is going to save us all, and other reasons why IoT is still insecure

TLS 1.3 is going to save us all, and other reasons why IoT is still insecure

As I’m writing this, four DDoS attacks are ongoing and being automatically mitigated by Gatebot. Cloudflare’s job is to get attacked. Our network gets attacked constantly.

Around the fall of 2016, we started seeing DDoS attacks that looked a little different than usual. One attack we saw around that time had traffic coming from 52,467 unique IP addresses. The clients weren’t servers or desktop computers; when we tried to connect to the clients over port 80, we got the login pages to CCTV cameras.

Obviously it’s important to lock down IoT devices so that they can’t be co-opted into evil botnet armies, but when we talk to some IoT developers, we hear a few concerning security patterns. We’ll dive into two problematic areas and their solutions: software updates and TLS.

The Trouble With Updates

With PCs, the end user is ultimately responsible for securing their devices. People understand that they need to update their computers and phones. Just 4 months after Apple released iOS 10, it was installed on 76% of active devices.

People just don’t know that they are supposed to update IoT things like they are supposed to update their computers because they’ve never had to update things Continue reading

TLS 1.3 is going to save us all, and other reasons why IoT is still insecure

TLS 1.3 is going to save us all, and other reasons why IoT is still insecure

As I’m writing this, four DDoS attacks are ongoing and being automatically mitigated by Gatebot. Cloudflare’s job is to get attacked. Our network gets attacked constantly.

Around the fall of 2016, we started seeing DDoS attacks that looked a little different than usual. One attack we saw around that time had traffic coming from 52,467 unique IP addresses. The clients weren’t servers or desktop computers; when we tried to connect to the clients over port 80, we got the login pages to CCTV cameras.

Obviously it’s important to lock down IoT devices so that they can’t be co-opted into evil botnet armies, but when we talk to some IoT developers, we hear a few concerning security patterns. We’ll dive into two problematic areas and their solutions: software updates and TLS.

The Trouble With Updates

With PCs, the end user is ultimately responsible for securing their devices. People understand that they need to update their computers and phones. Just 4 months after Apple released iOS 10, it was installed on 76% of active devices.

People just don’t know that they are supposed to update IoT things like they are supposed to update their computers because they’ve never had to update things Continue reading

Technical reading from the Cloudflare blog for the holidays

During 2017 Cloudflare published 172 blog posts (including this one). If you need a distraction from the holiday festivities at this time of year here are some highlights from the year.


CC BY 2.0 image by perzon seo

The WireX Botnet: How Industry Collaboration Disrupted a DDoS Attack

We worked closely with companies across the industry to track and take down the Android WireX Botnet. This blog post goes into detail about how that botnet operated, how it was distributed and how it was taken down.

Randomness 101: LavaRand in Production

The wall of Lava Lamps in the San Francisco office is used to feed entropy into random number generators across our network. This blog post explains how.

ARM Takes Wing: Qualcomm vs. Intel CPU comparison

Our network of data centers around the world all contain Intel-based servers, but we're interested in ARM-based servers because of the potential cost/power savings. This blog post took a look at the relative performance of Intel processors and Qualcomm's latest server offering.

How to Monkey Patch the Linux Kernel

One engineer wanted to combine the Dvorak and QWERTY keyboard layouts and did so by patching the Linux kernel using SystemTap. This blog explains Continue reading

Technical reading from the Cloudflare blog for the holidays

During 2017 Cloudflare published 172 blog posts (including this one). If you need a distraction from the holiday festivities at this time of year here are some highlights from the year.

CC BY 2.0 image by perzon seo

The WireX Botnet: How Industry Collaboration Disrupted a DDoS Attack

We worked closely with companies across the industry to track and take down the Android WireX Botnet. This blog post goes into detail about how that botnet operated, how it was distributed and how it was taken down.

Randomness 101: LavaRand in Production

The wall of Lava Lamps in the San Francisco office is used to feed entropy into random number generators across our network. This blog post explains how.

ARM Takes Wing: Qualcomm vs. Intel CPU comparison

Our network of data centers around the world all contain Intel-based servers, but we're interested in ARM-based servers because of the potential cost/power savings. This blog post took a look at the relative performance of Intel processors and Qualcomm's latest server offering.

How to Monkey Patch the Linux Kernel

One engineer wanted to combine the Dvorak and QWERTY keyboard layouts and did so by patching the Linux kernel using SystemTap. This blog explains Continue reading

1 2 3 24