RPKI and the RTR protocol

Today’s Internet requires stronger protection within its core routing system and as we have already said: it's high time to stop BGP route leaks and hijacks by deploying operationally-excellent RPKI!

Luckily, over the last year plus a lot of good work has happened in this arena. If you’ve been following the growth of RPKI’s validation data, then you’ll know that more and more networks are signing their routes and creating ROA’s or Route Origin Authorizations. These are cryptographically-signed assertions of the validity of an announced IP block and contribute to the further securing of the global routing table that makes for a safer Internet.

The protocol that we have not written much about is RTR. The Resource Public Key Infrastructure (RPKI) to Router Protocol - or RTR Protocol for short. Today we’re fixing that.

RPKI rewind

We have written a few times about RPKI (here and here). We have written about how Cloudflare both signs its announced routes and filters its routing inbound from other networks (both transits and peers) using RPKI data. We also added our efforts in the open-source software space with the release of the Cloudflare RPKI Toolkit.

The primary part of the RPKI (Resource Continue reading

50 Years of The Internet. Work in Progress to a Better Internet

It was fifty years ago when the very first network packet took flight from the Los Angeles campus at UCLA to the Stanford Research Institute (SRI) building in Palo Alto. Those two California sites had kicked-off the world of packet networking, of the Arpanet, and of the modern Internet as we use and know it today. Yet by the time the third packet had been transmitted that evening, the receiving computer at SRI had crashed. The “L” and “O” from the word “LOGIN” had been transmitted successfully in their packets; but that “G”, wrapped in its own packet, caused the death of that nascent packet network setup. Even today, software crashes, that’s a solid fact; but this historic crash, is exactly that — historic.

So much has happened since that day (October 29’th to be exact) in 1969, in fact it’s an understatement to say “so much has happened”! It’s unclear that one blog article would ever be able to capture the full history of packets from then to now. Here at Cloudflare we say we are helping build a “better Internet”, so it would make perfect sense for us to Continue reading

The deep-dive into how Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline Monday

A recap on what happened Monday

On Monday we wrote about a painful Internet wide route leak. We wrote that this should never have happened because Verizon should never have forwarded those routes to the rest of the Internet. That blog entry came out around 19:58 UTC, just over seven hours after the route leak finished (which will we see below was around 12:39 UTC). Today we will dive into the archived routing data and analyze it. The format of the code below is meant to use simple shell commands so that any reader can follow along and, more importantly, do their own investigations on the routing tables.

This was a very public BGP route leak event. It was both reported online via many news outlets and the event’s BGP data was reported via social media as it was happening. Andree Toonk tweeted a quick list of 2,400 ASNs that were affected.

This blog contains a large number of acronyms and those are explained at the end of Continue reading

The deep-dive into how Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline Monday

A recap on what happened Monday

On Monday we wrote about a painful Internet wide route leak. We wrote that this should never have happened because Verizon should never have forwarded those routes to the rest of the Internet. That blog entry came out around 19:58 UTC, just over seven hours after the route leak finished (which will we see below was around 12:39 UTC). Today we will dive into the archived routing data and analyze it. The format of the code below is meant to use simple shell commands so that any reader can follow along and, more importantly, do their own investigations on the routing tables.

This was a very public BGP route leak event. It was both reported online via many news outlets and the event’s BGP data was reported via social media as it was happening. Andree Toonk tweeted a quick list of 2,400 ASNs that were affected.

Using RIPE NCC archived data

The RIPE NCC operates a very useful archive of BGP routing. Continue reading

Lagos, Nigeria – Cloudflare’s 155th city

At just shy of 200 million, Nigeria is the most populous country in Africa (Ethiopia is second and Egypt is third). That’s a lot of people to communicate with the world - and communicate they all do!

According to a published report earlier this year, 84% of the Nigerian population own a mobile device (193 million population and 162 million mobile subscriptions). Again, that’s #1 for any country in Africa. But why so connected? Maybe because Nigeria (and Lagos specifically) is always on the move!

Lagos, as those that know the city say, never sleeps, it’s filled with color from the food to fashion to even the diverse people going about their business. The vibrancy of the city is like a hard slap to the face, no matter what you have been told, your first time here will still knock you out. In Lagos, anything is possible, from the sadness of poverty to the clearly visible upper class, the city sucks you in like a surfers dream wave. Visitor come into Lagos and leave feeling like they’ve been through a unique experience. The traffic is mind blowing and the same goes for the work pace.

Lagos, a city always on Continue reading

Ulaanbaatar, Mongolia

Whenever you get into a conversation about exotic travel or ponder visiting the four corners of the globe, inevitably you end up discussing Ulaanbaatar in Mongolia. Travelers want to experience the rich culture and vivid blue skies of Mongolia; a feature which gives the country its nickname of “Land of the Eternal Blue Sky”.

Ulaanbaatar (or Ulan Bator; but shortened to UB by many) is the capital of Mongolia and located nearly a mile above sea level just outside the Gobi Desert - a desert that spans a good percentage of Central Asia’s Mongolia. (The rest of the Gobi Desert extends into China). The country is nestled squarely between Russia to the north and China to the south. It’s also home to some of the richest and ancient customs and festivals around. It’s those festivals that successfully draw in the tourists who want to experience something quite unique. Luckily, even with all the tourists, Mongolia has managed to keep its local customs; both in the cities and within its nomadic tribes.

via Wikipedia

History also has drawn explorers and conquerors to and from the region; but more on that later.

Cloudflare is also drawn into Mongolia

Any avid reader of Continue reading

RPKI – The required cryptographic upgrade to BGP routing

We have talked about the BGP Internet routing protocol before. We have talked about how we build a more resilient network and how we can see outages at a country-level via BGP. We have even talked about the network community that is vital to the operation of the global Internet.

Today we need to talk about why existing operational practices for BGP routing and filtering have to significantly improve in order to finally stop route leaks and hijacks; which are sadly pervasive in today’s Internet routing world. In fact, the subtle art of running a BGP network and the various tools (both online and within your a networks subsystems) that are vital to making the Internet routing world a safe and reliable place to operate need to improve.

Internet routing and BGP and security along with its operational expertise must improve globally.

Nothing specific triggered today’s writing except the fact that Cloudflare has decided that it's high-time we took a leadership role to finally secure BGP routing. We believe that each and every network needs to change its mindset towards BGP security both on a day-by-day and a long-term basis.

It's time to stop Continue reading

Kathmandu, Nepal is data center 123

We said that we would head to the mountains for Cloudflare’s 123rd data center, and mountains feature prominently as we talk about Kathmandu, Nepal, home of our newest deployment and our 42nd data center in Asia!

Five and three quarter key facts to get started:

• Nepal is home to the highest mountain in the world.
• Kathmandu has more UNESCO heritage sites in its immediate area than any other capital!
• The Nepalese flag isn’t a rectangle. It’s not even close!
• Nepal has never been conquered or ruled by another country.
• Kathmandu, Nepal is where Cloudflare has placed its 123rd data center.
• Nepal’s timezone is 5 hours 45 minutes ahead of GMT.

Mountains

The mountainous nation of Nepal is home to Mount Everest, the highest mountain in the world, known in Nepali as Sagarmāthā. Most of us learn that at school; however there’s plenty of other mountains located in Nepal. Here’s the ones above 8,000 meters (extracted from the full list) to get you started:

• Mount Everest at 8,848 meters
• Kanchenjunga at 8,586 meters
• Lhotse at 8,516 meters
• Makalu at 8,463 meters
• Cho Oyu at 8,201 meters
• Dhaulagiri I at 8,167 meters
• Manaslu at 8,156 meters
• Annapurna I at 8,091 meters

Continue reading

McAllen, Texas: Cloudflare opens 119th Data Center just north of the Mexico border

Five key facts to know about McAllen, Texas

• McAllen, Texas is on the southern tip of the Rio Grande Valley
• The city is named after John McAllen, who provided land in 1904 to bring the St. Louis, Brownsville & Mexico Railway railway into the area
• McAllen, Texas is named the City of Palms
• The border between Mexico and the USA is less than nine miles away from the data center
• McAllen, Texas is where Cloudflare has placed its 119th data center

Second datacenter in Texas; first on the border with Mexico

While McAllen is close to the Mexican border, its importance goes well beyond that simple fact. The city is halfway between Dallas, Texas (where Cloudflare has an existing datacenter) and Mexico City, the center and capital of Mexico. This means that any Cloudflare traffic delivered into Mexico is better served from McAllen. Removing 500 miles from the latency equation is a good thing. 500 miles equates to around 12 milliseconds of round-trip latency and when a connection operates (as all connections should), as a secure connection, then there can be many round trip communications before the first page starts showing up. Improving latency is key, even if we have Continue reading

Cape Town (South Africa): Cloudflare Data Center #113

Five fun facts:

• Cape Town is where the Atlantic Ocean and the Indian Ocean meet deep in the Southern Hemisphere.
  
• The city is the start of the Garden Route, a 185 mile (300 km) glorious coastal drive brimming with native flowers and stunning vistas.
  
• Cape Town is the gateway into the Stellenbosch and Franschhoek wine districts that make for very popular weekend excursions.
  
• The imposing Table Mountain can be seen from most of Cape Town. When you take the cable car to the top of the mountain, the view from up there is even more stunning.
  
• Cape Town is where Cloudflare has placed its 113th data center.
  

Second data center in South Africa

Back in December 2014, Cloudflare opened our first data center in Africa and our 30th datacenter globally. That was in Johannesburg, which has since seen over 10x growth in traffic delivered to South Africa and surrounding countries.

Now, we are expanding into our second city in South Africa — Cape Town, bringing us 870 miles (1,400km) closer to millions of Internet users. Only 15% smaller than Johannesburg by population, Cape Town commands a majority of the tourism business for the country.

For Cloudflare, our newest Continue reading

Curaçao and Djibouti – two new Cloudflare datacenters located where undersea cables meet

Cloudflare has just turned up two new datacenters (numbers 108 and 109). Both are around halfway between the Tropic of Cancer and the Equator. They are located continents-apart, yet share something very-much in common as both of these new data centers are deployed and associated with where undersea cables reach land. Undersea cables have been and still are a growing part of the interconnected world that the Internet represents.

Curaçao, part of the Netherland Antilles in the Caribbean

CC-BY 2.0 image by Nelo Hotsuma

Curaçao is located in the Southern Caribbean Sea (just north of Venezuela) and has a strong Dutch heritage. Along with Aruba and Bonaire, Curaçao is part of the Lesser Antilles (they are called the ABC islands).

More importantly, Willemstad - the capital of Curaçao is where the Amsterdam Internet Exchange operates AMS-IX Caribbean. Why AMS-IX? Because of that Dutch relationship!

It’s AMS-IX’s goal (along with its local partners) to promote Curaçao as an interconnection location for the Caribbean. Cloudflare is there with all its services ready for that day!

Djibouti on the horn of Africa

Djibouti is a country of around 850,000 people with ~60% of the population living in the nation's capital, also Continue reading

NANOG – the art of running a network and discussing common operational issues

The North American Network Operators Group (NANOG) is the loci of modern Internet innovation and the day-to-day cumulative network-operational knowledge of thousands and thousands of network engineers. NANOG itself is a non-profit membership organization; but you don’t need to be a member in order to attend the conference or join the mailing list. That said, if you can become a member, then you’re helping a good cause.

The next NANOG conference starts in a few days (February 6-8 2017) in Washington, DC. Nearly 900 network professionals are converging on the city to discuss a variety of network-related issues, both big and small; but all related to running and improving the global Internet. For this upcoming meeting, Cloudflare has three network professionals in attendance. Two from the San Francisco office and one from the London office.

With the conference starting next week, it seemed a great opportunity to introduce readers of the blog as to why a NANOG conference is so worth attending.

Tutorials

While it seems obvious how to do some network tasks (you unpack the spiffy new wireless router from its box, you set up its security and plug it in); alas the global Internet is somewhat more complex. Continue reading

Not one, not two, but three undersea cables cut in Jersey

Sometime before midnight Monday (UK local time) a ship dropped its anchor and broke, not one, not two, but three undersea cables serving the island of Jersey in the English Channel. Jersey is part of the Channel Islands along with Guernsey and some smaller islands.

Image courtesy TeleGeography Submarine Cable Map

These things happen and that’s not a good thing. The cut was reported on the venerable BBC news website. For the telecom operators in Jersey (JT Global) this wasn’t good news. However looking at the traffic from Cloudflare’s point of view; we can see that while the cable cut removed the direct path from London to Jersey, it was replaced by the backup path from Paris to Jersey. The move was 100% under the control of the BGP routing protocol. It's a relief that there's a fallback for when these unpredictable events happen.

Here's a look at one network on the island.

The red traffic is being served from our London data center (the normal location for all Jersey traffic) and the blue traffic is coming from our Paris data center. The step could well be caused by either a delayed break in one of the cables or the Continue reading

98.01% of sites on Cloudflare now use IPv6

It's 2016 and almost every site using Cloudflare (more than 4 million of them) is using IPv6. Because of this, Cloudflare sees significant IPv6 traffic globally where networks have enabled IPv6 to the consumer.

The top IPv6 networks are shown here.

The chart shows the percentage of IPv6 within a specific network vs. the relative bandwidth of that network. We will talk about specific networks below.

Why IPv6? Because fast internet.

IPv6 is faster for two reasons. The first is that many major operating systems and browsers like iOS, MacOS, Chrome and Firefox impose anywhere from a 25ms to 300ms artificial delay on connections made over IPv4. The second is that some mobile networks won’t need to perform extra v4 -> v6 and v6 -> v4 translations to connect visitors to IPv6 enabled sites if the phone is only assigned an IPv6 address. (IPv6-only phones are becoming very common. If you have a phone on T-Mobile, Telstra, SK Telecom, Orange, or EE UK, to name a few, it’s likely you’re v6-only.)

How much faster is IPv6? Our data shows that visitors connecting over IPv6 were able to connect and load pages in 27% less time than visitors connecting Continue reading

python-cloudflare

Using the CloudFlare API via Python

Very early on in the company’s history we decided that everything that CloudFlare does on behalf of its customer-base should be controllable via an API. In fact, when you login to the CloudFlare control panel, you’re really just making API calls to our backend services. Over time that API has matured and improved. We are now on v4 of that API.

The current CloudFlare API is documented here and it’s used by both the CloudFlare control panel and directly by umpteen customers every minute of every day. The new API is designed with a clean naming structure and consistent data representation for data. It’s also extensible.

This blog entry introduces python-cloudflare, a Python wrapper providing full access to the CloudFlare v4 API.

An example

Let’s get right into the thick-of-it with the simplest coding example available to show python-cloudflare in action. This example lists all your domains (zones) and also checks some basic features for each zone.

#!/usr/bin/env python
import CloudFlare  
def main():  
    cf = CloudFlare.CloudFlare()
    zones = cf.zones.get(params={'per_page':50})
    for zone in zones:
        zone_name = zone['name']
        zone_id = zone['id']
        settings_ipv6 = cf.zones.settings.ipv6.get(zone_id)
        ipv6_on = settings_ipv6['value']
         Continue reading

CloudFlare launches new data centers in Oslo and Minneapolis

Four thousand miles (6,400 kilometers) separate CloudFlare’s latest two data centers: Oslo (#75) and Minneapolis (#76).

Oslo

In Oslo, we have now built our third data center in Scandinavia. This joins our existing facilities in Stockholm and Copenhagen. With a data center in Norway, we recognize an important country that stands above others with a staggering 95.05% of the population having Internet connectivity. This Internet penetration rate is the fourth best in the world. For reference, the Internet penetration rate in the US is 84%, the UK is 90% and Egypt, where we deployed our last data center it is only 50%

At 59.9500° N, Oslo is also the “northernmost” CloudFlare data center on our network map.

Oslo, according to the Norwegian Sagas is over 1,000 years old. CloudFlare has built itself into a facility just a handful of years old and while we respect all the wonderful history and tradition associated with Norway, we hope the locals appreciate our 21st century choice.

Norway has a very important position within the history of the Internet (well the ARPANET actually). In June 1973, the Royal Radar Establishment in Norway became one of the first international connections to Continue reading

Four years later and CloudFlare is still doing IPv6 automatically

Over the past four years CloudFlare has helped well over two million websites join the modern web, making us one of the fastest growing providers of IPv6 web connectivity on the Internet. CloudFlare's Automatic IPv6 Gateway allows IPv4-only websites to support IPv6-only clients with zero clicks. No hardware. No software. No code changes. And no need to change your hosting provider.

Image by Andrew D. Ferguson

A Four Year Story

The story of IPv6 support for customers of CloudFlare is about as long as the story of CloudFlare itself. June 6th, 2011 (four years ago) was the original World IPv6 Day, and CloudFlare participated. Each year since, the global Internet community has pushed forward with additional IPv6 deployment. Now, four years later, CloudFlare is celebrating June 6th knowing that our customers are being provided with a solid IPv6 offering that requires zero configuration to enable. CloudFlare is the only global CDN that provides IPv4/IPv6 delivery of content by default and at scale.

IPv6 has been featured in our blog various times over the last four years. We have provided support for legacy logging systems to handle IPv6 addresses, provided DDoS protection on IPv6 alongside classic IPv4 address space, and provided Continue reading