How a Nigerian ISP Accidentally Knocked Google Offline

Last Monday evening — 12 November 2018 — Google and a number of other services experienced a 74 minute outage. It’s not the first time this has happened; and while there might be a temptation to assume that bad actors are at work, incidents like this only serve to demonstrate just how much frailty is involved in how packets get from one point on the Internet to another.

Our logs show that at 21:12 UTC on Monday, a Nigerian ISP, MainOne, accidentally misconfigured part of their network causing a "route leak". This resulted in Google and a number of other networks being routed over unusual network paths. Incidents like this actually happen quite frequently, but in this case, the traffic flows generated by Google users were so great that they overwhelmed the intermediary networks — resulting in numerous services (but predominantly Google) unreachable.

You might be surprised to learn that an error by an ISP somewhere in the world could result in Google and other services going offline. This blog post explains how that can happen and what the Internet community is doing to try to fix this fragility.

What Is A Route Leak, And How Does One Happen?

Continue reading

Norfolk and Richmond, Virginia: Cloudflare’s 152nd and 153rd cities

Virginia has a very important place in Internet history, as well as the history of Cloudflare’s network.

Northern Virginia, in the area around Ashburn VA, has for a long time been core to Internet infrastructure. In the early 1990’s, MAE-East (Metropolitan-Area-Exchange East) , an Internet Exchange Point (IXP) was established. MAE-East and West were some of the earliest IXPs. Internet Exchange Points are crucial interconnection points for ISPs and other Internet Networks to interconnect and exchange traffic. Eco-systems have grown around these through new data center offerings and new Internet platforms. Like many pieces of the Internet, MAE-East had a humble beginning, though not many humble beginnings grew to handle around 50% of Internet traffic exchange.

Cloudflare’s second Data Center, and one that still plays a critical component in our Global Network was Ashburn, Virginia. Similarly across many organizations, the Northern Virginia area has become a Data Center mecca. Many of the largest Clouds have a substantial amount of their footprint in Northern Virginia. Although MAE-East no longer exists, other Internet Exchange Points have come and grown in its place.

Cloudflare’s network has grown beyond what was traditional Interconnection points, like Ashburn/Northern VA, to a new Edge of the Continue reading

IPv6 in China

Photo by chuttersnap / Unsplash

At the end of 2017, Xinhua reported that there will be 200 Million IPv6 users inside Mainland China by the end of this year. Halfway into the year, we’re seeing a rapid growth in IPv6 users and traffic originating from Mainland China.

Why does this matter?

IPv6 is often referred to the next generation of IP addressing. The reality is, IPv6 is what is needed for addressing today. Taking the largest mobile network in China today, China Mobile has over 900 Million mobile subscribers and over 670 Million 4G/LTE subscribers. To be able to provide service to their users, they need to provide an IP address to each subscriber’s device. This means close to a billion IP addresses would be required, which is far more than what is available in IPv4, especially as the available IP address pools have been exhausted.

What is the solution?

To solve the addressability of clients, many networks, especially mobile networks, will use Carrier Grade NAT (CGN). This allows thousands, possibly up to hundreds of thousands, of devices to be shared behind a single internet IP address. The CGN equipment can be very expensive to scale and further, given the Continue reading

Cebu City, Philippines: Cloudflare’s 138th Data Center

Cebu City, the second most populous metro area, but oldest city in the Philippines is the home of Cloudflare’s newest Data Center.

Located centrally in the Philippines, Cebu has had a long standing tradition of trade and business activity, the word itself “Cebu” meaning trade. It’s central location brings excellent coverage to central and southern Philippines, while our existing location in Manila, serving the Manila Metro and northern Philippines.

Photo by Zany Jadraque / Unsplash

Cebu’s history covers hundreds of years, with rich local culture and international influence dating back from the first Spanish visitors to modern trade and shipping. One of the more popular dishes is Lechon.

Cebu has infamous white sand beaches, in-between making millions of websites and applications faster and more secure for the Philippine internet users, we hope our servers can get some excellent R&R on the famous beaches.

Macau: Cloudflare Data Center 127

Cloudflare's 127th data center is now live in Macau, helping make over 7 million Internet facing applications safer and faster. This is our 44th data center in Asia.

Cloudflare 將在澳門啟用全球第127個數據中心, 幫助超過 7,000,000 客戶的互聯網資產運行得更快、更安全。澳門也是我們在亞洲的第44個數據中心。

O 127º centro de dados da Cloudflare agora está em funcionamento em Macau, ajudando a tornar mais de 7 milhões de aplicações voltadas para a Internet de forma mais segura e rápida. Estamos felizes em compartilhar que este é o nosso 44º centro de dados na Ásia.

CC BY-NC-ND 2.0 image by kidchen915

Blending Chinese and Portuguese culture, just last year, Macau welcomed over 30 million visitors. Visit Macau to experience its unique and extravagant entertainment scene, see scenic spots such as the Ruins of St Paul, Senado Square, attempt the world's highest bungy jump from Macau Tower, or enjoy the foodie paradise Macau delivers!

有著與眾不同的中國及葡萄牙文化融合景觀，澳門至去年為止已經吸引了三千萬遊客來一睹她的風采。你可以拜訪著名的娛樂景觀，像是聖保祿大教堂遺址，議事亭廣場，挑戰澳門的美食，亦可以選擇從全世界最高的高空彈跳地點ㄧ澳門旅遊塔上一躍而下。

Combinando cultura chinesa e portuguesa, o ano passado, Macau recebeu mais de 30 milhões de visitantes. Recomendamos que visite Macau para experimentar a cena de entretenimento única e extravagante, explore os pontos cénica, como ruínas de São Paulo, Praça do Senado, e ainda, tente o "bungee jumping" mais alto do mundo deste da Torre de Macau, ou aproveite Continue reading

Power outage hits the island of Taiwan. Here’s what we learned.

At approximately 4:50pm local time (8:50am UTC) August 15, a major unexpected power outage hit the island of Taiwan with a significant amount of its power generation facilities going down.

Blackout!

Most of the island was hit with power outages, shortages and rolling blackouts, with street lights not functioning, nor power in many of Taipei’s shopping malls, and much other infrastructure.

Blackouts of this scale are very rare. Usually, during an outage of this scale, it would be expected that Internet traffic would greatly drop, as houses and businesses lose power and are unable to connect to the Internet. I’ve experienced this in the past, working at consumer ISPs. As households and businesses lose power, so do their modems or routers which connect them to the Internet.

However, during yesterday's outage, something different happened. I'd like to share some insights from yesterday's outage.

Photo: Taipei 101 Dark during the Blackout -
Source: David Chang/EPA

Even when the power is out, the Internet still operates

Most Telecom and Data Center facilities are built with redundancy in mind and have backup power generation. Our Data Center partner, Chief, was able to switch to backup power generation without any service interruption, allowing Continue reading