On Thursday, May 10 at approximately 02:00 UTC, the SeaMeWe-3 (SMW-3) subsea cable suffered yet another cable break. The break disrupted connectivity between Australia and Singapore, causing latencies to spike as illustrated below in our Internet Intelligence tool, because traffic had to take a more circuitous path.

The SMW-3 cable has had a history of outages, which we have reported on multiple times in the past, including August 2017, December 2014, and January 2013.

Yesterday morning we posted a tweet (below) that Amazon’s authoritative DNS service had been impacted by a routing (BGP) hijack.  Little did we know this was part of an elaborate scheme to use the inherent security weaknesses of DNS and BGP to pilfer crypto currency, but that remarkable scenario appears to have taken place.

BGP hijack this morning affected Amazon DNS. eNet (AS10297) of Columbus, OH announced the following more-specifics of Amazon routes from 11:05 to 13:03 UTC today:
205.251.192.0/24
205.251.193.0/24
205.251.195.0/24
205.251.197.0/24
205.251.199.0/24

— InternetIntelligence (@InternetIntel) April 24, 2018

After posting the hijack tweet, I observed reports of a DNS hijack relating to the cryptocurrency website myetherwallet.com and thought the two things might be related:

Maybe related to this: https://t.co/6dOrmEuRAz

— Doug Madory (@DougMadory) April 24, 2018

Sure enough, it appears that eNet/XLHost (AS10297) suffered a breach enabling attackers to impersonate Amazon’s authoritative DNS service.  These attackers used AS10297 to announce five routes used by Amazon’s DNS:

The ACE (African Coast to Europe) submarine cable runs along the west coast of Africa between France and South Africa, connecting 22 countries. It extends over 17,000 km, and has a potential capacity of 5.12 Tbps. The cable system is managed by a consortium of 19 telecommunications operators & administrations, and the first phase entered service in December 2012. While it may not have been completely problem-free over the last 5+ years, online searches do not return any published reports of significant outages caused by damage to the cable.

However, on March 30, damage to the cable disrupted Internet connectivity to a number of connected countries, with reported problems posted to social media over the next several days. These posts indicated that the ACE submarine cable was cut near Noukachott, Mauritania, but did not provide any specific information about what severed the cable.

The Sierra Leone Cable Limited (SALCAB) says the data connection to #SierraLeone is partly down due to the ACE Submarine cable cut in Nouakchott, Mauritania. #SierraLeoneDecides

— Leanne de Bassompierre (@leannedb01) April 1, 2018

Of the 22 countries listed as having landing points for the ACE Submarine Cable, 10 had significant disruptions evident in Oracle’s Continue reading

On Wednesday, March 21, a massive power failure impacted large parts of northern Brazil, leaving tens of millions of people without electricity. Beginning at about 3:40pm local time (18:40 UTC), the outage was reportedly due to the failure of a transmission line near the Belo Monte hydroelectric station.

As occurred in a major power outage in Brazil in 2009, this power failure had a measureable impact on the country’s Internet. This is illustrated below through graphs from Oracle Dyn’s Internet Intelligence team based on BGP and traceroute data, as well as graphs from Akamai’s mPulse service, based on end user Web traffic.

The graphic below depicts the counts of available networks (lower graph) and unstable networks (upper graph) for Brazil in the latter half of March 21. The number of unstable networks spikes around 18:40 UTC as routers of ISPs in Brazil began re-routing traffic away from disabled connections, while the lower graph shows that the corresponding drop in available networks (i.e. routed prefixes) was minor when compared to the total number routes that define the Internet of Brazil.

In addition to aggregating BGP routing information from around the globe, the Internet Intelligence team also performs millions of Continue reading

The Oracle Cloud Infrastructure (OCI) team is proud of the data centers and network we are building for the next generation of cloud users and, in the spirit of transparency, we want to share with our users tools to better evaluate and measure the performance they will experience on our cloud. Today, we are pleased to announce two new network tools available in the Console to help you measure and analyze network performance.

OCI Market Performance is an interactive visualization tool that displays network performance metrics from OCI regions to cities around the globe.  Performance is measured over time to a carefully curated set of endpoint IP addresses within the top providers in each market, providing the user with aggregated performance data for markets and providers over the last day, week, month or three months. This latency data from our Phoenix, Ashburn, and Frankfurt regions can help you predict and manage network performance.  We will be adding metrics for more of our regions as they come online (including our twelve recently announced regional data centers).

OCI Market Performance can also assist in planning for growth, as you extend your footprint into new global markets.  For FastConnect customers, this tool can help to predict performance between an OCI region and a specific Continue reading

How did I use over a gigabyte of mobile data in a single day? Why is my phone as warm as a hot plate? If you have ever asked yourself either of these questions, you might be the victim of a malicious application that is using your device and consuming your mobile bandwidth to facilitate ad fraud. We have recently identified a large population of apps being distributed from the Google Play Store that support this behavior.  These apps are installed on devices on a majority of the major cell phone carriers around the world.  These carriers operate in the US (AT&T, Verizon, Sprint, and T-Mobile), Europe (KPN, Vodafone, Ziggo, Sky, Virgin, Talk Talk, BT, O2, and T-Mobile), and the Asia Pacific region (Optus, Telstra, iinet, and others) [Note: Mobile providers and Google have been notified]. Just this morning, before this article was published, Buzzfeed broke another ad fraud story.

The Mechanics of the Grift

Online advertising consists of a complex ecosystem of ad buyers, sellers, exchanges, and data providers. Operators of websites and application authors have available space in their content layout and interaction in the user experience that can be integrated to include various forms of Continue reading

On 10 January 2018, China Telecom activated a long-awaited terrestrial link to the landlocked country of Nepal.  The new fiber optic connection, which traverses the Himalayan mountain range, alters a significant aspect of Nepal’s exclusive dependency on India, shifting the balance of power (at least for international connectivity) in favor of Kathmandu.

Breaking India’s monopoly in providing Internet access to Nepal, China becomes their second service provider. #China #Internethttps://t.co/sQEM7aqCms

— The Hindu (@the_hindu) January 13, 2018

Following a number of brief trials since mid-November, Nepal Telecom fully activated Internet transit from China Telecom at 08:28 UTC on 10 January 2018, as depicted below.

In our 2015 coverage of the earthquake that devastated Nepal, I wrote:

Nepal, as well as Bhutan, are both South Asian landlocked countries wedged between India and China that are dependent on India for a number of services including telecommunications. As a result, each country has been courting Chinese engagement that would provide a redundant source of Internet connectivity.

In December 2016, executives Ou Yan of China Telecom Global (CTG) and Lochan Lal Amatya of Nepal Telecom (pictured below) signed an agreement to route IP service through a new terrestrial cable running between Continue reading

Last week, the IP address space belonging to several high-profile companies, including Google, Facebook and Apple, was briefly announced out of Russia, as was first reported by BGPmon.

Following the incident, Job Snijders of NTT wrote in a post entitled, “What to do about BGP hijacks”.  He stated that, given the inherent security weaknesses in BGP, things will only improve “the moment it becomes socially unacceptable to operate an Internet network without adequate protections in place” and thus customers would stop buying transit from providers that operate without proper route filtering.

Since Job has presented at NANOG about the various filtering methods employed by NTT, I decided to look into how well NTT (AS2914) did in this particular incident.  While a handful of the 80 misdirected routes were ultimately carried on by AS2914 to the greater internet, NTT didn’t contribute to the leaking of any of the major internet companies, such as Facebook, Google, Apple, etc.  In fact, when one analyzes the propagation of every one of these leaked routes, a pattern begins to emerge.

Route Leaks by AS39523

On 12 December 2017, AS39523 announced 80 prefixes (only one of which was theirs) for two different 3-4 Continue reading

With 2017 drawing to a close, year-end lookbacks litter media and the blogosphere like so many leaves on the ground. (Or piles of snow, depending on where you are.) Many tend to focus on pop culture, product/movie/music releases, or professional sports. However, given the focus of Oracle Dyn’s Internet Intelligence team on monitoring and measuring the Internet, we’re going to take a look back at significant Internet “events” of the past year, and how they have impacted connectivity for Internet users around the world.

Hurricanes Harvey, Irma, and Maria Cause Internet Disruptions

In late August, and through September, an active Atlantic hurricane season spawned a number of destructive storms that wreaked havoc across the Caribbean, as well as Florida and Texas in the United States. On the Caribbean islands that were hardest hit by the storms, the resulting physical damage was immense, severely impacting last-mile Internet infrastructure across the whole country. This was also the case in Florida and Texas, though on a much more localized basis. On September 25, we looked at the impacts of these hurricanes on Internet connectivity in the affected areas, noting that while some “core” Internet components remained available during these storms thanks to Continue reading

On 20 September 2017, Hurricane Maria made landfall in Puerto Rico.  Two and a half months later, the island is still recovering from the resulting devastation.  This extended phase of recovery is reflected in the state of the local internet and reveals how far Puerto Rico still has to go to make itself whole again.

While most of the BGP routes for Puerto Rico have returned, DNS query volumes from the island are still only a fraction of what they were on September 19^th  — the day before the storm hit.  DNS activity is a better indicator of actual internet use (or lack thereof) than the simple announcements of BGP routes.

We have been analyzing the impacts of natural disasters such as hurricanes and earthquakes going back to Hurricane Katrina in 2005.  Compared to the earthquake near Japan in 2011, Hurricane Sandy in 2012, or the earthquake in Nepal in 2015, Puerto Rico’s disaster stands alone with respect to its prolonged and widespread impact on internet access.  The following analysis tells that story.

DNS statistics

Queries from Puerto Rico to our Internet Guide recursive DNS service have still not recovered to pre-hurricane levels Continue reading

In January 2011, what was arguably the first significant disconnection of an entire country from the Internet took place when routes to Egyptian networks disappeared from the Internet’s global routing table, leaving no valid paths by which the rest of the world could exchange Internet traffic with Egypt’s service providers. It was followed in short order by nationwide disruptions in Bahrain, Libya, and Syria. These outages took place during what became known as the Arab Spring, highlighting the role that the Internet had come to play in political protest, and heralding the wider use of national Internet shutdowns as a means of control.

“How hard is it to disconnect a country from the Internet, really?”

After these events, and another significant Internet outage in Syria, this question led a blog post published in November 2012 by former Dyn Chief Scientist Jim Cowie that examined the risk of Internet disconnection for countries around the world, based on the number of Internet connections at their international border. “You can think of this, to [a] first approximation,” Cowie wrote, “as the number of phone calls (or legal writs, or infrastructure attacks) that would have to be performed in order to Continue reading

For a little more than 90 minutes yesterday, internet service for millions of users in the U.S. and around the world slowed to a crawl.  Was this widespread service degradation caused by the latest botnet threat?  Not this time.  The cause was yet another BGP routing leak — a router misconfiguration directing internet traffic from its intended path to somewhere else.

On Nov. 6, our network experienced a disruption affecting some IP customers due to a configuration error. All are restored.

— Level 3 Network Ops (@Level3NOC) November 6, 2017

While not a day goes by without a routing leak or misconfiguration of some sort on the internet, it is an entirely different matter when the error is committed by the largest telecommunications network in the world.

In this blog post, I’ll describe what happened in this routing leak and some of the impacts.  Unfortunately, there is no silver bullet to completely remove the possibility of these occurring in the future.  As long as we have humans configuring routers, mistakes will take place.

What happened?

At 17:47:05 UTC yesterday (6 November 2017), Level 3 (AS3356) began globally announcing thousands of BGP routes that had Continue reading

The Oracle Dyn team behind this blog have frequently covered ‘network availability’ in our blog posts and Twitter updates, and it has become a common topic of discussion after natural disasters (like hurricanes), man-made problems (including fiber cuts), and political instability (such as the Arab Spring protests). But what does it really mean for the Internet to be “available”? Since the Internet is defined as a network of networks, there are various levels of availability that need to be considered. How does the (un)availability of various networks impact an end user’s experience, and their ability to access the content or applications that they are interested in? How can this availability be measured and monitored?

Deriving Insight From BGP Data

Many Tweets from @DynResearch feature graphs similar to this one, which was included in a September 20 post that noted “Internet connectivity in #PuertoRico hangs by a thread due to effects of #HurricaneMaria.”

There are two graphs shown — “Unstable Networks” and “Number of Available Networks”, and the underlying source of information for those graphs is noted to be BGP Data. The Internet analysis team at Oracle Dyn collects routing information in over 700 locations around the world, giving us Continue reading

The CNAME resource record was defined in RFC 1035 as “the canonical name for an alias.” It plays the role of a pointer, for example, the CNAME informs the requestor that www.containercult.com is really this other name, instance001.couldbalancer.example.com.

The CNAME record provides a “configure once” point of integration for third party platforms and services. A CNAME is often used as opposed to an A/AAAA record for the same reason developers often use variables in their code as opposed to hard coded values. The CNAME can easily be redefined by the third party or service provider without requiring the end user to make any changes.

A stipulation that prevents use of the CNAME at the apex is that no other records can exist at or alongside a CNAME. This specification is what prevents an end user from being able to place a CNAME at the apex of their zone due to the other records, which must be defined at the apex such as the Start of Authority (SOA).

ALIAS / ANAME – The way of the future 

The Oracle ALIAS record allows for CNAME-like functionality at the apex of a zone. The Oracle implementation of the Continue reading

This past weekend, North Korea expert Martyn Williams and I spotted the activation of a new internet path out of North Korea.  At 09:07:51 UTC on 1 October 2017, the country’s single internet provider, Star JV (AS131269), gained a new connection to the global internet through Russian fixed-provider Transtelecom (AS20485), often referred to as TTK.  Martyn published his analysis on the US-Korea Institute‘s 38 North blog, named after the dividing line between North and South Korea.

The internet of North Korea is very small (four BGP routes) and reportedly only accessible by a few elites in the country.  Since the appearance of AS131279 in the global routing table almost 7 years ago, Star JV has almost exclusively relied on China Unicom for its connectivity to the global internet — the only exception was its partial usage of satellite service from Intelsat between 2012 and 2013.  In light of this history, a new internet connection out of North Korea is certainly a notable development.

Unsteady Connection

At 09:07:51 UTC, TTK (AS20485) appeared as a transit provider for three of the four BGP routes announced by AS131279, namely, 175.45.176.0/24, 175.45.178.0/24, and Continue reading

Devastation caused by several storms during the 2017 Atlantic hurricane season has been significant, as Hurricanes Harvey, Irma, and Maria destroyed property and took lives across a number of Caribbean island nations, as well as Texas and Florida in the United States. The strength of these storms has made timely communication of information all the more important, from evacuation orders, to pleas for help and related coordination among first responders and civilian rescuers, to insight into open shelters, fuel stations, and grocery stores. The Internet has become a critical component of this communication, with mobile weather applications providing real-time insight into storm conditions and locations, social media tools like Facebook and Twitter used to contact loved ones or ask for assistance, “walkie talkie” apps like Zello used to coordinate rescue efforts, and “gas tracker” apps like GasBuddy used to crowdsource information about open fuel stations, gas availability, and current prices.

As the Internet has come to play a more pivotal role here, the availability and performance of Internet services has become more important as well.  While some “core” Internet components remained available during these storms thanks to hardened data center infrastructure, backup power generators, and comprehensive disaster planning, local infrastructure Continue reading

The term “break[ing] the Internet” has taken hold over the last few years – it sounds significant, and given the role that the Internet has come to play in our daily lives, even a little scary. A Google search for “break the Internet” returns 14.6 million results, while “broke the Internet” returns just under a half million results.

Interestingly, Google Trends shows a spike in searches for the term in November 2014 (arguably representing its entry into mainstream usage), coincident with Kim Kardashian’s appearance in Paper Magazine, and on the magazine’s Web site. (Warning: NSFW) To that end, Time Magazine says “But in the context of viral media content, ‘breaking the Internet’ means engineering one story to dominate Facebook and Twitter at the expense of more newsworthy things.” Presumably in celebration of those efforts, there’s even now a “Break the Internet” Webby Award.

“Breaking the Internet” in this context represents, at best, the failure of a website to do sufficient capacity planning, such as using a content delivery network (CDN) to help improve the scalability and performance of the Web site in the face of increased traffic from a flash crowd from the viral Continue reading

At 03:22 UTC on Friday, 25 August 2017, the internet experienced the effects of another massive BGP routing leak.  This time it was Google who leaked over 160,000 prefixes to Verizon, who in turn accepted these routes and passed them on.  Despite the fact that the leak took place in Chicago, Illinois, it had devastating consequences for the internet in Japan, half a world away. Two of Japan’s major telecoms (KDDI and NTT’s OCN) were severely affected, posting outage notices (KDDI / OCN pictured below).

Massive routing leaks continue

In recent years, large-scale (100K+ prefix) BGP routing leaks typically fall into one of two buckets:  the leaker either 1) announces the global routing table as if it is the origin (or source) of all the routes (see Indosat in 2014), or 2) takes the global routing table as learned from providers and/or peers and mistakenly announced it to another provider (see Telekom Malaysia in 2015).

This case is different because the vast majority of the routes involved in this massive routing leak were not in the global routing table at the time but instead were more-specifics of routes that were.  This is an important Continue reading

Another development in the long-running conflict between Ukraine and Russia occurred in May of this year when Ukrainian President Petro Poroshenko enacted a ban on Russia’s four most prominent internet companies in the name of national security.  The ban included the two most widely used social media websites, VKontakte (often referred to as the “Russian Facebook“) and Odnoklassniki (“Classmates” in Russian), as well as email service provider Mail.ru and Russian search engine Yandex.

These websites have such a significant Ukrainian user base that Mail.ru says it expects to lose $13 million this year as a result of the ban and Yandex is appealing the ban through Ukraine’s Supreme Administrative Court.

And now it appears that this ban has spilled out into the global routing table.  On 27 July 2017, Ukrainian ISP UARNet (AS3255) began announcing several new BGP routes that were hijacks of the IP address space of these Russian internet companies.  On this day, AS3255 briefly announced more-specific hijacks of each of these four Russian internet companies including 94.100.180.0/24 (Mail.ru), 87.250.250.0/23 (Yandex), 87.240.165.0/24 (Vkontakte) and 217.20.159.0/24 (Odnoklassniki).  While Continue reading

Protecting end users starts with understanding their use and integration of services. For authoritative DNS, this includes human error when copying and pasting information between interfaces. After purchasing a new domain, such as “containercult.com,” the end user configures authoritative nameservers. Delegation is a “set it and forget it” operation; it is often made outside of scope of continuous integration pipelines and automated deployment systems. To quantify this risk and reconcile it with reality, we started to look at the existence of nameserver record typos in the .COM zone file.

There are typos in nameserver records for a number of authoritative DNS providers made across a number of zones, making it clear that end users make delegation typos. The existence of the typo is one thing, it’s another when the typo has been registered and another provider is serving responses. One of the typos of interest was dynect.ne, which was registered some time in February of 2016. At that time, it was delegated to a pair of authoritative nameservers operated by myhostadmin.net, a name related to a Chinese hosting provider. Sometime around January 2017, the authoritative nameservers changed over to Yandex, the Russian internet services provider, and Continue reading