Doug Madory

Author Archives: Doug Madory

Securing the Internet: Introducing Oracle Internet Intelligence IXP Filter Check

Oracle is an Organization Member of the Internet Society. We welcome this guest post announcing a new tool that complements our work to improve the security of the Internet’s routing infrastructure.

We are proud to announce the launch of the IXP Filter Check, which is designed to improve Internet routing security by monitoring route filtering at Internet Exchange Points (IXPs). Here we describe the origin of this project, how it works, and what it hopes to achieve.

Background

Last year, Oracle started partnering with the Internet Society to explore ways to make the Internet safer and more secure for our enterprise customers and users. Businesses – banks, insurance companies, pharmaceutical firms – as well as non-profit organizations and governments continue to turn to Internet-facing assets as key components of their critical infrastructure. Market research firm IDC estimates that 55.9 billion devices will be online by 2025. We believe it is incumbent upon us, as trusted partners and suppliers, to help make the global Internet as safe as possible.

Securing trust-based Internet routing is one such security challenge. Despite decades of research and engineering on the topic, securing Internet routing remains a notoriously difficult task. The challenge is evidenced by the fact that nearly every month there is another major story of a Continue reading

Cuba’s New 3G Service, Six Years After ALBA-1

Last month, ETECSA (Cuba’s state telecom) activated national 3G mobile service.  For the first time in the nation’s history, a very modest level of internet service is now available to anyone on the island with a 3G-capable device and the funds to pay for it (i.e., 45cuc per month or almost twice the monthly salary of a Cuban state worker).

The development was announced in a tweet from Cuba’s new president Miguel Díaz-Canel and came almost six years since the activation of the ALBA-1 submarine cable connecting Cuba to the global internet via Venezuela.

The activation of Cuba’s mobile internet service appeared in our Internet Intelligence Map as a dramatic increase in the number of authoritative DNS queries handled by Dyn’s servers, as we tweeted below.

Continue reading

China Telecom’s Internet Traffic Misdirection

In recent weeks, the Naval War College published a paper that contained a number of claims about purported efforts by the Chinese government to manipulate BGP routing in order to intercept internet traffic.

In this blog post, I don’t intend to address the paper’s claims around the motivations of these actions. However, there is truth to the assertion that China Telecom (whether intentionally or not) has misdirected internet traffic (including out of the United States) in recent years. I know because I expended a great deal of effort to stop it in 2017.

Traffic misdirection by AS4134

On 9 December 2015, SK Broadband (formerly Hanaro) experienced a brief routing leak lasting little more than a minute. During the incident, SK’s ASN, AS9318, announced over 300 Verizon routes that were picked up by OpenDNS’s BGPstream service:

The leak was announced exclusively through China Telecom (AS4134), one of SK Broadband’s transit providers. Shortly afterwards, AS9318 began transiting the same routes from Verizon APAC (AS703) to China Telecom (AS4134), who in turn began announcing them to international Continue reading

First Subsea Cable Across South Atlantic Activated

Yesterday marked the first time in recent Internet history that a new submarine cable carried live traffic across the South Atlantic, directly connecting South America to Sub-Saharan Africa.  The South Atlantic Cable System (SACS) built by Angola Cables achieved this feat around midday on 18 September 2018.

Our Internet monitoring tools noticed a change in latency between our measurement servers in various Brazilian cities and Luanda, Angola, decreasing from over 300ms to close to 100ms.  Below these are measurements to Angolan telecoms TVCABO (AS36907) and Movicel (AS37081) as the SACS cable came online yesterday.

A race to be first

In the past decade there have been multiple submarine cable proposals to full this gap in international connectivity, such as South Atlantic Express (SAEx) and South Atlantic Inter Link (SAIL) cables.

In recent weeks, the SAIL cable, financed and built by China, announced that they had completed construction of their cable and it was the first cable connecting Brazil to Africa (Cameroon). However, since we haven’t seen any changes in international connectivity for Cameroon, we don’t believe this cable is carrying any traffic yet.

What’s the significance?

In addition to directly connecting Brazil to Portuguese-speaking Angola, the cable offers Continue reading

Civil War in Yemen Begins to Divide Country’s Internet

The latest development in Yemen’s long-running civil war is playing out in the global routing table.  The country’s Internet is now being partitioned along the conflict’s battle lines with the recent activation of a new telecom in government-controlled Aden.

Control of YemenNet

The Iranian-backed Houthi rebels currently hold the nation’s capital Sana’a in the north, while Saudi-backed forces loyal to the president hold the port city of Aden in the south (illustrated in the map below from Al Jazeera).  One advantage the Houthis enjoy while holding Sana’a is the ability to control Yemen’s national operator YemenNet.  Last month, the Houthis cut fiber optic lines severing 80% of Internet service in Yemen.


Launch of AdenNet

In response to the loss of control of YemenNet, the government of President Hadi began plans to launch a new Yemeni telecom, AdenNet, that would provide service to Aden without relying on (or sending revenue to) the Houthi-controlled incumbent operator.  Backed with funding from UAE and built using Huawei gear, AdenNet (AS204317) went live in the past week exclusively using transit from Saudi Telecom (AS39386), as depicted below in a view from Dyn Internet Intelligence.

The new Aden-based telecom Continue reading

BGP/DNS Hijacks Target Payment Systems

In April 2018, we detailed a brazen BGP hijack of Amazon’s authoritative DNS service in order to redirect users of a crypto currency wallet service to a fraudulent website ready to steal their money.

In the past month, we have observed additional BGP hijacks of authoritative DNS servers with a technique similar to what was used in April. This time the targets included US payment processing companies.

As in the Amazon case, these more recent BGP hijacks enabled imposter DNS servers to return forged DNS responses, misdirecting unsuspecting users to malicious sites.  By using long TTL values in the forged responses, recursive DNS servers held these bogus DNS entries in their caches long after the BGP hijack had disappeared — maximizing the duration of the attack.

The Hijacks

At 23:37:18 UTC on 6 July 2018, Digital Wireless Indonesia (AS38146) announced the following prefixes for about thirty minutes.  These prefixes didn’t propagate very far and were only seen by a handful of our peers.

> 64.243.142.0/24 Savvis
> 64.57.150.0/24 Vantiv, LLC
> 64.57.154.0/24 Vantiv, LLC
> 69.46.100.0/24 Q9 Networks Inc.
> 216.220.36.0/24 Q9 Networks Continue reading

Iraq Protests Lead To Two-Day Blackout

After a week of widespread protests against corruption and poor government services, the Iraqi government declared a state of emergency last week.  And as part of that measure, the government ordered the disconnection of the fiber backbone of Iraq that carries traffic for most of the country.

On Monday, Internet services in Iraq were coming back online (however, social media site are still blocked according to independent measurement outfit NetBlocks). The blackout, which lasted almost 48hrs, was clearly visible in our Internet Intelligence Map (screenshot below):

A history of government-directed outages

Government-directed Internet outages have become a part of regular life in Iraq.  Just yesterday, the government ordered its latest national outage to coincide this year’s last 6th grade placement exam.

The first government-directed outage in Iraq that we documented occurred in the fall of 2013 and revolved around a pricing dispute between the Iraqi Ministry of Communications (MoC) and various telecommunications companies operating there.  While the intention of this outage was to enforce the MoC’s authority, it served mainly to reveal the extent to which Iraqi providers were now relying on Kurdish transit providers operating outside the control of the central government – a topic Continue reading

Shutting down the BGP Hijack Factory

It started with a lengthy email to the NANOG mailing list on 25 June 2018: independent security researcher Ronald Guilmette detailed the suspicious routing activities of a company called Bitcanal, whom he referred to as a “Hijack Factory.”  In his post, Ronald detailed some of the Portuguese company’s most recent BGP hijacks and asked the question: why Bitcanal’s transit providers continue to carry its BGP hijacked routes on to the global internet?

This email kicked off a discussion that led to a concerted effort to kick this bad actor, who has hijacked with impunity for many years, off the internet.

Transit Providers

When presented with the most recent evidence of hijacks, transit providers GTT and Cogent, to their credit, immediately disconnected Bitcanal as a customer.  With the loss of international transit, Bitcanal briefly reconnected via Belgian telecom BICS before being disconnected once they were informed of their new customer’s reputation.

The following graphic illustrates a BGP hijack by Bitcanal via Cogent before Cogent disconnected them. Bitcanal’s announcement of 101.124.128.0/18 (Beijing Jingdong 360 Degree E-commerce) was a more-specific hijack of 101.124.0.0/16, normally announced by AS131486 (Beijing Jingdong 360 Degree E-commerce).  Continue reading

Introducing the Internet Intelligence Map

Today, we are proud to announce a new website we’re calling the Internet Intelligence Map. This free site will help to democratize Internet analysis by exposing some of our internal capabilities to the general public in a single tool.

For over a decade, the members of Oracle’s Internet Intelligence team (first born as Renesys, more recently as Dyn Research, and now reborn with David Belson, former author of Akamai’s State of the Internet report) have helped to break some of the biggest stories about the Internet.  From the Internet shutdowns of the Arab Spring to the impacts of the latest submarine cable cut, our continuing mission is to help inform the public by reporting on the technical underpinnings of the Internet and its intersection with, and impact on, geopolitics and e-Commerce.

And since major Internet outages (whether intentional or accidental) will be with us for the foreseeable future, we believe offering a self-serve capability for some of the insights we produce is a great way to move towards a healthier and more accountable Internet.

The website has two sections: Country Statistics and Traffic Shifts.  The Country Statistics section reports any potential Internet disruptions Continue reading

BGP Hijack of Amazon DNS to Steal Crypto Currency

Yesterday morning we posted a tweet (below) that Amazon’s authoritative DNS service had been impacted by a routing (BGP) hijack.  Little did we know this was part of an elaborate scheme to use the inherent security weaknesses of DNS and BGP to pilfer crypto currency, but that remarkable scenario appears to have taken place.

After posting the hijack tweet, I observed reports of a DNS hijack relating to the cryptocurrency website myetherwallet.com and thought the two things might be related:

Sure enough, it appears that eNet/XLHost (AS10297) suffered a breach enabling attackers to impersonate Amazon’s authoritative DNS service.  These attackers used AS10297 to announce five routes used by Amazon’s DNS:


205.251.192.0/24 Amazon.com, Inc.
205.251.193.0/24 Amazon.com, Inc.
205.251.195.0/24 Amazon.com, Inc.
205. Continue reading

China Activates Historic Himalayan Link To Nepal

On 10 January 2018, China Telecom activated a long-awaited terrestrial link to the landlocked country of Nepal.  The new fiber optic connection, which traverses the Himalayan mountain range, alters a significant aspect of Nepal’s exclusive dependency on India, shifting the balance of power (at least for international connectivity) in favor of Kathmandu.

Following a number of brief trials since mid-November, Nepal Telecom fully activated Internet transit from China Telecom at 08:28 UTC on 10 January 2018, as depicted below.

Background

In our 2015 coverage of the earthquake that devastated Nepal, I wrote:

Nepal, as well as Bhutan, are both South Asian landlocked countries wedged between India and China that are dependent on India for a number of services including telecommunications. As a result, each country has been courting Chinese engagement that would provide a redundant source of Internet connectivity.

In December 2016, executives Ou Yan of China Telecom Global (CTG) and Lochan Lal Amatya of Nepal Telecom (pictured below) signed an agreement to route IP service through a new terrestrial cable running between Continue reading

Recent Russian Routing Leak was Largely Preventable

Last week, the IP address space belonging to several high-profile companies, including Google, Facebook and Apple, was briefly announced out of Russia, as was first reported by BGPmon.

Following the incident, Job Snijders of NTT wrote in a post entitled, “What to do about BGP hijacks”.  He stated that, given the inherent security weaknesses in BGP, things will only improve “the moment it becomes socially unacceptable to operate an Internet network without adequate protections in place” and thus customers would stop buying transit from providers that operate without proper route filtering.

Since Job has presented at NANOG about the various filtering methods employed by NTT, I decided to look into how well NTT (AS2914) did in this particular incident.  While a handful of the 80 misdirected routes were ultimately carried on by AS2914 to the greater internet, NTT didn’t contribute to the leaking of any of the major internet companies, such as Facebook, Google, Apple, etc.  In fact, when one analyzes the propagation of every one of these leaked routes, a pattern begins to emerge.

Route Leaks by AS39523

On 12 December 2017, AS39523 announced 80 prefixes (only one of which was theirs) for two different 3-4 Continue reading

Puerto Rico’s Slow Internet Recovery

On 20 September 2017, Hurricane Maria made landfall in Puerto Rico.  Two and a half months later, the island is still recovering from the resulting devastation.  This extended phase of recovery is reflected in the state of the local internet and reveals how far Puerto Rico still has to go to make itself whole again.

While most of the BGP routes for Puerto Rico have returned, DNS query volumes from the island are still only a fraction of what they were on September 19th  — the day before the storm hit.  DNS activity is a better indicator of actual internet use (or lack thereof) than the simple announcements of BGP routes.

We have been analyzing the impacts of natural disasters such as hurricanes and earthquakes going back to Hurricane Katrina in 2005.  Compared to the earthquake near Japan in 2011, Hurricane Sandy in 2012, or the earthquake in Nepal in 2015, Puerto Rico’s disaster stands alone with respect to its prolonged and widespread impact on internet access.  The following analysis tells that story.

DNS statistics

Queries from Puerto Rico to our Internet Guide recursive DNS service have still not recovered to pre-hurricane levels Continue reading

Widespread impact caused by Level 3 BGP route leak

For a little more than 90 minutes yesterday, internet service for millions of users in the U.S. and around the world slowed to a crawl.  Was this widespread service degradation caused by the latest botnet threat?  Not this time.  The cause was yet another BGP routing leak — a router misconfiguration directing internet traffic from its intended path to somewhere else.

While not a day goes by without a routing leak or misconfiguration of some sort on the internet, it is an entirely different matter when the error is committed by the largest telecommunications network in the world.

In this blog post, I’ll describe what happened in this routing leak and some of the impacts.  Unfortunately, there is no silver bullet to completely remove the possibility of these occurring in the future.  As long as we have humans configuring routers, mistakes will take place.

What happened?

At 17:47:05 UTC yesterday (6 November 2017), Level 3 (AS3356) began globally announcing thousands of BGP routes that had Continue reading

North Korea Gets New Internet Link via Russia

This past weekend, North Korea expert Martyn Williams and I spotted the activation of a new internet path out of North Korea.  At 09:07:51 UTC on 1 October 2017, the country’s single internet provider, Star JV (AS131269), gained a new connection to the global internet through Russian fixed-provider Transtelecom (AS20485), often referred to as TTK.  Martyn published his analysis on the US-Korea Institute‘s 38 North blog, named after the dividing line between North and South Korea.

The internet of North Korea is very small (four BGP routes) and reportedly only accessible by a few elites in the country.  Since the appearance of AS131279 in the global routing table almost 7 years ago, Star JV has almost exclusively relied on China Unicom for its connectivity to the global internet — the only exception was its partial usage of satellite service from Intelsat between 2012 and 2013.  In light of this history, a new internet connection out of North Korea is certainly a notable development.

Unsteady Connection

At 09:07:51 UTC, TTK (AS20485) appeared as a transit provider for three of the four BGP routes announced by AS131279, namely, 175.45.176.0/24, 175.45.178.0/24, and Continue reading

Large BGP Leak by Google Disrupts Internet in Japan

At 03:22 UTC on Friday, 25 August 2017, the internet experienced the effects of another massive BGP routing leak.  This time it was Google who leaked over 160,000 prefixes to Verizon, who in turn accepted these routes and passed them on.  Despite the fact that the leak took place in Chicago, Illinois, it had devastating consequences for the internet in Japan, half a world away. Two of Japan’s major telecoms (KDDI and NTT’s OCN) were severely affected, posting outage notices (KDDI / OCN pictured below).

Massive routing leaks continue

In recent years, large-scale (100K+ prefix) BGP routing leaks typically fall into one of two buckets:  the leaker either 1) announces the global routing table as if it is the origin (or source) of all the routes (see Indosat in 2014), or 2) takes the global routing table as learned from providers and/or peers and mistakenly announced it to another provider (see Telekom Malaysia in 2015).

This case is different because the vast majority of the routes involved in this massive routing leak were not in the global routing table at the time but instead were more-specifics of routes that were.  This is an important Continue reading

Ukraine Leaks Russian Social Media Ban

Another development in the long-running conflict between Ukraine and Russia occurred in May of this year when Ukrainian President Petro Poroshenko enacted a ban on Russia’s four most prominent internet companies in the name of national security.  The ban included the two most widely used social media websites, VKontakte (often referred to as the “Russian Facebook“) and Odnoklassniki (“Classmates” in Russian), as well as email service provider Mail.ru and Russian search engine Yandex.

These websites have such a significant Ukrainian user base that Mail.ru says it expects to lose $13 million this year as a result of the ban and Yandex is appealing the ban through Ukraine’s Supreme Administrative Court.

And now it appears that this ban has spilled out into the global routing table.  On 27 July 2017, Ukrainian ISP UARNet (AS3255) began announcing several new BGP routes that were hijacks of the IP address space of these Russian internet companies.  On this day, AS3255 briefly announced more-specific hijacks of each of these four Russian internet companies including 94.100.180.0/24 (Mail.ru), 87.250.250.0/23 (Yandex), 87.240.165.0/24 (Vkontakte) and 217.20.159.0/24 (Odnoklassniki).  While Continue reading

Telecom Heroics in Somalia

Internet service in and around Mogadishu, Somalia suffered a crippling blow recently as the East African Submarine System (EASSy) cable, which provides service to the area, was cut by the anchor of a passing ship.  The government of Somalia estimated that the impact of the submarine cable cut was US$10 million per day and detained the MSC Alice, the cargo vessel that reportedly caused the damage.

The cable was repaired on 17 July. The incident is the latest in a series of recent submarine cable breaks (see Nigeria, Ecuador, Congo-Brazzaville and Vietnam) that remind us how dependent much of the world remains on a limited set of physical connections which maintain connectivity to the global Internet.

Internet in Mogadishu

 

The story of how high-speed Internet service came to Mogadishu is nothing short of remarkable.  It involved Somali telecommunications personnel staring down the threat of a local terrorist group (Al-Shabaab) in order to establish Somalia’s first submarine cable connection.  This submarine cable link would be vital if Mogadishu were to have any hope of improving its local economy and ending decades of violence and hunger.  However, in January 2014, Al-Shabaab Continue reading

Iraq Downs Internet To Combat Cheating…Again!

Earlier this morning, the national fiber backbone of Iraq was taken offline in an effort to combat cheating on 6th grade placement exams.  It was the fourth such outage in the past five days.   2017 marks the third year Iraq has used government-directed internet blackouts to combat cheating on student exams.

These recent outages are a continuation of a growing (and somewhat puzzling) trend by governments in many developing parts of the world to cut communications services in a desperate attempt to staunch rampant cheating on high-stakes student exams.

Iraq’s Exam Blackouts

In the summer of 2015, we broke the story of periodic early-morning outages of the national backbone of Iraq’s internet.  These were the first such government-directed national internet outages to combat cheating on exams and were subsequently covered by publications such as Ars Technica and The Daily Beast.

Last year, Iraq shut down the national fiber backbone ten times in May.  Iraq then conducted a second round of outages in August for make-up exams and then finally directed a third round of outages to coincide with yet another round of make-up exams in October.  (A copy of the government order is pictured above. Continue reading

Cuba Up, Vietnam Down: Recent Subsea Cable Developments

Recent submarine cable-related developments have impacted internet connectivity in locales as diverse as Vietnam, Cuba, India, the Marshall Islands and Russia’s Kamchatka Peninsula.  In this blog post, we report on positive developments in Cuba and Russia and a few notable cable failures in other parts of the world.

Vietnam

The internet of Vietnam got off to a shaky start in 2017 when, on 8 January, the America-Asia Gateway (AAG) submarine cable experienced yet another of its many failures.  In September of last year, Tuoi Tre News reported that AAG had suffered its 10th failure in three years, prompting VietnamNet to ask the question: Why does the AAG underwater cable have to be repaired so often?  Over the years, we have frequently analyzed these cable breaks.  (For example, see this, this or this.)

Internet performance in Ho Chi Minh City suffers greatly during these unfortunate episodes.  For Saigontourist Cable Television (SCTV), the recent break meant a brief disruption in connectivity and the loss of NTT transit as illustrated below.

saigon_tourist
Following this year’s loss of AAG, the TGN-Intra Asia (IA) cable also suffered a failure, compounding the problems for Vietnamese internet connectivity.  Tuoi Continue reading

1 2 3