This post is presented in conjunction with The Internet Society.

February was a surprisingly quiet month for major Internet disruptions. In contrast to previous months, we observed few full outages or multi-day disruptions in the Oracle Internet Intelligence Map during the month. As always, there were a number of brief and unattributed disruptions observed over the course of the month, but the issues highlighted below were related to fiber cuts (and repairs) and likely problems with satellite connectivity. And while not yet a visible disruption, reportssurfaced in February that Russian authorities and major Internet providers are planning to disconnect the country from the global Internet as part of a planned experiment.

Fiber

Kicking off the month, Burkina Faso experienced brief partial disruptions to its Internet connectivity on February 1 & 2, as shown in the Country Statistics graphs below. The disruptions are also evident in the Traffic Shifts graphs below for AS25543 (Onatel), which is the country’s National Office of Telecommunications, holding a monopoly on fixed-line telecommunications there. Facebook posts from Onatel (February 1, 2) indicated that road work between the towns of Sabou and Boromo had resulted in a fiber cut, and subsequent posts made Continue reading

This post is presented in conjunction with The Internet Society.

During the second half of 2018, the causes of significant Internet disruptions observed through the Oracle Internet Intelligence Map could be clustered into a few overarching areas: government-directed, cable problems, power outages, distributed denial of service (DDoS) attacks, and general technical issues. Little changed heading into 2019, with two new government-directed Internet disruptions observed in Africa, alongside disruptions caused by fiber cuts and other network issues that impacted a number of countries around the world.

Government Directed

Initially covered in last month’s overview, the Internet disruption in the Democratic Republic of Congo continued into January, lasting through the third week of the month. Government authorities reportedly cut off Internet access in the country in December to prevent “rumor mongering” in the run-up to presidential elections.

An attempted military coup in Gabon led to a day-long Internet disruption in the country. The disruption started just after 07:00 UTC on January 7, as seen in the figure below, which shows clear declines in the Traceroute Completion Ratio and BGP Routes metrics, as well as a disruption to the usual diurnal pattern seen in the DNS Query Rate metric. Although the coup Continue reading

Closing out 2018, in December the Oracle Internet Intelligence team observed Internet disruptions in countries around the world due to power outages, government direction, technical faults, and possible issues relating to satellite connectivity. While these causes have become relatively common, it is interesting to note that other common reasons for Internet disruptions, including severe weather (such as typhoons and hurricanes), concerns over cheating on exams, and denial-of-service attacks did not appear to drive significant Internet disruptions observed in Oracle’s Internet Intelligence Map during the month. And while we tend to focus on Internet disruptions, it is also important to highlight that after several rounds of testing, nationwide mobile Internet access was finally activated across Cuba.

Cuba

In three tranches (based on the first two digits of a subscriber’s mobile phone number) over December 6, 7, and 8, ETECSA, Cuba’s national telecommunications company, enabled nationwide mobile Internet access. The rollout was reportedly stable, in contrast to the congestion experienced during the trials conducted several months prior. The figure below shows the gradual adoption of this newly available connectivity through changes in the DNS Query Rate. As seen in the graph, the query rate was comparatively low in the days ahead of Continue reading

Last month, ETECSA (Cuba’s state telecom) activated national 3G mobile service.  For the first time in the nation’s history, a very modest level of internet service is now available to anyone on the island with a 3G-capable device and the funds to pay for it (i.e., 45cuc per month or almost twice the monthly salary of a Cuban state worker).

The development was announced in a tweet from Cuba’s new president Miguel Díaz-Canel and came almost six years since the activation of the ALBA-1 submarine cable connecting Cuba to the global internet via Venezuela.

Hoy martes el  Ministro de Comunicaciones anunciará y explicará en la Mesa Redonda el servicio de Internet en los teléfonos. Seguimos avanzando en la informatización de la sociedad #SomosContinuidad #SomosCuba

— Miguel Díaz-Canel Bermúdez (@DiazCanelB) December 4, 2018

The activation of Cuba’s mobile internet service appeared in our Internet Intelligence Map as a dramatic increase in the number of authoritative DNS queries handled by Dyn’s servers, as we tweeted below.

Last week Cuba rolled out its first nationwide mobile internet service. DNS query volume up for Cuba, visible on @Oracle @Internetintel map. https://t.co/tOpkt7hME7 pic.twitter.com/i0uKS2nk7u

— InternetIntelligence (@InternetIntel) December 10, 2018

Continue reading

In November, we saw fewer significant Internet disruptions in the Oracle Internet Intelligence Map as compared to prior months. As usual, there were hundreds of brief issues with limited impact and generally unknown causes, but the most notable issues last month were due to reported DDoS attacks, problems with terrestrial and submarine cables, and general network issues.

DDoS Attacks

On November 4 and 5, several Cambodian ISPs were targeted by DDoS attacks described as the “biggest attacks in Cambodian history.” Published reports indicated that ISPs including EZECOM, SINET, Telcotech, and Digi were targeted by DDoS attacks totaling nearly 150 Gbps, causing subscriber downtime lasting as much as half a day. Disruption from the attacks was visible in the Country Statistics view for Cambodia in the Internet Intelligence Map, as shown in the figure below. However, because Internet connectivity remained generally available (albeit impaired) across the country, the impact appears nominal in the graphs.

However, when viewed at a network level, the impact of the attacks appears to be more significant. SINET, one of the ISPs targeted by the DDoS attacks, posted a Tweet on November 5 letting users know that they were under attack, and followed up Continue reading

The level of significant Internet disruptions observed through the Oracle Internet Intelligence Map was lower in October, though the underlying reasons for these disruptions remained generally consistent compared to prior months. For enterprises, the importance of redundant Internet connectivity and regularly exercised failover plans is clear. Unfortunately, for state-imposed Internet outages, such planning and best practices may need to include failsafes for operations while periodically offline.

Directed disconnection

On October 10, Ethiopian Prime Minister Abiy Ahmed met with several hundred soldiers who had marched on his office to demand increased pay. The Ethiopian Broadcasting Corporation (formerly known as ETV) did not cover the soldiers marching but noted that Internet connectivity within the country had been shut off for several hours to prevent “fake news” from circulating on social media. This aligned with residents’ reports of a three-hour Internet outage. The figure below shows that the disruption began around 12:00 GMT, significantly impacting both traceroutes to, and DNS query traffic from, Ethiopia for several hours.

The impact of the Internet shutdown is also clearly evident in the figure below, which shows traceroutes into Ethio Telecom, the state-owned telecommunications service provider. Similar to the country-level graph shown above, the number of Continue reading

In recent weeks, the Naval War College published a paper that contained a number of claims about purported efforts by the Chinese government to manipulate BGP routing in order to intercept internet traffic.

In this blog post, I don’t intend to address the paper’s claims around the motivations of these actions. However, there is truth to the assertion that China Telecom (whether intentionally or not) has misdirected internet traffic (including out of the United States) in recent years. I know because I expended a great deal of effort to stop it in 2017.

Traffic misdirection by AS4134

On 9 December 2015, SK Broadband (formerly Hanaro) experienced a brief routing leak lasting little more than a minute. During the incident, SK’s ASN, AS9318, announced over 300 Verizon routes that were picked up by OpenDNS’s BGPstream service:

Woah, an ASN in Korea just hijacked a bunch of other ASNs across APAC. pic.twitter.com/46Ih5CaVmi

— Compose Button               Richard Westmoreland (@RSWestmoreland) December 9, 2015

The leak was announced exclusively through China Telecom (AS4134), one of SK Broadband’s transit providers. Shortly afterwards, AS9318 began transiting the same routes from Verizon APAC (AS703) to China Telecom (AS4134), who in turn began announcing them to international Continue reading

Over the course of a given month, hundreds of Internet-impacting “events” are visible within the Oracle Internet Intelligence Map. Many are extremely short-lived, lasting only minutes, while others last for hours or days; some have a minor impact on a single metric, while others significantly disrupt all three metrics. In addition, for some events, the root cause is publicly available/known, while for other events, digging into the underlying data helps us make an educated guess about what happened. Ultimately, this creates challenges in separating the signal from the noise, triaging and prioritizing that month’s events for review in this blog post.

Having said that, in September we observed Internet disruptions due to exams, power outages, extreme weather, and submarine cable issues, as well as a number of others with unknown causes. Additionally, a third test of nationwide mobile Internet connectivity took place in Cuba.

Cuba

As noted in our August post, ETECSA (the Cuban state telecommunications company) carried out two tests of nationwide mobile Internet connectivity, which were evident as spikes in the DNS query rates from Cuba. In a Facebook post, they noted, “On August 14th was a first test that measured levels of traffic congestion and Continue reading

Yesterday marked the first time in recent Internet history that a new submarine cable carried live traffic across the South Atlantic, directly connecting South America to Sub-Saharan Africa.  The South Atlantic Cable System (SACS) built by Angola Cables achieved this feat around midday on 18 September 2018.

Our Internet monitoring tools noticed a change in latency between our measurement servers in various Brazilian cities and Luanda, Angola, decreasing from over 300ms to close to 100ms.  Below these are measurements to Angolan telecoms TVCABO (AS36907) and Movicel (AS37081) as the SACS cable came online yesterday.

A race to be first

In the past decade there have been multiple submarine cable proposals to full this gap in international connectivity, such as South Atlantic Express (SAEx) and South Atlantic Inter Link (SAIL) cables.

In recent weeks, the SAIL cable, financed and built by China, announced that they had completed construction of their cable and it was the first cable connecting Brazil to Africa (Cameroon). However, since we haven’t seen any changes in international connectivity for Cameroon, we don’t believe this cable is carrying any traffic yet.

What’s the significance?

In addition to directly connecting Brazil to Portuguese-speaking Angola, the cable offers Continue reading

During August 2018, the Oracle Internet Intelligence Map surfaced Internet disruptions around the world due to familiar causes including nationwide exams, elections, maintenance, and power outages. A more targeted disruption due to a DDoS attack was also evident, as were a number of issues that may have been related to submarine cable connectivity. In addition, in a bit of good news, the Internet Intelligence Map also provided evidence of two nationwide trials of mobile Internet services in Cuba.

Cuba

On August 15, the Oracle Internet Intelligence Twitter account highlighted that a surge in DNS queries observed the prior day was related to a nationwide test of mobile Internet service, marking the first time that Internet services were available nationwide in Cuba’s history. The figure below shows two marked peaks in DNS query rates from resolvers located in Cuba during the second half of the day (GMT) on the 14^th. Paul Calvano, a Web performance architect at Akamai, also observed a roughly 25% increase in their HTTP traffic to Cuba during the trial period.

This testing was reported by ETECSA (the Cuban state telecommunications company) in a Facebook post in which they noted:

The Telecommunications company of Cuba S.A. Continue reading

Much like air travel, the internet has certain hubs that play important relay functions in the delivery of information. Just as Heathrow Airport serves as a hub for passengers traveling to or from Europe, AMS-IX (Amsterdam Internet Exchange) is a key hub for information getting in or out of Europe. Instead of airline companies gathering in one place to drop off or pick up passengers, it’s internet service providers coming together to swap data – lots and lots of data.

Where the world’s largest internet exchange points (IXPs) reside are mostly where you would expect to find them: advanced economies with sophisticated internet infrastructure. As internet access reached new populations around the world, however, growth in IXPs lagged and traffic tended to make some roundabout, and seemingly irrational, trips to the more established IXPs.

For example, users connected to a server just a few miles away may be surprised to discover that data will cross an entire ocean, turn 180 degrees, and cross that ocean again to arrive at its destination. This occurrence, known as the “boomeranging” or “hair-pinning” (or “trombone effect” due to the path’s shape), is especially true for emerging markets, where local ISPs are less interconnected and Continue reading

The latest development in Yemen’s long-running civil war is playing out in the global routing table.  The country’s Internet is now being partitioned along the conflict’s battle lines with the recent activation of a new telecom in government-controlled Aden.

Control of YemenNet

The Iranian-backed Houthi rebels currently hold the nation’s capital Sana’a in the north, while Saudi-backed forces loyal to the president hold the port city of Aden in the south (illustrated in the map below from Al Jazeera).  One advantage the Houthis enjoy while holding Sana’a is the ability to control Yemen’s national operator YemenNet.  Last month, the Houthis cut fiber optic lines severing 80% of Internet service in Yemen.

Launch of AdenNet

In response to the loss of control of YemenNet, the government of President Hadi began plans to launch a new Yemeni telecom, AdenNet, that would provide service to Aden without relying on (or sending revenue to) the Houthi-controlled incumbent operator.  Backed with funding from UAE and built using Huawei gear, AdenNet (AS204317) went live in the past week exclusively using transit from Saudi Telecom (AS39386), as depicted below in a view from Dyn Internet Intelligence.

The new Aden-based telecom Continue reading

In April 2018, we detailed a brazen BGP hijack of Amazon’s authoritative DNS service in order to redirect users of a crypto currency wallet service to a fraudulent website ready to steal their money.

In the past month, we have observed additional BGP hijacks of authoritative DNS servers with a technique similar to what was used in April. This time the targets included US payment processing companies.

As in the Amazon case, these more recent BGP hijacks enabled imposter DNS servers to return forged DNS responses, misdirecting unsuspecting users to malicious sites.  By using long TTL values in the forged responses, recursive DNS servers held these bogus DNS entries in their caches long after the BGP hijack had disappeared — maximizing the duration of the attack.

The Hijacks

At 23:37:18 UTC on 6 July 2018, Digital Wireless Indonesia (AS38146) announced the following prefixes for about thirty minutes.  These prefixes didn’t propagate very far and were only seen by a handful of our peers.

> 64.243.142.0/24 Savvis
> 64.57.150.0/24 Vantiv, LLC
> 64.57.154.0/24 Vantiv, LLC
> 69.46.100.0/24 Q9 Networks Inc.
> 216.220.36.0/24 Q9 Networks Continue reading

After a week of widespread protests against corruption and poor government services, the Iraqi government declared a state of emergency last week.  And as part of that measure, the government ordered the disconnection of the fiber backbone of Iraq that carries traffic for most of the country.

On Monday, Internet services in Iraq were coming back online (however, social media site are still blocked according to independent measurement outfit NetBlocks). The blackout, which lasted almost 48hrs, was clearly visible in our Internet Intelligence Map (screenshot below):

A history of government-directed outages

Government-directed Internet outages have become a part of regular life in Iraq.  Just yesterday, the government ordered its latest national outage to coincide this year’s last 6th grade placement exam.

The first government-directed outage in Iraq that we documented occurred in the fall of 2013 and revolved around a pricing dispute between the Iraqi Ministry of Communications (MoC) and various telecommunications companies operating there.  While the intention of this outage was to enforce the MoC’s authority, it served mainly to reveal the extent to which Iraqi providers were now relying on Kurdish transit providers operating outside the control of the central government – a topic Continue reading

In June, we launched the Internet Intelligence microsite (home of this blog), featuring the new Internet Intelligence Map.  As the associated blog post noted, “This free site will help to democratize Internet analysis by exposing some of our internal capabilities to the general public in a single tool. …. And since major Internet outages (whether intentional or accidental) will be with us for the foreseeable future, we believe offering a self-serve capability for some of the insights we produce is a great way to move towards a healthier and more accountable Internet.”

While we will continue to share information about Internet disruptions and events as they occur via @InternetIntel, we also plan to provide a monthly roundup in a blog post, allowing readers to learn about Internet disruptions and events that they may have missed, while enabling us to provide additional context and insight beyond what fits within Twitter’s character limit.

Exams

In the past, countries including Iraq, Syria, and Ethiopia have implemented partial or complete national Internet shutdowns in an effort to prevent student cheating on exams. This past month saw Iraq implement yet another round of Internet shutdowns, and Algeria began Continue reading

It started with a lengthy email to the NANOG mailing list on 25 June 2018: independent security researcher Ronald Guilmette detailed the suspicious routing activities of a company called Bitcanal, whom he referred to as a “Hijack Factory.”  In his post, Ronald detailed some of the Portuguese company’s most recent BGP hijacks and asked the question: why Bitcanal’s transit providers continue to carry its BGP hijacked routes on to the global internet?

This email kicked off a discussion that led to a concerted effort to kick this bad actor, who has hijacked with impunity for many years, off the internet.

Transit Providers

When presented with the most recent evidence of hijacks, transit providers GTT and Cogent, to their credit, immediately disconnected Bitcanal as a customer.  With the loss of international transit, Bitcanal briefly reconnected via Belgian telecom BICS before being disconnected once they were informed of their new customer’s reputation.

The following graphic illustrates a BGP hijack by Bitcanal via Cogent before Cogent disconnected them. Bitcanal’s announcement of 101.124.128.0/18 (Beijing Jingdong 360 Degree E-commerce) was a more-specific hijack of 101.124.0.0/16, normally announced by AS131486 (Beijing Jingdong 360 Degree E-commerce).  Continue reading

Today, we are proud to announce a new website we’re calling the Internet Intelligence Map. This free site will help to democratize Internet analysis by exposing some of our internal capabilities to the general public in a single tool.

For over a decade, the members of Oracle’s Internet Intelligence team (first born as Renesys, more recently as Dyn Research, and now reborn with David Belson, former author of Akamai’s State of the Internet report) have helped to break some of the biggest stories about the Internet.  From the Internet shutdowns of the Arab Spring to the impacts of the latest submarine cable cut, our continuing mission is to help inform the public by reporting on the technical underpinnings of the Internet and its intersection with, and impact on, geopolitics and e-Commerce.

And since major Internet outages (whether intentional or accidental) will be with us for the foreseeable future, we believe offering a self-serve capability for some of the insights we produce is a great way to move towards a healthier and more accountable Internet.

The website has two sections: Country Statistics and Traffic Shifts.  The Country Statistics section reports any potential Internet disruptions Continue reading

On September 28, 2010, Vivek Kundra, Federal CIO at the time, issued a “Transition to IPv6” memorandum noting that “The Federal government is committed to the operational deployment and use of Internet Protocol version 6 (IPv6).” The memo described specific steps for agencies to take to “expedite the operational deployment and use of IPv6”, and laid out target deadlines for key milestones. Of specific note, it noted that agencies shall “Upgrade public/external facing servers and services (e.g. web, email, DNS, ISP services, etc) to operationally use native IPv6 by the end of FY 2012.”

For this sixth “launchiversary” of the World IPv6 Launch event, we used historical Internet Intelligence data collected from Oracle Dyn’s Internet Guide recursive DNS service to examine IPv6 adoption trends across federal agencies both ahead of the end of FY 2012 (September 2012) deadline, as well as after it.

Background

The data set used for this analysis is similar to the one used for the recent “Tracking CDN Usage Through Historical DNS Data” blog post, but in this case, it only includes .gov hostnames. While the memorandum calls out the use of IPv6 for ‘web, email, DNS, ISP services, etc.’, in order Continue reading

With Mother’s Day having just passed, some e-commerce sites likely saw an associated boost in traffic. While not as significant as the increased traffic levels seen around Black Friday and Cyber Monday, these additional visitors can potentially impact the site’s performance if it has not planned appropriately.  Some sites have extra infrastructure headroom and can absorb increased traffic without issue, but others turn to CDN providers to ensure that their sites remain fast and available, especially during holiday shopping periods.

To that end, I thought that it would be interesting to use historical Internet Intelligence data (going back to 2010) collected from Oracle Dyn’s Internet Guide recursive DNS service, to examine CDN usage. As a sample set, I chose the top 50 “shopping” sites listed on Alexa, and looked at which sites are being delivered through CDNs, which CDN providers are most popular, and whether sites change or add providers over time. Although not all of the listed sites would commonly be considered “shopping” sites, as a free and publicly available list from a well-known source, it was acceptable for the purposes of this post.

The historical research was done on the www hostname of the listed Continue reading