David Belson

Author Archives: David Belson

Last Month In Internet Intelligence: August 2018

During August 2018, the Oracle Internet Intelligence Map surfaced Internet disruptions around the world due to familiar causes including nationwide exams, elections, maintenance, and power outages. A more targeted disruption due to a DDoS attack was also evident, as were a number of issues that may have been related to submarine cable connectivity. In addition, in a bit of good news, the Internet Intelligence Map also provided evidence of two nationwide trials of mobile Internet services in Cuba.

Cuba

On August 15, the Oracle Internet Intelligence Twitter account highlighted that a surge in DNS queries observed the prior day was related to a nationwide test of mobile Internet service, marking the first time that Internet services were available nationwide in Cuba’s history. The figure below shows two marked peaks in DNS query rates from resolvers located in Cuba during the second half of the day (GMT) on the 14th. Paul Calvano, a Web performance architect at Akamai, also observed a roughly 25% increase in their HTTP traffic to Cuba during the trial period.

This testing was reported by ETECSA (the Cuban state telecommunications company) in a Facebook post in which they noted:

The Telecommunications company of Cuba S.A. Continue reading

Last Month in Internet Intelligence: July 2018

In June, we launched the Internet Intelligence microsite, including the new Internet Intelligence Map. In July, we published the inaugural “Last Month in Internet Intelligence” overview, covering Internet disruptions observed during the prior month. The first summary included insights into exam-related outages and problems caused by fiber cuts. In this month’s summary, covering July, we saw power outages and fiber cuts, as well as exam-related and government-directed shutdowns, disrupt Internet connectivity. In addition, we observed Internet disruptions in several countries where we were unable to ascertain a definitive cause.

Power Outages

It is no surprise that power outages can wreak havoc on Internet connectivity – not every data center or router is connected to backup power, and last mile access often becomes impossible as well.

At approximately 20:00 GMT on July 2, the Internet Intelligence Map Country Statistics view showed a decline in the traceroute completion ratio and DNS query rate for Azerbaijan, related to a widespread blackout. These metrics gradually recovered over the next day. Published reports (Reuters, Washington Post) noted that the blackout was due to an explosion at a hydropower station, following an overload of the electrical system due to increased use Continue reading

Last Month In Internet Intelligence: June 2018

In June, we launched the Internet Intelligence microsite (home of this blog), featuring the new Internet Intelligence Map.  As the associated blog post noted, “This free site will help to democratize Internet analysis by exposing some of our internal capabilities to the general public in a single tool. …. And since major Internet outages (whether intentional or accidental) will be with us for the foreseeable future, we believe offering a self-serve capability for some of the insights we produce is a great way to move towards a healthier and more accountable Internet.”

While we will continue to share information about Internet disruptions and events as they occur via @InternetIntel, we also plan to provide a monthly roundup in a blog post, allowing readers to learn about Internet disruptions and events that they may have missed, while enabling us to provide additional context and insight beyond what fits within Twitter’s character limit.

Exams

In the past, countries including Iraq, Syria, and Ethiopia have implemented partial or complete national Internet shutdowns in an effort to prevent student cheating on exams. This past month saw Iraq implement yet another round of Internet shutdowns, and Algeria began Continue reading

IPv6 Adoption Still Lags In Federal Agencies


On September 28, 2010, Vivek Kundra, Federal CIO at the time, issued a “Transition to IPv6” memorandum noting that “The Federal government is committed to the operational deployment and use of Internet Protocol version 6 (IPv6).” The memo described specific steps for agencies to take to “expedite the operational deployment and use of IPv6”, and laid out target deadlines for key milestones. Of specific note, it noted that agencies shall “Upgrade public/external facing servers and services (e.g. web, email, DNS, ISP services, etc) to operationally use native IPv6 by the end of FY 2012.”

For this sixth “launchiversary” of the World IPv6 Launch event, we used historical Internet Intelligence data collected from Oracle Dyn’s Internet Guide recursive DNS service to examine IPv6 adoption trends across federal agencies both ahead of the end of FY 2012 (September 2012) deadline, as well as after it.

Background

The data set used for this analysis is similar to the one used for the recent “Tracking CDN Usage Through Historical DNS Data” blog post, but in this case, it only includes .gov hostnames. While the memorandum calls out the use of IPv6 for ‘web, email, DNS, ISP services, etc.’, in order Continue reading

Tracking CDN Usage Through Historical DNS Data

With Mother’s Day having just passed, some e-commerce sites likely saw an associated boost in traffic. While not as significant as the increased traffic levels seen around Black Friday and Cyber Monday, these additional visitors can potentially impact the site’s performance if it has not planned appropriately.  Some sites have extra infrastructure headroom and can absorb increased traffic without issue, but others turn to CDN providers to ensure that their sites remain fast and available, especially during holiday shopping periods.

To that end, I thought that it would be interesting to use historical Internet Intelligence data (going back to 2010) collected from Oracle Dyn’s Internet Guide recursive DNS service, to examine CDN usage. As a sample set, I chose the top 50 “shopping” sites listed on Alexa, and looked at which sites are being delivered through CDNs, which CDN providers are most popular, and whether sites change or add providers over time. Although not all of the listed sites would commonly be considered “shopping” sites, as a free and publicly available list from a well-known source, it was acceptable for the purposes of this post.

The historical research was done on the www hostname of the listed Continue reading

SeaMeWe-3 Experiences Another Cable Break

On Thursday, May 10 at approximately 02:00 UTC, the SeaMeWe-3 (SMW-3) subsea cable suffered yet another cable break. The break disrupted connectivity between Australia and Singapore, causing latencies to spike as illustrated below in our Internet Intelligence tool, because traffic had to take a more circuitous path.

The SMW-3 cable has had a history of outages, which we have reported on multiple times in the past, including August 2017, December 2014, and January 2013.

The incident summary posted by cable owner Vocus Communications for this most recent break noted that “There is no ETR at this stage.” However, based on our observations of past outages, time to recovery has been measured on the order of weeks.

While this subsea cable is currently the only one carrying traffic from Western Australia to South East Asia, there are several additional cable projects in process that will help address this long-standing issue. The Australia-Singapore Cable (ASC) Continue reading

ACE Submarine Cable Cut Impacts Ten Countries

The ACE (African Coast to Europe) submarine cable runs along the west coast of Africa between France and South Africa, connecting 22 countries. It extends over 17,000 km, and has a potential capacity of 5.12 Tbps. The cable system is managed by a consortium of 19 telecommunications operators & administrations, and the first phase entered service in December 2012. While it may not have been completely problem-free over the last 5+ years, online searches do not return any published reports of significant outages caused by damage to the cable.

However, on March 30, damage to the cable disrupted Internet connectivity to a number of connected countries, with reported problems posted to social media over the next several days. These posts indicated that the ACE submarine cable was cut near Noukachott, Mauritania, but did not provide any specific information about what severed the cable.

Of the 22 countries listed as having landing points for the ACE Submarine Cable, 10 had significant disruptions evident in Oracle’s Continue reading

Power Failure Leaves Brazilian Internet In The Dark

On Wednesday, March 21, a massive power failure impacted large parts of northern Brazil, leaving tens of millions of people without electricity. Beginning at about 3:40pm local time (18:40 UTC), the outage was reportedly due to the failure of a transmission line near the Belo Monte hydroelectric station.

As occurred in a major power outage in Brazil in 2009, this power failure had a measureable impact on the country’s Internet. This is illustrated below through graphs from Oracle Dyn’s Internet Intelligence team based on BGP and traceroute data, as well as graphs from Akamai’s mPulse service, based on end user Web traffic.

The graphic below depicts the counts of available networks (lower graph) and unstable networks (upper graph) for Brazil in the latter half of March 21. The number of unstable networks spikes around 18:40 UTC as routers of ISPs in Brazil began re-routing traffic away from disabled connections, while the lower graph shows that the corresponding drop in available networks (i.e. routed prefixes) was minor when compared to the total number routes that define the Internet of Brazil.

In addition to aggregating BGP routing information from around the globe, the Internet Intelligence team also performs millions of Continue reading

2017 Internet Intelligence Roundup

With 2017 drawing to a close, year-end lookbacks litter media and the blogosphere like so many leaves on the ground. (Or piles of snow, depending on where you are.) Many tend to focus on pop culture, product/movie/music releases, or professional sports. However, given the focus of Oracle Dyn’s Internet Intelligence team on monitoring and measuring the Internet, we’re going to take a look back at significant Internet “events” of the past year, and how they have impacted connectivity for Internet users around the world.

Hurricanes Harvey, Irma, and Maria Cause Internet Disruptions

In late August, and through September, an active Atlantic hurricane season spawned a number of destructive storms that wreaked havoc across the Caribbean, as well as Florida and Texas in the United States. On the Caribbean islands that were hardest hit by the storms, the resulting physical damage was immense, severely impacting last-mile Internet infrastructure across the whole country. This was also the case in Florida and Texas, though on a much more localized basis. On September 25, we looked at the impacts of these hurricanes on Internet connectivity in the affected areas, noting that while some “core” Internet components remained available during these storms thanks to Continue reading

The Migration of Political Internet Shutdowns

In January 2011, what was arguably the first significant disconnection of an entire country from the Internet took place when routes to Egyptian networks disappeared from the Internet’s global routing table, leaving no valid paths by which the rest of the world could exchange Internet traffic with Egypt’s service providers. It was followed in short order by nationwide disruptions in Bahrain, Libya, and Syria. These outages took place during what became known as the Arab Spring, highlighting the role that the Internet had come to play in political protest, and heralding the wider use of national Internet shutdowns as a means of control.

“How hard is it to disconnect a country from the Internet, really?”

After these events, and another significant Internet outage in Syria, this question led a blog post published in November 2012 by former Dyn Chief Scientist Jim Cowie that examined the risk of Internet disconnection for countries around the world, based on the number of Internet connections at their international border. “You can think of this, to [a] first approximation,” Cowie wrote, “as the number of phone calls (or legal writs, or infrastructure attacks) that would have to be performed in order to Continue reading

What Does “Internet Availability” Really Mean?

The Oracle Dyn team behind this blog have frequently covered ‘network availability’ in our blog posts and Twitter updates, and it has become a common topic of discussion after natural disasters (like hurricanes), man-made problems (including fiber cuts), and political instability (such as the Arab Spring protests). But what does it really mean for the Internet to be “available”? Since the Internet is defined as a network of networks, there are various levels of availability that need to be considered. How does the (un)availability of various networks impact an end user’s experience, and their ability to access the content or applications that they are interested in? How can this availability be measured and monitored?

Deriving Insight From BGP Data

Many Tweets from @DynResearch feature graphs similar to this one, which was included in a September 20 post that noted “Internet connectivity in #PuertoRico hangs by a thread due to effects of #HurricaneMaria.”

There are two graphs shown — “Unstable Networks” and “Number of Available Networks”, and the underlying source of information for those graphs is noted to be BGP Data. The Internet analysis team at Oracle Dyn collects routing information in over 700 locations around the world, giving us Continue reading

Internet Impacts of Hurricanes Harvey, Irma, and Maria

Hurricane Irene - Dyn Uptime

Devastation caused by several storms during the 2017 Atlantic hurricane season has been significant, as Hurricanes Harvey, Irma, and Maria destroyed property and took lives across a number of Caribbean island nations, as well as Texas and Florida in the United States. The strength of these storms has made timely communication of information all the more important, from evacuation orders, to pleas for help and related coordination among first responders and civilian rescuers, to insight into open shelters, fuel stations, and grocery stores. The Internet has become a critical component of this communication, with mobile weather applications providing real-time insight into storm conditions and locations, social media tools like Facebook and Twitter used to contact loved ones or ask for assistance, “walkie talkie” apps like Zello used to coordinate rescue efforts, and “gas tracker” apps like GasBuddy used to crowdsource information about open fuel stations, gas availability, and current prices.

As the Internet has come to play a more pivotal role here, the availability and performance of Internet services has become more important as well.  While some “core” Internet components remained available during these storms thanks to hardened data center infrastructure, backup power generators, and comprehensive disaster planning, local infrastructure Continue reading

Breaking the Internet: Swapping Backhoes for BGP

The term “break[ing] the Internet” has taken hold over the last few years – it sounds significant, and given the role that the Internet has come to play in our daily lives, even a little scary. A Google search for “break the Internet” returns 14.6 million results, while “broke the Internet” returns just under a half million results.

Interestingly, Google Trends shows a spike in searches for the term in November 2014 (arguably representing its entry into mainstream usage), coincident with Kim Kardashian’s appearance in Paper Magazine, and on the magazine’s Web site. (Warning: NSFW) To that end, Time Magazine says “But in the context of viral media content, ‘breaking the Internet’ means engineering one story to dominate Facebook and Twitter at the expense of more newsworthy things.” Presumably in celebration of those efforts, there’s even now a “Break the Internet” Webby Award.

“Breaking the Internet” in this context represents, at best, the failure of a website to do sufficient capacity planning, such as using a content delivery network (CDN) to help improve the scalability and performance of the Web site in the face of increased traffic from a flash crowd from the viral Continue reading