Route Leak Causes Major Google Outage
Google recently faced a major outage in many parts of the world thanks to a BGP leak. This incident that was caused by a Nigerian ISP – Mainone – occurred on 12 November 2018 between 21.10 and 22.35 UTC, and was identified in tweets from the BGP monitoring service BGPMon, as well as the network monitoring provider Thousand Eyes.
Google also announced the problem through their status page:
We’ve received a report of an issue with Google Cloud Networking as of Monday, 2018-11-12 14:16 US/Pacific. We have reports of Google Cloud IP addresses being erroneously advertised by internet service providers other than Google. We will provide more information by Monday, 2018-11-12 15:00 US/Pacific.
In order to understand this issue, MainOne Inc (AS37282) is peering at IXPN (Internet Exchange Point of Nigeria) in Lagos where Google (AS151169) and China Telecom (AS4809) are also members.
Google (AS15169) advertise their prefixes (more than 500) through the IXPN Route Server, where PCH (Packet Clearing House) collects a daily snapshot of BGP announcements of IXPN. Unfortunately, 212 prefixes (aggregates of those 500+ announcements) from Google were leaked, which was recorded by BGPMon and RIPEstat.
Looking at the RIPE stats it is evident Continue reading
Popular Destinations rerouted to Russia
Early this morning (UTC) our systems detected a suspicious event where many prefixes for high profile destinations were being announced by an unused Russian Autonomous System.Starting at 04:43 (UTC) 80 prefixes normally announced by organizations such Google, Apple, Facebook, Microsoft, Twitch, NTT Communications and Riot Games were now detected in the global BGP routing tables with an Origin AS of 39523 (DV-LINK-AS), out of Russia.
Looking at timeline we can see two event windows of about three minutes each. The first one started at 04:43 UTC and ended at around 04:46 UTC. The second event started 07:07 UTC and finished at 07:10 UTC.
Even though these events were relatively short lived, they were significant because it was picked up by a large number of peers and because of several new more specific prefixes that are not normally seen on the Internet. So let’s dig a little deeper.
One of the interesting things about this incident is the prefixes that were affected are all network prefixes for well known and high traffic internet organizations. The other odd thing is that the Origin AS 39523 (DV-LINK-AS) hasn’t been seen announcing any prefixes for many years (with one exception below), so why Continue reading
Today’s BGP leak in Brazil
Earlier today several people noticed network reachability problems for networks such as Twitter, Google and others. The root cause turned out to be another BGP mishap.Between 11:09 and 11:27 UTC traffic for many large CDN was rerouted through Brazil. Below an example for the Internet’s most famous prefix 8.8.8.0/24 (Google DNS)Some Google services seem to have been hijacked for roughly 15 minutes. Seen anything? @atoonk @bgpmon @bgpstream
— Fusl Neko Shy Dash (@OhNoItsFusl) October 21, 2017
MTR: https://t.co/RyCoE7zMld pic.twitter.com/DCT2JpKgsc
At 2017-10-21 11:09:59 UTC, AS33362, an US based ISP saw the path towards Google’s 8.8.8.0/24 like this:
33362 6939 16735 263361 15169
This shows the US based network AS33362, would have sent traffic to Google via 6939 (HE) to 16735 (Algar Telecom, Brazil), to 263361 infovale telecom which would have tried to delivered it to Google. The successful delivery of packets would have been unlikely, typically due to congestion which would have been the result of the increase in attracted traffic or an ACL blocking the unexpected traffic.
Below an example from a network connected to the Toronto Internet Exchange, trying to reach 8.8.8.8 at the time Continue reading
BGP leak causing Internet outages in Japan and beyond.
Yesterday some Internet users would have seen issues with their Internet connectivity, experiencing slowness or parts of the Internet as unreachable. This incident hit users in Japan particularly hard and it caused the Internal Affairs and Communications Ministry of Japan to start an investigation into what caused the large-scale internet disruption that slowed or blocked access to websites and online services for dozens of Japanese companies. In this blog post we will take a look at the root cause of these outages, who was affected and what networks were involved. Starting at 03:22 UTC yesterday (aug 25) followers of @BGPstream would have seen an increase in alerts involving Google. The BGPstream alerts were informing us that Google was announcing the peering lan prefixes of a few well known Internet exchanges. This in itself is actually a fairly common type of incident and typically indicates something isn’t quite right within the networks hijacking those prefixes and so these alerts were the first clues that something wasn’t quite right with Google’s BGP advertisements.
A closer look at our data shows not only BGP hijack incidents but also a high number of BGP leak events. A random example is this one: 171. Continue reading
BGPstream and The Curious Case of AS12389
The world of BGP routing is a fascinating place with lots of interesting BGP events happening every day. It can be challenging to keep track of it all and so two years ago we started the BGPstream website where we keep track of large scale outages and BGP hijacks. We list the events, basic info and visualize it with one of my favorite tools: BGPlay. For those who keep an eye on @bgpstream , you probably noticed a curious series of BGP hijacks today all by the same Autonomous system affecting many well known networks.
Starting at April 26 22:36 UTC till approximately 22:43 UTC AS12389 (PJSC Rostelecom) started to originate 50 prefixes for numerous other Autonomous systems. The 50 hijacked prefixes included 37 unique autonomous systems and the complete list of affected networks can be found below. If your organization is in this list feel free to reach out and we can provide more details if needed. Keep in mind that many of these hijacks are already published on BGPstream.com as well.
So back to this incident, what happened here? What makes the list of affected networks ‘curious’ is the high number of financial institutions such as for example: MasterCard,Visa, Fortis,
Large hijack affects reachability of high traffic destinations
Starting today at 17:09 UTC our systems detected a large scale routing incident affecting hundreds of Autonomous systems. Many BGPmon users have received an email informing them of this change.
Our initial investigation shows that the scope of this incident is widespread and affected 576 Autonomous systems and 3431 prefixes. Amongst the networks affected are high traffic prefixes including those of Google, Amazon, Twitter, Apple, Akamai, Time Warner Cable Internet and more.
All these events have either AS200759 “innofield AG” or private AS 65021 as the origin AS. In the cases where AS65021 appears as the origin AS, AS200759 is again the next-hop AS.
AS200759 “innofield AG” is a provider based out of Switzerland and normally only announces one IPv4 and one IPv6 prefix.
These are 2 example events:
Prefix 66.220.152.0/21 Is normally announced by Facebook AS32934 and during this event was announced by AS200759 as a more specific /22
Detected prefix: 66.220.152.0/22
Example aspath: 4608 24130 7545 6939 200759
And AS origin: 65021 behind AS 200759
Detected prefix: 66.220.152.0/22
Example aspath: 133812 23948 4788 6939 200759 65021
We saw the announcements via the following peers and providers of AS200759 “innofield Continue reading
Country wide outage in Azerbaijan
It doesn’t happen often that a country with hundreds of prefixes is affected by a massive outage, however earlier today this unfortunately happened to Azerbaijan. Starting at 12:04 UTC approximately 94% of the prefixes out of Azerbaijan became unreachable. At the time of writing the outage is still active.
The event was reported on @bgpstream and details plus a replay can be found here: https://bgpstream.com/event/7981
The image below shows the impact on traffic from Azerbaijan to OpenDNS. It’s clear that almost all of the traffic from Azerbaijan disappeared at the time of outage.
Traffic from Azerbaijan
The main Internet Service provider in Azerbaijan is AS29049, Delta Telecom Ltd. The majority of the country relies on Delta Telecom for connectivity to the rest of the Internet. The outage is reportedly the result of a fire damaging the equipment of Delta Telecom. As a result all routes for AS29049, Delta Telecom Ltd. and all of the networks that rely on Delta Telecom disappeared from the Internet.
The graph below shows the number of prefixes observed in the BGP routing tables for Azerbaijan. Clearly visible is the drop in reachable networks starting at 12:04.
Visible BGP prefixes for Azerbaijan
Networks out of Continue reading
Large scale BGP hijack out of India
BGP hijacks happen every day, some of them affect more networks than others and every now and then there’s a major incident that affects thousands of networks. Our monitoring systems keep an eye out for our users and if you would like to have a general idea of what’s going on in the world of BGP incidents, keep an eye on BGPstream.com. Earlier today we detected one of those major incidents that affected thousands of networks.
Starting at 05:52 UTC, AS9498 (BHARTI Airtel Ltd.) started to claim ownership for thousands of prefixes by originating them in BGP. This affected prefixes for over two thousand unique organizations (Autonomous systems).
Our systems detected origin AS changes (hijacks) for 16,123 prefixes. The scope and impact was different per prefix but to give you an idea, about 7,600 of these announcements were seen by five or more of our peers (unique peers ASns) and 6,000 of these were seen by more than 10 of our peers.
One of the reasons this was so widespread is because large networks such as AS174 (Cogent Communications) and AS52320 (GlobeNet Cabos Submarinos VZLA) accepted and propagated these prefixes to their peers and customers.
The BGPplay visualization Continue reading
How to Make a Science Reasonable Research Paper
They are simply not a reality of what faculty is really like although movies like Dog House are unquestionably engaging. Some college students might differ with me, but faculty is not about becoming successful socially, all, instead, it’s about being successful academically. Your freshman year is your vital year of school. For all, it fails them as university students or makes them. Students who wander onto university their year thinking that faculty is one occasion that is large come in to get a rude awakening, particularly if they’ve been hearing upperclassmen reveal reports about waiting before the last second to create that report and not going to course. Continue reading
How to Publish a Two Page Paper in a Single Day
A dog photo in a “poop argument” has enraged visitors of a SWAT group standoff when nearly several users of the Wisconsin police SWAT team turned up, positioned themselves behind an armored vehicle, and pumped two fits right into a little, end-wagging puppy. The SWAT were termed following the pet, who lowered a deuce within the wrong position, started a defecation conflict between his neighbors and an man. Writes the NY Daily Information on Nov. 3: ” SWAT crew delivered to manage a town argument over waste ended up igniting a firestorm of criticism from the nearby police force, and harming canine. Continue reading
How Hacking Team Helped Italian Special Operations Group with BGP Routing Hijack
As part of the Hacking Team fall out and all the details published on wikileaks, it became public knowledge that Hacking Team helped one of their customers Special Operations Group (ROS), regain access to Remote Access Tool (RAT) clients. ROS recommended using BGP hijacking and Hacking Team helped with the setup of new RAT CnC servers.
In this post we’ll take a closer look at the exact details of this incident and support the wikileaks findings with BGP data.
Raggruppamento Operativo Speciale and Hacking Team
The Raggruppamento Operativo Speciale or ROS is the Special Operations Group of the Italian National Military police. The group focuses on investigating organized crime and terrorism. Hacking Team sells its RAT software known as Remote Control System (RCS) to law enforcement and intelligence agencies, ROS included.
ROS infected and installed the RCS client on the machines of persons of interest (referred to in the emails as targets). These Remote Access Tools can provide ROS with all kinds of information and typically provide the tool’s operator with full access over a victim’s machine. The RCS clients normally need to check in with a server —for example a machine the clients can get their commands (orders) from— Continue reading
Massive route leak causes Internet slowdown
Earlier today a massive route leak initiated by Telekom Malaysia (AS4788) caused significant network problems for the global routing system. Primarily affected was Level3 (AS3549 – formerly known as Global Crossing) and their customers. Below are some of the details as we know them now.
Starting at 08:43 UTC today June 12th, AS4788 Telekom Malaysia started to announce about 179,000 of prefixes to Level3 (AS3549, the Global crossing AS), whom in turn accepted these and propagated them to their peers and customers. Since Telekom Malaysia had inserted itself between these thousands of prefixes and Level3 it was now responsible for delivering these packets to the intended destinations.
This event resulted in significant packet loss and Internet slow down in all parts of the world. The Level3 network in particular suffered from severe service degradation between the Asia pacific region and the rest of their network. The graph below for example shows the packet loss as measured by OpenDNS between London over Level3 and Hong Kong. The same loss patterns were visible from other Level3 locations globally to for example Singapore, Hong Kong and Sydney.
Packet loss London to Hong Kong over Level3
At the same time the round trip time between these Continue reading
BGP Optimizer Causes Thousands Of Fake Routes
Earlier today many BGPmon users received one or more alerts informing them that their autonomous system (AS) started to announce a more-specific prefix. BGPmon classified many of these alerts as possible BGP man-in-the-middle (MITM) attacks. Here is an example alert:
====================================================================
Possible BGP MITM attack (Code: 21)
====================================================================
Your prefix: 23.20.0.0/15:
Prefix Description: acxiom-online.com --- Amazon EC2 IAD prefix
Update time: 2015-03-26 11:27 (UTC)
Detected by #peers: 24
Detected prefix: 23.21.112.0/20
Announced by: AS14618 (AMAZON-AES - Amazon.com, Inc.,US)
Upstream AS: AS3257 (TINET-BACKBONE Tinet SpA,DE)
ASpath: 4608 24130 7545 6939 40633 18978 3257 14618
The alert shows the user was monitoring 23.20.0.0/15, normally announced by Amazon, Inc. (AS14618). In this case however, the detected prefix was the more specific 23.21.112.0/20. The netblock owners would have verified their BGP announcements and quickly recognized they did not originate this more-specific prefix. Further analysis pointed to the suspicion that a bad actor was impersonating Amazon. BGPmon algorithms alerted to this as well, and–within moments of the initial change–marked these events as a possible BGP MITM attack.
One reason for this classification is the way BGPmon understands and interprets AS Continue reading
BGPMon Joins OpenDNS
Dear BGPmon.net user,
I’m excited to announce that BGPmon has been acquired by OpenDNS. OpenDNS is a leading cloud-delivered network security company known for engineering predictive intelligence technology that stops malicious activity before it can threaten a network.
Over the last few years BGPmon has grown from a community service into a successful business that helps thousands of network engineers from around the world monitor their networks. Throughout this journey, we’ve developed close relationships with many of you and together, worked on some truly fascinating cases.
Becoming a part of OpenDNS is a logical next step for BGPmon. With its engineering resources, massive scale and cloud delivery model, OpenDNS is the right direction to continue growing the BGPmon service. I’m confident that moving forward BGPmon will only get better.
The transition plan is straightforward. OpenDNS will invest in building out the service even more but also is committed to keeping the free features free. Simply put, nothing regarding the service will change other than we’ll continue adding new functionality.
On a personal note, I’d like to thank all of you for your continued support and encouragement. I am excited for the changes ahead and personally being a part of Continue reading
What caused the Google service interruption?
This morning people on twitter reported that they were unable to reach Google services. Businessinsider followed up with a story in which they mentioned that the Google service interruption primarily involved European and Indian users.
In this blog we’ll take a quick look at what exactly happened by looking at our BGP data. The first clue comes from David Roy and Franck Klopfenstein on twitter who noticed traffic was re-routed towards AS9498 in India. Digging through our BGP data we are able to indeed confirm that routing paths for many google prefixes changed to a path that includes the Indian AS 9498 between 08:58 UTC and 09:14 UTC.
Let’s take a look at an example. In my case www.google.com resolves to the following addresses:
www.google.com has address 74.125.226.19
www.google.com has address 74.125.226.20
www.google.com has address 74.125.226.17
www.google.com has address 74.125.226.16
www.google.com has address 74.125.226.18
www.google.com has IPv6 address 2607:f8b0:4006:806::1014
The IPv4 addresses are all in the 74.125.226.0/24 range. If we now look at the BGP announcements for that Continue reading
BGP routing incidents in 2014, malicious or not?
Over the last year we have seen and written about numerous BGP routing incidents that looked out of the ordinary, straight-up suspicious or were just configuration mistakes. In this blog post we will highlight a few of them and look at the impact and cause of each of the observed incidents and try to determine if there was any malicious intent.
I presented the same data last week at NANOG 63, in San Antonio, a recording of this presentation can be found below:
BGP hijacking for monetary gain.
We have all heard of Bitcoin, it’s been in the news quite a bit and chances are that some of you are mining Bitcoins right now. There are now computing devices optimized for Bitcoin mining and even dedicated Bitcoin mining data centers. In addition to the dedicated data centers, many Bitcoin miners use cloud compute instances from Amazon, OVH, Digital Ocean, etc. So it’s obvious that there is a lot of money spent on Bitcoin mining & trading; and as such there is also an opportunity to make a quick buck.
This summer we blogged about a series of BGP hijacks where an attacker cleverly misused the Bitcoin stratum protocol. By Continue reading
BGP hijack incident by Syrian Telecommunications Establishment
The Syrian national Telecommunications Establishment (STE) has been in the news numerous times over the last few years, mostly because of the long lasting large scale Internet outages in Syria. This morning however we observed a new incident involving the two Autonomous systems for STE (AS29386 and AS29256). Starting at 08:33 UTC we detected that hundreds of new prefixes were being announced by primarily AS 29386. The new BGP announcements by STE (AS29386) were for prefixes that are not owned or operates by the Syrian Telco and as a result triggered ‘hijack / origin AS’ alerts for numerous BGPmon users. The announcements lasted for a few minutes only and we saw paths changing back to the original origin AS at about 08:37 UTC.
RIPE stat has some great tools that visualize the event, this example shows what happened to the youtube prefix 208.117.232.0/24
Propagation
STE buys upstream connectivity to the rest of the Internet via three providers, AS3491 (PCCW Global), AS3320 (Deutsche Telekom AG) and AS6762 (Telecom Italia Sparkle). The ‘bad’ BGP updates from this morning were only seen via Telecom Italia. This is either because STE only announced it to Telecom Italia, or because the other two providers filtered Continue reading
Using BGP data to find Spammers
It’s long been assumed that Spammers use a technique called IP squatting to get around IP reputation lists and to make it harder to find the real source of the spammers. In this blog we’ll take a closer look at Spam operations and their techniques.
IP Squatting
We’ve all read the reports about IPv4 running out of free address space and while that is certainly true there’s still a lot of address space that has been allocated but is not actually routed on the Internet today. A significant portion of that space are prefixes that were allocated a long time ago and folks are no longer using these allocations, forgot about it or have other reasons to not use their IP address space on the Internet. IP squatters look for space that hasn’t been routed for a while and will claim ownership of the space. This can then be used for things such as Spamming. There is vast range address space that is not currently announced on the Internet and it is not uncommon for IP squatters to cycle through this space using one or more prefixes at a time for a brief period.
Below we’ll expose two actual Spam Continue reading
What caused today’s Internet hiccup
Like others, you may have noticed some instability and general sluggishness on the Internet today. In this post we’ll take a closer look at what happened, including some of the BGP details!
At around 8am UTC Internet users on different mailing lists, forums and twitter, reported slow connectivity and intermediate outages. Examples can be found on the Outages mailing list company support site such as liquidweb and of course on Nanog.
How stable is the Internet
So how do we know if the Internet was really unstable today? One way to look at this is by looking at the outages visible in BGP over the last 12 months. On average we see outages for about 6,033 unique prefixes per day, affecting on average 1470 unique Autonomous Systems. These numbers are global averages and it’s worth noting that certain networks or geographical areas are more stable than others.
BGP stability and outages differ per country.
If we look at the number of detected outages by BGPmon today we see outage for 12,563 unique prefixes affecting 2,587unique Autonomous Systems. This is well above the daily average and indeed both the unique prefixes and the unique Autonomous Systems count are Continue reading
The Canadian Bitcoin Hijack
A few days ago researchers at Dell SecureWorks published the details of an attacker repeatedly hijacking BGP prefixes for numerous large providers such as Amazon, OVH, Digital Ocean, LeaseWeb, Alibaba and more. The goal of the operation was to intercept data between Bitcoin miners and Bitcoin mining pools. They estimated that $83,000 was made with this attack in just four months. The original post has many of details which we won’t repeat here, instead will take a closer look at the BGP details of this specific attack.
Attack details
Our friends at Dell SecureWorks decided not to name the network from which the hijacks originated. As a result we won’t name the exact Autonomous System either, instead we will suffice by saying that the originator of this hijack is a network operating in Eastern Canada.
Initial experiment
BGPmon detected the first hijack by this Canadian Autonomous System on October 8th 2013. For about 14 minutes a more specific /24 prefix for a Palestinian network was hijacked. Looking at geographical scope of the announcements and the probes that saw this route, we believe that in this case the route was only announced over the Toronto Internet Exchange.
Bitcoin hijack
On Feb Continue reading