Google recently faced a major outage in many parts of the world thanks to a BGP leak. This incident that was caused by a Nigerian ISP – Mainone – occurred on 12 November 2018 between 21.10 and 22.35 UTC, and was identified in tweets from the BGP monitoring service BGPMon, as well as the network monitoring provider Thousand Eyes.
Google also announced the problem through their status page:
We’ve received a report of an issue with Google Cloud Networking as of Monday, 2018-11-12 14:16 US/Pacific. We have reports of Google Cloud IP addresses being erroneously advertised by internet service providers other than Google. We will provide more information by Monday, 2018-11-12 15:00 US/Pacific.
Google (AS15169) advertise their prefixes (more than 500) through the IXPN Route Server, where PCH (Packet Clearing House) collects a daily snapshot of BGP announcements of IXPN. Unfortunately, 212 prefixes (aggregates of those 500+ announcements) from Google were leaked, which was recorded by BGPMon and RIPEstat.
Looking at the RIPE stats it is evident Continue reading
The world of BGP routing is a fascinating place with lots of interesting BGP events happening every day. It can be challenging to keep track of it all and so two years ago we started the BGPstream website where we keep track of large scale outages and BGP hijacks. We list the events, basic info and visualize it with one of my favorite tools: BGPlay. For those who keep an eye on @bgpstream , you probably noticed a curious series of BGP hijacks today all by the same Autonomous system affecting many well known networks.
Starting at April 26 22:36 UTC till approximately 22:43 UTC AS12389 (PJSC Rostelecom) started to originate 50 prefixes for numerous other Autonomous systems. The 50 hijacked prefixes included 37 unique autonomous systems and the complete list of affected networks can be found below. If your organization is in this list feel free to reach out and we can provide more details if needed. Keep in mind that many of these hijacks are already published on BGPstream.com as well.
So back to this incident, what happened here? What makes the list of affected networks ‘curious’ is the high number of financial institutions such as for example: MasterCard,Visa, Fortis,
Starting today at 17:09 UTC our systems detected a large scale routing incident affecting hundreds of Autonomous systems. Many BGPmon users have received an email informing them of this change.
Our initial investigation shows that the scope of this incident is widespread and affected 576 Autonomous systems and 3431 prefixes. Amongst the networks affected are high traffic prefixes including those of Google, Amazon, Twitter, Apple, Akamai, Time Warner Cable Internet and more.
All these events have either AS200759 “innofield AG” or private AS 65021 as the origin AS. In the cases where AS65021 appears as the origin AS, AS200759 is again the next-hop AS.
AS200759 “innofield AG” is a provider based out of Switzerland and normally only announces one IPv4 and one IPv6 prefix.
These are 2 example events:
Prefix 126.96.36.199/21 Is normally announced by Facebook AS32934 and during this event was announced by AS200759 as a more specific /22
Detected prefix: 188.8.131.52/22
Example aspath: 4608 24130 7545 6939 200759
And AS origin: 65021 behind AS 200759
Detected prefix: 184.108.40.206/22
Example aspath: 133812 23948 4788 6939 200759 65021
We saw the announcements via the following peers and providers of AS200759 “innofield Continue reading
It doesn’t happen often that a country with hundreds of prefixes is affected by a massive outage, however earlier today this unfortunately happened to Azerbaijan. Starting at 12:04 UTC approximately 94% of the prefixes out of Azerbaijan became unreachable. At the time of writing the outage is still active.
The event was reported on @bgpstream and details plus a replay can be found here: https://bgpstream.com/event/7981
The image below shows the impact on traffic from Azerbaijan to OpenDNS. It’s clear that almost all of the traffic from Azerbaijan disappeared at the time of outage.
The main Internet Service provider in Azerbaijan is AS29049, Delta Telecom Ltd. The majority of the country relies on Delta Telecom for connectivity to the rest of the Internet. The outage is reportedly the result of a fire damaging the equipment of Delta Telecom. As a result all routes for AS29049, Delta Telecom Ltd. and all of the networks that rely on Delta Telecom disappeared from the Internet.
The graph below shows the number of prefixes observed in the BGP routing tables for Azerbaijan. Clearly visible is the drop in reachable networks starting at 12:04.
Networks out of Continue reading
BGP hijacks happen every day, some of them affect more networks than others and every now and then there’s a major incident that affects thousands of networks. Our monitoring systems keep an eye out for our users and if you would like to have a general idea of what’s going on in the world of BGP incidents, keep an eye on BGPstream.com. Earlier today we detected one of those major incidents that affected thousands of networks.
Starting at 05:52 UTC, AS9498 (BHARTI Airtel Ltd.) started to claim ownership for thousands of prefixes by originating them in BGP. This affected prefixes for over two thousand unique organizations (Autonomous systems).
Our systems detected origin AS changes (hijacks) for 16,123 prefixes. The scope and impact was different per prefix but to give you an idea, about 7,600 of these announcements were seen by five or more of our peers (unique peers ASns) and 6,000 of these were seen by more than 10 of our peers.
One of the reasons this was so widespread is because large networks such as AS174 (Cogent Communications) and AS52320 (GlobeNet Cabos Submarinos VZLA) accepted and propagated these prefixes to their peers and customers.
The BGPplay visualization Continue reading
They are simply not a reality of what faculty is really like although movies like Dog House are unquestionably engaging. Some college students might differ with me, but faculty is not about becoming successful socially, all, instead, it’s about being successful academically. Your freshman year is your vital year of school. For all, it fails them as university students or makes them. Students who wander onto university their year thinking that faculty is one occasion that is large come in to get a rude awakening, particularly if they’ve been hearing upperclassmen reveal reports about waiting before the last second to create that report and not going to course. Continue reading
A dog photo in a “poop argument” has enraged visitors of a SWAT group standoff when nearly several users of the Wisconsin police SWAT team turned up, positioned themselves behind an armored vehicle, and pumped two fits right into a little, end-wagging puppy. The SWAT were termed following the pet, who lowered a deuce within the wrong position, started a defecation conflict between his neighbors and an man. Writes the NY Daily Information on Nov. 3: ” SWAT crew delivered to manage a town argument over waste ended up igniting a firestorm of criticism from the nearby police force, and harming canine. Continue reading
As part of the Hacking Team fall out and all the details published on wikileaks, it became public knowledge that Hacking Team helped one of their customers Special Operations Group (ROS), regain access to Remote Access Tool (RAT) clients. ROS recommended using BGP hijacking and Hacking Team helped with the setup of new RAT CnC servers.
In this post we’ll take a closer look at the exact details of this incident and support the wikileaks findings with BGP data.
Raggruppamento Operativo Speciale and Hacking Team
The Raggruppamento Operativo Speciale or ROS is the Special Operations Group of the Italian National Military police. The group focuses on investigating organized crime and terrorism. Hacking Team sells its RAT software known as Remote Control System (RCS) to law enforcement and intelligence agencies, ROS included.
ROS infected and installed the RCS client on the machines of persons of interest (referred to in the emails as targets). These Remote Access Tools can provide ROS with all kinds of information and typically provide the tool’s operator with full access over a victim’s machine. The RCS clients normally need to check in with a server —for example a machine the clients can get their commands (orders) from— Continue reading
Earlier today a massive route leak initiated by Telekom Malaysia (AS4788) caused significant network problems for the global routing system. Primarily affected was Level3 (AS3549 – formerly known as Global Crossing) and their customers. Below are some of the details as we know them now.
Starting at 08:43 UTC today June 12th, AS4788 Telekom Malaysia started to announce about 179,000 of prefixes to Level3 (AS3549, the Global crossing AS), whom in turn accepted these and propagated them to their peers and customers. Since Telekom Malaysia had inserted itself between these thousands of prefixes and Level3 it was now responsible for delivering these packets to the intended destinations.
This event resulted in significant packet loss and Internet slow down in all parts of the world. The Level3 network in particular suffered from severe service degradation between the Asia pacific region and the rest of their network. The graph below for example shows the packet loss as measured by OpenDNS between London over Level3 and Hong Kong. The same loss patterns were visible from other Level3 locations globally to for example Singapore, Hong Kong and Sydney.
At the same time the round trip time between these Continue reading
Earlier today many BGPmon users received one or more alerts informing them that their autonomous system (AS) started to announce a more-specific prefix. BGPmon classified many of these alerts as possible BGP man-in-the-middle (MITM) attacks. Here is an example alert:
Possible BGP MITM attack (Code: 21)
Your prefix: 220.127.116.11/15:
Prefix Description: acxiom-online.com --- Amazon EC2 IAD prefix
Update time: 2015-03-26 11:27 (UTC)
Detected by #peers: 24
Detected prefix: 18.104.22.168/20
Announced by: AS14618 (AMAZON-AES - Amazon.com, Inc.,US)
Upstream AS: AS3257 (TINET-BACKBONE Tinet SpA,DE)
ASpath: 4608 24130 7545 6939 40633 18978 3257 14618
The alert shows the user was monitoring 22.214.171.124/15, normally announced by Amazon, Inc. (AS14618). In this case however, the detected prefix was the more specific 126.96.36.199/20. The netblock owners would have verified their BGP announcements and quickly recognized they did not originate this more-specific prefix. Further analysis pointed to the suspicion that a bad actor was impersonating Amazon. BGPmon algorithms alerted to this as well, and–within moments of the initial change–marked these events as a possible BGP MITM attack.
One reason for this classification is the way BGPmon understands and interprets AS Continue reading
Dear BGPmon.net user,
I’m excited to announce that BGPmon has been acquired by OpenDNS. OpenDNS is a leading cloud-delivered network security company known for engineering predictive intelligence technology that stops malicious activity before it can threaten a network.
Over the last few years BGPmon has grown from a community service into a successful business that helps thousands of network engineers from around the world monitor their networks. Throughout this journey, we’ve developed close relationships with many of you and together, worked on some truly fascinating cases.
Becoming a part of OpenDNS is a logical next step for BGPmon. With its engineering resources, massive scale and cloud delivery model, OpenDNS is the right direction to continue growing the BGPmon service. I’m confident that moving forward BGPmon will only get better.
The transition plan is straightforward. OpenDNS will invest in building out the service even more but also is committed to keeping the free features free. Simply put, nothing regarding the service will change other than we’ll continue adding new functionality.
On a personal note, I’d like to thank all of you for your continued support and encouragement. I am excited for the changes ahead and personally being a part of Continue reading
This morning people on twitter reported that they were unable to reach Google services. Businessinsider followed up with a story in which they mentioned that the Google service interruption primarily involved European and Indian users.
In this blog we’ll take a quick look at what exactly happened by looking at our BGP data. The first clue comes from David Roy and Franck Klopfenstein on twitter who noticed traffic was re-routed towards AS9498 in India. Digging through our BGP data we are able to indeed confirm that routing paths for many google prefixes changed to a path that includes the Indian AS 9498 between 08:58 UTC and 09:14 UTC.
Let’s take a look at an example. In my case www.google.com resolves to the following addresses:
www.google.com has address 188.8.131.52
www.google.com has address 184.108.40.206
www.google.com has address 220.127.116.11
www.google.com has address 18.104.22.168
www.google.com has address 22.214.171.124
www.google.com has IPv6 address 2607:f8b0:4006:806::1014
The IPv4 addresses are all in the 126.96.36.199/24 range. If we now look at the BGP announcements for that Continue reading
Over the last year we have seen and written about numerous BGP routing incidents that looked out of the ordinary, straight-up suspicious or were just configuration mistakes. In this blog post we will highlight a few of them and look at the impact and cause of each of the observed incidents and try to determine if there was any malicious intent.
I presented the same data last week at NANOG 63, in San Antonio, a recording of this presentation can be found below:
We have all heard of Bitcoin, it’s been in the news quite a bit and chances are that some of you are mining Bitcoins right now. There are now computing devices optimized for Bitcoin mining and even dedicated Bitcoin mining data centers. In addition to the dedicated data centers, many Bitcoin miners use cloud compute instances from Amazon, OVH, Digital Ocean, etc. So it’s obvious that there is a lot of money spent on Bitcoin mining & trading; and as such there is also an opportunity to make a quick buck.
This summer we blogged about a series of BGP hijacks where an attacker cleverly misused the Bitcoin stratum protocol. By Continue reading
The Syrian national Telecommunications Establishment (STE) has been in the news numerous times over the last few years, mostly because of the long lasting large scale Internet outages in Syria. This morning however we observed a new incident involving the two Autonomous systems for STE (AS29386 and AS29256). Starting at 08:33 UTC we detected that hundreds of new prefixes were being announced by primarily AS 29386. The new BGP announcements by STE (AS29386) were for prefixes that are not owned or operates by the Syrian Telco and as a result triggered ‘hijack / origin AS’ alerts for numerous BGPmon users. The announcements lasted for a few minutes only and we saw paths changing back to the original origin AS at about 08:37 UTC.
STE buys upstream connectivity to the rest of the Internet via three providers, AS3491 (PCCW Global), AS3320 (Deutsche Telekom AG) and AS6762 (Telecom Italia Sparkle). The ‘bad’ BGP updates from this morning were only seen via Telecom Italia. This is either because STE only announced it to Telecom Italia, or because the other two providers filtered Continue reading
It’s long been assumed that Spammers use a technique called IP squatting to get around IP reputation lists and to make it harder to find the real source of the spammers. In this blog we’ll take a closer look at Spam operations and their techniques.
We’ve all read the reports about IPv4 running out of free address space and while that is certainly true there’s still a lot of address space that has been allocated but is not actually routed on the Internet today. A significant portion of that space are prefixes that were allocated a long time ago and folks are no longer using these allocations, forgot about it or have other reasons to not use their IP address space on the Internet. IP squatters look for space that hasn’t been routed for a while and will claim ownership of the space. This can then be used for things such as Spamming. There is vast range address space that is not currently announced on the Internet and it is not uncommon for IP squatters to cycle through this space using one or more prefixes at a time for a brief period.
Below we’ll expose two actual Spam Continue reading
Like others, you may have noticed some instability and general sluggishness on the Internet today. In this post we’ll take a closer look at what happened, including some of the BGP details!
At around 8am UTC Internet users on different mailing lists, forums and twitter, reported slow connectivity and intermediate outages. Examples can be found on the Outages mailing list company support site such as liquidweb and of course on Nanog.
How stable is the Internet
So how do we know if the Internet was really unstable today? One way to look at this is by looking at the outages visible in BGP over the last 12 months. On average we see outages for about 6,033 unique prefixes per day, affecting on average 1470 unique Autonomous Systems. These numbers are global averages and it’s worth noting that certain networks or geographical areas are more stable than others.
If we look at the number of detected outages by BGPmon today we see outage for 12,563 unique prefixes affecting 2,587unique Autonomous Systems. This is well above the daily average and indeed both the unique prefixes and the unique Autonomous Systems count are Continue reading
A few days ago researchers at Dell SecureWorks published the details of an attacker repeatedly hijacking BGP prefixes for numerous large providers such as Amazon, OVH, Digital Ocean, LeaseWeb, Alibaba and more. The goal of the operation was to intercept data between Bitcoin miners and Bitcoin mining pools. They estimated that $83,000 was made with this attack in just four months. The original post has many of details which we won’t repeat here, instead will take a closer look at the BGP details of this specific attack.
Our friends at Dell SecureWorks decided not to name the network from which the hijacks originated. As a result we won’t name the exact Autonomous System either, instead we will suffice by saying that the originator of this hijack is a network operating in Eastern Canada.
BGPmon detected the first hijack by this Canadian Autonomous System on October 8th 2013. For about 14 minutes a more specific /24 prefix for a Palestinian network was hijacked. Looking at geographical scope of the announcements and the probes that saw this route, we believe that in this case the route was only announced over the Toronto Internet Exchange.
On Feb Continue reading