Video: Outages Caused by Bugs in BGP Implementations

The previous BGP-related videos described how fat fingers and malicious actors cause Internet outages.

Today, we’ll focus on the impact of bugs in BGP implementations, from malformed AS paths to mishandled transitive attributes. The examples in the video are a few years old, but you can see similar things in the wild in 2023.

Worth Reading: Cloudflare Control Plane Outage

Cloudflare experienced a significant outage in early November 2023 and published a detailed post-mortem report. You should read the whole report; here are my CliffsNotes:

• Regardless of how much redundancy you have, sometimes all systems will fail at once. Having redundant systems decreases the probability of total failure but does not reduce it to zero.
• As your systems grow, they gather hidden- and circular dependencies.
• You won’t uncover those dependencies unless you run a full-blown disaster recovery test (not a fake one)
• If you don’t test your disaster recovery plan, it probably won’t work when needed.

Also (unrelated to Cloudflare outage):

BGP Labs: Using Multi-Exit Discriminator (MED)

In the previous labs we used BGP weights and Local Preference to select the best link out of an autonomous system and thus change the outgoing traffic flow.

Most edge (end-customer) networks face a different problem – they want to influence the incoming traffic flow, and one of the tools they can use is BGP Multi-Exit Discriminator (MED).

Is Anyone Using netlab on Windows?

Tomas wants to start netlab with PowerShell, but it doesn’t work for him, and I don’t know anyone running netlab directly on Windows (I know people running it in a Ubuntu VM on Windows, but that’s a different story).

In theory, netlab (and Ansible) should work fine with Windows Subsystem for Linux. In practice, there’s often a gap between theory and practice – if you run netlab on Windows (probably using VirtualBox with Vagrant), I’d love to hear from you. Please leave a comment, email me, add a comment to Tomas’ GitHub issue, or fix the documentation and submit a PR. Thank you!

LAN Data Link Layer Addressing

Last week, we discussed Fibre Channel addressing. This time, we’ll focus on data link layer technologies used in multi-access networks: Ethernet, Token Ring, FDDI, and other local area- or Wi-Fi technologies.

The first local area networks (LANs) ran on a physical multi-access medium. The first one (original Ethernet) started as a thick coaxial cable¹ that you had to drill into to connect a transceiver to the cable core.

Later versions of Ethernet used thinner cables with connectors that you put together to build whole network segments out of pieces of cable. However, even in that case, we were dealing with a single multi-access physical network – disconnecting a cable would bring down the whole network.

Git Rebase: What Can Go Wrong?

Julia Evans wrote another must-read article (if you’re using Git): git rebase: what can go wrong?

I often use git rebase to clean up the commit history of a branch I want to merge into a main branch or to prepare a feature branch for a pull request. I don’t want to run it unattended – I’m always using the interactive option – but even then, I might get into tight spots where I can only hope the results will turn out to be what I expect them to be. Always have a backup – be it another branch or a copy of the branch you’re working on in a remote repository.

Video: Kubernetes Calico Plugin

November is turning out to be the Month of BGP on my blog. Keeping in line with that theme, let’s watch Stuart Charlton explain the Calico plugin (which can use BGP to advertise the container networking prefixes to the outside world) in the Kubernetes Networking Deep Dive webinar.

BGP Labs: Using BGP Local Preference

A while ago we used BGP weights to select the best link out of an autonomous system. In this lab we’ll use BGP local preference to implement a consistent network-wide routing policy:

Open BGP Daemons: There’s So Many of Them

A while ago, the Networking Notes blog published a link to my “Will Network Devices Reject BGP Sessions from Unknown Sources?” blog post with a hint: use Shodan to find how many BGP routers accept a TCP session from anyone on the Internet.

The results are appalling: you can open a TCP session on port 179 with over 3 million IP addresses.

Rapid Progress in BGP Route Origin Validation

In 2022, I was invited to speak about Internet routing security at the DEEP conference in Zadar, Croatia. One of the main messages of the presentation was how slow the progress had been even though we had had all the tools available for at least a decade (RFC 7454 was finally published in 2015, and we started writing it in early 2012).

At about that same time, a small group of network operators started cooperating on improving the security and resilience of global routing, eventually resulting in the MANRS initiative – a great place to get an overview of how many Internet Service Providers care about adopting Internet routing security mechanisms.

Fibre Channel Addressing

Whenever we talk about LAN data-link-layer addressing, most engineers automatically switch to the “must be like Ethernet” mentality, assuming all data-link-layer LAN framing must somehow resemble Ethernet frames.

That makes no sense on point-to-point links. As explained in Early Data-Link Layer Addressing article, you don’t need layer-2 addresses on a point-to-point link between two layer-3 devices. Interestingly, there is one LAN technology (that I’m aware of) that got data link addressing right: Fibre Channel (FC).

Worth Reading: Confusing Git Terminology

Julia Evans wrote another great article explaining confusing git terminology. Definitely worth reading if you want to move past simple recipes or reminiscing about old days.

Video: Hacking BGP for Fun and Profit

At least some people learn from others’ mistakes: using the concepts proven by some well-publicized BGP leaks, malicious actors quickly figured out how to hijack BGP prefixes for fun and profit.

Fortunately, those shenanigans wouldn’t spread as far today as they did in the past – according to RoVista, most of the largest networks block the prefixes Route Origin Validation (ROV) marks as invalid.

Notes:

• ROV cannot stop all the hijacks, but it can identify more-specific-prefixes hijacks (assuming the origin AS did their job right).
• You’ll find more Network Security Fallacies videos in the How Networks Really Work webinar.

BGP Labs: Build a Transit Network with IBGP

Last time we built a network with two adjacent BGP routers. Now let’s see what happens when we add a core router between them:

Worth Reading: Taming the BGP Reconfiguration Transients

Almost exactly a decade ago I wrote about a paper describing how IBGP migrations can cause forwarding loops and how one could reorder BGP reconfiguration steps to avoid them.

One of the paper’s authors was Laurent Vanbever who moved to ETH Zurich in the meantime where his group keeps producing great work, including the Chameleon tool (code on GitHub) that can tame transient loops while reconfiguring BGP. Definitely something worth looking at if you’re running a large BGP network.

Weird: vJunos Evolved 23.2R1.5 Declines DHCP Address

It’s time for a Halloween story: imagine the scary scenario in which a DHCP client asks for an address, gets it, and then immediately declines it. That’s what I’ve been experiencing with vJunos Evolved release 23.2R1.15.

Worth Exploring: BGP from Theory to Practice

My good friend Tiziano Tofoni finally created an English version of his evergreen classic BGP from theory to practice with co-authors Antonio Prado and Flavio Luciani.

I had the Italian version of the book since the days I was running SDN workshops with Tiziano in Rome, and it’s really nice to see they finally decided to address a wider market.

Also, you know what would go well with that book? Free open-source BGP configuration labs of course 😉

Early Data-Link Layer Addressing

After covering the theoretical part of network addressing (part 2, part 3), let’s go into some practical examples. I’ll start with data link layer and then move on to networking and higher layers.

The earliest data link implementations that were not point-to-point links were multi-drop links and I mentioned them in the networking challenges part of the webinar. Initially, we implemented multi-drop links with modems, but even today you can see multi-drop in satellite communications, Wi-Fi, or in cable modems.

BGP Labs: Multivendor External Routers

A quick update BGP Labs project status update: now that netlab release 1.6.4 is out I could remove the dependency on using Cumulus Linux as the external BGP router.

You can use any device that is supported by bgp.session and bgp.policy plugins as the external BGP router. You could use Arista EOS, Aruba AOS-CX, Cisco IOSv, Cisco IOS-XE, Cumulus Linux or FRR as external BGP routers with netlab release 1.6.4, and I’m positive Jeroen van Bemmel will add Nokia SR Linux to that list.

If you’re not ready for a netlab upgrade, you can keep using Cumulus Linux as external BGP routers (I’ll explain the behind-the-scenes magic in another blog post, I’m at the Deep Conference this week).

For more details read the updated BGP Labs Software Installation and Lab Setup guide.

netlab 1.6.4: Support for Multi-Lab Projects; More BGP Goodies

Features in netlab release 1.6.4 were driven primarily by the needs of my BGP labs project:

• bgp.session plugin (formerly known as ebgp.utils plugin) got support for BFD, passive BGP peers and remove-private-as option.
• bgp.policy plugin implements basic BGP routing policy tools, including per-neighbor weights, local preference and MED.
• You can enable external tools in user defaults and use default groups to create user- or project-wide groups in the defaults files.
• Version-specific lab topology files allow netlab to select a lab topology that is a best fit for the netlab release you’re running.

Numerous platforms already support the new BGP nerd knobs: