Worth Reading: Cloudflare Control Plane Outage

Cloudflare experienced a significant outage in early November 2023 and published a detailed post-mortem report. You should read the whole report; here are my CliffsNotes:

• Regardless of how much redundancy you have, sometimes all systems will fail at once. Having redundant systems decreases the probability of total failure but does not reduce it to zero.
• As your systems grow, they gather hidden- and circular dependencies.
• You won’t uncover those dependencies unless you run a full-blown disaster recovery test (not a fake one)
• If you don’t test your disaster recovery plan, it probably won’t work when needed.

Also (unrelated to Cloudflare outage):

BGP Labs: Using Multi-Exit Discriminator (MED)

In the previous labs we used BGP weights and Local Preference to select the best link out of an autonomous system and thus change the outgoing traffic flow.

Most edge (end-customer) networks face a different problem – they want to influence the incoming traffic flow, and one of the tools they can use is BGP Multi-Exit Discriminator (MED).

BGP Labs: Using Multi-Exit Discriminator (MED)

In the previous labs, we used BGP weights and Local Preference to select the best link out of an autonomous system and thus change the outgoing traffic flow.

Most edge (end-customer) networks face a different problem – they want to influence the incoming traffic flow, and one of the tools they can use is BGP Multi-Exit Discriminator (MED).

Is Anyone Using netlab on Windows?

Tomas wants to start netlab with PowerShell, but it doesn’t work for him, and I don’t know anyone running netlab directly on Windows (I know people running it in a Ubuntu VM on Windows, but that’s a different story).

In theory, netlab (and Ansible) should work fine with Windows Subsystem for Linux. In practice, there’s often a gap between theory and practice – if you run netlab on Windows (probably using VirtualBox with Vagrant), I’d love to hear from you. Please leave a comment, email me, add a comment to Tomas’ GitHub issue, or fix the documentation and submit a PR. Thank you!

Is Anyone Using netlab on Windows?

Tomas wants to start netlab with PowerShell, but it doesn’t work for him, and I don’t know anyone running netlab directly on Windows (I know people running it in a Ubuntu VM on Windows, but that’s a different story).

In theory, netlab (and Ansible) should work fine with Windows Subsystem for Linux. In practice, there’s often a gap between theory and practice – if you run netlab on Windows (probably using VirtualBox with Vagrant), I’d love to hear from you. Please leave a comment, email me, add a comment to Tomas’ GitHub issue, or fix the documentation and submit a PR. Thank you!

LAN Data Link Layer Addressing

Last week, we discussed Fibre Channel addressing. This time, we’ll focus on data link layer technologies used in multi-access networks: Ethernet, Token Ring, FDDI, and other local area- or Wi-Fi technologies.

The first local area networks (LANs) ran on a physical multi-access medium. The first one (original Ethernet) started as a thick coaxial cable¹ that you had to drill into to connect a transceiver to the cable core.

Later versions of Ethernet used thinner cables with connectors that you put together to build whole network segments out of pieces of cable. However, even in that case, we were dealing with a single multi-access physical network – disconnecting a cable would bring down the whole network.

LAN Data Link Layer Addressing

Last week, we discussed Fibre Channel addressing. This time, we’ll focus on data link layer technologies used in multi-access networks: Ethernet, Token Ring, FDDI, and other local area- or Wi-Fi technologies.

The first local area networks (LANs) ran on a physical multi-access medium. The first one (original Ethernet) started as a thick coaxial cable¹ that you had to drill into to connect a transceiver to the cable core.

Later versions of Ethernet used thinner cables with connectors that you put together to build whole network segments out of pieces of cable. However, even in that case, we were dealing with a single multi-access physical network – disconnecting a cable would bring down the whole network.

Git Rebase: What Can Go Wrong?

Julia Evans wrote another must-read article (if you’re using Git): git rebase: what can go wrong?

I often use git rebase to clean up the commit history of a branch I want to merge into a main branch or to prepare a feature branch for a pull request. I don’t want to run it unattended – I’m always using the interactive option – but even then, I might get into tight spots where I can only hope the results will turn out to be what I expect them to be. Always have a backup – be it another branch or a copy of the branch you’re working on in a remote repository.

Git Rebase: What Can Go Wrong?

Julia Evans wrote another must-read article (if you’re using Git): git rebase: what can go wrong?

I often use git rebase to clean up the commit history of a branch I want to merge into a main branch or to prepare a feature branch for a pull request. I don’t want to run it unattended – I’m always using the interactive option – but even then, I might get into tight spots where I can only hope the results will turn out to be what I expect them to be. Always have a backup – be it another branch or a copy of the branch you’re working on in a remote repository.

Video: Kubernetes Calico Plugin

November is turning out to be the Month of BGP on my blog. Keeping in line with that theme, let’s watch Stuart Charlton explain the Calico plugin (which can use BGP to advertise the container networking prefixes to the outside world) in the Kubernetes Networking Deep Dive webinar.

Video: Kubernetes Calico Plugin

November is turning out to be the Month of BGP on my blog. Keeping in line with that theme, let’s watch Stuart Charlton explain the Calico plugin (which can use BGP to advertise the container networking prefixes to the outside world) in the Kubernetes Networking Deep Dive webinar.

BGP Labs: Using BGP Local Preference

A while ago we used BGP weights to select the best link out of an autonomous system. In this lab we’ll use BGP local preference to implement a consistent network-wide routing policy:

BGP Labs: Using BGP Local Preference

A while ago, we used BGP weights to select the best link out of an autonomous system. In this lab, we’ll use BGP local preference to implement a consistent network-wide routing policy:

Open BGP Daemons: There’s So Many of Them

A while ago, the Networking Notes blog published a link to my “Will Network Devices Reject BGP Sessions from Unknown Sources?” blog post with a hint: use Shodan to find how many BGP routers accept a TCP session from anyone on the Internet.

The results are appalling: you can open a TCP session on port 179 with over 3 million IP addresses.

Open BGP Daemons: There’s So Many of Them

A while ago, the Networking Notes blog published a link to my “Will Network Devices Reject BGP Sessions from Unknown Sources?” blog post with a hint: use Shodan to find how many BGP routers accept a TCP session from anyone on the Internet.

The results are appalling: you can open a TCP session on port 179 with over 3 million IP addresses.

Rapid Progress in BGP Route Origin Validation

In 2022, I was invited to speak about Internet routing security at the DEEP conference in Zadar, Croatia. One of the main messages of the presentation was how slow the progress had been even though we had had all the tools available for at least a decade (RFC 7454 was finally published in 2015, and we started writing it in early 2012).

At about that same time, a small group of network operators started cooperating on improving the security and resilience of global routing, eventually resulting in the MANRS initiative – a great place to get an overview of how many Internet Service Providers care about adopting Internet routing security mechanisms.

Rapid Progress in BGP Route Origin Validation

In 2022, I was invited to speak about Internet routing security at the DEEP conference in Zadar, Croatia. One of the main messages of the presentation was how slow the progress had been even though we had had all the tools available for at least a decade (RFC 7454 was finally published in 2015, and we started writing it in early 2012).

At about that same time, a small group of network operators started cooperating on improving the security and resilience of global routing, eventually resulting in the MANRS initiative – a great place to get an overview of how many Internet Service Providers care about adopting Internet routing security mechanisms.

Fibre Channel Addressing

Whenever we talk about LAN data-link-layer addressing, most engineers automatically switch to the “must be like Ethernet” mentality, assuming all data-link-layer LAN framing must somehow resemble Ethernet frames.

That makes no sense on point-to-point links. As explained in Early Data-Link Layer Addressing article, you don’t need layer-2 addresses on a point-to-point link between two layer-3 devices. Interestingly, there is one LAN technology (that I’m aware of) that got data link addressing right: Fibre Channel (FC).

Fibre Channel Addressing

Whenever we talk about LAN data-link-layer addressing, most engineers automatically switch to the “must be like Ethernet” mentality, assuming all data-link-layer LAN framing must somehow resemble Ethernet frames.

That makes no sense on point-to-point links. As explained in Early Data-Link Layer Addressing article, you don’t need layer-2 addresses on a point-to-point link between two layer-3 devices. Interestingly, there is one LAN technology (that I’m aware of) that got data link addressing right: Fibre Channel (FC).

Worth Reading: Confusing Git Terminology

Julia Evans wrote another great article explaining confusing git terminology. Definitely worth reading if you want to move past simple recipes or reminiscing about old days.