Distributed Systems Resources

Distributed systems are complicated. Add networking to the mix, and you get traumatic challenges like the CAP theorem and Byzantine fault tolerance. Most of those challenges are unknown to engineers who have to suffer through the vendor marketing presentations, making it hard to determine whether the latest shiny gizmo works outside of PowerPoint.

I started collecting articles describing distributed-system gotchas years ago, wrote numerous blog posts on the topic in the heydays of the SDN Will Save the World lemming run, and organized them into the Distributed Systems Resources page.

EIGRP Third-Party Next Hops

EIGRP routing updates have always contained the next hop field (similar to BGP updates), which was unused until Cisco IOS release 12.3 when the no ip next-hop-self eigrp AS-number interface configuration command was implemented.

EIGRP does not set the next hop field by default. An EIGRP router receiving a routing update thus assumes that the next hop of the received routes is the sending router. This behavior usually works well, but prevents site-to-site shortcuts to be established in DMVPN networks, and results in suboptimal routing in some route redistribution scenarios.

EIGRP Third-Party Next Hops

EIGRP routing updates have always contained the next hop field (similar to BGP updates), which was unused until Cisco IOS release 12.3 when the no ip next-hop-self eigrp AS-number interface configuration command was implemented.

EIGRP does not set the next hop field by default. An EIGRP router receiving a routing update thus assumes that the next hop of the received routes is the sending router. This behavior usually works well, but prevents site-to-site shortcuts to be established in DMVPN networks, and results in suboptimal routing in some route redistribution scenarios.

Default EBGP Policy (RFC 8212)

One of the most common causes of Internet routing leaks is an undereducated end-customer configuring EBGP sessions with two (or more) upstream ISPs.

Without basic-level BGP knowledge or further guidance from the service providers, the customer network engineer¹ might start a BGP routing process and configure two EBGP sessions, similar to the following industry-standard CLI² configuration:

Default EBGP Policy (RFC 8212)

One of the most common causes of Internet routing leaks is an undereducated end-customer configuring EBGP sessions with two (or more) upstream ISPs.

Without basic-level BGP knowledge or further guidance from the service providers, the customer network engineer¹ might start a BGP routing process and configure two EBGP sessions, similar to the following industry-standard CLI² configuration:

Worth Reading: Internet WAN Edge Design

Brandon Hitzel published a detailed document describing various Internet WAN edge designs. Definitely worth reading and bookmarking.

Worth Reading: Internet WAN Edge Design

Brandon Hitzel published a detailed document describing various Internet WAN edge designs. Definitely worth reading and bookmarking.

Worth Reading: Unbounded TCP Memory Usage

Another phenomenal detective story published on Cloudflare blog: Unbounded memory usage by TCP for receive buffers, and how we fixed it.

TL&DR: Moving TCP window every time you acknowledge a segment doesn’t work well with scaled window sizes.

The interesting takeaways:

Worth Reading: Unbounded TCP Memory Usage

Another phenomenal detective story published on Cloudflare blog: Unbounded memory usage by TCP for receive buffers, and how we fixed it.

TL&DR: Moving TCP window every time you acknowledge a segment doesn’t work well with scaled window sizes.

The interesting takeaways:

Video: Link State Routing Protocol Basics

After introducing the routing protocols and explaining the basics of link-state routing it was time for implementation considerations including:

• Collecting local endpoint reachability information
• Finding neighbors and exchanging the collected information (hint: a link-state topology database is just a distributed key-value store)
• Running the SPF algorithm (including partial SPF details) and installing the results

Video: Link State Routing Protocol Basics

After introducing the routing protocols and explaining the basics of link-state routing it was time for implementation considerations including:

• Collecting local endpoint reachability information
• Finding neighbors and exchanging the collected information (hint: a link-state topology database is just a distributed key-value store)
• Running the SPF algorithm (including partial SPF details) and installing the results

Source IP Address in Multicast Packets

One of my readers sent me this (paraphrased) question:

What I have seen in my network are multicast packets with the IP source address set to 0.0.0.0 and source port set to 0. Is that considered acceptable? Could I use a multicast IP address as a source address?

TL&DR: **** NO!!!

It also seemed like a good question to test ChatGPT, and this time it did a pretty good job.

Source IP Address in Multicast Packets

One of my readers sent me this (paraphrased) question:

What I have seen in my network are multicast packets with the IP source address set to 0.0.0.0 and source port set to 0. Is that considered acceptable? Could I use a multicast IP address as a source address?

TL&DR: **** NO!!!

It also seemed like a good question to test ChatGPT, and this time it did a pretty good job.

NTP in a Nutshell

Years ago I’ve been involved in an interesting discussion focusing on NTP authentication and whether you can actually implement it reliably on Cisco IOS. What I got out of it (apart from a working example) was the feeling that NTP and it’s implementation in Cisco IOS was under-understood and under-documented, so I wrote an article about it. Of course the web version got lost in the mists of time but I keep my archives handy.

Last weekend I migrated that article to blog.ipSpace.net. I hope you’ll still find it useful; while it’s pretty old, the fundamentals haven’t changed in the meantime.

NTP in a Nutshell

Years ago I’ve been involved in an interesting discussion focusing on NTP authentication and whether you can actually implement it reliably on Cisco IOS. What I got out of it (apart from a working example) was the feeling that NTP and it’s implementation in Cisco IOS was under-understood and under-documented, so I wrote an article about it. Of course the web version got lost in the mists of time but I keep my archives handy.

Last weekend I migrated that article to blog.ipSpace.net. I hope you’ll still find it useful; while it’s pretty old, the fundamentals haven’t changed in the meantime.

Path Failure Detection on Multi-Homed Servers

TL&DR: Installing an Ethernet NIC with two uplinks in a server is easy¹. Connecting those uplinks to two edge switches is common sense². Detecting physical link failure is trivial in Gigabit Ethernet world. Deciding between two independent uplinks or a link aggregation group is interesting. Detecting path failure and disabling the useless uplink that causes traffic blackholing is a living hell (more details in this Design Clinic question).

Want to know more? Let’s dive into the gory details.

Path Failure Detection on Multi-Homed Servers

TL&DR: Installing an Ethernet NIC with two uplinks in a server is easy¹. Connecting those uplinks to two edge switches is common sense². Detecting physical link failure is trivial in Gigabit Ethernet world. Deciding between two independent uplinks or a link aggregation group is interesting. Detecting path failure and disabling the useless uplink that causes traffic blackholing is a living hell (more details in this Design Clinic question).

Want to know more? Let’s dive into the gory details.

Goodbye Twitter. It Was Fun While It Lasted

I joined Twitter in October 2008 (after noticing everyone else was using it during a Networking Field Day event), and eventually figured out how to automate posting the links to my blog posts in case someone uses Twitter as their primary source of news.

This week, I got a nice email from IFTTT (the solution I used) telling me they had to disable the post-to-Twitter applet. Twitter started charging for the API, and I was using their free service – obviously the math didn’t work out.

That left me with three options:

Goodbye Twitter. It Was Fun While It Lasted

I joined Twitter in October 2008 (after noticing everyone else was using it during a Networking Field Day event), and eventually figured out how to automate posting the links to my blog posts in case someone uses Twitter as their primary source of news – an IFTTT applet that read my RSS feed and posted links to new entries to Twitter.

This week, I got a nice email from IFTTT telling me they had to disable the post-to-Twitter applet. Twitter started charging for the API, and I was using their free service – obviously the math didn’t work out.

That left me with three options:

Worth Reading: Cargo Cult AI

Before we managed to recover from the automation cargo cults, a tsunami wave of cargo cult AI washed over us as Edlyn V. Levine explained in an ACM Queue article. Enjoy ;)

Also, a bit of a historical perspective is never a bad thing:

Impressive progress in AI, including the recent sensation of ChatGPT, has been dominated by the success of a single, decades-old machine-learning approach called a multilayer (or deep) neural network. This approach was invented in the 1940s, and essentially all of the foundational concepts of neural networks and associated methods—including convolutional neural networks and backpropagation—were in place by the 1980s.