Twilight Zone: File Transfer Never Completes

Ages ago when we were building networks using super-expensive 64kbps WAN links, a customer sent us a weird bug report:

Everything works fine, but we cannot transfer one particular file between two locations – the file transfer stalls and eventually times out. At the same time, we’re seeing increased number of CRC errors on the WAN link.

My chat with the engineer handling the ticket went along these lines:

Twilight Zone: File Transfer Never Completes

Ages ago when we were building networks using super-expensive 64kbps WAN links, a customer sent us a weird bug report:

Everything works fine, but we cannot transfer one particular file between two locations – the file transfer stalls and eventually times out. At the same time, we’re seeing increased number of CRC errors on the WAN link.

My chat with the engineer handling the ticket went along these lines:

Worth Exploring: Akvorado Flow Collector and Visualizer

The results you can get when you know how to apply proper glue to a bunch of open-source tools never cease to amaze me. The latest entrant in that category: Akvorado, a Netflow/IPFIX collector and analyzer by Vincent Bernat.

Some of the sample graphs (shown in the GitHub repo) are not far off from those that knocked our socks off during the first Kentik Networking Field Day presentation. Definitely a tool worth exploring ;)

Worth Exploring: Akvorado Flow Collector and Visualizer

The results you can get when you know how to apply proper glue to a bunch of open-source tools never cease to amaze me. The latest entrant in that category: Akvorado, a Netflow/IPFIX collector and analyzer by Vincent Bernat.

Some of the sample graphs (shown in the GitHub repo) are not far off from those that knocked our socks off during the first Kentik Networking Field Day presentation. Definitely a tool worth exploring ;)

Twilight Zone: File Transfer Causes Link Drop

Long long time ago, we built a multi-protocol WAN network for a large organization. Everything worked great, until we got the weirdest bug report I’ve seen thus far:

When trying to transfer a particular file with DECnet to the central location, the WAN link drops. That does not happen with any other file, or when transferring the same file with TCP/IP. The only way to recover is to power cycle the modem.

Try to figure out what was going on before reading any further ;)

Twilight Zone: File Transfer Causes Link Drop

Long long time ago, we built a multi-protocol WAN network for a large organization. Everything worked great, until we got the weirdest bug report I’ve seen thus far:

When trying to transfer a particular file with DECnet to the central location, the WAN link drops. That does not happen with any other file, or when transferring the same file with TCP/IP. The only way to recover is to power cycle the modem.

Try to figure out what was going on before reading any further ;)

Worth Reading: On the Dangers of Cryptocurrencies…

Bruce Schneier wrote an article on the dangers of cryptocurrencies and the uselessness of blockchain, including this gem:

From its inception, this technology has been a solution in search of a problem and has now latched onto concepts such as financial inclusion and data transparency to justify its existence, despite far better solutions to these issues already in use.

Please feel free to tell me how he’s just another individual full of misguided opinions… after all, what does he know about crypto?

Worth Reading: On the Dangers of Cryptocurrencies…

Bruce Schneier wrote an article on the dangers of cryptocurrencies and the uselessness of blockchain, including this gem:

From its inception, this technology has been a solution in search of a problem and has now latched onto concepts such as financial inclusion and data transparency to justify its existence, despite far better solutions to these issues already in use.

Please feel free to tell me how he’s just another individual full of misguided opinions… after all, what does he know about crypto?

Repost: Buffers, Congestion, Jitter, and Shapers

Béla Várkonyi left a great comment on a blog post discussing (among other things) whether we need large buffers on spine switches. I don’t know how many people read the comments; this one is too valuable to be lost somewhere below the fold

You might want to add another consideration. If you have a lot of traffic aggregation even when the ingress and egress port are roughly at the same speed or when the egress port has more capacity, you could still have congestion. Then you have two strategies, buffer and suffer jitter and delay, or drop and hope that the upper layers will detect it and reduce the sending by shaping.

Repost: Buffers, Congestion, Jitter, and Shapers

Béla Várkonyi left a great comment on a blog post discussing (among other things) whether we need large buffers on spine switches. I don’t know how many people read the comments; this one is too valuable to be lost somewhere below the fold

You might want to add another consideration. If you have a lot of traffic aggregation even when the ingress and egress port are roughly at the same speed or when the egress port has more capacity, you could still have congestion. Then you have two strategies, buffer and suffer jitter and delay, or drop and hope that the upper layers will detect it and reduce the sending by shaping.

Worth Reading: Smart Highways or Smart Cars?

I stumbled upon an interesting article in one of my RSS feeds: should be build smart highways or smart cars?

The article eloquently explains how ridiculous and expensive it would be to put the smarts in the infrastructure, and why most everyone is focused on building smart cars. The same concepts should be applied to networking, but of course the networking vendors furiously disagree – the network should be as complex, irreplaceable, and expensive as possible. I collected a few examples seven years ago, and nothing changed in the meantime.

Worth Reading: Smart Highways or Smart Cars?

I stumbled upon an interesting article in one of my RSS feeds: should we build smart highways or smart cars?

The article eloquently explains how ridiculous and expensive it would be to put the smarts in the infrastructure, and why most everyone is focused on building smart cars. The same concepts should be applied to networking, but of course the networking vendors furiously disagree – the network should be as complex, irreplaceable, and expensive as possible. I collected a few examples seven years ago, and nothing changed in the meantime.

VLAN Module in netsim-tools Is Complete

One of the last things I did before starting the 2022 summer break was to push out the next release of netsim-tools.

It includes support for routed VLAN subinterfaces (needed to implement router-on-a-stick) and routed VLANs (needed to implement multi-hop VRF lite), completing the lengthy (and painful) development of the VLAN configuration module. Stefano Sasso added VLAN support for Mikrotik RouterOS and VyOS, and Jeroen van Bemmel completed VLAN implementation for Nokia SR Linux. Want to see VLANs on other platforms? Read the contributor guidelines and VLAN developer docs, and submit a PR.

I’ll be back in September with more blog posts, webinars, and cool netsim-tools features. In the meantime, automate everything, get away from work, turn off the Internet, and enjoy a few days in your favorite spot with your loved ones!

netlab VLAN Module Is Complete

One of the last things I did before starting the 2022 summer break was to push out the next netlab release.

It includes support for routed VLAN subinterfaces (needed to implement router-on-a-stick) and routed VLANs (needed to implement multi-hop VRF lite), completing the lengthy (and painful) development of the VLAN configuration module. Stefano Sasso added VLAN support for Mikrotik RouterOS and VyOS, and Jeroen van Bemmel completed VLAN implementation for Nokia SR Linux. Want to see VLANs on other platforms? Read the contributor guidelines and VLAN developer docs, and submit a PR.

I’ll be back in September with more blog posts, webinars, and cool netlab features. In the meantime, automate everything, get away from work, turn off the Internet, and enjoy a few days in your favorite spot with your loved ones!

MLAG Deep Dive: Layer-3 Forwarding

The layer-2 forwarding and flooding in an MLAG cluster are intricate but still reasonably easy to understand. Layer-3 gets more interesting; its quirks depend heavily on layer-2 implementation. While most MLAG implementations exhibit similar bridging behavior, expect interesting differences in routing behavior.

We’ll have to expand by-now familiar network topology to cover layer-3 edge cases. We’ll still work with two switches in an MLAG cluster, but we’ll have an external router attached to both of them. The hosts connected to the switches belong to two subnets (red and blue).

MLAG Deep Dive: Layer-3 Forwarding

The layer-2 forwarding and flooding in an MLAG cluster are intricate but still reasonably easy to understand. Layer-3 gets more interesting; its quirks depend heavily on layer-2 implementation. While most MLAG implementations exhibit similar bridging behavior, expect interesting differences in routing behavior.

We’ll have to expand by-now familiar network topology to cover layer-3 edge cases. We’ll still work with two switches in an MLAG cluster, but we’ll have an external router attached to both of them. The hosts connected to the switches belong to two subnets (red and blue).

VXLAN-to-VXLAN Bridging in DCI Environments

Almost exactly a decade ago I wrote that VXLAN isn’t a data center interconnect technology. That’s still true, but you can make it a bit better with EVPN – at the very minimum you’ll get an ARP proxy and anycast gateway. Even this combo does not address the other requirements I listed a decade ago, but maybe I’m too demanding and good enough works well enough.

However, there is one other bit that was missing from most VXLAN implementations: LAN-to-WAN VXLAN-to-VXLAN bridging. Sounds weird? Supposedly a picture is worth a thousand words, so here we go.

VXLAN-to-VXLAN Bridging in DCI Environments

Almost exactly a decade ago I wrote that VXLAN isn’t a data center interconnect technology. That’s still true, but you can make it a bit better with EVPN – at the very minimum you’ll get an ARP proxy and anycast gateway. Even this combo does not address the other requirements I listed a decade ago, but maybe I’m too demanding and good enough works well enough.

However, there is one other bit that was missing from most VXLAN implementations: LAN-to-WAN VXLAN-to-VXLAN bridging. Sounds weird? Supposedly a picture is worth a thousand words, so here we go.

Help Appreciated: netsim-tools Device Features

There are (at least) two steps to get new functionality (like VLANs) implemented in netsim-tools:

• We have to develop a data transformation module that takes high-level lab-, node-, link- or interface attributes and transforms them into device data.
• Someone has to create Jinja2 templates for each supported device that transform per-device netsim-tools data into device configurations.

I usually implement new features on Cisco IOSv and Arista EOS¹, Stefano Sasso adds support for VyOS, Dell OS10, and Mikrotik RouterOS, and Jeroen van Bemmel adds Nokia SR Linux and/or SR OS support. That’s less than half of the platforms supported by netsim-tools, and anything you could do to help us increase the coverage would be highly appreciated.

Help Appreciated: netsim-tools Device Features

There are (at least) two steps to get new functionality (like VLANs) implemented in netsim-tools:

• We have to develop a data transformation module that takes high-level lab-, node-, link- or interface attributes and transforms them into device data.
• Someone has to create Jinja2 templates for each supported device that transform per-device netsim-tools data into device configurations.

I usually implement new features on Cisco IOSv and Arista EOS¹, Stefano Sasso adds support for VyOS, Dell OS10, and Mikrotik RouterOS, and Jeroen van Bemmel adds Nokia SR Linux and/or SR OS support. That’s less than half of the platforms supported by netsim-tools, and anything you could do to help us increase the coverage would be highly appreciated.