Docker Swarm Services behind the Scenes

Remember the claim that networking is becoming obsolete and that everyone else will simply bypass the networking teams (source)?

Good news for you – there are many fast growing overlay solutions that are adopted by apps and security teams and bypass the networking teams altogether.

That sounds awesome in a VC pitch deck. Let’s see how well that concept works out in reality using Docker Swarm as an example (Kubernetes is probably even worse).

Worth Reading: Hardware Packet Capture Failures

Greg Ferro is back with some great technical content, this time explaining why hardware-based packet capture might return unexpected results.

MUST READ: What I’ve learned about scaling OSPF in Datacenters

Justin Pietsch published a fantastic recap of his experience running OSPF in AWS infrastructure. You MUST read what he wrote, here’s the TL&DR summary:

• Contrary to popular myths, OSPF works well on very large leaf-and-spine networks.
• OSPF nuances are really hard to grasp intuitively, and the only way to know what will happen is to run tests with the same codebase you plan to use in production environment.

Dinesh Dutt made similar claims on one of our podcasts, and I wrote numerous blog posts on the same topic. Not that anyone would care or listen, it’s so much better to watch vendor slide decks full of latest unicorn dust… but in the end, it’s usually not the protocol that’s broken, but the network design.

Podcast: Trusting Routing Protocols

The can we trust routing protocols series of blog posts I wrote in April 2020 (part 1, part 2, response from Jeff Tantsura) culminated in an interesting discussion with Russ White and Nick Russo now published as The Hedge Episode 43.

Which Public Cloud Should I Master First?

I got a question along these lines from a friend of mine:

Google recently announced a huge data center build in country to open new GCP regions. Does that mean I should invest into mastering GCP or should I focus on some other public cloud platform?

As always, the right answer is “it depends”, for example:

Worth Reading: NetDevOps Concepts – Minimum Viable Product

Brett Lykins published an excellent description of what an automation Minimum Viable Product could be.

Not surprisingly, he’s almost perfectly in sync with what we’ve been telling networking engineers in ipSpace.net Network Automation online course:

• Start small
• Go for quick wins
• Do read-only stuff before modifying device configurations
• Test, test, test…

Worth Reading: Redistributing Your Entire IS-IS Network By Mistake

Here’s an interesting factoid: when using default IS-IS configuration (running L1 + L2 on all routers in your network), every router inserts every IP prefix from anywhere in your network into L2 topology… at least on Junos.

For more details read this article by Chris Parker.

Worth Reading: Seamless Suffering

When someone sent me a presentation on seamless MPLS a long while ago my head (almost) exploded just by looking at the diagrams… or in the immortal words of @amyengineer:

“If it requires a very solid CCIE on an obscure protocol mix at 4am, it is a bad design” - Peter Welcher, genius crafter of networks, granter of sage advice.

Turns out I was not that far off… Dmytro Shypovalov documented the underlying complexity and a few things that can go wrong in Seamless Suffering.

MUST READ: IPv4, IPv6, and a Sudden Change in Attitude

Avery Pennarun continued his if only IPv6 would be less academic saga with a must-read IPv4, IPv6, and a sudden change in attitude article in which he (among other things) correctly identified IPv6 as a typical example of second-system effect:

If we were feeling snarky, we could perhaps describe IPv6 as “the String Theory of networking”: a decades-long boondoggle that attracts True Believers, gets you flamed intensely if you question the doctrine, and which is notable mainly for how much progress it has held back.

In the end, his conclusion matches what I said a decade ago: if only the designers of the original Internet wouldn’t be too stubborn to admit a networking stack needs a session layer. For more details, watch The Importance of Network Layers part of Networks Really Work webinar

Must Watch: How NOT to Measure Latency

A while ago someone pointed me to an interesting talk explaining why 99th percentile represents a pretty good approximation of user-experienced latency on a typical web page (way longer version: Understanding Latency and Application Responsiveness, also How I Learned to Stop Worrying and Love Misery)

If you prefer reading instead of watching videos, there’s also everything you know about latency is wrong.

MUST READ: SR(x)6 – Snake Oil Or Salvation?

I wanted to write a “SRv6 makes no little sense” blog post for a long while, but there were always more relevant topics to focus on. Fortunately I won’t have to write it anytime soon; Ethan Banks did a fantastic job with SR(x)6 - Snake Oil Or Salvation?. Make sure you read it before attending the next “SRx6 will save the world” vendor presentation.

Worth Reading: How CEOs think

Robert Graham wrote a great article explaining why CEOs don’t care much about cybersecurity or any other non-core infrastructure (including networking, unless you happen to be working for a service provider). It’s a must-read if you want to understand the **** you have to deal with in enterprise environments.

OMG, Not Again: New Mobile Internet Protocol Vulnerabilities

Every now and then a security researcher “discovers” a tunneling protocol designed to be used over a protected transport core and “declares it vulnerable” assuming the attacker can connect to that transport network… even though the protocol was purposefully designed that way, and everyone with a bit of clue knew the whole story years ago (and/or it’s even documented in the RFC).

It was MPLS decades ago, then VXLAN a few years ago, and now someone “found” a “high-impact vulnerability” in GPRS Tunnel Protocol. Recommended countermeasures: whitelist-based IP filtering. Yeah, it’s amazing what a wonderful new tool they found.

Unfortunately (for the rest of us), common sense never generated headlines on Hacker News (or anywhere else).

Worth Reading: entr: Rerun Your Build when Files Change

Julia Evans recently described another awesome Linux tool: entr allows you to run a bash command every time a watched file changes (and it works on Linux and OSX).

I wish I found it years ago…

Worth Reading: Written communication is remote work super power

Snir David wrote a great article explaining why you should focus on documenting stuff you do instead of solving other people’s challenges (or putting out fires) on Slack/Zoom/whatever. Enjoy ;)

Worth Reading: Working with TC on Linux systems

Here’s one of the weirdest ideas I’ve found recently: patch together two dangling ends of virtual Ethernet cables with PBR.

To be fair, Jon Langemak used that example to demonstrate how powerful tc could be. It’s always fun to see a totally-unexpected aspect of Linux networking… even though it looks like the creators of those tools believed in Perl mentality of creating a gazillion variants of line noise to get the job done.

Worth Reading: Lies, Damned Lies, and Keynotes

Got sick and tired of conference keynotes? You might love the Lies, Damned Lies, and Keynotes rant by Corey Quinn. Here are just two snippets:

They’re selling a fantasy, and you’ve been buying it all along.

We’re lying to ourselves. But it feels better than the unvarnished truth.

Enjoy!

Summer Break 2020

Almost 30 webinars, an online course, and over 140 blog posts later it’s time for another summer break.

While we’ll do our best to reply to support and sales requests (it might take us a bit longer than usual), don’t expect anything deeply technical for the new two months… but of course you can still watch over 280 hours of existing content, listen to over 100 podcast episodes, or read over 3500 blog posts.

We’ll be back with tons of new content in early September.

In the meantime, automate everything, get away from work, turn off the Internet, and enjoy a few days in your favorite spot with your loved ones!

BGP Navel Gazing on Software Gone Wild

This podcast introduction was written by Nick Buraglio, the host of today’s podcast.

As we all know, BGP runs the networked world. It is a protocol that has existed and operated in the vast expanse of the internet in one form or another since early 1990s, and despite the fact that it has been extended, enhanced, twisted, and warped into performing a myriad of tasks that one would never have imagined in the silver era of internetworking, it has remained largely unchanged in its operational core.

The world as we know it would never exist without BGP, and because of the fact that it is such a widely deployed protocol with such a solid track record of “just working”, the transition to a better security model surrounding it has been extraordinarily slow to modernize.

Adapting Network Design to Support Automation

This blog post was initially sent to the subscribers of my SDN and Network Automation mailing list. Subscribe here.

Adam left a thoughtful comment addressing numerous interesting aspects of network design in the era of booming automation hype on my How Should Network Architects Deal with Network Automation blog post. He started with:

A question I keep tasking myself with addressing but never finding the best answer, is how appropriate is it to reform a network environment into a flattened design such as spine-and-leaf, if that reform is with the sole intent and purpose to enable automation?

A few basic facts first: