Sometimes It’s Not the Network

Marek Majkowski published an awesome real-life story on CloudFlare blog: users experienced occasional short-term sluggish performance and while everything pointed to a network problem, it turned out to be a garbage collection problem in Linux kernel.

Takeaway: It might not be the network's fault.

Also: How many people would be able to troubleshoot that problem and fix it? Technology is becoming way too complex, and I don’t think software-defined-whatever is the answer.

Is Flow-Based Forwarding Just Marketing Fluff?

When writing the Packet- and Flow-Based Forwarding blog post, I tried to find a good definition of flow-based forwarding (and I was not the only one being confused), and the one from Junos SRX documentation is as good as anything else I found, so let’s use it.

TL&DR: Flow-based forwarding is a valid technical concept. However, when mentioned together with OpenFlow, it’s mostly marketing fluff.

Detecting NAT64 Prefix

If you’re a host running on an IPv6-only network, you might want to detect the IPv6 prefix used for NAT64 (for example, to transform IPv4 literals a clueless idiot embedded into a URL into IPv6 addresses).

Apple has a wonderful developer-focused page describing NAT64 and DNS64, including the way they synthesize IPv6 addresses from IPv4 literals. You (RFC 6919) MUST read it.

Packet- and Flow-Based Forwarding

I don’t like to correct my friends in public, but if someone says “I still believe that flow-based technologies will exceed the capabilities of packet-based technologies” (see Network Break 53), it’s time to revisit the networking fundamentals.

What is a flow?

According to Wikipedia (but what do they know…):

Fibbing: OSPF-Based Traffic Engineering with Laurent Vanbever

You might be familiar with the idea of using BGP as an SDN tool that pushes forwarding entries into routing and forwarding tables of individual devices, allowing you to build hop-by-hop path across the network (more details in Packet Pushers podcast with Petr Lapukhov).

Researchers from University of Louvain, ETH Zürich and Princeton figured out how to use OSPF to get the same job done and called their approach Fibbing. For more details, listen to Episode 45 of Software Gone Wild podcast with Laurent Vanbever (one of the authors), visit the project web site, or download the source code.

Thank you for your trust!

Wow, another year swooshed by. I can’t believe it’s almost gone. Maybe it’s all the travels I had throughout the year, and I MUST start with a huge THANK YOU to whoever is watching after me – there wasn’t a single major SNAFU.

Next, I’d like to thank the people who caused all that travel: attendees of my workshops.

Ethernet Checksums Are Not Good Enough for Storage (Updated)

A while ago I described why some storage vendors require end-to-end layer-2 connectivity for iSCSI replication.

TL&DR version: they were too lazy to implement iSCSI checksums and rely on Ethernet checksums because TCP/IP checksums are not good enough.

It turns out even Ethernet checksums fail every now and then.

2015-12-06: I misunderstood the main technical argument in Evan’s post. The real problem is that switches recalculate CRC, so the Ethernet CRC is no longer end-to-end protection mechanism.

Blogging Rule#1: Be Useful

I love stumbling upon new networking-focused blogs. Many of my old friends switched to the dark side vendors and stopped blogging, others simply gave up, and it seems like there aren’t that many engineers that would like to start this experiment.

One of the obvious first questions is always “what should I write about” and my reply is always “it doesn’t really matter – make sure it’s useful.”

Can You Afford to Reformat Your Data Center?

I love listening to the Datanauts podcast (Ethan and Chris are fantastic hosts), starting from the very first episode (hyper-converged infrastructure) in which Chris made a very valid comment along the lines of “with the hyper-converged infrastructure it’s possible to get so many things done without knowing too much about any individual thing…” and I immediately thought “… and what happens when it fails?”

Video: Cumulus Linux Architecture

Do you want to know more about Cumulus Linux after learning what data center architectures it supports, what base technologies it uses, and how you can use it to simplify network configurations? It’s time to explore Cumulus Linux architecture (part 5 of the presentation Dinesh Dutt had during the Data Center Fabrics webinar).

DMVPN Split Default Routing and Internet Access

One of the engineers listening to my DMVPN webinars sent me a follow-up question (yes, I always try to reply to them) asking how to implement direct Internet access from the spoke sites (aka local exit) in combination with split default routing you have to use in DMVPN Phase 2 or Phase 3 networks.

It’s really simple: either you have a design requirement that requires split default routing, or you don’t.

IP over Ultrasound

One of my readers read the Ars Technica article on ads communicating with other devices via ultrasound and wondered whether something similar could be done for IP.

Not surprisingly, someone already did it. A quick google search found this tutorial which explains how to run IP stack over Gnuradio (at speeds that were last experienced with dial-up modems 30 years ago).

Presentation: All You Need Are Two Switches

I was asked to present a data-center-related talk last week and decided to focus on one of my favorite topics: because most people don’t have more than a few hundred servers in their data center, they don’t need more than two switches (or a rack of servers).

Not surprisingly, an equipment reseller sitting in the room was not amused.

The video and the slide deck are already online, but there’s a minor challenge: the whole event was in Slovenian ;) However, I plan to record the same topic in English once my SDN travels stop.

Test-Driven Network Development with Michael Kashin on Software Gone Wild

Imagine you’d design your network by documenting the desired traffic flow across the network under all failure conditions, and only then do a low-level design, create configurations, and deploy the network… while being able to use the desired traffic flows as a testing tool to verify that the network still behaves as expected, both in a test lab as well as in the live network.

Control-Plane Protocols for Overlay Virtual Networking – the Madness Continues

You might remember all the fuss about various encapsulations used in overlay virtual networking… just because one wouldn’t be good enough (according to Andrew Lerner “we provide users with choice” actually means “we can’t decide which product to offer you”).

Junos Fusion: the First Steps (updated)

I was really excited when Juniper announced Junos Fusion. I hoped for QFabric Done Right, but after watching the NFD10 video describing the architecture, I was disappointed: they reinvented Fabric Extenders.

The blog post was slightly updated on November 14th 2015 based on feedback received from Juniper engineers.

Andrew Lerner on Vendorspeak

Andrew Lerner, my favorite Gartner analyst, recently published a hilarious blog post describing what vendors mean when they say “our product is software-defined” or “we’ll make it work”. Enjoy!

Need more vendorspeak? Try eight levels of vendor acceptance (carefully documented during a particularly stressful on-site test in Poland).

The Numerous Levels of SDN Reality

A newbie exploring the mythical lands of SDN might decide to start at the ONF definition of SDN, which currently (November 2015) starts with a battle cry:

The physical separation of the network control plane from the forwarding plane, and where a control plane controls several devices.

The rest of that same page is what I’d call the marketing definition of SDN: directly programmable, agile, centrally managed, programmatically configured, open standards based and vendor-neutral.

Video: Control Plane Protocols in OpenFlow-Based Networks

One of the typical questions I get in my SDN workshops is “how do you run control-plane protocols like LACP or OSPF in OpenFlow networks?”.

I wrote a blog post describing the process two years ago and we discussed the details of this challenge in the OpenFlow Deep Dive webinar. That part of the webinar is now public: you’ll find the OpenFlow Use Cases: Control-Plane Protocols video on the ipSpace.net Free Content web site.

There’s a Problem with IPv6 Multihoming

In an amazing turn of events, at least one IETF working group recognized we have serious problems with IPv6 multihoming. According to the email Fred Baker sent to a number of relevant IETF working groups:

PI multihoming demonstrably works, but PA multihoming when the upstreams implement BCP 38 filtering requires the deployment of some form of egress routing - source/destination routing in which the traffic using a stated PA source prefix and directed to a remote destination is routed to the provider that allocated the prefix. The IETF currently has no such recommendation, or consensus that it should have.

Here are a few really old blog posts just in case you don’t know what I’m talking about (and make sure you read the comments as well):