How Microsoft Azure Orchestration System Crashed My Demos

One of the first things I realized when I started my Azure journey was that the Azure orchestration system is incredibly slow. For example, it takes almost 40 seconds to display six routes from per-VNIC routing table. Imagine trying to troubleshoot a problem and having to cope with 30-second delay on every single SHOW command. Cisco IGS/R was faster than that.

If you’re old enough you might remember working with VT100 terminals (or an equivalent) connected to 300 baud modems… where typing too fast risked getting the output out-of-sync resulting in painful screen repaints (here’s an exercise for the youngsters: how long does it take to redraw an 80x24 character screen over a 300 bps connection?). That’s exactly how I felt using Azure CLI - the slow responses I was getting were severely hampering my productivity.

Feedback: Ansible for Networking Engineers

I always love to hear from networking engineers who managed to start their network automation journey. Here’s what one of them wrote after watching Ansible for Networking Engineers webinar (part of paid ipSpace.net subscription, also available as an online course).

This webinar helped me a lot in understanding Ansible and the benefits we can gain. It is a big area to grasp for a non-coder and this webinar was exactly what I needed to get started (in a lab), including a lot of tips and tricks and how to think. It was more fun than I expected so started with Python just to get a better grasp of programing and Jinja.

In early 2019 we made the webinar even better with a series of live sessions covering new features added to recent Ansible releases, from core features (loops) to networking plugins and new declarative intent modules.

Running OSPF in a Single Non-Backbone Area

One of my subscribers sent me an interesting puzzle:

>One of my colleagues configured a single-area OSPF process in a customer VRF customer, but instead of using area 0, he used area 123 nssa. Obviously it works, but I was thinking: “What the heck, a single OSPF area MUST be in Area 0”

Not really. OSPF behaves identically within an area (modulo stub/NSSA behavior) regardless of the area number…

Switch Buffer Sizes and Fermi Estimates

In my quest to understand how much buffer space we really need in high-speed switches I encountered an interesting phenomenon: we no longer have the gut feeling of what makes sense, sometimes going as far as assuming that 16 MB (or 32MB) of buffer space per 10GE/25GE data center ToR switch is another $vendor shenanigan focused on cutting cost. Time for another set of Fermi estimates.

Let’s take a recent data center switch using Trident II+ chipset and having 16 MB of buffer space (source: awesome packet buffers page by Jim Warner). Most of switches using this chipset have 48 10GE ports and 4-6 uplinks (40GE or 100GE).

Use Per-Link Prefixes in Network Data Models

We got pretty far in our data deduplication in network data model journey, from initial attempts to network modeled as a graph… but we still haven’t got rid of all the duplicate information.

For example, if we have multiple devices connected to the same subnet, why should we have to specify IP address and subnet mask for every device (literally begging the operators to make input errors). Wouldn’t it be better (assuming we don’t care about exact IP addresses on core links) to assign IP addresses automatically?

Repost: Automation Without Simplification

The No Scripting Required to Start Your Automation Journey blog post generated lively discussions (and a bit of trolling from the anonymous peanut gallery). One of the threads focused on “how does automation work in real life IT department where it might be challenging to simplify operations before automating them due to many exceptions, legacy support…”

Here’s a great answer provided by another reader:

As Expected: Where Have All the SDN Controllers Gone?

Roy Chua (SDx Central) published a blog post titled “Where Have All the SDN Controllers Gone” a while ago describing the gradual disappearance of SDN controller hype.

No surprise there - some of us were pointing out the gap between marketing and reality years ago.

It was evident to anyone familiar with how networking actually works that in a generic environment the drawbacks of orthodox centralized control plane SDN approach far outweigh its benefits. There are special use cases like intelligent patch panels where a centralized control plane makes sense.

Stop Using GUI to Configure SDN or Intent-Based Products

This blog post was initially sent to subscribers of my SDN and Network Automation mailing list. Subscribe here.

At the end of my vNIC 2018 keynote speech I made a statement along these lines:

The moment you start using GUI with an SDN product you’re back to square one.

That claim confused a few people – Mark left this comment on my blog:

Do Packet Drops Matter for TCP Performance?

Approximately two years ago I tried to figure out whether aggressive marketing of deep buffer data center switches makes sense, recorded a few podcasts on the topic and organized a webinar with JR Rivers.

Not surprisingly, the question keeps popping up, so it seems it’s time for another series of TL&DR articles. Let’s start with the basics:

Generalize the Network-as-Graph Data Model

Remember the avoid duplicate data in network automation data models challenge and the restructuring we did to represent a network as a graph.

Well, I was not happy with the end result - I hated the complexity of supporting Jinja2 templates that had to check left- and right nodes of a link, so I generalized the data structure a bit, and all of a sudden I could model stub interfaces, P2P links and multi-access networks.

Know Thy Environment Before Redesigning It

A while ago I had an interesting consulting engagement: a multinational organization wanted to migrate off global Carrier Ethernet VPN (with routers at the edges) to MPLS/VPN.

While that sounds like the right thing to do (after all, L3 must be better than L2, right?) in that particular case they wanted to combine the provider VPN with Internet-based IPsec VPN… and doing that in parallel with MPLS/VPN tends to become an interesting exercise in “how convoluted can I make my design before I give up and migrate to BGP”.

MUST READ: Thou shalt not commit logical fallacies

Found a fantastic list of common logical fallacies. It's a must read for anyone having at least occasional interaction with non-Vulcans... and when you stop laughing (or screaming, or both) make sure you go through the companion web site to understand bugs in your wetware that sabotage your attempts at being perfectly logical.

Upcoming Webinars and Events (June 2019)

I’m always amazed at how fast the time flies. I have no idea where May disappeared to, it seems like it was only yesterday when I was writing about webinar plans in 2019… and yet it’s only a month till ipSpace.net Summer Break™.

During June 2019 I’ll continue updating Designing the Private Cloud Infrastructure webinar, and start a new pet project: How Networks Really Work – I’m literally minutes away from traveling to a quiet spot in the middle of nowhere where I’ll work on the materials. In between these webinars you’ll find me in Zurich where I’ll run Microsoft Azure Networking workshop on June 12^th in parallel with SIGS Technology Conference.

As you might expect we have plenty of things already lined up for autumn 2019… more about that in a week or two.

Remember: Don’t Panic

I hate listening to “this is what we were doing this year” podcasts as they usually turn into pointless blabbering, self-congratulations and meaningless plans (think New Year resolutions). The Full Stack Journey Episode 28 with Scott Lowe was an amazing deviation from this too-common template.

If you don’t have time to listen to the podcast (but you OUGHT TO do it) here’s what I loved most: “When faced with the onslaught of new technologies, don’t panic. Wait a few months to see which ones survive”.

IPv6 Support in Microsoft Azure

TL&DR: MIA

Six years ago, when I was talking about overlay virtual networks at Interop, I loved to joke that we must be living on a weird planet where Microsoft has the best overlay virtual networking implementation… at least as far as IPv6 goes.

Even then, their data plane implementation which was fully dual-stack-aware on both tenant- and underlay level was way ahead of what System Center could do.

Model Your Network as a Graph not a Set of Boxes

Last week I explained how you could take a typical first attempt at a network automation data model and reduce the amount of duplicate data… but the data model we used was still describing a set of seemingly disconnected boxes.

How about restructuring the whole thing and describing what networks really are - graphs made of nodes (network devices) and links?

Keep reading

It’s Time for Another Pet Project

More than a decade ago I decided to start a pet project: a blog describing interesting details of networking technologies. The idea quickly morphed into vendor-neutral webinars - the first one took place in February 2010. A year or two later I had my first guest speaker and as of today we had more than 50 industry experts participating in ipSpace.net webinars and online courses.

In the meantime the ipSpace.net team grew: I had video and audio editors for years, Irena Marčetič took over marketing, logistics, and production in 2018, and we got a team of webinar moderators that will help us with guest speaker webinars (last week we ran the first guest speaker webinar where I didn’t have to be involved - hooray ;)

Worth Reading: Overly Attached

Kate Matsudaira wrote a nice article explaining how to deal with emotional attachment to a project you spent a lot of time working on. While she's focusing on software development, the same fallacies apply to networking - sometimes it's time to let the old pile of **** die and replace it with something created in this decade.

If You Worry About 768K Day, You’re Probably Doing Something Wrong

A few years ago we “celebrated” 512K day - the size of the full Internet routing table exceeded 512K (for whatever value of K ;) prefixes, overflowing TCAMs in some IP routers and resulting in interesting brownouts.

We’re close to exceeding 768K mark and the beware 768K day blog posts have already started appearing. While you (RFC 2119) SHOULD check the size of your forwarding table and the maximum capabilities of your hardware, the more important question should be “Why do I need 768K forwarding entries if I’m not a Tier-1 provider”

How Hard Is It to Manage Your Intent?

This blog post was initially sent to subscribers of my SDN and Network Automation mailing list. Subscribe here.

Remember the “every device configuration is really an expression of our intent” discussion? Forgetting the wrong level of abstraction (we mostly don’t want to deal with all the idiosyncratic stuff network devices want to see in their configurations) and box-oriented thinking caused by device-level intent for the moment, let’s focus on another aspect: how hard is it to manage your intent?