Disaster Recover and Failure Domains

One of the responses to my Disaster Recovery Faking blog post focused on failure domains:

What is the difference between supporting L2 stretched between two pods in your DC (which everyone does for seamless vMotion), and having a 30ms link between these two pods because they happen to be in different buildings?

I hope you agree that a single broadcast domain is a single failure domain. If not, let agree to disagree and move on - my life is too short to argue about obvious stuff.

Tuning BGP Convergence in High-Availability Firewall Cluster Design

Two weeks ago Nicola Modena explained how to design BGP routing to implement resilient high-availability network services architecture. The next step to tackle was obvious: how do you fine-tune convergence times, and how does BGP convergence compare to the more traditional FHRP-based design.

You Still Need a Networking Engineer for a Successful Cloud Deployment

You’ve probably heard cloudy evangelists telling CIOs how they won’t need the infrastructure engineers once they move their workloads into a public cloud. As always, whatever sounds too good to be true usually is. Compute resources in public clouds still need to be managed, someone still needs to measure application performance, and backups won’t happen by themselves.

Even more important (for networking engineers), network requirements don’t change just because you decided to use someone else’s computers:

Questions to Ask About Product Using Overhyped Technology

I stumbled upon a great MIT Technology Review article (warning: regwall ahead) with a checklist you SHOULD use whenever considering a machine-learning-based product.

While the article focuses on machine learning at least some of the steps in that list apply to any new product that claims to use a brand new technology in a particular problem domain like overlay virtual networking with blockchain:

Worth Reading: Resilience Engineering

Starting with my faking disaster recovery tests blog post Terry Slattery wrote a great article delving into the intricacies of DR testing, types of expected disasters, and resilience engineering. As always, a fantastic read from Terry.

IP Fabric with Gian-Paolo Boarina on Software Gone Wild

No, we were not talking about IP fabrics in general - IP Fabric is a network management software (oops, network assurance platform) Gian Paolo discovered a while ago and thoroughly tested in the meantime.

He was kind enough to share what he found in Episode 107 of Software Gone Wild, and as Chris Young succinctly summarized: “it’s really sad what we still get excited about something 30 years after it was first promised”… but maybe this time it really works ;)

Upcoming Events and Webinars (December 2019)

The registration is still open for the Using VXLAN to Build Active-Active Data Centers workshop on December 3rd, but if you can’t make it to Zurich you might enjoy these live sessions we’ll run in December 2019:

• In the last session of autumn 2019 network automation course (December 5th) we’ll have Jeremy Schulman talking about ChatOps and Slack integration;
• On December 10th Matthias Luft will continue the Cloud Security series with a session focused on identity and access management (IAM);
• Remember the Business Aspects of Networking Technologies idea I started in April? Finally I found time for the second session (on December 12th), this time focused on addressing the business challenges instead of getting excited about technology;
• We’ll conclude the Autumn 2019 webinar season with another session in the EVPN Deep Dive saga on December 17th. This time we’ll have Krzysztof Szarkowicz, the author of MPLS in the SDN Era talk about MPLS-based EVPN.

All webinars I mentioned above are accessible with Standard ipSpace.net Subscription, and you’ll need Expert Subscription to enjoy the automation course contents.

Is There a Future for Networking Engineers?

Someone sent me this observation after reading my You Cannot Have Public Cloud without Networking blog post:

As much as I sympathize with your view, scales matter. And if you make ATMs that deal with all the massive client population, the number of bank tellers needed will go down. A lot.

Based on what I read a while ago a really interesting thing happened in financial industry: while the number of tellers went down, number of front-end bank employees did not go down nearly as dramatically, they just turned into “consultants”.

The EVPN Dilemma

Got an interesting set of questions from a networking engineer who got stuck with the infamous “let’s push the **** down the stack” challenge:

So I am a rather green network engineer trying to solve the typical layer two stretch problem.

I could start the usual “friends don’t let friends stretch layer-2” or “your business doesn’t really need that” windmill fight, but let’s focus on how the vendors are trying to sell him the “perfect” solution:

Upcoming Workshops: NSX, ACI, VXLAN, EVPN, DCI and More

I’m running two workshops in Zurich in the next 10 days:

• Comparing VMware NSX and Cisco ACI (and how EVPN and VXLAN fit into the big picture) on Thursday, November 28th;
• Explaining how you could use VXLAN with EVPN to build infrastructure for active-active data centers on Tuesday, December 3rd.

I published the slide deck for the NSX versus ACI workshop a few days ago (and you can already download it if you have a paid ipSpace.net subscription) and it’s full of new goodness like ACI vPod, multi-pod ACI, multi-site ACI, ACI-on-AWS, and multi-site NSX-V and NSX-T.

Worth Reading: Early History of Usenet

Steve Bellovin wrote a great series of articles describing the early history of Usenet. The most interesting part in the “security and authentication” part was probably this gem:

That left us with no good choices. The infrastructure for a cryptographic solution was lacking. The uux command rendered illusory any attempts at security via the Usenet programs themselves. We chose to do nothing. That is, we did not implement fake security that would give people the illusion of protection but not the reality.

A lot of other early implementers chose the same route, resulting in SMTP, BGP… which wouldn’t be a problem if someone kept track of that and implemented security a few years later. Unfortunately we considered those problems solved and moved on to chase other squirrels. We’re still paying the interest on that technical debt.

Worth Reading: Going Back to Being an Engineer

Another great advice from Charity Majors: does it make sense to go back to being an engineer after being a manager for a few years?

Personal note: finding a great replacement for my CTO role was probably the best professional decision I ever made ;)

Video: Breaking the End-to-End Principle

Original TCP/IP and OSI network stacks had relatively clean layered architecture (forgetting the battle scars for the moment) and relied on end-to-end principle to keep the network core simple.

As always, no good deed goes unpunished - “creative” individuals trying to force-fit their mis-designed star-shaped pegs into round holes, and networking vendors looking for competitive advantage quickly destroyed the idea with tons of middlebox devices, ranging from firewalls and load balancers to NAT, WAN optimization, and DPI monstrosities.

You need free ipSpace.net subscription to watch the video, or a paid ipSpace.net subscriptions to watch the whole How Networks Really Work webinar.

Guest Speakers in the Networking in Public Cloud Deployments Online Course

We are proud to announce a great lineup of guest speakers for the first Networking in Public Cloud Deployments course that will run in Spring 2020:

• Joep Piscaer will dive into what changes public clouds bring and what these changes mean for you, as well as what developers and other consumers of cloud resources expect from you in the new public cloud, DevOps and Infrastructure-as-Code world.
  
• Ned Bellavance will review the principles of Infrastructure as Code (IaC) and how they apply to public cloud solutions. Then he will take a look at the landscape of IaC tools that exist and examine their pros and cons.
  
• Howard Marks will review the types of storage available across public clouds, how they differ between cloud providers and the applications and pitfalls associated with each of them.
  
• Connecting on-premises data centers or office locations to a public cloud has some unique challenges. Ed Horley will help you create a framework and a checklist to make sure you have the required redundancy, throughput, routing, and security all baked in from day one.
  
• Matthias Luft will cover the aspects of securing your public cloud deployments.
  
• Justin Warren will explain how to make good tradeoffs between Continue reading

Automation Solution: L3VPN Topology

Here’s another “let’s use network automation tools to create reports we couldn’t get in the past” (like IP multicast trees) solution coming from an attendee in our network automation course: Paddy Kelly created L3VPN graphs detailing PE-to-CE connectivity using Cisco’s pyATS to parse the Cisco IOS printouts.

You’ll find dozens of other interesting solutions on our Sample Network Automation Solutions page - all of them were created by networking engineers who knew almost nothing about network automation or open-source automation tools when they started our automation course.

Using BGP for Firewall High Availability: Design and Software Upgrades

Remember the “BGP as High Availability Protocol” article Nicola Modena wrote a few months ago? He finally found time to extend it with BGP design considerations and a description of a seamless-and-safe firewall software upgrade procedure.

Facts and Fiction: BGP Is a Hot Mess

Every now and then a smart person decides to walk away from their competence zone, and start spreading pointless clickbait opinions like BGP is a hot mess.

Like any other technology, BGP is just a tool with its advantages and limitations. And like any other tool, BGP can be used sloppily… and that’s what’s causing the various problems and shenanigans everyone is talking about.

Just in case you might be interested in facts instead of easy-to-digest fiction:

Worth Reading: the Lure of the Easy Button

Russ White wrote a great blog post explaining why you have to understand the problem you’re solving instead of blindly believing the $vendor slide deck… or as I said a long time ago, think about how you’ll troubleshoot your network in because you won’t be able to reformat it once it crashes.

Your First Public Cloud Deployment Should Be Small

I’ve seen successful public (infrastructure) cloud deployments… but also spectacular failures. The difference between the two usually comes down to whether the team deploying into a public cloud environment realizes they’re dealing with an unfamiliar environment and acts accordingly.

Please note that I’m not talking about organizations migrating their email to Office 365. While that counts as public cloud deployment when an industry analyst tries to paint a rosy picture of public cloud acceptance, I’m more interested in organizations using compute, storage, security and networking public cloud infrastructure.

Can We Really Use Millions of VXLAN Segments?

One of my readers sent me a question along these lines…

VXLAN Network Identifier is 24 bit long, giving 16 us million separate segments. However, we have to map VNI into VLANs on most switches. How can we scale up to 16 million segments when we have run out of VLAN IDs? Can we create a separate VTEP on the same switch?

VXLAN is just an encapsulation format and does not imply any particular switch architecture. What really matters in this particular case is the implementation of the MAC forwarding table in switching ASIC.