Rama Darbha

Author Archives: Rama Darbha

UCMP: Augmenting L3 only designs

An aspiration of modern web scale networking is to leverage a pure L3 solution and integrate it with anycast addresses to allow for load balancing functionality. So why is this design aspirational? Well it requires discipline in the way that applications are architected— specifically around tenancy requirements and application redundancy. In this blog I’ll discuss a recent augmentation that was introduced into Cumulus Linux 4.1 that makes this style of design much more flexible in web scale networks.

Common challenges

Two common challenges when using anycast addressing in layer 3 only solutions are:

  • Resilient hashing to support the change in ECMP paths
  • Balancing traffic across unequal cost advertisements

Solutions

The first solution was implemented back in the early version of Cumulus Linux and is well documented. This solution is also known as “RASH” as the colloquial term.

The second solution addresses an interesting artifact of the way Layer 3 routes are advertised and learned, specifically with regards to next hop selection. Let us imagine the following simplified design:

The IP address of 192.168.1.101 is an anycast address that is being advertised by 3 different hosts in our environment. These three hosts are all serving the exact Continue reading

Capex vs Opex – Hidden complexity by making an unmanageable network

Many networking solutions purport great Opex savings through automation, simulation and continuous integration. Similarly, there is a school of thought where network designs will have a single point in a network perform multiple roles. This will short change an initial Capex cost of purchasing additional switches with the intention of overlapping features on that single device.

Let’s take the simplest example. We have a 3 rack environment with dual-leaf per rack and 2 spines for inter-rack connectivity. In this design, we are leveraging VXLAN as the data plane overlay with BGP/EVPN as the control plane. Additionally, all 3 racks are compute, leaving no additional leafs to act as the service/border/exit leafs.

A network designer will look at the infrastructure and try to overlap features by repurposing the spines as exit leafs. Why will they think this way, you ask? Well, this is only an 8 switch design. Spending money on an additional 2 switches to act as dedicated border leafs uplifts my capex cost by 25 percent! I would then be required to buy 10 total switches instead of 8.

So instead, we end up overlaying the VXLAN onto the spines. So now the spines act as both interconnections between Continue reading

Using ACLs for security vs compliance

Compliance exists in many forms, with tenancy, traffic isolation and access restriction. Compliance is mostly due to regulatory needs, which really comes down to security needs. Compliance applies to networking, fundamentally ensuring that resources can only be accessed from allowed locations. The actual media and content normally isn’t pertinent, as they merely just influence the scope of access for the data.

As a result, this post doesn’t delve into more complex compliance requirements such as deep packet inspection or encryption. Rather, this is to discuss how networking engineers use their existing toolset to enforce compliance requirements.

The most fundamental of these requests is networking access permissions. Most customers will allocate subnets for functional sections of their network.

ACLs are some of the most classic and undemanding forms of permission. The greatest part of ACLs is that they can be applied on nearly every networking device, and provide somewhat of a base level of security. But there are hidden complexities with ACLs that may not make it the ideal choice for most compliance solutions:

1. Connection State Tracking

ACLs are unidirectional elements, examining packets one at a time and only blocking traffic in a single direction. This can create some logical Continue reading

A new era for Cumulus in the Cloud

When we launched Cumulus in the Cloud (CitC) over two years ago, we saw it as a way for our customer base to test out Cumulus Linux in a safe sandboxed environment. Looking back, September 2017 feels like an eternity ago.

Since then, CitC has become a place where we’ve been able to roll out new functionality and solutions to customers and Cumulus-curious alike — and we’ve done some really interesting things (some of our favs include integrating it with an Openstack demo and Mesos demo). It’s pretty much become a Cumulus technology playground.

As our CitC offering has evolved, we’ve also taken stock of the requirements from our customers and realized the direction we want to take CitC. So where is it heading? We’re excited to share that with the launch of our production-ready automation solution last week, CitC will have a new user experience and user interface.

Out with the old:

In with the new:

This redesigned UI comes with some really great enhancements:

  • Customized external connectivity to oob-mgmt-server to run user customized applications

  • Default lifetime increased to 12 hours

  • NetQ native integration within the demo

Cumulus Networks launches the industry’s first open source and fully packaged automation solution — making open networking easier to deploy and manage and enabling infrastructure-as-code models

Today, Cumulus Networks is announcing the release of its production-ready automation solution for organizations moving towards fully automated networks in order to take advantage of infrastructure-as-code deployment models.

At the forefront of the networking industry, we see our customers caught in the shifting tides as the modern data center moves toward fully automated networking. As they look to take advantage of innovative technology like 5G, cloud, IoT and more, organizations are looking to innovative networking deployments that incorporate new ways of thinking about automation like infrastructure-as-code, CI/CD and more. As network traffic continues to grow at an exponential rate, organizations are left with infrastructure that is harder to manage and deploy. Bogged down by the cost and time it takes to build out bits and pieces of fully automated solutions, these organizations are in need of a solution to help them innovate their networks at the speed business demands.

Cumulus is now offering the first open source, out-of-the-box, robust, end-to-end automated configuration and testing solution using Ansible. Customers no longer have to piece together their network automation from disparate and untested scripts and proof-of-concept playbooks. Cumulus is offering a framework for an elegant push-button solution for those looking for cutting-edge Continue reading

VXLAN/EVPN vs VRF Lite

A little background information first.

When having a business requirement of tenancy, most solutions will tend to lean towards VRF. That is because VLANs require a distributed L2 environment, which comes with spanning tree, mlag and a whole other glut of inefficient network control plane protocols. Upleveling the infrastructure to L3 ends up requiring VRF technology to enforce tenancy.

Once you’ve settled on this feature as the solution for the business requirement, the next question is: How do I successfully deploy VRFs in a large distributed environment at scale, that also allows me to minimize the burden of management while still enforcing tenancy in all the important parts of my network? Most conversations surrounding this question will lead down two solution paths:

  1. VXLAN with EVPN
  2. VRF Lite

Definitions of VXLAN with EVPN and VRF Lite.

VXLAN with EVPN leverages VRFs at every border and leaf switch, while all the intermediate devices (ie. spines, super spines) only see the encapsulated VXLAN traffic, and hence do not need any VRF intelligence or visibility.

A VRF Lite solution is fundamentally simpler since it uses less moving parts. The thought of enabling the EVPN address family and encapsulating traffic into a VXLAN tunnel Continue reading

Tenancy and the importance of defining it.

One of the common starting points during the IT architecture process is trying to define and identify tenancy. Part of the problem is that the term “tenant” has various definitions depending on the business unit. As a network engineer, I don’t care about one of those definitions of the word: “Layer 3 subnet isolation with a unified set of routing and security policies.” Trying to communicate that to technical leaders in other IT organizations is hard enough, but trying to communicate this to a non-technical business leader can feel like an impossible task.

The goal then becomes trying to distill the complexities of a tenant into a consumable morsel.

What happened?

In my most recent consulting engagement, I worked with a company that had three key players during the design stage:

  • Server team
  • Network team
  • Business team

We kept using the word “tenancy” thinking we were speaking the same language , but we slowly realized that we weren’t speaking with the same definitions. In my network-first definition, I only cared about the subnets allocated to the servers and whether the servers were able to talk to each other. From my definition of tenancy, any server allocated to a tenant Continue reading

Upgrade Scripts

We need to do “upgrades in the network” is one of those phrases that chills the bones of all IT engineers. Upgrades don’t have to be so painful and in this blog, we’re going to discuss the upgrade process recommended by Cumulus and leave you with some example automation to make the process as efficient as possible.

Upgrades are necessary to maintain stable and secure code but bring the risk of new bugs and sustained outages due to unforeseen circumstances, and they’re generally not very easy to perform. Anyone who has worked network operations knows that upgrade windows could run as quickly as an hour or as long as all night (and maybe for the next three nights). Even as I write this I am remembering experiences from upgrade windows of old where things did not go according to plan. But before we get into the specifics of the upgrade process with Cumulus, it is worth discussing why upgrades in the network are so fraught with peril.

DISCLAIMER: Rant Incoming

The biggest impediment to network upgrades is complexity. When we say complexity we mean the conscious choice to add complexity into the design of the network that most folks undertake Continue reading

Operations guide

One of the most common requests we, as consultants, get from our customers is for an operations guide as the final deliverable for any data center build out. There are a few goals for such a guide:

  • Allow the customer to better transfer knowledge as their teams grow and change.
  • Provide an “as built” guide that explains step-by-step how to deploy and manage the infrastructure.
  • Tie together the operational workflow for all the new components that are leveraged in the modern open-networking paradigm.

Since Scott and I have been working on many operations guides, we thought it would be great to document our process so that customers can write their own operations guides.

The operations guide for web scale networking goes beyond just documenting configuration backups, user account access and change requests though. Web scale networking integrates proven software development processes and as such, the operations guide needs to account for these workflows.

As Built

The starting point of all operations guides is the initial build. Most of the cabling architecture, traffic flows and features, along with decision making and architectural choices, are captured within the High level Design and Low Level Design document. The operations guide on the other Continue reading

Lessons learned from Black Friday and Cyber Monday

 

If you’re a consumer-facing business, Black Friday and Cyber Monday are the D-Day for IT operations. Low-level estimates indicate that upwards of 20% of all revenues for companies can occur within these two days. The stakes are even higher if you’re a payment processor as you aggregate the purchases across all consumer businesses. This means that the need to remain available during these crucial 96 hours is paramount.

My colleague, David, and I have been working the past 10 months preparing for this day.  In January 2018 we started a new deployment with a large payment processor to help them build out capacity for their projected 2018 holiday payment growth. Our goal was to create a brand new, 11 rack data center to create a third region to supplement the existing two regions used for payment processing. In addition, we helped deploy additional Cumulus racks and capacity at the existing two regions, which were historically built with traditional vendors.

Now that both days have come and gone, read on to find out what we learned from this experience.

Server Interop Testing

Payment processing has most of its weight on the payment applications running in the data center. As with Continue reading

EVPN behind the curtains

Is EVPN magic? Well, like Arthur C Clarke said, any considerable leap in technology is indistinguishable from magic. On that premise, moving from a traditional layer 2 environment to VXLAN driven by EVPN has much of that same hocus pocus feeling. To help demystify the sorcery, this blog aims to help users new to EVPN create some step-by-step understanding of how EVPN works and how the control plane converges. In this blog post, we’ll focus on basic layer 2 (L2) building blocks then work our way up to layer 3 (L3) connectivity and the control plane.

We’ll be using the “reference topology” as our cable plan and foundation to build our understanding of the traffic flow. Our infrastructure will try to demystify a symmetric mode EVPN environment using distributed gateways. All the configurations are defined in this github repo. 

If you’d like to follow along as we go, feel free to launch your own CITC blank slate and deploy the above playbook:

EVPN message types

Like any good protocol, EVPN has a robust process for exchanging information with its peers. In EVPN this process uses message types. If you already know OSPF and the LSA messages you can Continue reading

How to manage access in a web-scale data center

One of the consistent questions that arises during the web-scale transition is the impact of managed access to networking infrastructure. How do we take traditional management techniques and adapt them to the new operational paradigm of web-scale networking, where automation drives the majority of changes and the infrastructure is treated as a holistic entity rather than node-by-node?

Local privileges

In the most basic way, we can migrate existing workflows to the new paradigm. Though inefficient, the old way of doing things still works with the new web-scale paradigm. The easiest way to do this is to restrict access to your switches using local privileges. In Linux, users are controlled using the adduser command, and the permissions for that user are controlled using the chmod commands.

A list of all users is stored in the /etc/passwd folder of Linux:

 cumulus@leaf02:~$ cat /etc/passwd
 root:x:0:0:root:/root:/bin/bash
 daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
 bin:x:2:2:bin:/bin:/usr/sbin/nologin
 sys:x:3:3:sys:/dev:/usr/sbin/nologin
 sync:x:4:65534:sync:/bin:/bin/sync
 games:x:5:60:games:/usr/games:/usr/sbin/nologin
 man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
 lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
 mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
 news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
 uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
 proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
 www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
 backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
 list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
 irc:x:39:39:ircd:/var/run/ircd:/usr/sbin/nologin
 gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
 nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
 systemd-timesync:x:100:103:systemd Time Synchronization,,,:/run/systemd:/bin/false
 systemd-network:x:101:104:systemd Network Management,,,:/run/systemd/netif:/bin/false
 systemd-resolve:x:102:105:systemd Resolver,,,:/run/systemd/resolve:/bin/false
 systemd-bus-proxy:x:103:106:systemd Bus Proxy,,,:/run/systemd:/bin/false
 frr:x:104:109:Frr routing suite,,,:/var/run/frr/:/bin/false
 ntp:x:105:110::/home/ntp:/bin/false
 uuidd:x:106:111::/run/uuidd:/bin/false
 messagebus:x:107:112::/var/run/dbus:/bin/false
 sshd:x:108:65534::/var/run/sshd:/usr/sbin/nologin
 snmp:x:109:114::/var/lib/snmp:/usr/sbin/nologin
 dnsmasq:x:110:65534:dnsmasq,,,:/var/lib/misc:/bin/false
 _lldpd:x:111:115::/var/run/lldpd:/bin/false
 cumulus:x:1000:1000:cumulus,,,:/home/cumulus:/bin/bash

Users can be added and deleted using the adduser and deluser commands:

 cumulus@leaf02:~$ sudo  Continue reading

NetDevOpEd: Demand more from your vendors

I was at the London Network Automation meetup this past week. London has a great community of network engineers that are eager to improve their current skill set and make their work lives easier. This meetup was geared towards learning new technologies around automation in the networking industry. There were talks about new ways to implement various existing automation tools, as well as real world discussions around how automation was implemented in the industry.

One common theme I experienced was how eager all network engineers are to open the scripts and code they write to administer their network devices. The sense of community and camaraderie amongst peers in this industry is one of the reasons I really enjoy working in this field. This is a real change in the industry. Even one year ago, the majority of network engineers did not have accounts on github or gitlab. Now, all these same network engineers exchange github and gitlab accounts like business cards. I couldn’t be happier with the direction this industry has taken.

However, I also couldn’t help but think about the irony of this situation. These creative and intelligent engineers are creating innovative scripts to shoehorn solutions into closed systems. Continue reading

NetDevOpEd: Abstracting configuration management

I was talking to a banking customer in Northern Europe the other day and they asked me about configuration management. They had many different vendors with different management methods in their infrastructure and wanted to know how they could speed up management.
This specific customer had an outsourced infrastructure. They picked what hardware they wanted to run, but then paid a managed services company to deploy the infrastructure in a colocation facility and perform day-to-day operations.

The issue arose in the speed of deployment. When they launched a new application that required a new service in their data center, the application engineers would need to contact the network team in this bank. The network team would then open up a ticket with the managed services company to provision VLANs and open up ports on their firewalls to allow access to the application. The issue was that this process took up to one week to complete.

This bank contacted us with the hope we could help them unify their management under one framework, so that they could insource the firewall configuration to accelerate their application deployments. They asked me about automated management best practices.
Normally when I have this conversation, we Continue reading

VXLAN designs: 3 ways to consider routing and gateway design (part 2)

In my previous post, I focused on the concepts of what is called off box routing and centralized routing. They were two different yet similar solutions. The first one being the simplest solution leveraging an external gateway to route between VXLANs. The second solution integrated the edge device to be both an external gateway and VXLAN end point (VTEP).

To expand on my previous post, the next logical place to put a gateway in VXLAN designs is to distribute them all on the top of rack (TOR), also known as the leaf. This TOR acts as a VTEP in the VXLAN solution. Its primary purpose is to encapsulate and decapsulate traffic. This solution is also colloquially known as Anycast Gateway VXLAN Routing. Anycast Gateway VXLAN Routing can only be performed on ASICs that support routing in and out of tunnels (RIOT), as discussed in the previous post. For the rest of this post, when I refer to VXLAN Routing, I specifically mean Anycast Gateway VXLAN Routing unless otherwise noted.

In the simplest form, VXLAN Routing allows the TOR to perform a route lookup on the inbound packet before encapsulating the traffic into a VXLAN tunnel. There are two ways that Continue reading

VXLAN designs: 3 ways to consider routing and gateway design (part 1)

With VXLAN design, the easiest thing to overlook is how communication occurs between subnets. I think many times, network engineers take for granted that our traffic will flow in a VXLAN environment. And it’s also easy to get confused when trying to figure out traffic routing path between your overlay and underlay.

As I work with customers in designing VXLAN infrastructures, one of the first questions I always ask is: “Where do you expect the gateway of the servers?”

This always leads to one of three designs, which I will outline over the next two posts. Before we start, know that all these designs leverage BGP EVPN. Ethernet Virtual Private Networks (EVPN) are an address family within BGP that are used to exchange VXLAN related information. This blog won’t go into detail about EVPN, but we have previous blogs to help fill in the gap.

With that said, let’s get started with the first VXLAN design example.

The first case is the simplest environment, and that is the gateway on an internet edge service. In this case, the VXLAN acts as a strict L2 overlay, and the L3 routed BGP underlay is hidden from the end hosts and servers.

VXLAN designs

Continue reading

NetDevOpEd: The right tool for the right job

I’ve been traveling to northern Europe these past few months to meet with various customers, deliver onsite trainings and speak at meetups. I’ve noticed some common themes that crop up no matter with whom I speak. IT professionals are exhausted by the complexity required to manage and maintain their infrastructure. Somehow, networking and server interconnectivity has become this unmanageable complex mess over the past 20 years. And I don’t blame them. As networking has layered on more and more solutions, it becomes hard to separate out the complexity from the deployment.

Normally when I have these conversations, I start at the most basic levels. I focus on two topics that create the most grief for the vast majority of networks:

  1. Layer 1 cabling issues
  2. Typos and misconfigurations

The reason I start there is because resolving these two issues in the data center would eradicate over 90% of all issues that cause late maintenance windows and urgent midnight troubleshooting calls.

At Cumulus Networks, we resolve these issues by rethinking what it means to configure a network device. The most effective solution to both of these issues is simplification of configuration. Because we focus on integration and solution first, we are able Continue reading

OpenStack and CITC – Creating and learning about OpenStack networking

Part 1 – Examining the existing network

In my previous post, I was playing around with Cumulus in the Cloud (CITC) and how it was integrated with OpenStack. Now that I was playing with OpenStack in CITC, I wanted to dive deeper into the networking specific technology.

In this blog post I will be discussing how I leveraged a flat network to initially create simple instance deployments. Then I’ll dive more deeply into how I created a VXLAN network for my OpenStack instances to create more scalable east-west communication. In the previous post I used the CITC console as my primary interface for configuration. This time I will be using an SSH client and the direct SSH information, as the outputs I’m gathering have wider width that is easier to obtain via an SSH client.

To do so, I just clicked the SSH access button on the right hand side of the GUI. This provided me with the username, password and IP address that would allow me to use my own SSH client to connect to the CITC infrastructure.

For the uninitiated, here is a great intro doc into OpenStack networking. In addition, my colleague Eric Pulvino pointed me Continue reading

OpenStack now featured in Cumulus in the Cloud

First of all, we’re thrilled to announce that today we launched OpenStack with Cumulus in the Cloud. That means that you can now test out Cumulus Networks technology with an OpenStack environment easily and at zero cost to you.

I’ve written previously about Cumulus In The Cloud (CitC) when we first released it a month ago with Mesos as the initial release flavor. Since then, JR Rivers and his team have been diligently working on adding additional flavors to the CitC offering. I could not have been happier to hear the good news that they had integrated an OpenStack solution with the cloud testing framework.

I immediately launched my own free instance of Cumulus in the Cloud using the standard steps. I was greeted with a new option where I could pick the flavor of CitC I wanted to initiate:

OpenStack Cumulus 1

Since I had already experimented with Mesos, I was eager to tinker with OpenStack to better learn this technology.

To be upfront, I am not an OpenStack expert. I have been diligently learning it over the past six months ever since a majority of my customer engagements have involved private cloud deployments leading with OpenStack. As a network engineer first, Continue reading

1 2