Kernel of Truth season 3 episode 6: Building modern campus networks
Subscribe to Kernel of Truth on iTunes, Google Play, Spotify, Cast Box and Sticher!
Click here for our previous episode.
In this episode we talk about trends, architectures and technologies for building modern Campus networks. Joining Kernel of Truth podcast hosts Brian O’Sullivan and Roopa Prabhu are two of our senior consultants, Eric Pulvino and David Marshall, who know what they’re talking about because they are in the field working with customers building these networks. They share their first hand knowledge here so be sure to take a listen!
Guest Bios
Brian O’Sullivan: Brian currently heads Product Management for Cumulus Linux. For 15 or so years he’s held software Product Management positions at Juniper Networks as well as other smaller companies. Once he saw the change that was happening in the networking space, he decided to join Cumulus Networks to be a part of the open networking innovation. When not working, Brian is a voracious reader and has held a variety of jobs, including bartending in three countries and working as an extra in a German soap opera. You can find him on Twitter at @bosullivan00.
Roopa Prabhu: Roopa Prabhu is Chief Linux Architect at Cumulus Networks. Continue reading
Cumulus content roundup: April 2020
April has come and gone but you know what hasn’t? New content and useful resources from us! If you missed out on it earlier, catch up with this month’s edition of the Cumulus Content Roundup. We’ve got all the details on recent announcements like our NetQ 3.0 product launch, new podcast episodes for you listen, and so much more.
From Cumulus Networks
Modernizing your network has never been easier with Cumulus NetQ Lifecycle Management: To help network operations, we’ve added new lifecycle management capabilities to Cumulus NetQ 3.0, offering a simple GUI-driven workflow for provisioning, operating, maintaining & retiring network switches running Cumulus Linux. Scott Ciccone explains more about the recent product release in his blog here.
Linux Network Observability: Building Blocks: As Linux reigns the “Open-Distributed-Virtualized-Software-Driven-Cloud-Era”— understanding what is available within Linux in terms of observability is essential to our jobs & careers. Linux expert Roopa Prabhu shares what you need to know in this blog.
Kernel of Truth season 3 episode 5: Routing protocols in the datacenter fabric: Special guest Russ White joins the Kernel of Truth podcast to discuss what we should think about when it comes to routing protocols in the datcenter. What are Continue reading
UCMP: Augmenting L3 only designs
An aspiration of modern web scale networking is to leverage a pure L3 solution and integrate it with anycast addresses to allow for load balancing functionality. So why is this design aspirational? Well it requires discipline in the way that applications are architected— specifically around tenancy requirements and application redundancy. In this blog I’ll discuss a recent augmentation that was introduced into Cumulus Linux 4.1 that makes this style of design much more flexible in web scale networks.
Common challenges
Two common challenges when using anycast addressing in layer 3 only solutions are:
- Resilient hashing to support the change in ECMP paths
- Balancing traffic across unequal cost advertisements
Solutions
The first solution was implemented back in the early version of Cumulus Linux and is well documented. This solution is also known as “RASH” as the colloquial term.
The second solution addresses an interesting artifact of the way Layer 3 routes are advertised and learned, specifically with regards to next hop selection. Let us imagine the following simplified design:
The IP address of 192.168.1.101 is an anycast address that is being advertised by 3 different hosts in our environment. These three hosts are all serving the exact Continue reading
Linux Network Observability: Building Blocks
As developers, operators and devops people, we are all hungry for visibility and efficiency in our workflows. As Linux reigns the “Open-Distributed-Virtualized-Software-Driven-Cloud-Era”— understanding what is available within Linux in terms of observability is essential to our jobs and careers.
Linux Community and Ecosystem around Observability
More often than not and depending on the size of the company it’s hard to justify the cost of development of debug and tracing tools unless it’s for a product you are selling. Like any other Linux subsystem, the tracing and observability infrastructure and ecosystem continues to grow and advance due to mass innovation and the sheer advantage of distributed accelerated development. Naturally, bringing native Linux networking to the open networking world makes these technologies readily available for networking.
There are many books and other resources available on Linux system observability today…so this may seem no different. This is a starter blog discussing some of the building blocks that Linux provides for tracing and observability with a focus on networking. This blog is not meant to be an in-depth tutorial on observability infrastructure but a summary of all the subsystems that are available today and constantly being enhanced by the Linux networking community for networking. Continue reading
Modernizing your network has never been easier with Cumulus NetQ Lifecycle Management
Upgrade and configuration management with push button simplicity
Network operations is all about keeping the network running smoothly and without problems. It includes monitoring the network for performance problems and faults, as well as fixing problems before they affect end users, or at least in the timeliest manner possible. Managing network assets correctly is critical to avoiding application-impacting network outages, performance degradation and security incidents.
However, network operators often struggle with operational challenges such as network disruptions caused by maintenance and configuration changes. Furthermore, business networks are often fairly large and complex, which means the set of tasks a network administrator will need to perform can quickly overwhelm manual efforts. This requires a shift, not only to modern networking, but also to modern operational tools as well.
To help solve these issues, Cumulus Networks has added new lifecycle management (LCM) capabilities to Cumulus NetQ 3.0, offering a simple GUI-driven workflow for provisioning, operating, maintaining and retiring network switches running Cumulus Linux.
Figure 1: NetQ Lifecycle Management
With the addition of full lifecycle management functionality, NetQ 3.0 now combines the ability to easily upgrade, configure and deploy network elements with a full suite of operations capabilities, such as visibility, Continue reading
Kernel of Truth season 3 episode 5: Routing protocols in the datacenter fabric
Subscribe to Kernel of Truth on iTunes, Google Play, Spotify, Cast Box and Sticher!
Click here for our previous episode.
Hosts Roopa Prabhu and Pete Lumbis are joined by a special guest to the podcast, Russ White! The group come together virtually to discuss what we should think about when it comes to routing protocols in the datcenter. What are the tradeoffs when using traditional protocols like OSPF or BGP? What about new protocols like RIFT or a hybrid approach with things like BGP-link state? Spoiler alert: it depends.
Guest Bios
Roopa Prabhu: Roopa Prabhu is Chief Linux Architect at Cumulus Networks. At Cumulus she and her team work on all things kernel networking and Linux system infrastructure areas. Her primary focus areas in the Linux kernel are Linux bridge, Netlink, VxLAN, Lightweight tunnels. She is currently focused on building Linux kernel dataplane for E-VPN. She loves working at Cumulus and with the Linux kernel networking and debian communities. Her past experience includes Linux clusters, ethernet drivers and Linux KVM virtualization platforms. She has a BS and MS in Computer Science. You can find her on Twitter at @__roopa.
Pete Lumbis: Pete, CCIE R&S #28677 and CCDE 2012::3, is Continue reading
As networks grow, web-scale automation is the only way to keep up
Networks just keep growing, don’t they? They’ve evolved from a few machines on a LAN to the introduction of Wi-Fi—and with the Internet of Things (IoT), we’ve now got a whole new class of devices. Throw in the rise of smartphones and tablets, cloud and edge computing, and network management starts to get a little unwieldy. Managing a network with 300 devices manually might be possible—300,000 devices, not so much.
What is web-scale automation?
Network automation has been around awhile now, in various names from various vendors, using a number of proprietary protocols. The key word being “proprietary.” Many traditional network vendors design a well-functioning network automation system, but participate in vendor lock-in by ensuring that the associated automation stack, and its requisite protocols, only run on their hardware.
Web-scale automation is different. It relies on open, extendable standards like HTTPS, JSON, and netconf, among an ever-increasing number of systems and solutions. With web-scale automation in your organization, network management can over time become a background function; something that only notifies you in exceptional circumstances.
This does not, in any way, reduce the need for those who know networks to be employed at your organization—it simply reduces the amount Continue reading
Kernel of Truth season 3 episode 4: Production Ready Automation
Subscribe to Kernel of Truth on iTunes, Google Play, Spotify, Cast Box and Sticher!
Click here for our previous episode.
In this episode hosts Roopa Prabhu and Brian O’Sullivan chat with Justin Betz about production grade automation and CI/CD workflows for continuous maintenance and deployment of your “infrastructure as code.” They also discuss how like any other software code, code to manage and automate your infrastructure (IAC) has to be maintained, fixed, withstand hardware and software upgrades — so how do you do that? Finally, the group talks about Open Source production quality automation code. Enjoy!
Guest Bios
Roopa Prabhu: Roopa Prabhu is Chief Linux Architect at Cumulus Networks. At Cumulus she and her team work on all things kernel networking and Linux system infrastructure areas. Her primary focus areas in the Linux kernel are Linux bridge, Netlink, VxLAN, Lightweight tunnels. She is currently focused on building Linux kernel dataplane for E-VPN. She loves working at Cumulus and with the Linux kernel networking and debian communities. Her past experience includes Linux clusters, ethernet drivers and Linux KVM virtualization platforms. She has a BS and MS in Computer Science. You can find her on Twitter at @__roopa.
Capex vs Opex – Hidden complexity by making an unmanageable network
Many networking solutions purport great Opex savings through automation, simulation and continuous integration. Similarly, there is a school of thought where network designs will have a single point in a network perform multiple roles. This will short change an initial Capex cost of purchasing additional switches with the intention of overlapping features on that single device.
Let’s take the simplest example. We have a 3 rack environment with dual-leaf per rack and 2 spines for inter-rack connectivity. In this design, we are leveraging VXLAN as the data plane overlay with BGP/EVPN as the control plane. Additionally, all 3 racks are compute, leaving no additional leafs to act as the service/border/exit leafs.
A network designer will look at the infrastructure and try to overlap features by repurposing the spines as exit leafs. Why will they think this way, you ask? Well, this is only an 8 switch design. Spending money on an additional 2 switches to act as dedicated border leafs uplifts my capex cost by 25 percent! I would then be required to buy 10 total switches instead of 8.
So instead, we end up overlaying the VXLAN onto the spines. So now the spines act as both interconnections between Continue reading
Load balancer with BGP and UCMP
Uniform load distribution to anycast servers using BGP bandwidth community and unequal cost multipath forwarding
Scalable modern applications are deployed as clusters of server instances and load balancers are needed to distribute client requests across server instances. In order to ensure positive user experience an application needs to be always responsive and no instance should get bogged down with overload.
Sophisticated load balancing solutions help but often involve expensive and proprietary components. These also are additional point of failure requiring maintenance and are often overkill for most use cases.
Here is one solution to this complex problem that can achieve spreading client requests evenly across application servers with network switches running Cumulus Linux. All this is achieved without adding any additional device or component.
The case in point is that the user has a large number of anycast services running in a multipod Clos network. The number of service endpoints can dynamically change and user expectation is that service endpoints get uniformly loaded.
This solution works well for both cases, one where Clos fabric is Layer-3 only network and another where we have Evpn vxlan overlay network. Care must be taken though to select switch hardware that can support overlay Continue reading
Cumulus content roundup: March 2020
Spring has sprung! With the change of the seasons, we’ve kept busy pumping out new content and useful resources. If you’re looking for a quick mental vacation, get ready to cozy up with this month’s edition of the Cumulus Content Roundup. We’ve got exciting announcements, fresh podcast episodes for your listening enjoyment, as well as blog posts packed with open networking and data center goodness.
From Cumulus Networks
Cumulus Networks launches the industry’s first open source and fully packaged automation solution — making open networking easier to deploy and manage and enabling infrastructure-as-code models: Cumulus Networks announces the release of its production-ready automation solution for organizations moving towards fully automated networks! Read all about how we are taking the next step in network automation in this blog post.
Production-ready automation — the how and why: So we’ve announced the industry’s first open source and fully packaged automation solution– but how exactly did we get there? This blog post dives into the challenges that customers were facing and the reason we wanted to help.
A new era for Cumulus in the Cloud: Can you believe that Cumulus in the Cloud was launched over two years ago? Yeah, we’re also Continue reading
Using ACLs for security vs compliance
Compliance exists in many forms, with tenancy, traffic isolation and access restriction. Compliance is mostly due to regulatory needs, which really comes down to security needs. Compliance applies to networking, fundamentally ensuring that resources can only be accessed from allowed locations. The actual media and content normally isn’t pertinent, as they merely just influence the scope of access for the data.
As a result, this post doesn’t delve into more complex compliance requirements such as deep packet inspection or encryption. Rather, this is to discuss how networking engineers use their existing toolset to enforce compliance requirements.
The most fundamental of these requests is networking access permissions. Most customers will allocate subnets for functional sections of their network.
ACLs are some of the most classic and undemanding forms of permission. The greatest part of ACLs is that they can be applied on nearly every networking device, and provide somewhat of a base level of security. But there are hidden complexities with ACLs that may not make it the ideal choice for most compliance solutions:
1. Connection State Tracking
ACLs are unidirectional elements, examining packets one at a time and only blocking traffic in a single direction. This can create some logical Continue reading
768K day: the importance of adaptable software in the growing Internet
The Internet is big. Moreover, the Internet is bigger now than when that first sentence was written, and keeps increasing in size. The growth of the Internet from its humble beginnings as a DARPA research project was unprecedented and almost entirely unexpected. This—as well as the widespread usage of older routers and switches as crucial connection points in the Internet—has resulted in real-world scaling issues.
One of these issues, known commonly as “512K day,” occurred on Aug. 12, 2014. On that day, Verizon, a large United States-based Internet provider and Internet exchange point (IXP), submitted an extra 15,000 routes to the global BGP routing table. As these routes propagated across the network, they were accepted by some routers—the ones that had new firmware, or were configured to only store a subset of the global routing table.
But in other routers, this additional route load overran the 512,000 route maximum expected by the firmware designers, causing widespread Internet outages and degradation of service. In many cases, the issue was resolved quickly, but not as quickly as it could have been. Proprietary vendors were required to push out firmware updates for hundreds of router and switch models—a process that can take months. Continue reading
Virtual Data Centers, SDN, and Multitenancy
When you aren’t the size of Netflix, you may not be guaranteed dedicated infrastructure within a data center; you have to share. Even in larger organizations, multitenancy may be required to solve regulatory compliance issues. So what is multitenancy, how does it differ from other forms of resource division, and what role do networks play?
Gartner Inc. defines multitenancy as “a reference to the mode of operation of software where multiple independent instances of one or multiple applications operate in a shared environment. The instances (tenants) are logically isolated, but physically integrated.” This is basically a fancy way of saying “cutting up IT infrastructure so that more than one user/department/organization/and so on can share the same physical IT infrastructure, without being able to see one another’s data.”
That “without being able to see one another’s data” is the critical bit. Allowing multiple users to use a single computer has been possible for decades. Multi-user operating systems, for example, can allow multiple users to log in to a single computer at the same time. While this approach does allow multiple users to share a physical piece of IT infrastructure, it isn’t multitenancy.
In a multi-user OS, the multiple users Continue reading
Automate, orchestrate, survive: treating your network as a holistic entity
Organizations need to learn to think about networks as holistic entities. Networks are more than core routers or top-of-rack (ToR) switches. They’re composed of numerous connectivity options, all of which must play nice with one another. What role does automation play in making network heterogeneity viable? And does getting all the pieces from a single vendor really make management easier if that vendor has 15 different operating systems spread across their lineup of network devices?
Most network administrators are used to thinking about their networks in terms of tiers. Access is different from branch, which is different from campus, and so forth. Datacenter is something different again, and then there’s virtual networking complicating everything.
With networks being so big and sprawling that they frequently occupy multiple teams, it’s easy to focus on only one area at a time. Looking at the network holistically—both as it exists, and as it’s likely to evolve—is a much more complicated process, and increasingly important.
Networks grow, evolve and change. Some of this is organic; growth of the organization necessitates the acquisition of new equipment. Other times growth is more unmanaged; something that’s especially common with mergers and acquisitions (M&As).
Regardless of reason, change in Continue reading
Kernel of Truth season 3 episode 3: Linux networking with eBPF
Subscribe to Kernel of Truth on iTunes, Google Play, Spotify, Cast Box and Sticher!
Click here for our previous episode.
This podcast is all about Linux and to talk about it, we have two of the top Linux kernel experts. Kernel of Truth host Roopa Prabhu is one and chats with our special guest David Ahern about eBPF. If you haven’t heard of eBPF, it’s the hottest Linux kernel technology bringing programmability and acceleration to many Linux subsystems. In this podcast we focus on eBPF’s impact on networking and the million possibilities it brings to the table.
Guest Bios
Roopa Prabhu: Roopa Prabhu is Chief Linux Architect at Cumulus Networks. At Cumulus she and her team work on all things kernel networking and Linux system infrastructure areas. Her primary focus areas in the Linux kernel are Linux bridge, Netlink, VxLAN, Lightweight tunnels. She is currently focused on building Linux kernel dataplane for E-VPN. She loves working at Cumulus and with the Linux kernel networking and debian communities. Her past experience includes Linux clusters, ethernet drivers and Linux KVM virtualization platforms. She has a BS and MS in Computer Science. You can find her on Twitter at @__roopa.
A new era for Cumulus in the Cloud
When we launched Cumulus in the Cloud (CitC) over two years ago, we saw it as a way for our customer base to test out Cumulus Linux in a safe sandboxed environment. Looking back, September 2017 feels like an eternity ago.
Since then, CitC has become a place where we’ve been able to roll out new functionality and solutions to customers and Cumulus-curious alike — and we’ve done some really interesting things (some of our favs include integrating it with an Openstack demo and Mesos demo). It’s pretty much become a Cumulus technology playground.
As our CitC offering has evolved, we’ve also taken stock of the requirements from our customers and realized the direction we want to take CitC. So where is it heading? We’re excited to share that with the launch of our production-ready automation solution last week, CitC will have a new user experience and user interface.
Out with the old:
In with the new:
This redesigned UI comes with some really great enhancements:
- Auto-deployed production ready demos
- Customized external connectivity to oob-mgmt-server to run user customized applications
- Default lifetime increased to 12 hours
- NetQ native integration within the demo
- Ability to store and wake up Continue reading
Cumulus content roundup: February 2020
Are you looking for a break from the general news? Our content round-ups are a great source for all the recent blogs (and more!) that we’ve shared. From podcasts, to product updates, to industry thought leadership and even customer examples of how they’re implementing open networking technology— we’ve got it all here for you.
Dive into the content below and update yourself on what’s been happening. It may just be that needed break from the rest of the news that you were looking for.
From Cumulus Networks:
VXLAN/EVPN vs VRF Lite: In this blog Rama Darbha, Senior Consulting Engineer at Cumulus Networks, talks through the difference between VXLAN with EVPN and VRFLite and offers up his conclusion as to which solution is better.
Kernel of Truth season 3 episode 1: FRRouting update: Season three jumps right into the deep end of the pool with a discussion on FRRouting. Listen as hosts Brian O’Sullivan & Roopa Prabhu chat about what’s new with the FRR community with a new guest to the podcast, FRR expert Donald Sharp.
Topology matters: how port-per-workload management strategies no longer hold up: A new way of looking at switching—as a logical, rather Continue reading
Production-ready automation — the how and why
Last week Cumulus announced the launch of our exciting production-ready solution. This suite of automation scripts provides customers with a quick and validated way to leverage automation for day 1 deployment and day 2 operations. Plus, it’s open source. So it’s completely free to access and use, and it will only expand and improve over time.
Amidst all of the excitement, I wanted to take an opportunity to dive into some of the details of why and how we ended up with such a unique solution. So here we go.
Let’s start with what brought us here
Like most good technology solutions, production-ready automation started with an evaluation of customer challenges.
Challenge #1: First and foremost, we want to produce features and products that help our customers build better networks — networks that are scalable, agile, flexible and efficient. Automation is a huge part of the story and we believe having a feature-rich, Linux-based operating system makes automation even better.
That said, no matter what type of operating system you’re running, most engineers have to piece together scripts and playbooks to build something custom that will hopefully (fingers crossed!) work with their new operating system. This is tedious at Continue reading
Cumulus Networks launches the industry’s first open source and fully packaged automation solution — making open networking easier to deploy and manage and enabling infrastructure-as-code models
Today, Cumulus Networks is announcing the release of its production-ready automation solution for organizations moving towards fully automated networks in order to take advantage of infrastructure-as-code deployment models.
At the forefront of the networking industry, we see our customers caught in the shifting tides as the modern data center moves toward fully automated networking. As they look to take advantage of innovative technology like 5G, cloud, IoT and more, organizations are looking to innovative networking deployments that incorporate new ways of thinking about automation like infrastructure-as-code, CI/CD and more. As network traffic continues to grow at an exponential rate, organizations are left with infrastructure that is harder to manage and deploy. Bogged down by the cost and time it takes to build out bits and pieces of fully automated solutions, these organizations are in need of a solution to help them innovate their networks at the speed business demands.
Cumulus is now offering the first open source, out-of-the-box, robust, end-to-end automated configuration and testing solution using Ansible. Customers no longer have to piece together their network automation from disparate and untested scripts and proof-of-concept playbooks. Cumulus is offering a framework for an elegant push-button solution for those looking for cutting-edge Continue reading