Cumulus Linux: First Impressions

Typically, when you buy a network router or switch, it comes bundled with some version of the manufacturer's operating system. Cisco routers come with IOS (or some derivative), Juniper routers come with Junos, and so on. But with the recent proliferation of merchant silicon, there seem to be fewer and fewer differences between competing devices under the hood. For instance, the Juniper QFX3500, the Cisco Nexus 3064, and the Arista 7050S are all powered by an off-the-shelf Broadcom chipset rather than custom ASICs developed in-house. Among such similar hardware platforms, the remaining differentiator is the software.

One company looking to benefit from this trend is Cumulus Networks. Cumulus does not produce or sell hardware, only a network operating system: Cumulus Linux. The Debian-based OS is built to run on whitebox hardware you can purchase from a number of partner Original Device Manufacturers (ODMs). (Their hardware compatability list includes a number of 10GE and 40GE switch models from different vendors.)

Cumulus Linux is, as the name implies, Linux. There is no "front end" CLI as on, for example, Arista platforms. Upon login you are presented with a Bash terminal and all the standard Linux utilities (plus a number of Continue reading

Preliminary Book Topics

As I announced earlier this summer, I'm working on writing a book targeted to people entering the field of computer networking. I've got a fair amount of content fleshed out already, but figured it might help to get some feedback on the tentative structure. The book is being written in a question-and-answer style, organized into chapters by subject.

Below is the preliminary table of contents. It's still very much a work in progress, but I'm curious what people think of this approach. Constructive criticism and suggestions for additional content are welcome!

Continue reading · 45 comments

Replacing an MPLS WAN with an Internet VPN Overlay

I received an email last week from a reader seeking advice on a fairly common predicament:

Our CIO has recently told us that he wants to get rid of MPLS because it is too costly and is leaning towards big internet lines running IPSEC VPNs to connect the whole of Africa.

As you can imagine, this has caused a huge debate between the networks team and management, we run high priority services such as Lync enterprise, SAP, video conferencing etc. and networks feel we need MPLS for guaranteed quality for these services but management feels the Internet is today stable enough to run just as good as MPLS.

What is your take on the MPLS vs Internet debate from a network engineer's point of view? And more so, would running those services over Internet work?

This is something I struggled with pretty frequently in a prior job working for a managed services provider. MPLS WANs are great because they provide flexible, private connectivity with guaranteed throughput. Most MPLS providers also allow you to choose from a menu of QoS schemes and classify your traffic so that real-time voice and video services are treated higher preference during periods of congestion.

Unfortunately, Continue reading

Beyond the Blog

I'm thinking about writing a book.

Obviously, there are a lot of networking books on the market today. Search for any mainstream certification on Amazon and you'll find titles from half a dozen publishers. The majority of these are oriented toward specific vendors (most commonly Cisco) and many parallel a given certification exam. These books are overall pretty great. Most of them.

There also exists a minority of books which cover topics outside of the vendor-driven mainstream, like Gary A. Donahue's Network Warrior published by O'Reilly, now in its second edition. I love this kind of independent title because its content isn't constrained to a particular mold. The author finds stuff he thinks is relevant and interesting, and he writes about it. This is the correct way to write a book.

But over the past few years it has become painfully evident to me that there are many areas of this field we simply don't talk about in print, at least not at the entry level where perhaps it would be most helpful. If you want a thirty-page lecture on subnetting or a terrible mnemonic for the OSI model, pick any CCNA book from the pile and you're good to Continue reading

PSA: Global IPv4 Routing Table Hits 500k Routes

Last week, the global IPv4 routing table has surpassed the 500 thousand route benchmark, according to the CIDR Report. The graph below shows its progression since the early nineties:

I last wrote about global IPv4 growth in August of 2009, when the table size was at a mere 300 thousand routes. While that benchmark was largely ceremonial, this one crosses a threshold which should may be of grave concern for many.

As has been pointed out on the NANOG mailing list, we are quickly approaching the hard forwarding plane capacity limits which exists on several very popular platforms, namely the Cisco 7600/6500 and RSP720/Sup720. The default TCAM partitioning scheme of these platforms allows for a maximum of 512 thousand IPv4 routes.

If you accept full Internet routes anywhere on your network, you'll want to verify the maximum table sizes for those platforms. On the 6500/7600 platform, the current partitioning scheme can be inspected with show mls cef maximum-routes:

Router# show mls cef maximum-routes
FIB TCAM maximum routes :
=======================
Current :
---------
 IPv4 + MPLS         - 512k (default)
 IPv6 + IP Multicast - 256k (default)

The good news is that it's easy to repartition the default scheme (e. Continue reading

Deploying Datacenter MPLS/VPN on Junos

One of my recent projects has been deploying an MPLS/VPN architecture across a pair of smallish datacenters comprised entirely of Juniper gear. While I'm no stranger to MPLS/VPN, I am still a bit green to Junos, so it was a good learning exercise. My previous articles covering MPLS/VPN on Cisco IOS have been fairly popular, so I figured it would be worthwhile to cover a similar implementation in the Juniper world.

For our datacenters, we decided to implement a simple spine and leaf topology with a pair of core routers functioning as IBGP route reflectors and a pair of layer three ToR switches in each server rack. The spine is comprised of four layer three switches which run only MPLS and OSPF; they do not participate in BGP.

This article assume some basic familiarity with MPLS/VPN, so if you're new to the game, consider reading through these previous articles for some background before continuing:

• Intro to VRF Lite
• Inter-VRF Routing with VRF Lite
• Getting to know MPLS
• Creating an MPLS VPN

Continue reading · 12 comments

The Value of a Microsecond

While perusing vendor datasheets, have you ever questioned the inclusion of seemingly insignificant latency specifications? Take a look at Arista's line-up, for instance. Their 7500 series chassis lists a port-to-port latency of up to 13 microseconds (that's thirteen thousandths of a millisecond) whereas their "ultra-low latency" 7150 series switches provide sub-microsecond latency.

But who cares? Both values can be roughly translated as "zero" for us wetware-powered humans. (For reference, 8,333 microseconds pass in the time it takes your shiny new 120 Hz HDTV to complete one screen refresh.) So, does anyone really care about such obscenely low latency?

For a certain few organizations involved in high-frequency stock trading, those shaved microseconds can add up to billions of dollars in profit. The New York Times recently published an article titled The Wolf Hunters of Wall Street by Michael Lewis, which reveals how banks have leveraged low network latency to manipulate stock prices in open markets. (Thanks to @priscillaoppy for the tip!)

The increments of time involved were absurdly small: In theory, the fastest travel time, from Katsuyama’s desk in Manhattan to the BATS exchange in Weehawken, N.J., was about two milliseconds, and the slowest, from Continue reading

Learn Python

Around six years ago, I decided to start a website called packetlife.net. Maybe you've heard of it. Most people turn to a purpose-built content management system like Wordpress or Drupal for such an endeavor, but I needed greater flexibility to achieve some of the projects I had in mind. This meant I needed to learn a programming language and write a good amount of the site's logic myself.

I already had some experience dabbling in PHP, but wasn't thrilled with it. I figured if I was going to learn a new language, it should be useful as a general purpose language and not just for building a web site. After a bit of research and deliberation, I chose Python (and the Django web framework).

The purpose of this post is to convince networkers with little to no experience writing code to learn Python. In the past I've encouraged fellow networkers to pick up any programming language, as it's more important to think like a programmer than it is to gain proficiency in a particular language. However, I've realized that many people get stuck on which language they want to learn, lose motivation, and end up not growing proficient Continue reading

Packet Life Turns Six

Today marks Packet Life's sixth birthday, and I'm celebrating by launching the new site format I talked about in January. The relaunched site is hosted on an entirely new server from Linode, which means you can (finally) access packetlife.net via native IPv6! The entire code base has been rewritten on Django 1.6, and should feel lighter and more responsive. The layout has been rewritten as well using the Bootstrap CSS framework.

You might have noticed that some components of the old site are now gone: The discussion forums and wiki have been axed in favor of focusing more on the site's core content. The tools armory, which was initially in jeopardy, has been maintained in response to community interest (although I do intend to spend a good amount of time cleaning it up).

There are no doubt bits of code here and there that need a tweak or three, but generally speaking the site is up and running. If you do encounter an error, rest assured that I've been alerted and should have it fixed in little time. If you feel that something is terribly amiss, give me a shout on Twitter and I'll look into Continue reading

Selecting Shapes by Layer

Selecting shapes and connectors one-by-one in Visio can be tedious, especially when working with large or repetitive drawings. If you've been drawing for a while, you've probably gotten the hang of selecting just the right subset of shapes using the rectangular select tool, and employing the control key to add or remove any outliers as desired. This can be time-consuming though, especially when you want to pick out just a few connectors from a jumble of criss-crossing lines.

Here's a trick to try next time you find yourself excessively control-clicking: Identify each logical group of shapes or connectors that you'll likely want to tweak, and bundle them up into to their own layer. You can then use Visio's "select by layer" option to grab them all at once later. Take the drawing below, for instance.

Continue reading · 6 comments

Overhauling PacketLife.net for 2014

Regular readers no doubt have noticed that I haven't posted anything new in the past few months. I've been pretty busy with the holidays, home projects, and adjusting to a new job, and haven't had much time or motivation to devote to writing. Good news though: I have started on a long-overdue refresh of the Packet Life design and code base.

When I originally debuted Packet Life, I ultimately wanted it to serve as major community hub, so I built in features like the wiki and discussion forum. Although Packet Life has grown quite popular over the last few years, these areas of the site have seen little activity. Acknowledging that there are more active and useful sites out there which serve these functions, I've decided to chop off some of the bloat in favor of focusing on the blog and the site's other more popular features.

Here's the fate I've outlined for each function of the site:

Blog: The blog is the heart of the site and will remain mostly unchanged, albeit refreshed and optimized. I'm considering allow guest posts but haven't committed to the idea.

Lab: No, there are no plans to bring the community lab back online Continue reading

Last Day to Buy a Poster is October 31

7 comments

VRF Export Maps

VRFs are an excellent tool for maintaining segregated routing topologies for separate customers or services. I've previously covered inter-VRF routing using route targets, but what if we only want to export a subset of the routes within a VRF? Here's a scenario in which this would be desirable.

Customers A and B each have a site network and a colocation network, and both customers need access to the 192.168.0.0/24 network in the Services VRF. The customers must utilize unique IP space in order to prevent overlapping networks, so each customer has been allocated dedicated IP space from their common provider out o 10.0.0.0/8. Unfortunately, customer A is still has some networks within the 172.16.0.0/16. These networks need to access services in the host colo, but the service provider can't allow this space to be advertised into the Services VRF as it's not approved IP space.

Our goal is to export only the networks within the 10.0.0.0/8 space from the customer VRFs to the Services VRF. How can we accomplish this?

Let's have a look at the initial network state. (This lab was performed using a single router for Continue reading

Announcing Labkeeper

It's been nearly a year since I had to take the community lab offline to relocate my home, and I still get frequent emails asking when it will be returned. Sadly, my plan to host it with my current employer didn't pan out, and I'm still searching for a suitable host in the Raleigh-Durham, North Carolina area. (If you might be able to give the lab a good physical home, please let me know!)

I also get a lot of emails asking if I can share the scheduling application I used to make the lab available. In short, no, I can't. Not because I don't want to or because it's secret, but simply because that code was written specifically for the packetlife.net site and is not in the least bit portable. But these requests got me thinking: A lot of people obviously would like to be able to share their labs, they just need a platform. Maybe I could rewrite and improve upon the scheduling application to spin it off as its own service.

If you follow me on Twitter, you may have caught one or two references to a new project recently. Indeed, this is exactly Continue reading

Lessons Learned Writing a Custom Config Builder

Continue reading · 25 comments

Route Distinguishers and Route Targets

People new to MPLS VPN are often unclear on what functions route distinguishers and route targets serve, and the difference between the two. Let's see if we can clear up some of that confusion. If you could use a refresher on VRF fundamentals, I encourage you to first check out my earlier articles on the topic, Intro to VRF lite and Inter-VRF Routing with VRF Lite.

Route Distinguisher

As you know, VRFs allow IP address space to be reused among isolated routing domains. For example, assume you have to connect to three customer sites, all of which are using 192.168.0.0/24 as their local network. We can assign each customer its own VRF so that the overlapping networks are kept isolated from one another in their respective routing domains.

This works well, but we need a way to keep track of which 192.168.0.0/24 route belongs to which customer. This is where route distinguishers come in. As its name implies, a route distinguisher (RD) distinguishes one set of routes (one VRF) from another. It is a unique number prepended to each route within a VRF to identify it as belonging to that particular VRF or customer. Continue reading