Lifting the Hood on Cisco Software Defined Access

If you’re an IT professional and you have at least a minimal awareness of what Cisco is doing in the market and you don’t live under a rock, you would’ve heard about the major launch that took place in June: “The network. Intuitive.” The anchor solution to this launch is Cisco’s Software Defined Access (SDA) in which the campus network becomes automated, highly secure, and highly scalable.

The launch of SDA is what’s called a “Tier 1” launch where Cisco’s corporate marketing muscle is fully exercised in order to generate as much attention and interest as possible. As a result, there’s a lot of good high-level material floating around right now around SDA. What I’m going to do in this post is lift the hood on the solution and explain what makes the SDA network fabric actually work.

SDA’s (Technical) Benefits

Let’s examine the benefits of SDA through a technical lens (putting aside the business benefits we’ve been hearing about since the launch).

• Eliminates STP (!!). How many years have we been hearing about this in the data center?? Now the same is true in the campus network as well. STP can finally be left in the Continue reading

How I’ve Attempted to Blog More in 2017

This post has been sitting in the “drafts” folder for a while now. Clearly, since it’s August and is therefore a little late to be deciding on a plan that is supposed to carry through all 12 months of 2017. Regardless, I think it’s still worth sharing how I’ve attempted to increase the frequency of my blogging. My basic goal for 2017 is:

Create more content in 12 months than I ever have before in order to a) significantly build up the depth and breadth of knowledge on my blog, b) increase my skills as a writer, and c) continue to build this blog and the readership as a key part of my online persona and brand.

In order to achieve this goal, I’ve identified a couple of tactical objectives:

1. Reduce the friction between me and the keyboard; make it possible to “just write”.
2. Be able to “just write” anywhere. At home. On vacation. In a waiting room. On an airplane. I should also be able to start a post in one location and pick it up again in another. Indirectly this means I need to be able to write on any of my computers or mobile devices.

In order Continue reading

Troubleshooting Cisco Network Elements with the USE Method

I want to draw some attention to a new document I’ve written titled “Troubleshooting Cisco Network Elements with the USE Method“. In it, I explain how I’ve taken a model for troubleshooting a complex system–the USE Method, by Brendan Gregg–and applied it to Cisco network devices. By applying the USE Method, a network engineer can perform methodical troubleshooting of a network element in order to determine why the NE is not performing/acting/functioning as it should.

I ask that if you’re familiar with a given Cisco network platform (or platforms), that you please contribute commands that would also fit into the USE Method! My list is just a start and I welcome contributions from others in order to make it a stronger, more valuable reference.

Please check out the guide: Troubleshooting Cisco Network Elements with the USE Method

Original article: Troubleshooting Cisco Network Elements with the USE Method

Copyright © 2017 Joel Knight . All Rights Reserved.

Tools for TE with EIGRP

In response to my article about what would cause a directly connected route to be overridden, Matt Love (@showflogi) made a good observation:

Good stuff – LPM rule can be a useful tool if you want to manipulate paths without mucking with metrics, esp if using multiple protocols

— Matt Love (@showflogi) July 13, 2017

What Matt is saying is that longest prefix match (LPM) is a mechanism that can be used to steer traffic around the network in order to meet a technical or business need. This type of traffic steering is called traffic engineering (TE).

LPM refers to how route lookups work on a Layer 3 device: the longest, most-specific match is always chosen. Like I explained in the prior post, if the routing table contains 10.10.10.0/24 and 10.10.10.64/26, the latter route will be used to forward traffic to 10.10.10.100 (as an example) because a /26 is longer (ie, has a longer prefix length) and is therefore more specific. We can use this behavior to direct traffic towards 10.10.10.100 over a specific interface or via a specific path (ie, a path with Continue reading

When is a Connected Route Not Used?

I ran into this situation on a recent project and thought it would make an excellent question on an exam. It could be worded something like this:

What is the behavior of a router or Layer 3 switch when a dynamic route is learned that partially overlaps with a directly connected network?

a. The router reboots
b. The network reboots
c. That’s um-possible
d. None of the above

The answer, of course, is “d” but the specifics of what does happen is what’s interesting. First, this is the scenario I’m trying to describe in the question above:

R12#show ip route
...
Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 6 subnets, 3 masks
D        10.1.14.0/24 [90/1024640] via 123.1.1.14, 00:14:37, Ethernet0/1
C        10.10.10.0/24 is directly connected, Ethernet0/0
L        10.10.10.12/32 is directly connected, Ethernet0/0
D        10.10.10.64/26 [90/1024640] via 123.1.1.14, 00:14:05, Ethernet0/1

R12 has a directly connected network 10.10.10.0/24 on its e0/0 interface. It has also learned a route for 10.10.10.64/26 via an EIGRP neighbor on its e0/1 interface. We can see both networks Continue reading

Reflecting On My First Cisco Live! Presentation

Well, I got to tick a big item off my list of goals last week. I successfully delivered a presentation at Cisco Live! in front of a large group of people. It didn’t kill me and I didn’t trip over anything and embarrass myself so no matter what, I have those two points to feel good about :-)

All joking aside, it actually went a whole lot better than that.

I’ve recently realized that I really enjoy teaching. Not in the sense that I want to be a trainer full time or have a job in a classroom, more like I feel that’s a big part of what drives me to write this blog and is why I feel (relatively) comfortable talking in front of people. As long as the subject is something that I feel I can weave some teaching/learning into, I’m comfortable to deliver it. By contrast, I would feel far less comfortable delivering something like a keynote speech or a toast at a wedding.

So along those lines, that was a big goal I set for myself in delivering my Cisco Live! (CLUS) presentation: empower the audience by sharing targeted, high-value knowledge and Continue reading

OpenVPN 2.3.17 on OpenBSD 6.0

On Jun 21, the OpenVPN team released an update for the 2.3.x and 2.4.x branches that resolved some newly discovered security vulnerabilities. The OpenVPN team recommends that users “upgrade to OpenVPN 2.4.3 or 2.3.17 as soon as possible“.

OpenBSD 6.0–which was released Sep 1 2016 and is still receiving security updates to the base system as per OpenBSD’s policy–shipped with a package for OpenVPN 2.3.11. Below you will find a patch and instructions for using the ports system to upgrade to version 2.3.11. Note that if you’re running OpenBSD 6.1, the ports tree has been updated to 2.4.3 so all you need to do is “cvs up” and “make install”.

Instructions:

1. Follow the OpenBSD FAQ for instructions on how to download, verify, and extract the ports tree on your machine.
2. Then:

% cd ports/net/openvpn
% patch < ~/openvpn-2.3.17p0.diff
% make install

Original article: OpenVPN 2.3.17 on OpenBSD 6.0

Copyright © 2017 Joel Knight . All Rights Reserved.

I Will Be Presenting For the First Time at CLUS 2017!

Well, it looks like another major item will get struck from my bucket list this year. I’ve been accepted to present at Cisco Live in Las Vegas this summer!

This session is designed to walk through an enterprise network and look at how EIGRP can be engineered with purpose to best suit the needs of the different areas of the network. I will focus a lot on stability and scaling EIGRP and will show the audience how, where, and when to leverage common EIGRP features such as summarization, fast timers, BFD, and wide metrics. Before getting into the nuts and bolts, I will be doing a bit of a level-set on certain EIGRP features such as queries, going active, summarization, and support for flexible network hierarchies. I will round out the session by talking about how EIGRP has been optimized for use in Cisco’s Intelligent WAN (IWAN) solution and even touch on a not-so-commonly seen application of EIGRP: EIGRP Over-The-Top. The full session agenda looks like this:

I’m actually inheriting this session from a fellow CPOC engineer, Steve Moore who, un-coincidentally, is the same S. Moore whose name is on the EIGRP RFC. Steve will be presenting a sister session Continue reading

Five Functional Facts About OSPF

It’s funny, in my exerperience, OSPF is the most widely used interior gateway protocol because it “just works” and it’s an IETF standard which means it interops between different vendors and platforms. However, if you really start to look at how OSPF works, you realize it’s actually a highly complex protocol. So on the one hand you get a protocol that likely works across your whole environment, regardless of vendor/platform, but on the other you’re implementing a lot of complexity in your control plane which may not be intuitive to troubleshoot.

This post isn’t a judgement about OSPF or link-state protocols in general. Instead it will detail five functional aspects of OSPF in order to reveal–at least in part–how this protocol works, and indirectly, some of the complexity lying under the hood.

1. OSPF Has Its Own Best Path Decision Process

Ever looked closely at OSPF routes in the show ip route output? You’ll notice flags such as O or O IA beside the route.

O     10.1.14.0 255.255.255.0
        [110/21] via 123.1.0.18, 00:00:07, Ethernet0/0
O IA  11.11.11.0 [110/20] via 123.1.0.18, 00:00:07, Ethernet0/0
O IA  123.1. Continue reading

Why I Enthusiastically Switched from Cacti to Zabbix for System Monitoring

Cacti is a “complete network graphing solution” according to their website. It has also been a thorn in my side for a long time.

See what I did there? Thorn… because it’s a cactus… never mind.

When Cacti is in a steady state–when I could get it to a steady state–it was good. Not great, because there was a lot of effort to get it into what I consider “steady state”, but good. The rest of the time… thorny.

There are five major things that have driven me up the wall. In no particular order:

Round Robin Database (RRD) sucks

The concept behind RRD is cool: a fixed-size, circular database (oldest data overwritten by the newest data) makes good sense for the type of data that a network graphing solution collects. In practice, using RRD means:

• Another software dependency that needs to be updated, patched, and integrated in the Cacti ecosystem
• Manually managing all of the RRD files that are generated for all of the data sources you’re collecting. RRD stores its data in individual files on the file system, you see, and the more data sources you collect with Cacti, the more RRD files you have Continue reading

How I Relearned the Consequences of Improper Monitoring

I had just lost the RAID array that hosts my ESXi data store. I didn’t yet know that’s what had happened, but with some investigation, some embarrassment, and a bit of swearing, I would find out that an oversight on my part three years ago would lead to this happening.

I first realized there was trouble when every VM on the host became unresponsive. Most notably, the Plex Media Server fell off the network which caused the episode of Modern Family that we were watching to immediately freeze. What was odd to me is that while the VMs were unreachable, the ESXi host itself was fine. I could ping it, ssh to it and load it up with the vSphere client. The first wave of panic hit me when I found messages like this in the host’s event log:

This was quickly confirmed from the ssh shell by looking for the data store and finding that a) the symlink for the volume (RAID1) pointed to a non-existent directory and b) the reported size of the volume was a paltry 450MB compared to the 930GB I expected.

Since I knew from prior experience Continue reading

Big Changes in 2017

This past June when I was in North Carolina at Cisco’s CPOC lab, I learned that there was a chance–albeit a slim one, but a chance nonetheless–that a position would be opening up on the CPOC team in the fall. By that point I had been to CPOC three times and knew many of the engineers who worked there. I spoke to them to get their feedback, met with the newly-hired manager of the team, and just generally did all the things I thought I should be doing to take advantage of my time being face to face with these folks.

Then I flew home, subscribed to the “new jobs at Cisco mailing list” and waited.

And then, one day, it was posted: CPOC Technical Projects Systems Engineer. I immediately sent a message to my wife who responded as only she knows how:

Five short interviews later I was offered the job!

This brings me to change #1: As of this month (January), I am no longer a Systems Engineer with Cisco Systems Canada. I am now a Systems Engineer on the CPOC team reporting to a manager in the US.

Beyond the basic level of Continue reading

My Personal Look Back on 2016

I haven’t ever written a “year in review” type of post before. Sure, I do a post to summarize how the blog has done over the year but I’ve never done a personal look back. Last night–New Years Eve–I was thinking about everything that I was involved in during 2016 and I realized “I should write this down! I was involved in or a participant of some amazing things last year!”

So here we go. In an effort to show a more personal side and not just my geeky side, here is my personal 2016 year in review.

Got Married

In February, my then-girlfriend and I got married! I know everyone says their wedding was the best, but ours totally was! Trust me! In all honesty, it was one of the funnest days of my life. Full credit to my wife for planning what was essentially an awesome party with our families and closest friends. Oh, and the venue and staff were absolutely amazing as well which sealed the deal as the greatest wedding ever .

Launched a Second Blog

For a while now I’ve been in the habit of snapping photos of wireless access points Continue reading

2016 End of Year Blog Statistics

Happy New Year! I just realized the other day that this blog turned 5 years old in 2016. It’s been a lot of fun and has paid me back for my time in terms of building my brand and being a means to explore and learn new topics. I have plans to put more focus on my writing in 2017 and reduce the friction between starting with a blank page and hitting that “Publish” button.

Anyways! Here’s a look back at 2016 on packetmischief.ca.

Hmm. Basically flat growth in terms of views and visitors. I feel like this is to be expected based on how much writing and promotion I did throughout the year. I can improve these numbers for 2017.

Just like last year, the new vs returning visitor numbers are basically unchanged.

The 5 most popular posts in 2016 are:

Quick links:

• 4 Types of Port Channels and When They’re Used
• Resetting Admin Password on a Cisco ISE Appliance
• What the Fex is a FEX Anyways
• Five Functional Facts about FabricPath
• Airplay, VLANs, and an Open Source Solution

And the top 5 posts in 2016 that Continue reading

OpenBSD on the Sixth Generation Intel NUC

I recently decided it would be fun to upgrade the hardware on my main OpenBSD machine at home (because, you know, geek). These Intel NUC machines are pretty interesting. They are pretty powerful, support a decent amount of RAM, certain models support internal storage, and they are very low power and low noise. Perfect for a machine that is a shell/email/development box.

The model I chose is the NUC6i3SYH.

• Core i3 processor (because my machine is not at all CPU bound)
• Very low power consumption (15W)
• Supports a 2.5″ SSD

OpenBSD 6.0 boots with the GENERIC kernel; no tuning or tweaking required. Full dmesg is at the end of this post. Hightlights of the hardware include:

• Wired network: Intel I219-V using the em(4) driver
• Wireless network: Intel Dual Band Wireless AC 8260 using the iwm(4) driver (no support for 802.11ac in OpenBSD at the time of this writing so it’s 802.11n only)
• Dual-core CPU with hyperthreading (be sure to boot GENERIC.MP)

The kernel recognizes the Intel SpeedStep capabilities of the CPU and will adjust the CPU’s clock speed as needed (further keeping the power consumption of the machine at a very Continue reading

L3 vPC Support on Nexus 5k

So… I’m a little embarrased to admit this but I only very recently found out that there are significant differences in how Virtual Port Channels (vPC) behave on the Nexus 5k vs the Nexus 7k when it comes to forming routing adjacencies over the vPC.

I’ve read the vPC Best Practice whitepaper and have often referred
others to it and also referred back to it myself from time to time. What I failed to realize is that I should’ve been taking the title of this paper more literally: it is 100% specific to the Nexus 7k. The behaviors the paper describes, particularly around the data plane loop prevention protections for packets crossing the vPC peer-link, are specific to the n7k and are not necessarily repeated on the n5k.

To that end, there are some topologies for Layer 3 peering over a vPC which are not supported on the n7k but are supported on the n5k. For example, peering a third-party Layer 3 device to the SVIs on the two n5ks in the vPC is supported.

The third-party device has an IP address on its port-channel interface and forms an Layer 3 adjacency/neighborship with Continue reading

So Your Username and Password Where in a Data Dump. Now What?

Whether it’s Dropbox, LinkedIn, MySpace, PlayStation, or whatever the latest breach happens to be, it’s almost inevitable that you will be caught up in one of these breaches and have your username, password and possibly other information exposed in a data dump. Here’s how to respond when that happens.

How Does This Happen?

A data dump is what often happens after a website has been breached and information about that site’s users/customers is stolen. All that stolen data is often “dumped” on the Internet for all to see. Once the data is dumped, it’s at that point that all this information becomes public and along with it, your information.

Sometimes, as in the case of the Ashely Madison dump, that information can be personally damaging. Other times the information is limited to usernames and passwords.

This article is going to focus on how to respond if your username and password are part of a data dump.

Step 1 – Reset Your Password

This is obvious, but go and change your password. Do it right now, before something comes along and distracts you. Even if you’re a security concious person and you’re using Two-Factor Authentication Continue reading

SSH Agent on OS X

There’s a lot of information on the intertoobs about getting ssh-agent “working” in OS X and even more articles about when and how the stock behavior of ssh-agent changed (mostly with respect to how ssh-agent interacted with the Keychain).

This article doesn’t cover or care about any of that.

This article is concerned with:

• Enabling ssh-agent in such a way that I can “ssh-add” in one terminal window and that same agent (and the loaded keys) is available in all of my other terminal windows.
• Enabling use of ssh-agent from MacPorts and/or Homebrew and not the older ssh-agent that OS X ships with in /usr/bin.
• To avoid having to put my keys in the Keychain (just a matter of preference).

Compatibility

Beware, reader. There’s an awful lot of outdated, inaccurate information out there on how to modify ssh-agent behavior on OS X. Guess what? OS X changes from version to version! Many articles out there cater to older versions of the OS and are either no longer applicable (due to changes in OS X behavior) or plain don’t work (due to functional changes in the software).

The steps below have been tested with OS X El Capitan (10.11).

What’s Continue reading

Cisco DevNet Scavenger Hunt at GSX 17

At Cisco’s GSX conference at the start of FY17, the DevNet team made a programming scavenger hunt by posting daily challenges that required using things like containers, Cisco Shipped, Python, and RESTful APIs in Cisco software in order to solve puzzles. In order to submit an answer, the team created an API that contestants had to use (in effect creating another challenge that contestants had to solve).

This post contains the artifacts I created while solving some of the challenges.

Tools Used

• Postman
• Browser (for reading docs and such)
• Python 2.7 (for Challenge 2)

Challenge 2: UCS Manager API

Your customer has been asked by their Help Desk Manager to make it easier on his first line engineers when requests come in related to UCS Server problems. Today, they have to train engineers on UCS Manager to get details like blade status, firmware levels, and uptime.  He is asking how they could embed those details in their help desk system.  Create some sample code leveraging the Python SDK for UCS Manager.

• There is a Service Profile with a dn of org-root/ls-gsx-minihack1. Find its int_id.

Answer

Challenge 5: ACI REST API

Congratulations! Your customer has selected ACI Continue reading

Auto Renew Let’s Encrypt Certificates

I’m a big fan of Let’s Encrypt (free, widely trusted SSL certificates) but not a big fan of most of the client software available for requesting and renewing certificates. Unlike a typical certificate authority, Let’s Encrypt doesn’t have a webui for requesting/renewing certs; everything is driven via an automated process that is run between a Let’s Encrypt software client and the Let’s Encrypt web service.

Since the protocols that Let’s Encrypt uses are standards-based, there are many open source clients available. Being security conscious, I have a few concerns with most of the clients:

• Complication. Many of the clients are hundreds of lines long and unnecessarily complicated. This makes the code really hard to audit and since this code is playing with my crypto key material, I do want to audit it.
• Elevated privilege. At least one of the clients I saw required root permission. That’s a non starter.

I can’t remember how, but I discovered a very clean, very simple client called acme-tiny at github.com/diafygi/acme-tiny. This script was obviously written by someone who shares the same concerns as I do and I highly recommend it to others.

I used acme-tiny to request my initial certificates — and it Continue reading