0

Walking with Packets: Traceroute Through MPLS Cloud

Think about this for a minute: An MPLS network with a two Provider Edge (PE) routers and some Provider (P) routers. The P routers have no VRFs configured on them and therefore have no routes whatsoever for any of the customer networks. A customer then does a traceroute from one of their sites, across the MPLS cloud, and into one of their other sites. The traceroute output shows the P routers as hops along the path.

How is it possible for the P routers to reply to the traceroute if they don’t have routes back to the customer network?

The Lab Setup

Here’s the network:

 

Here’s the traceroute output from R21’s loopback0 to R8’s loopback0 (the last octet of each IP address corresponds to the name of each router):

R21#traceroute 10.1.8.8 source loopback0
Type escape sequence to abort.
Tracing the route to 10.1.8.8
VRF info: (vrf in name/id, vrf out name/id)
  1 10.4.4.4 21 msec 18 msec 17 msec
  2 10.2.45.5 [MPLS: Labels 21/24 Exp 0] 19 msec 18 msec 18 msec
  3 10.2.15.1 [MPLS: Labels 21/24 Exp  Continue reading

0

Five Functional Facts about EIGRP

Normally for these FFF articles I’ve taken to writing about new protocols as a way of introducing others to it and also edumacating myself about it. For this post I get all nostalgic and look at good old Enhanced Interior Gateway Routing Protocol (EIGRP).

1 – The EIGRP Metric is Calculated From a Formula

Unlike RIP with its simple hop count or OSPF with its simple bandwidth metric, the EIGRP metric is actually derived by plugging a number of values into a formula and solving the formula. The formula looks like this:

Let’s talk about the k values first. The k values are constants that are configured in IOS and fed into the formula. They have the affect of basically turning on and off the variables that are used in the calculation: bandwidth, delay, load, reliability. They also have the affect of giving more or less emphasis to a variable. For example, setting k3 to 50 would give the “delay” variable more emphasis than if k3 is set to 1. The default settings for the k values are:

• k1 and k3 = 1
• k2, k4, k5 = 0

This has the net result of simplifying the Continue reading

0

EIGRP “FD is Infinity”

Let’s take a look at EIGRP and the state a route can get into where EIGRP tells you “FD is Infinity”.

First of all, every EIGRP speaker maintains a local database called the EIGRP topology table which holds a copy of every route received from every neighbor and every route being advertised by the local system. EIGRP performs its best-path decision process on the entries in this table in order to determine which routes are the best and then hands those best routes to the Routing Information Base (the RIB). By inspecting the entries in this table, you can see things like:

1. Bandwidth, Load, Delay, Reliability – the values to go into computing the composite metric
2. The actual composite metric that the local system has calculated
3. The composite metric that the neighbor calculated
4. Whether the route is internal or external
5. A list of all neighbors that advertised the route (neighbor’s router ID)
6. The feasible distance (FD) for the route
7. The metric for the route as seen in the RIB

I’ve used superscript numbers (^x) in the output below to indicate where each item in the list above is found.

R12#show ip eigrp topology 10.1.11.0/24
EIGRP-IPv4  Continue reading

0

OS X – Outlook Search “No Results”

The worst feeling for a geek:

This has happened to me twice now: upgrading Mac OS X from one release to another and after the dust settles, the search function in Outlook 2011 totally breaks and always returns “no results”. As we all know, email sucks and being able to deftly search through that mound of crap in your mail client is the only thing that makes it somewhat bearable.

When I upgraded from 10.8 to 10.9, I was the guy in the cartoon above. I had to resort to uninstalling and reinstalling all of Office to get this repaired. Urgh.

Well, I just upgraded from 10.9 to 10.10 and lo, the same problem with Outlook search. However this time my karma must be right topped off because I found the solution buried in a message board after an hour or so of searching.

The post is from the macrumors.com forum and exactly described the issue and how to fix it on my machine. As stated, the permissions on my Microsoft Office 2011 directory allowed only my account to open the directory:

jknight@mac:~% ls -ld /Applications/Microsoft Office  Continue reading

0

Lab: iBGP and OSPF Traffic Engineering

Here’s the scenario: An enterprise network with an MPLS core and two branch locations connected to their own Provider Edge (PE) router. In addition to the MPLS link, the PEs are also connected via a DMVPN tunnel. The PEs are peering via iBGP (of course) and are also OSPF neighbors on the DMVPN. Both Customer Edge (CE) routers at the branch are OSPF neighbors with their local PE.

Task: Use the high speed MPLS network as the primary path between the CE routers and only use the DMVPN network if the MPLS network becomes unavailable.

Question: Is the solution as simple as adjusting the Admin Distance (AD) so that the iBGP routes are more preferred?

Default State

The obvious first issue is the default AD for iBGP (200) is higher than the default AD of OSPF (110) which means the OSPF path over the DMVPN is going to be preferred. This is confirmed if we do a traceroute from R5 to R6:

R5#traceroute 6.6.6.6 source lo5
 1 10.0.45.4 2 msec 0 msec 1 msec
 2 10.10.10.7 17 msec 17 msec 17 msec
 3 10.0.67.6 18  Continue reading

0

Choosing a Route: Order of Operations

In Cisco IOS packets are forwarded through the router (or Layer 3 switch) by Cisco Express Forwarding (CEF). A data structure called the CEF table contains a list of known IP prefixes and the outgoing interface that packets should be put on in order to get them onwards to their destination. That’s well and good. But how do the IP prefixes make it into the CEF table? To answer that question you have to work backwards and understand the order of operations that IOS goes through in order for a prefix to make it into the CEF table.

The answer to the question of what makes it into the CEF table confused me a bit, particularly when working with complex redistribution schemes. I would end up concentrating so much on admin distance (AD) that I would overlook the other, more important elements that went into determining what went into the CEF table. In order to improve my understanding I came up with this order of operations which helps me not only with redistribution, but in pretty much any situation where I’m trying to do traffic engineering.

You will not find this order of operations on cisco.com or in any Continue reading

0

Role Based Access Control in IOS

I don’t believe this is well known: Cisco IOS has Role Based Access Control (RBAC) which can be used to create and assign different levels of privileged access to the device. Without RBAC there are two access levels in IOS: a read-only mode with limited access to commands and no ability to modify the running config (also called privilege level 1) and enable mode with full administrative access. There is no middle ground; it’s all or nothing. RBAC allows creation of access levels somewhere between nothing and everything. A common use case is creating a role for the first line NOC analyst which might allow them to view the running config, configure interfaces, and configure named access-lists.

A “role” in IOS is called a “view” and since views control which commands are available in the command line parser, they are configured under the parser. A view can be assigned a password which allows users to “enable” into the view. More typically, the view is assigned by the RADIUS/TACACS server as part of the authorization process when a user is logging into the device.

A view is configured with the “parser view <view-name>” config command after which commands are added/removed to/from Continue reading

0

What the *, traceroute?

If you’ve ever done a traceroute from one IOS box to another, you’ve undoubtedly seen output like this:

R8# traceroute 192.168.100.7
Tracing the route to 192.168.100.7
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.0.1 4 msec 3 msec 4 msec
  2 192.168.100.7 4 msec *  0 msec

That “msec * msec” output. Why is the middle packet always lost?? And why only on the last hop??

This was always something curious to me but not something I ever bothered to learn about. Well it turns out that IOS has a rate limiter that meters the generation of ICMP Unreachable messages. The default setting for the rate limiter is 1 ICMP Unreach every 500ms. Since IOS’s traceroute doesn’t put a delay between its probe packets, the delay between when 192.168.100.7 receives the first and second probe packets is much less than 500ms. The second packet violates the rate limiter and so 192.168.100.7 drops it.

Why isn’t the third packet also dropped? Because the traceroute command waits for 3 seconds (by default) before deciding that a probe packet was lost and Continue reading

0

IPv4 QoS Markings Calculator

This is a quick calculator I came up that I could use in the CCIE lab to translate between various IPv4 header QoS markings. As long as I could remember how to draw out the calculator, all I had to do was some basic math and I could translate between markings quite easily.

This post does not explain what the header fields are, why there’s so many or what the significance of one QoS value is over another. I’m making an assumption that the reader is already familiar with QoS concepts.

Here’s the calculator:

x	x	x	x	x	x	x	x
128	64	32	16	8	4	2	1	ToS
32	16	8	4	2	1	-	-	DSCP
4	2	1	2	1	-	-	-	DSCP AF values (AFxy)
4	2	1	-	-	-	-	-	IP Precedence
								Enter 0 or 1 in each position

Each column represents one bit from the Type of Service (ToS) field in the IPv4 header. The right-most column describes what each row is for. The very last row is meant to be filled in by us; it’s the input into the calculator.

As an example, let’s say we Continue reading

0

CCIE R&S – By the Numbers

When I started studying in earnest for my CCIE, I started a log of how I was spending my time studying, which books and papers I’d read, videos I’d watched, and so on. I thought it would be a neat exercise to look back afterwards at what it took to achieve this goal. I’m also somewhat self-deprecating and tend to minimize my accomplishments, so having this data is a way for me to remember that this wasn’t a small accomplishment at all.

1,041,248 bytes of digital notes taken

13,916 km traveled (8,711 mi)

1,432 total study hours

652 hours in the lab

321 hours of just reading

223 videos watched

161 hours spent watching those videos

128 PDF documents read

23 books read

5 figures worth of expenses and costs

1 completely trashed USB mouse

Beyond these numbers there’s the intangibles that went into this goal too. No vacations. Giving up free time on evenings and weekends. Not seeing friends as much. Not spending as much time with family. Maintaining focus constantly on the end goal. Constantly staying up late and getting up early.

Looking back at all of this, I realize that becoming CCIE certified isn’t just about Continue reading

0

Hello, I’m 47321

And now the big reveal. The reason I haven’t been blogging or doing much of anything for some time now is because I’ve had a teeny tiny side project going on:

And this week I passed the lab exam! I am CCIE 47321 (Routing and Switching).

---

Copyright Joel Knight. All Rights Reserved.
www.packetmischief.ca

0

2014 End of Year Blog Statistics

Although I wasn’t actively blogging this year, I’ve still been tracking views on the site and keeping an eye on number of visitors. I am looking forward to blogging again in 2015 so I wanted to collect the 2014 viewership statistics so I had something to compare to at the end of 2015.

Despite the lack of new content, I’m pretty happy that people are reading the existing articles and continue to post comments and email me questions. Please keep them coming!

Here are the year-over-year statistics for Jan 1 2014 – Dec 30 2014.

Overall YoY comparison:

Once again these stats are courtesy of Google Analytics. If you compare this chart to the 2013 equivalent, you’ll see that Google has changed some of the metric names. Visits are now called Sessions and Unique Visitors are now Users. And like last year’s chart, the first number in gray is the current year’s number, the second number the previous year.

While the number of Users and Sessions grew at a slower rate than in 2013, they still grew! Very cool.

As was the case in 2013, the users visiting the site are primarily made up of brand new visitors. It Continue reading

0

Will I Be Blogging Again?

Dan wrote in with a question:

Hey, I like your site as well… are you going to be doing any more posts? It just seems odd that your last blog post was the analyzation of the site and how it portends to your future work. anyway, good luck

I know it’s tacky to write a blog post about how you’re not writing enough blog posts… but here goes.

Yes, I will be blogging again. And I have lots of content ideas. And I’m actually itching to get back to writing. I’ve been working on something else for the past few months and I decided I couldn’t take that on and blog at the same time. So for now my writing is on hold, however I do see and respond to all comments in the articles and am reachable via email as well.

Thanks to everyone who reads and posts comments. I look forward to writing more posts in the new year!

---

Copyright Joel Knight. All Rights Reserved.
www.packetmischief.ca

0

2013 End of Year Blog Statistics

I debated whether to write an article like this. It seems to be the “in vogue” thing to do if you’re a blog author but I wasn’t inclined to do it until I started looking at the data (I heart data). When I started looking at the data, I saw not only number of visitors and so on to the blog, but a breakdown of browser versions and operating systems (thank you Google Analytics for the rich reports). As is often the case, your data set can tell you more than you initially bargained on.

All of the reports below are comparing year-over-year stats: Jan 1 2012 – Dec 29 2012 vs Jan 1 2013 – Dec 29 2013.

First, the overall picture of visits to the blog:

38% more overall visitors and 53% more unique visitors in 2013. Pretty cool! And over 58 thousand people put eyes on my blog in the past year. That’s something I hope to build on in 2014.

The first unexpected data point I saw is that in 2013, the blog saw more new visitors than in 2012. That means the readership is expanding to include new people which is excellent.

 

Next up Continue reading

0

IOS and NX-OS Platform Identifiers

I was preparing a presentation the other day about the high level differences between IOS, IOS-XE and NX-OS and one of the things I included in the presentation was the various platform and branch identifiers that’s used in each OS. It’s just a bit of trivia that I thought would be interesting and might come in handy one day. I’m posting the information I collected below so everyone can reference it.

IOS

For IOS, you can see the platform identifier in the filename and the “show version” output. If we take an example version string, 15.1(2)SG2, and break it down, here’s what each digit in the string signifies:

15	.1	(2)	SG	2
Major release number	Minor release number	New feature release number	Branch/train/platform identifier	Maintenance rebuild number

The branch/platform identifier can also be seen in the IOS filename:

cat4500e-entservicesk9-mz.151-2.SG2.bin

Here’s a summary of the branch identifiers in IOS 15:

• M/T – 800 Series ISR, ISRg2 (1900, 2900, 3900), Connected Grid Router 2000
• S – 7600 Series Router, ME 3600X, ME 3800X
• GC – 5900 Series Embedded Router
• SE – Catalyst 2960, 3560, 3750, IE 3000 Switches
• SG – Catalyst 4500E (Sup 6E and earlier)
• Continue reading

0

Five Functional Facts about VXLAN

It seems appropriate to write a FFF post about Virtual Extensible LAN (VXLAN) now since VXLAN is the new hotness in the data center these days. With VMware’s NSX using VLXAN (among other overlays) as a core part of its overall solution and the recent announcement of Cisco’s Application Centric Infrastructure (ACI) and the accompanying Nexus 9000 switch, both of which leverage VXLAN for delivering a network fabric, it seems inevitable that network engineers will have to use and understand VXLAN in the not too distant future.

As usual, this post is not meant to be an introduction to the technology; I assume you have at least a passing familiarity with VXLAN. Instead, I will jump right into 5 operational/technical/functional aspects of the protocol.

For more information on VXLAN, check out the draft at the IETF.

1 – VXLAN Use Cases

Despite the apparent ubiquity and fervent hype around VXLAN, it’s actually been designed to solve specific problems. It has not been designed to be “everything to everyone”.

The first, and most often cited, use case is for data center operators that require more than ~4000 logical partitions in the network. These 4000 partitions equate to the maximum number of Continue reading

0

Why I Use MediaWiki for Taking Notes

I was prompted to write this when I observed someone the other day who was sitting in the same training as me taking notes in a self-addressed email. No offense to people who do this, but W. T. F. How are you going to keep track of that email among the dozens/hundreds you receive every single day?

I take a lot of notes for research, certification study, and training. I use MediaWiki for almost all of these notes. Here’s why.

What is MediaWiki?

First off, MediaWiki is not a text editor. This may seem strange but after reading this whole article, I hope you’ll understand why that doesn’t matter and in fact, why that makes it more powerful. As the name implies, MediaWiki is actually software for running a wiki. In fact, it’s the same software that runs the most famous wiki, Wikipedia.

MediaWiki runs on UNIX (including OS X) and Windows machines. It’s written in PHP and runs under almost any web server (Apache, lighttpd, nginx, IIS). By its very nature, it’s web-based, which plays nicely into one of the reasons I like using it so much.

Simple Markup Language With Rich Rendering

When I’m taking study notes or Continue reading

0

Five Functional Facts about OTV

Following on from my previous “triple-F” article (Five Functional Facts about FabricPath), I thought I would apply the same concept to the topic of Overlay Transport Virtualization (OTV). This post will not describe much of the foundational concepts of OTV, but will dive right into how it actually functions in practice. A reasonable introduction to OTV can be found in my series on Data Center Interconnects.

So without any more preamble, here are five functional facts about OTV.

#1 – OTV Adds 42 Bytes of Overhead

OTV, being an encapsulation technology, adds additional headers to the encapsulated payload. Without rehashing too much of the basics, OTV extends a Layer 2 domain across a Layer 3 cloud. In order to preserve the Layer 2 semantics on either side of the cloud, OTV scoops up the entire Layer 2 packet on one side, transports it across the cloud in the middle, and puts it on the LAN in the other side. This preserves the entire Ethernet header including the original source/dest MAC, and even the CoS bits and VLAN tag.

So to begin with, we’re putting a (potentially) full-sized Ethernet frame – with headers – inside another Ethernet frame. That Continue reading

0

Understanding CME Overlays with Dual-Line DNs

Normally I talk about overlays in the context of data center/SDN/cloud but today I’m going out into left field and am going to talk about voice! :-)

I freely admit that I’m a noob when it comes to Cisco voice so I’m not sure if the behavior I’m about to describe is obvious or not. It wasn’t obvious to me and I only figured it out after running into the issue for real and troubleshooting it to resolution.

The issue stems from my misunderstanding about how dual-line ephone-dns function when used in an overlay.

The Desired Behavior

Here’s the scenario: Cisco Communications Manager Express (CME) with a handful of IP phones registered to it. There’s no Communications Manager (UCM); the dial plan lives entirely within CME. PSTN connectivity is via (4) POTS lines terminated on FXO ports on the router running CME.

The desired behavior is to have incoming calls from the PSTN ring on all the IP phones simultaneously. If a second call comes in, it should ring on all the remaining phones, and so on.

The Original CME Config

Here’s what the original CME config looked like. At least, the parts of the config relevant to this blog Continue reading

0

Plumbing OpenBSD Software with gdb(1)

This post is about finding and fixing a memory leak I discovered in the SNMP daemon, snmpd(8), in OpenBSD. This sort of analysis is foreign territory for me; I’m not a software hacker by day. However, using instructions written by Otto Moerbeek as my Rosetta stone and Google to fill in the blanks when it came to usage of the GNU debugger, gdb(1), I was able to find and fix the memory leak.

I’m documenting the steps I used for my future self and for others.

The Problem

When walking the pfTblAddrTable in the OPENBSD-PF-MIB, the unprivileged snmpd process would grow in terms of SIZE and RES. Querying other parts of PF-MIB or other MIBS altogether resulted in no memory usage increase.

Since I knew roughly which code path must have the leak, I first examined it manually. I could not see where memory wasn’t being given back. I needed to instrument the process as it was running in order to find the leak.

Before Starting

This set of instructions from Otto Moerbeek was my guide. As per his guide, you have to rebuild libc with MALLOC_STATS enabled. This enables statistics collection that is used later on.

Edit /usr/src/lib/libc/stdlib/malloc. Continue reading