chris marget

Author Archives: chris marget

Inspecting SCEP enrollment traffic

SCEP is a protocol which facilitates client enrollment with a Certificate Authorities (CA), delivery and renewal of certificates and delegation of identity verification from a CA to a trusted Registration Authoritie (RA)

A project I'm working on requires me to generate a Certificate Signing Request (CSR) on behalf client which doesn't exist yet, and deliver of those requests to the CA via an RA that I'm building. I'll then set aside the certificate and keys for installation onto the client system when it becomes available.

It seemed like ripping apart a request from a real client, as delivered by a real RA would be a good place to start, so that's what I did. I set up a CA (R1), an RA (R2) and a client (R3), performed the enrollment and captured the traffic between the R2 and R1.

There's a nice diagram detailing how a client delivers its  to a CA on this Cisco page, so have a quick peek at the breakdown listed under Client Enrollment there.

A CSR delivered by an RA (rather than the client) is similarly encapsulated, except that both of the PKCS7 functions are performed by the RA (with the RA's private key), Continue reading

udevadm, systemd and a barcode scanner

I've been fooling around with a Symbol LS2208 barcode scanner attached to a CentOS 7 machine as part of a network automation project. I learned a bit about the scanner, udev and systemd along the way.


The LS2208
I chose the LS2208 because there were lots of them on eBay and because documentation was available. So far I'm happy with the LS2208, but wish it didn't require a physical PC to be nearby. A USB Anywhere box may be in my future (nope, Windows only). If I'd been able to find a WiFi scanner that would POST scans directly to a REST API over TLS, I'd have gone with that instead, but it seems that this guy and I are out of luck in that regard. I've got zero interest in fooling around with WinCE or similar mobile devices with built-in scanners.
The LS2208 gets configured by scanning barcodes. Special codes found in the manual. So far, the ones I've found most interesting are:
  • Set Factory Defaults
  • Simple COM Port Emulation
  • Low Volume
  • Beep on <BEL> (still need to fool with this - seems like it could provide useful feedback to the operator)
  • Do Not Beep After Good Decode

Cisco Debug Persists Through Reboot

Normal boot time messages from a C881 router look something like this:
 System Bootstrap, Version 15.4(1r)T, RELEASE SOFTWARE (fc1)  
 Technical Support: http://www.cisco.com/techsupport  
 Copyright (c) 2013 by cisco Systems, Inc.  
   
 Total memory size = 1024 MB  
 C881-K9      platform with 1048576 Kbytes of main memory  
 Main memory is configured to 32 bit mode   
   
 Readonly ROMMON initialized  
   
   
 IOS Image Load Test   
 ___________________   
 Digitally Signed Production Software   
 Self decompressing the image : ###<snip>### [OK]

But there's one router in the fleet which does this instead:
 System Bootstrap, Version 15.4(1r)T, RELEASE SOFTWARE (fc1)  
 Technical Support: http://www.cisco.com/techsupport  
 Copyright (c) 2013 by cisco Systems, Inc.  
   
 Total memory size = 1024 MB  
 C881-K9      platform with 1048576 Kbytes of main memory  
 Main memory is configured to 32 bit mode  
   
 Readonly ROMMON initialized  
 Using monlib version 2  
 Using version info 2  
   
  dfs_openfile: Opening file.....  
  dfs_openfile: Opened file / with fib = 4019e5c  
 Reading cluster = 126, offset = 0, nsecs = 8  
 Reading cluster = 133, offset = 0, nsecs = 8  
 Reading cluster = 17013, offset = 0, nsecs = 8  
 Reading cluster = 17458, offset = 0, nsecs = 8  
 Reading cluster = 18056, offset = 0, nsecs = 8  
 Reading cluster  Continue reading

Bailed out by Comware’s python interpreter

Something funny happened to an IRF stack the other day. The gear was moved to a remote location and, as part of the move, the cables/transceivers were moved around.

The IRF (10Gb/s) and uplink (1Gb/s) ports switched places.

The new IRF ports (5 and 6) got added to the configuration just fine, leaving me with:
 #  
 irf-port 1/1  
  port group interface Ten-GigabitEthernet1/1/1  
  port group interface Ten-GigabitEthernet1/1/2  
  port group interface Ten-GigabitEthernet1/1/5  
  port group interface Ten-GigabitEthernet1/1/6  
 #  
 irf-port 2/2  
  port group interface Ten-GigabitEthernet2/1/1  
  port group interface Ten-GigabitEthernet2/1/2  
  port group interface Ten-GigabitEthernet2/1/5  
  port group interface Ten-GigabitEthernet2/1/6  
 #

But IRF would't release the old ports making it impossible to repurpose them as uplinks:
 [switch]irf-port 1/1  
 [switch-irf-port1/1]undo port group interface Ten-GigabitEthernet1/1/1  
 Check failed for reason:  
  Can't support IRF on a port with 1000M speed!  
 [switch]

Can't remove a port from the IRF group because it's ineligible to participate in IRF. Okaaaaay...

Without functioning uplinks, it was impossible to transfer the saved configuration away for an off-box edit.

Python to the rescue!
I split the IRF, used python to edit the stored configuration in-place on each IRF member, then rebooted the switches individually. Here's the edit-by-python:
 <switch>python   
 Python 2.7.3 (default, Apr  Continue reading

Amazon Dash Button Events On A Catalyst

Lots of folks are detecting Amazon Dash button events by watching for ARP traffic with python.

I took a slightly different approach by watching for the button's MAC address with an EEM applet.

My Mac 'n Cheese button speaks on the network twice with each push: once right when it's pushed, and then a second time about 40 seconds later.

The applet sleeps for 60 seconds after it's fired to ensure that the button only creates a single event with each press.

 event manager applet macNcheese  
  event mat mac-address 00bb.3a4b.5a01 type add maxrun 90  
  action 1 syslog msg "It's Mac N Cheese time!"  
  action 2 cli command "enable"  
  action 3 cli command "copy https://username:password@some_server/path/to/events.php^V?eventtype=MAC%20N%20CHEESE%20TIME! null:"  
  action 4 wait 60  
  action 5 cli command "clear mac address-table dynamic address 00bb.3a4b.5a01"

event mat refers to "mac address table" changes. This applet fires only when the button's address is added to the table. Without the add keyword, the event would fire twice, once when the entry is added, and again when the entry is removed from the switch L2 filtering table.

I'm triggering an external event by hitting a web server that's already configured to receive Continue reading

Protocol Spotlight: DLEP

Dynamic Link Exchange Protocol is a mechanism by which link layer devices (probably radio modems) can communicate neighbor reachability information to IP routers using those radios.

Radio interfaces are frequently variable sub-rate interfaces. Path selection is a huge challenge with this sort of handoff, because not only is the available bandwidth less than the speed of the handoff interface, it's a moving target based on RF conditions from moment-to-moment. DLEP provides a flexible framework for communicating link performance and other parameters to the router so that it can make good path selection decisions.

It's obviously handy for point-to-point links, but that's not where it gets really interesting.

Consider the following network topology:


We have four routers sharing a broadcast network (10.0.0.0/24), each with a satellite backup link. Simple stuff, right?

But what if that 10.0.0.0/24 network isn't an Ethernet segment, but was really an ad-hoc mesh of microwave radio modems, and the routers were scattered among various vehicles, drones and robots?

The radios know the topology of the mesh in real time, but the routers plugged into those radios do not.

Wasting microwave bandwidth with BFD packets would be silly because it won't tell Continue reading

Anycast For DMVPN Hubs

Dynamic assignment of DMVPN spoke tunnel addresses isn't just a matter of convenience. It provided the foundation for a recent design which included the following fun requirements:
  • There are many hub sites.
  • Spokes will be network-near exactly one hub site.
  • Latency between hub sites is high.
  • Bandwidth between hub sites is low.
  • Spoke routers don't know where they are in the network.
  • Spoke routers must connect only to the nearest hub.
The underlay topology in this environment1 made it safe for me to anycast the DMVPN hubs, so that's what I did. This made the "connect to the nearest hub" problem easy to solve, but introduced some new complexity.

Hub Anycast Interface
Each DMVPN router has a loopback interface with address 192.0.2.0/32 assigned to the front-door VRF. It's configured something like this:

 interface loopback 192020
  description DMVPN hub anycast target  
  ip vrf forwarding LTE_TRANSIT  
  ip address 192.0.2.0 255.255.255.255

The 192.0.2.0 /32 prefix was redistributed into the IP backbone. If this device were to fail, then the next-nearest instance of 192.0.2.0 would be selected by the IGP.

Spoke Configuration
Spokes look pretty much exactly like Continue reading

OpenSwitch: Exciting Stuff

It was about a month ago that HP (along with several partners) announced OpenSwitch, a new network OS for white box switching hardware.

This week, HPE brought OpenSwitch Chief Architect Michael Zayats to present to TFDx delegates at the ONUG conference. I was fortunate to be one of these delegates and the usual disclaimers apply.

What is OpenSwitch?
It's an open source network OS for whitebox switching platforms. The code is open, and so is the development process. They're actively encouraging people to get involved. Coordination is done over IRC, bug tracking is open, documentation is available for edit, etc... Open. Open. Open.

Who is behind OpenSwitch?
Well, first there's the vendor consortium. To a large degree, it's that new company with the boxy logo: HPE. They employ the chief architect and a handful of developers. There are some other vendors, notably Broadcom (without whom this couldn't happen because of their NDA policies around silicon drivers), switch manufacturers (ODMs), etc...

Also of critical importance are the users: There are already some large end-user companies playing with, using, and contributing to OpenSwitch.

Wait how many OSes is HPE shipping/supporting now?
Yeah... Awkward! That's a couple of versions of Comware, Provision, Continue reading

Musings on Datanauts #9

I listened to episode 9 of the excellent Datanauts podcast with Ethan Banks and Chris Wahl recently.

Great job with this one, guys. I can tell how engaged I am in a podcast by how often I want to interrupt you :)

For this episode, that was lots of times!

Since I couldn't engage during the podcast, I'm going to have a one-sided discussion here, about the topics that grabbed my attention.

RARP?
Chris explained that the 'notify switches' feature of an ESXi vSwitch serves to update the L2 filtering table on upstream physical switches. This is necessary any time a VM moves from one physical link (or host) to another.

Updating the tables in all of the physical switches in the broadcast domain can be accomplished with any frame that meets the following criteria:

  • Sourced from the VM's MAC address
  • Destined for an L2 address that will flood throughout the broadcast domain
  • Specifies an Ethertype that the L2 switches are willing to forward
VMware chose to do it with a RARP frame, probably because it's easy to spoof, and shouldn't hurt anything. What's RARP? It's literally Reverse ARP. Instead of a normal ARP query, which asks: "Who has IP Continue reading

HP Is Shipping Unicorns Now: 10GBASE-T SFP+ Module

It's long been said that we'll never see an SFP+ transceiver for 10GBASE-T media. Too much power, too small package, too much heat, etc...

I'm not sure that never is quite right. There's this wonderful/horrible contraption:
Dawnray SFP+ module. Photo found here.
It's huge. It's ugly. Its covered with fins, so it must be hot. The data sheet says it consumes 7 Watts. Where's it getting 7W? Not from the SFP+ interface on the switch... Note the power cord attached to the module. It uses a wall wart!

This is not an elegant solution, but 10GBASE-T is hard, and this is the best we've got.

Until now.

/u/asdlkf recently pointed out on reddit that HP have published a data sheet1 for a much more elegant SFP+ module for 10GBASE-T.

There were rumors that this module was going to have a giant heatsink and protrude far beyond the SFP+ slot, but it turns out that's not the case. It looks really good, and it's only a bit longer than some 1000BASE-T modules that I have kicking around the office.

The module uses only 2.3W (no wall wart required, but plugging in lots of them will still tax most switches), Continue reading

Assigning DMVPN tunnel interface addresses with DHCP

I posted previously about some of the inner workings of DHCP. The three key points from that post are critical building blocks for this discussion:
  • DHCP requests get modified in flight by the DHCP relay.
  • DHCP relay determines L2 destination by inspecting contents of relayed packets.
  • DHCP clients, relays and (sometimes) servers use raw sockets because the end-to-end protocol stack isn't yet available.
The basic steps to converting a DMVPN from static address assignment scheme to dynamic are:
  1. Configure a DHCP server. I'm using an external server1 in this example so that we can inspect the relayed packets while they're on the wire.
  2. Configure the hub router. There are some non-intuitive details we'll go over.
  3. Configure the spoke router. Ditto on the non-intuitive bits.
My DHCP server is running on an IOS router (because it's convenient - it could be anywhere) and it has the following configuration:
    1     no ip dhcp conflict logging  
    2     ip dhcp excluded-address 172.16.1.1  
    3     !  
    4     ip dhcp pool DMVPN_POOL  
    5      network 172.16.1.0 255.255.255.0

So, that's pretty straightforward.

The Hub Router has the following relevant configuration:
    1     ip dhcp support tunnel unicast  
    2     interface Tunnel0  
    3       Continue reading

Cisco DHCP client bummer

It looks to me like the Cisco IOS DHCP client mis-handles the DNS server option when it's working in a VRF.

I'm working on an IOS 15.4 router with an empty startup-config and only the following configuration applied:
 interface FastEthernet4  
  ip address dhcp  
  no shutdown

debug dhcp detail produces the following when the DHCP lease is claimed:
 Sep 25 19:48:23.316: DHCP: Received a BOOTREP pkt  
 Sep 25 19:48:23.316: DHCP: Scan: Message type: DHCP Offer  
 ...  
 Sep 25 19:48:23.316: DHCP: Scan: DNS Name Server Option: 192.168.100.4

Indeed, we can resolve DNS. We can also see that the DNS server learned from DHCP has been configured (is there a better way to see this?):
 lab-C881#ping google.com  
 Translating "google.com"...domain server (192.168.100.4) [OK]  
 Type escape sequence to abort.  
 Sending 5, 100-byte ICMP Echos to 205.158.11.53, timeout is 2 seconds:  
 !!!!!  
 Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/8 ms  
 lab-C881#show hosts summary  
 Default domain is fragmentationneeded.net  
 Name/address lookup uses domain service  
 Name servers are 192.168.100.4  
 Cache entries: 5  
 Cache prune timeout: 50  
 lab-C881#

If I put the interface into a VRF, Continue reading

Just some quick points about DHCP

Okay, so everybody knows DHCP pretty well.

I just want to point out a few little details as background for a future post:

DHCP Relays Can Change Things
The first point is about those times when the DHCP client and server aren't on the same segment.

In these cases, a DHCP relay (usually running on a router) scoops up the helpless client's broadcast packets and fires them at the far away DHCP server. The server's replies are sent back to the relay, and the relay transmits them onto the client subnet.

The DHCP relay can change several things when relaying these packets:
  • It increments the bootp hop counter.
  • It populates the relay agent field in the bootp header (The DHCP server uses this to identify the subnet where the client is looking for a lease).
  • It can introduce additional DHCP options to the request.
The last one is particularly interesting. When a DHCP relay adds information to a client message, it can be used by the DHCP server for decision-making or logging purposes. Alternatively, the added information can be used by the DHCP relay itself: Because the relay's addition will be echoed back by the server, the relay can parse Continue reading

Path MTU Discovery with DMVPN Tunnels

Ivan Pepelnjak's excellent article on IP fragmentation from 2008 is very thorough, but it doesn't cover the functionality of Cisco's tunnel path-mtu-discovery feature when applied to mGRE (DMVPN) interfaces.

I played with it a bit, and was delighted to discover that the dynamic tunnel MTU mechanism operates on a per-NBMA neighbor basis, much the same as ip pim nbma-mode on the same interface type. Both features do all the right things, just like you'd hope they would.

Here's the topology I'm using:
Constrained MTU in path between R1 and R4


The DMVPN tunnel interface on R1 is configured with a 1400-byte MTU. With GRE headers, it will generate packets that can't reach R4. It's also configured with tunnel MTU discovery.
 interface Tunnel0  
  ip address 192.168.1.1 255.255.255.0  
  no ip redirects  
  ip mtu 1400  
  ip pim sparse-mode  
  ip nhrp map multicast dynamic  
  ip nhrp network-id 1  
  tunnel source FastEthernet0/0  
  tunnel mode gre multipoint  
  tunnel path-mtu-discovery  
  tunnel vrf TRANSIT  
 end

The two spokes are online with NBMA interfaces (tunnel source) using 10.x addressing. Both routers have their NBMA interfaces configured with 1500 byte MTU, and their tunnel MTU set at 1400 bytes:
 R1#show dmvpn  
 Legend:  Continue reading

Link Aggregation on HP Moonshot – A Neat Trick

The Broadcom switching OS running on HP's Moonshot 45G and 180G switches can do a neat trick1 that I haven't seen on other platforms.

Background: LACP-Individual
The trick revolves around interfaces that are sometimes aggregated, and sometimes run as individuals. Lots of platforms don't support this behavior. On those platforms, if an interface is configured to attempt aggregation but doesn't receive LACP PDUs, the interface won't forward traffic at all. Less broken platforms make this behavior configurable or have some goofy in-between mode which allows one member of the aggregation to forward traffic.

If the Moonshot were saddled with one of these broken2 switching OSes, we'd be in a real pickle: Moonshot cartridges (my m300s, anyway) require PXE in order to become operational, and PXE runs in the option ROM of an individual network interface. Even if that interface could form an one-member aggregation, it wouldn't be able to coordinate its operation with the other interface, and neither of their LACP speaker IDs would match the one chosen by the operating system that eventually gets loaded.

I suppose we could change the switch configuration: Add and remove individual interfaces from aggregations depending on the mode required by the Continue reading

Failing to the Cloud – and Back!

I attended Virtualization Field Day 5 last week! The usual Field Day disclaimers apply.

This network guy found himself way outside his comfort zone at a Virtualization event, but I had a fantastic time, and I learned a lot.

One of the things that really struck me was just how much virtualization platforms depend on mucking around with block storage in use by VMs. Half or more of the presentations hinged on it. Frankly, this notion terrifies the UNIX admin in me. I realize that we're not talking about UFS filesystems on SunOS4, but it seems those fragile old systems have really imprinted on me!

One of the VFD presenters was OneCloud Software, which presented a DR-via-Public-Cloud offering. The following bullets describing their solution came from here:

  • Auto discovers your on-premise assets; data and applications
  • Provides you with a simple policy engine to set RPO and RTO
  • Automatically provisions a fully functioning virtual data center in the cloud that mirrors your on-premise data center
  • Optimizes the economics of your data center in the cloud by eliminating unneeded compute costs and using the most cost-effective storage
  • Executes on-going data replication to keep the virtual data center in sync with the Continue reading

PSA: Linux Does RPF Checking

Twice now I've "discovered" that Linux hosts (even those that aren't doing IP forwarding) do Reverse Path Forwarding checks on incoming traffic.

Both times this has come up was in the context of a multicast application. It resulted in a conversation that went like this:
Application Person: Hey Chris, what's up with the network? My application isn't receiving any traffic.
Me: Um... The routers indicate they're sending it to you. The L3 forwarding counters are clicking. The L2 gear indicates it has un-filtered all of the ports between the router and your access port. Are you sure?
Application Person: My application says it's not arriving.
Me: I now have tcpdump running on your server. The traffic is arriving. Here are the packets. Do they look okay?
In the end, it turns out that the network was operating perfectly fine. The requested traffic was being delivered to the server, on the interface that requested it. It was the routing table within the Linux host that was screwed up.

RPF Checks
Reverse Path Flow checking is a feature that checks to make sure that a packet's ingress interface is the one that would be used to reach the packet's source. If a Continue reading

Controlling HP Moonshot with ipmitool

I've been driving the HP Moonshot environment over the network with ipmitool, and found it not altogether straightforward. One of the HP engineers told me:
Yeah, we had to jump through some hoops to extend IPMI’s single-system view of the world into our multi-node architecture.
That is exactly why it's confusing. Everything here works reasonably well, but users have to jump through all of the hoops that the product engineers lined up for us.

Compatibility
The build of ipmitool that ships with OS X (2.5b1) doesn't support the Moonshot's double-bridged topology, so I'm using the one that ships with macports (1.8.12). To check whether your version of ipmitool is compatible, run ipmitool -h and look to see whether it supports both the single-bridge (-b, -t) and double-bridge (-B, -T) command line options. If it does, then it's probably okay.

Bridging
Using IPMI over the network with a regular rack server is pretty straightforward. You specify the device by name or IP, the user credentials and the command/query you want to run. That's about it. Such a command might look like this:

 ipmitool —H <IPMI_IP> -U <user> —P <password> —I lanplus chassis identify force

The command above Continue reading

The Verizon SuperCookie Won’t Go Away

Update 4/21/2015:
It's been pointed out to me that Relevant Mobile Advertising (RMA - the thing responsible for the SuperCookie) and Customer Proprietary Network Information (CPNI) are not the same thing. That may be, but the link in the opt out instructions on Verizon's RMA info page goes to the CPNI settings below. If there's an RMA opt-out lever available to me somewhere on verizonwireless.com, I sure can't find it. I spoke with a new Verizon phone rep today. She claims to have sorted things out. My HTTP traffic still has the extra header attached. We'll see if that changes in the next few days...
Verizon Wireless made the news a few months ago when somebody noticed that they were adding extra HTTP headers which uniquely identified subscribers to every web request which traversed their network.

There was something of an uproar about it. I checked at least one of my phones, and was disappointed to find the tracking header attached to my traffic.

Then, less than two weeks ago, Verizon announced that customers would be allowed to opt out of having their web requests marked in this way. Many news outlets covered the announcement, Twitter rejoiced, and I Continue reading
1 2 3 4