chris marget

Author Archives: chris marget

HP Moonshot – Stuff I Wish I’d Known

Yeah, it looks just like this.
I've been working with HP Moonshot for some months now. It's a neat box with a lot of interesting features. There are plenty of press releases and (possibly paid-for) "reviews" available out there, but not much frank commentary from actual end users, and that's disappointing.

What is it?
Briefly, Moonshot is a miniature blade enclosure. I'm sure there are marketing folks who would like me to use different terminology, but it boils down to servers, Ethernet switches and power all rolled into one box.

There are some key differentiators between this enclosure and some of its larger cousins:

Low Power - The whole package is tuned for high density and low power. There are no monstrously fast multi-socket servers available, but the density is amazing. With 8 cores per node and 180 nodes per chassis we're talking about 300+ cores per rack unit!

Less Redundancy - Unlike the C-class enclosures which sport redundant "Onboard Administrator" modules, Moonshot has a single "Chassis Manager". I do not view this as a problem for two reasons: First, Moonshot is mostly suited for massively horizontally scalable applications which should tolerate failure of a whole chassis. Second, failure of Continue reading

Exporting RSA keys from Cisco ASA: Harder than it should be

Unlike Cisco IOS routers, which by default don't allow RSA private keys to be exported from NVRAM, Cisco ASAs don't protect private keys. But there's no command (of which I'm aware) to directly export the keys either.

Sometimes you need to squirrel away those keys. You can do it by getting a certificate that uses the keys, then exporting a certificate bundle (with private key included). Here's how.

First, create a key:
 crypto key generate rsa label mykey modulus 2048

Next, create a trustpoint which references the key, and generate a self-signed certificate:
 crypto ca trustpoint throwaway  
  keypair mykey  
  enrollment self  
 crypto ca enroll throwaway noconfirm

Now the throwaway trustpoint has a certificate. Export that certificate to the terminal.
 no terminal pager  
 crypto ca export throwaway pkcs12 <passphrase>

Save the blob of text including the begin/end lines. The blob is a PKCS12 bundle encrypted using the passphrase above and then base64 encoded. Be sure to save the encryption passphrase.
 -----BEGIN PKCS12-----  
 MIIJZwIBAzCCCSEGCSqGSIb3DQEHAaCCCRIEggkOMIIJCjCCCQYGCSqGSIb3DQEH  
 BqCCCPcwggjzAgEAMIII7AYJKoZIhvcNAQcBMBsGCiqGSIb3DQEMAQMwDQQI4KTD  
 ...etc...  
 ru1WrVnO7wFa+83BK8D+aQ7UedzQuU6NOiDrjPR0w8uWSLwKmmSVgnZN4BEwPTAh  
 MAkGBSsOAwIaBQAEFGA2bfp4y+a/R29RZ9TA8sCUSZ+jBBRvppgVbM8rBbW62096  
 L/HnJErexgICBAA=  
 -----END PKCS12-----

We no longer need the certificate or the throwaway trustpoint in which it's stored. Kill it. The private key will survive.
 no crypto ca trustpoint throwaway noconfirm

The easiest way to Continue reading

Set Up PKI Service on a Cisco Router

This is how I tend to create a PKI service on a Cisco router. Some of the details here were non-obvious to me after reading the documentation several times. Maybe I can save somebody else a headache or two.

First, create a directory for the PKI server to work in. This step may be optional if the router is going to be using some network-based storage for all of its elements, but I find it handy to have, and it's easy to move things around afterward. I like using removable media when keeping things on routers, so that it's easy to snag the critical data if there's a hardware failure.
 mkdir usbflash:/MY_ROOT_CA

Next, generate an RSA keypair. It needs to be exportable, which is the reason I'm doing it manually, rather than let the router generate it automatically at CA startup. Name it the same as the CA will be named in the crypto pki server <whatever> configuration section.
 crypto key generate rsa label MY_ROOT_CA modulus 2048 exportable storage nvram:

Now export the keys. Having a copy of them squirreled away somewhere will be absolutely critical if you ever need to replace the CA. Replacing the CA will be Continue reading

Happy Birthday SR629007199!

It is with disappointment and frustration that I'm celebrating the 1st birthday of an unresolved Cisco support case. I'm not happy about it, plan to do some complaining in this post.

Now, don't get me wrong, I think the people at Cisco TAC are great. They're an absolutely first class support organization, the standard by which other vendors are judged, and they consistently give me great service.

In spite of their efforts, sometimes things just don't work out. This is one of those times.

I opened SR629007199 on January 31st 2014 after noticing a peculiar problem with an ISR G2 router: Servers couldn't receive packets intended for them because the router was screwing up their traffic. The Ethernet frames carrying these packets included the wrong destination MAC address, so the servers ignored them.

Specifically, the router was screwing up the IP->L2 address mapping required for IPv4 multicast packets. Instead of using 23 bits of the multicast group in the L2 header, that portion of the L2 header was all zeros. It looked like this:

Bogus dMAC on most of these frames

Those two HSRP packets originated by the router looked okay, and traffic from local sources was okay, but every Continue reading

Manually calculating MST digests

Switches sharing an MST region must agree on three things:
  • The region name
  • The region revision level
  • The region's mapping of VLANs to STP instances
The first two are exchanged directly inside BPDUs, so they are easy to validate. The third item is about 6KB of data at a minimum. It doesn't fit in a BPDU.

Rather than exchange the table directly, MST switches calculate a 128-bit hash of the table and exchange that instead. If the hashes match, the VLAN-to-stp-instance mapping database is assumed to match.

Most platforms will show you the calculated digest.

Catalyst:
 lab-catalyst#show spanning-tree mst configuration  
 Name   [lab]  
 Revision 3   Instances configured 4  
 Instance Vlans mapped  
 -------- ---------------------------------------------------------------------  
 0     4-9,40-99,101-199,201-299,301-4094  
 1     1,10-19,100  
 2     2,20-29,200  
 3     3,30-39,300  
 -------------------------------------------------------------------------------  
 lab-catalyst#show spanning-tree mst configuration digest  
 Name   [lab]  
 Revision 3   Instances configured 4  
 Digest     0x37D94E0098E3418C046F217A71077FB1  
 Pre-std Digest 0xFC2190275BBB19CD9A6F1BB116DB10E7  
 lab-catalyst#

Procurve: 
 lab-procurve# show spanning-tree mst-config  
  MST Configuration Identifier Information  
  MST Configuration Name : different              
  MST Configuration Revision : 4    
  MST Configuration Digest : 0x37D94E0098E3418C046F217A71077FB1  
  IST Mapped VLANs : 4-9,40-99,101-199,201-299,301-4094  
  Instance ID Mapped VLANs  
  ----------- ---------------------------------------------------------  
  1      1,10-19,100  
  2      2,20-29,200  
  3      3,30-39,300  
 lab-procurve#

Because their VLAN-to-instance mapping is the same, both switches arrived at the same digest value. Note that Continue reading

L2 DCI failure remediation with proxy ARP

I had the pleasure of taking an interstate road trip with Ethan Banks last weekend. Naturally, we talked shop a bit.

Somewhere in Connecticut we were speculating about the possibility of using host routes and proxy ARP to restore connectivity between members of a subnet when an L2 DCI fails.

Would a router produce proxy ARP replies for destinations which are members of a directly connected network?

I labbed it up to find out.
Don't do it!
The routers run different HSRP groups. One per data center.

Hosts are configured to use the local HSRP address as their default gateway. There is no FHRP localization other than the configuration on the hosts. This is not a scheme that facilitates VM mobility between sites.

Each router uses an IP SLA monitor to check the L2 interconnect. This is accomplished by pinging the other site's HSRP address. The IP SLA monitor drives a tracking object (track 1). The tracking object drives a second tracking object (track 2) with inverted logic: when the routers can't ping each other track 2 transitions to "up":
 ! IP SLA monitor on R1
 ip sla 1
  icmp-echo 192.168.1.2 source-interface FastEthernet0/1
 ip  Continue reading

New fiber connector is nifty

Corning has recently teamed up with Intel in introducing some new optical equipment. Corning's contribution (fibers, connectors) likely mean there will be some unfamiliar looking optical infrastructure in your data center soon.

The fiber is a new 1310nm singlemode variety that Corning touts as "bend-insensitive". The minimum allowable bend radius of this fiber is 7.5mm. This is impressive, but expected under ITU-T G.657.B.

More interesting is the MXC connector. This is a push-on connector with a locking tab like the 8P8C connectors used for twisted pair Ethernet. It supports up to 64 fiber strands, each running at 25Gb/s.
MXC connector. Image from Corning-Intel Whitepaper.

The only place I've seen this fiber or connector in use is on a prototype 100G CLR4 transceiver shot by Greg Ferro at the Intel Developer Forum a couple of weeks ago.
Greg's shot of CLR4 transceivers with MXC connectors.
The CLR4 alliance explains that their approach puts four channels running at 25Gb/s each onto a single pair of single mode fiber, and specifically calls for LC connectors on the transceiver, so I'm a little confused about why these transceivers are sporting MXC connectors.

It seems the MXC connector will be used not Continue reading

Making better use of libvirt hooks

Libvirtd includes handy hooks for doing management work at various phases in the lifecycle of the libvirt daemon, attached networks, and virtual machines. I've been using these hooks for various things and have found them particularly useful for management of short-lived Linux containers. Some of my use cases for these hooks include:
  • changing network policy
  • instantiating named routing tables
  • creating ramdisks for use by containers 
  • pre-loading data before container startup
  • archiving interesting data at container shutdown 
  • purging data at container destruction

Here's how the hooks work on a system with RedHat lineage:

The hook scripts live in /etc/libvirt/hooks. The scripts are named according to their purpose. I'm focusing right now on the LXC hook which is named /etc/libvirt/hooks/lxc. Note that neither the directory, nor the scripts exist by default.

The lxc script is called several times in each container's lifecycle, and is passed arguments that specify the libvirt domain id and the lifecycle phase. During startup and shutdown of one of my LXC systems, the script gets called five times, like this:

 /etc/libvirt/hooks/lxc MyAwesomeContainer prepare begin -  
 /etc/libvirt/hooks/lxc MyAwesomeContainer start begin -  
 /etc/libvirt/hooks/lxc MyAwesomeContainer started begin -  
 /etc/libvirt/hooks/lxc MyAwesomeContainer stopped end -  
 /etc/libvirt/hooks/lxc MyAwesomeContainer release end -

In addition to having those Continue reading

No more duplicate frames with Gigamon Visibility Fabric

disclaimer

Gigamon presented their Visibility Fabric Architecture at Network Field Day 8.  You can watch the presentation at Tech Field Day.

CC BY-NC-ND 2.0 licensed photo by smif
Needs deduplication
One of the interesting facets of Gigamon's solution was it's ability to do real-time de-duplication of captured traffic as it traverses the Visibility Fabric (a hierarchy of monitoring data sources and advanced aggregation switches). I've spent some time around proactively-deployed network taps, but never seen this capability before, and I think it's pretty nifty.

The Problem
Let's say you've got taps and mirror ports deployed throughout your network. They're on the uplinks from data center access switches, virtually attached to vSwitches for collecting intra-host VM traffic, at the WAN and Internet edge, on the User distribution tier, etc...  All of these capture points feed various analysis tools via Gigamon's Visibility Fabric. It's likely that a given flow will be captured and fed into the monitoring infrastructure at more several points. Simplistic capture-port-to-tool-port forwarding rules will result in a given packet being delivered to each interested tool more than once, possibly several times.

This can be problematic because it confuses the analysis tool (ZOMG, look at all Continue reading

Cisco 881 or Cisco 881?

There are two versions of the Cisco 881 branch router:
  • Part numbers beginning with CISCO881, which have been end of lifed.
  • Part numbers beginning with C881, which are newly available.
There are a bunch of differences between these models, but it's hard to tell that a difference even exists, let alone what the differences are by looking at the available documentation. I just got my hands on a new C881 for the first time. Here's what I've noticed.

Physical Differences
New C881 on top, old CISCO881 (not wireless - don't believe the stickers) on bottom.
New C881 on top
Twin screw holes on the new C881...

...make the ACS-890-RM-19= work on the C881.
  • The USB port has been moved from one side to the other.
  • The "fake" screw hole on the side is now a threaded hole, which means that the C881 will accept the 891's rack mount hardware.
  • The Fa4 port has moved a bit.
  • The C881 is lead free, which seems to be what prompted all of these gyrations.
    Power Differences
    • We have a power switch!
    • There's no longer a dedicated PoE brick.
    • There's still a required internal PoE module, and it's got a different part number.
    Licensing Differences

    Wireshark on Android

    The other day I found myself wishing I could run wireshark in realtime on an Android phone, but use the familiar GUI on my laptop. After a few minutes tinkering around, I was doing exactly that.

    The phone belonged to an Android developer, so he'd already rooted it, enabled developer tools, etc... He'd also installed a packet capture application which worked, but didn't allow me to see things in real time.

    The Android SDK bundle contains the adb binary, which is required for connecting to the phone. Extract adb and drop it somewhere in $PATH
     # run adb as root:  
     adb root  
     # connect to the phone over WiFi (the phone's owner had
     # already enabled this feature with 'adb tcpip' via USB):  
     adb connect <phone's wifi ip address>  
     # check that we get a root shell on the phone:  
     adb shell 'id'

    It turns out that the packet capture application included a tcpdump binary at /data/data/lv.n3o.shark/files/tcpdump, and invoking it from the adb shell worked normally. It produced the usual startup message, and then a one line summary of each packet.
     adb shell '/data/data/lv.n3o.shark/files/tcpdump -c 2'
     tcpdump: verbose output suppressed, use -v or -vv for  Continue reading

    Relaying email with postfix + TLS through gmail

    I needed to relay email from appliances in my house, and wanted to use my gmail domain + TLS to do it. Following are my notes from setting up a postfix server to do that job. All email relayed by this server appear to be sourced from the gmail account I created for it.

    I wouldn't use this for anything customer-facing, but it's a reasonable way to get messages out of closed environments without worrying about how the messages were sourced, who they appear to be from, will SPF records screw things up, etc...

    Create gmail account
    I'm using an account named [email protected]. I set that guy up, and gave him a password.

    Install Linux somewhere
    I'm using a minimal installation of CentOS 6.5 for this project, installed with some automated nonsense I've long used for this sort of thing.

    Tweak hostname
     sed -i 's/localhost.localdomain/postfix-relay.marget.com/' /etc/sysconfig/network

    NFS mount my CentOS repository
    The next little bit uses automounter to hang my CentOS repository on /CentOS and configure it as a repository. Skip it.
     yum install -y nfs-utils wget tcpdump unzip autofs  
     service rpcbind start  
     service autofs restart  
     ln -s /net/my_nfs_server/path/to/CentOS/ /CentOS  
     cp /etc/yum.repos.d/CentOS-Media.repo  Continue reading

    SDN: Where Everything is a Honeypot

    Beware the honeypot army!
    HP Networking introduced one of their SDN App Store partners to the Tech Field Day crowd at the ONUG spring conference a few weeks ago. If you don't know about ONUG, but you're interested in real-world SDN options and operator experience free of vendor lies, you should probably check out the upcoming fall ONUG conference.1 

    GuardiCore's Active Honeypot SDN offering really captured my imagination in ways that other SDN demonstrations have failed to do.

    The objective is to detect/ensnare an intruder who has already compromised an asset in your datacenter and is now attempting to move on from there. Honeypots are one way of doing this, but the likelihood of an attacker finding the honeypot, rather than a real server with real vulnerabilities is pretty low in a large data center. How can we improve the odds?

    The solution assumes that during normal operations, clients know where servers are and don't waste time attempting to connect to services which don't exist. An attacker, on the other hand, will be looking to find vulnerable services, and will probably attempt lots of connections to services that don't exist.

    Because the attacker doesn't know where Continue reading

    Custom DHCP server configuration on Opengear

    For the last few years I've run my home DHCP service on a virtual private server at AWS.

    This was not a great idea. It's a pain to resolve issues with my Internet service when those issues cause my laptop to stop getting an IP address assigned because the path between my house and the DHCP server has been interrupted.

    The service is at Amazon because I wanted to purge "server" like things from the house, but its clear that I needed to bring DHCPd back home. I started investigating moving the service to one of my Opengear ACM5000 units, which is always running anyway because it keeps tabs on my generator and home security system, sends me text messages about interesting events and whatnot.

    The Opengear web UI doesn't offer too many DHCP service configuration options, but I didn't expect that to be a problem. One of the things I love about Opengear is that most anywhere you look, the baked-in scripts and configuration elements can be replaced with user-supplied versions of those things.

    I'd expected to find something like include /etc/config/dhcpd-user.conf in the automagically-generated DHCPd configuration, but it wasn't there.

    So, I strings-ed every file in Continue reading

    Tab Completion on Cumulus Linux

    This film could have ended much differently

    if Jerry were running Cumulus Linux 

    The TAB key on my keyboard gets a lot of use. Whether I'm looking at a bash prompt on a *NIX system or logged into a router's CLI, I almost never type whole commands.

    In the bash shell, tab completion capabilities are usually limited to helping complete:

    • shell built-in commands
    • external executables found in $PATH
    • file names
    • directory names
    Completion in bash doesn't help with things like command line arguments to various commands, but it is (sometimes) smart enough to not offer filenames as completion options to the 'cd' command, choosing instead to only offer directories.

    Network devices, on the other hand, tend to have really rich inline help / command completion stuff, and I live by it.

    Rather than typing abbreviated commands, I prefer to let the system help me type the whole thing, partly because it eliminates errors, and partly because I usually can't remember the exact syntax for a given platform. Cisco's godawful platform-dependent mac-address-table vs. mac address-table comes immediately to mind as something that always seems to take more than one attempt.

    So, rather than typing this:
    ROUTER#sh ip bg vpnv4 vr Continue reading

    SDN Themes from ONUG – Let the ASIC go

    Edit: I banged this out on the flight home from ONUG four months ago. Just found it in the drafts folder. ONUG's spring 2014 conference in New York is just 3 months away.

    I was privileged to attend the Open Networking User Group (ONUG) Conference, ONUG Academy and mini Tech Field Day event hosted by JP Morgan Chase on October 29 and 30.
    I attended at someone else's expense. Disclaimer.

    ASICs came up a lot during these couple of days. Following are some ASIC-related things I heard and overheard at ONUG.

    Sun Microsystems was overly attached to their SPARC processor (and so was I!) Folks inside Sun made efforts to derail Solaris x86, in order to protect their favorite server platform, and contributed to killing the company altogether. Sad story.

    As good as your ASIC is, you'll never keep up with the performance of commodity chipsets. If the whitebox stuff is faster and still good enough to do the job, then it's probably going to win. It's certainly going to cost less. The proprietary ASIC may be better and have more features, but better is the enemy of good enough.

    People used to route packets using general-purpose servers Continue reading

    ACI Launch

    Tech Field Day brought me to the Cisco Application Centric Infrastructure launch event last week in New York. I attended at someone else's expense, but that doesn't mean my opinions are for sale, etc...

    If you're totally unfamiliar with ACI (formerly Insieme), I recommend listening to Episode 12 of the Class C Block podcast with guest Joe Onisick. This was far more informative than anything I encountered at the actual launch event, probably because the Tech Field Day crew went straight from the John Chambers presentation into a room where we recorded a roundtable discussion. There may have been some technical discussion going on next door, but I missed it.

    There's no shortage of people expressing opinions about ACI and what it will or won't do for you, most of whom have beaten me to the punch by several days. I'm going to post instead about a few details of the launch that I found interesting.

    Defining Policy Might Not Be Easy
    ACI requires that applications (really application owners) express to it the relationships between nodes before any traffic is allowed to flow. There are countless ways this might happen, but they all boil down to figuring out which ports Continue reading

    Stuff Spirent didn’t cover at NFD6

    Spirent presented their Avalanche NEXT product at Gestalt IT's Network Field Day 6 event in September of this year.

    Disclaimers:
    1. I attended the event, somebody else picked up the tab, yada yada.
    2. I like Spirent. I've used their products, worked with their people and been to their offices several times, and have gradually become a fan. This fact might be coloring my opinions :)
    What's Avalanche NEXT?
    Avalanche NEXT is a software frontend for performing tests of security hardware (firewalls and whatnot). It's a modular system that allows simple creation of test components (clients, servers, subnets, protocols, etc...), mixing of modules to create various tests, scaling of the test, application fuzzing, etc... Check out 1:14 - 1:30 in the video below to get a flavor of how slick the interface is. I love that interactive pie chart.


    This sort of testing is critical because security devices have some of the most misleading data sheets of anything we're likely to work with as network folks. Security box performance numbers depend heavily on what you're asking them to do. Will your current device survive if you enable SSL offload, application inspection, or similar features? You can try it out (fingers crossed! Continue reading

    SDN Themes from ONUG – Community Matters!

    I was privileged to attend the Open Networking User Group (ONUG) Conference, ONUG Academy and mini Tech Field Day event hosted by JP Morgan Chase on October 29 and 30.
    I attended at someone else's expense. Disclaimer. Additional disclaimer: I have a personal relationship with one of the people behind the ONUG conference. That fact will not color the opinions I express about ONUG. If I say I like something, it's because I like it, okay? :)

    SDN Joke from Brent Salisbury's awesome ONUG
    presentation    .
    I don't know these cats nor the owner of the photo.
    Community Matters
    ONUG is founded on the idea that the Software Defined Network (SDN) user community needs to stand up for itself. Prior to ONUG the direction of SDN was set by a handful of players including:
    • Vendors who are interested in shaping the SDN marketplace and standards bodies around the capabilities of their products, rather than around the problems being faced by their customers.
    • Powerful end users who needed SDN to solve their own peculiar problems. The problems they're solving, and the techniques they're using do not align well with the challenges nor capabilities of mere enterprise users.
    • Researchers, who took SDN Continue reading

    Calculating distances in meatspace

    I'm working on an automated provisioning system for a very large VPN network. For each new VPN client, I need to select a headend site where VPN tunnels should land. The only data available is that which I can get from the sales and billing systems. This system offers me the zip code of the install site.

    Using the zip code of the install site, and the known zip codes of my various head-end sites, I'm able to select the destination for the primary and secondary VPN tunnels.

    It's not perfect (physical location often has little to do with network path), but it's better than nothing. I haven't decided how to handle non-US sites yet.

    I'm using a database of US zip codes found here, and a very dirty perl script. The script grabs the latitude and longitude of two zip codes from the database, and prints the mileage between them as calculated using the Haversine formula for great circle distance.

    It runs like this:


    Christophers-MacBook-Pro:scripts chris$ zipdistance.pl 95134 60614
    1837 miles
    Christophers-MacBook-Pro:scripts chris$

    The script:


    #!/opt/local/bin/perl
    use GIS::Distance;

    my $dbfile="/Users/chris/Downloads/zipcode.csv";
    my $lat1,$lon1,$lat2,$lon2;

    sub usage{
      printf "Usage: $0 <zipcode> <zipcode>n";
      exit;
    }

    if (@ARGV ! Continue reading
    1 2 3 4