Operational Annoyances: SSL Intermediate Certificates

You’re asked to update the SSL certificate for movingpackets.net on a load balancer. The requestor (me, I suppose) gives you the certificate, the private key and passphrase, and the intermediate bundle file provided by the certificate authority.

movingpackets.net.crt
movingpackets.net.key
movingpackets.net-intermediate-chain.crt

You faithfully go to the load balancer, upload the files, enter the passphrase, and create a client SSL profile referencing the cert/key/chain combination I provided, and all is well. The only thing is, you have 200 VIPs on the load balancer, mostly issued by the same certification authority (CA), so don’t they nominally share the same intermediate chain? (Hint: Almost certainly, yes)

Operational Annoyance

Here is the operational annoyance. The fact that the same intermediate certificate/chain has been uploaded 200 times with different names doesn’t stop things working, but it does seem rather inefficient. As far as I can determine, the F5 LTM load balancers (for example) actually concatenate all the uploaded certificates into a single bundle file and search the bundle when a certificate is referenced. I have no idea if there’s a huge performance gain here (unlikely), but it seems logical to want to minimize that file size regardless. On other Continue reading

Multi-datacenter Firewall Automation – Part 2

In my last post, I described the logical model I decided to use in order that I might be able to compute the path between endpoints using the Dijkstra shortest path algorithm. I’ve already discovered the handy Perl module module Paths::Graph that implements Dijkstra for me, so now all I need to do is to turn the network model into a data format that Paths::Graph will accept.

The Model

As a reminder, this is the model I’m using for my two-datacenter network:

To model this network, I need to allocate a cost to every network link. It turns out that Paths::Graph wants me to create separate links in each direction; this is a huge pain, and makes an already risk-laden process doubly painful. After initially creating the data model directly in the code, I realized that it would be much simpler (hah!) to model the network using YAML, and to remove the requirement to include both directions of any link. In my new YAML mode, a link from A->B automatically implies a link from B->A with the same cost; this halves the number of link descriptions required, which I’ll take as a win. The format is dead simple:

 Continue reading

Multi-datacenter Firewall Policy Automation

Back in that world where I reinvent the commercially available wheel, I’ve been wondering for a while about how to automate the creation of firewall policies in a multi-datacenter environment. This week I started tinkering with possible ways to achieve this, and knocked up some proof of concept code in my favorite untrendy, archaic language, Perl. Don’t say it.

The key issue is that given a firewall request (source IP, destination IP, port) it’s necessary to identify the firewalls and zones to which those rules apply, in order that rules can be automatically built in the right place(s). An additional twist I’ve seen is firewalls that have multiple routing instances, each of which maintains its own set of zones, effectively isolated from each other, even though they’re all on the same firewall.

Graphing Firewalls

I spent a while thinking about ways to model the firewall architecture, and kept on getting caught up on the firewalls with multiple routing instances. Because of the routing isolation, they behave like two separate firewalls, which makes it a little tricky to figure out the correct paths. I also wanted a solution that might also tell me which specific firewall zones were involved in the path; Continue reading

A Look Back at ONUG NYC 2015

I had a great time at the Open Networking Users Group meeting in NYC last month. Shortly afterward, I took a moment to think back on what I’d seen there, and that post was published on Gestalit.com. I won’t reproduce it here, so please take a moment to read my post at the source (gestaltit.com) and let me know what you think (here or there; either is fine). Topics covered include:

• Containerized microservices
• Software Defined WAN
• Cross-functional teams
• Adoption rates of new technologies

ONUG covered some cool stuff; I hope you enjoy the post.

Disclosure

My post at GestaltIT is a sponsored post as part of the ONUG Spring 2015 Tech Talk Series, part of the larger Tech Talks series.

If you liked this post, please do click through to the source at A Look Back at ONUG NYC 2015 and give me a share/like. Thank you!

The IOS XR CLI

If you liked this post, please do click through to the source at The IOS XR CLI and give me a share/like. Thank you!

NetBeez – Private Distributed Monitoring

I have a T-shirt from the nice folks at NetBeez, and on the back it says “Ask me about my Beez!” Naturally people can’t resist asking and I’ll happily explain, but on the off chance that you’re not lucky enough to be standing behind me I’m going to share my thoughts on NetBeez and the product sample they were kind enough to let me play with.

NetBeez at NFD9

I met NetBeez at Networking Field Day 9 where they presented their distributed monitoring solution. I’m not going to repeat what they said for the most part, so if you have time, it’s worth watching their full presentations from the NetBeez page at Tech Field Day.

The high level overview of the NetBeez solution is that they offer small, cheap, plug-and-play network agents that you can put on your network, from where they will perform connectivity and performance tests and report back to a central dashboard which can generate reports, statistics and alerts. The agents are designed to be cheap enough that you could put one on every remote site and monitor, say, access to a critical corporate server and find out quickly which remote sites are seeing Continue reading

Nuage Announces Underlay/Overlay Event Management Platform

During the ONUG event I met with Dimitri Stiliadis, the Co-Founder & Chief Technology Officer of Nuage Networks, who was excited to tell us about the latest product release, the Virtualized Services Assurance Platform (VSAP).

Virtualized Services Assurance Platform

The history of Nuage products has been fairly straightforward; they began with a virtualized networking solution targeted at data centers. More recently, Nuage Networks announced an expansion of that product into the branch office space. What was missing though, was a good way to monitor and manage the complex environment that was built, from underlay to overlay, from the WAN all the way to from virtual switch. When failures occur, they can be difficult to track down or, worse, you are flooded with alerts and left to figure out which ones are actually important, and which ones are the true root cause. To that end, Nuage Networks’ VSAP aims to provide visibility of the network and event correlation so that you can see what might be affected by a given network event.

VSAP is composed of two main components:

Route Monitor

The Nuage Route Monitor uses route protocols to peer with the production network in the data center, backbone, and anywhere else Continue reading

Liveblog from ONUG Day 2

Yesterday’s Liveblog was a success, so let’s try again. Today I’ll be covering the ONUG Town Hall meeting on the topic “Will the DevOps Model Deliver in the Enterprise?”, featuring such luminaries as:

• Najam Ahmad (Facebook)
• Mike Dvorkin (Cisco)
• Tim Gerla (Ansible)
• Dimitri Stiliadis (Nuage Networks)
• Marc Woolward (vArmor)

This has the potential to be a great discussion; based on lunch with Dvorkin I can confirm that he is in great form. Please join me by following along below!

If you liked this post, please do click through to the source at Liveblog from ONUG Day 2 and give me a share/like. Thank you!

Liveblog from ONUG!

We’re going to try out a new thing today – liveblogging from the ONUG Spring 2015 presentations here in NYC. If it doesn’t work, I apologize – but it’ll be fun trying!

If you liked this post, please do click through to the source at Liveblog from ONUG! and give me a share/like. Thank you!

Juniper EX4550. EX1113. EX063. What?

I came across some weird behavior (I think) in Junos recently. Nothing major, but an annoyance nonetheless.

Juniper EX4550

Let’s check out some statistics on this beautiful looking EX4550 stack, shall we?

john@EX4550> show pfe statistics bridge

Slot 0

PFE:                         0
----------------------------------------
----  Ingress Counters ----
-- Set0 --
Received:            654383803
VLAN Filtered:               0
Security Filtered:           0
Other Discards:            324
-- Set1 --
Received:            654383797
VLAN Filtered:               0
Security Filtered:           0
Other Discards:            324
[...]

Oh dear look, 324 “Other” discards on my EX4550. I sure hope they aren’t incrementing. Let’s issue the command again and check if the 324 figure has increased?

{master:0}
john@EX4550> show pfe statistics bridge

Slot 0

PFE:                         0
----------------------------------------
----  Ingress Counters ----
-- Set0 --
Received:               913272
VLAN Filtered:               0
Security Filtered:           0
Other Discards:              0
-- Set1 --
Received:               913267
VLAN Filtered:               0
Security Filtered:           0
Other Discards:              0
[...]

No it hasn’t. In fact it has mysteriously reset itself along with all the other PFE statistics. That is, well, suboptimal to say the least.

It’s probably a known bug, though it’s not important enough for me to go look it up. Maybe it’s even intentional (though for the life of Continue reading

Curious About ONUG? Wonder What It Is?

In case you’re interested in ONUG or wondering what the heck ONUG stands for, I write a short post called What Is An “ONUG” over at GestaltIT. I’m not going to duplicate it here, so please do the clicky over to GestaltIT and have a read. Bonus: there’s a registration discount code over there as well! Hope you enjoy it; ONUG looks like it’ll be very interesting.

Disclosure

My post at GestaltIT is a sponsored post as part of the ONUG Spring 2015 Tech Talk Series, part of the larger Tech Talks series.

If you liked this post, please do click through to the source at Curious About ONUG? Wonder What It Is? and give me a share/like. Thank you!

Cisco Introduces Two New Cisco CCIE Logos

Thanks to my contacts in Cisco PR I’m really excited to be able to share the news that Cisco is introducing two new CCIE Logos for 2015, under the title of “CCIE ABL”. If you’re wondering what “ABL” is, it’s not another specialization track, but rather it signifies progress towards the ultimate goal of a CCIE number.

Phillip A. Prull is the Senior VP with the responsibility to oversee the valuable CCIE® brand for Cisco, and he explained to me why Cisco felt that they needed an additional logo:

“One of the things we recognized was that many engineers were working for many months – even years – towards passing our rigorous CCIE® lab exam, but until they finally achieved that success, many of them told us that they felt unable to demonstrate to potential employers that they had made progress towards that goal. In response, we have created a new logo – the Cisco CCIE® ABL™.

Beginning in April 2015, any candidate who has passed a CCIE qualification exam (aka the “written”) will be able to download the new CCIE® ABL™ logo from our Certification Tracking System, and use it on their resumés and business cards, subject to Continue reading

Telling OpenSSL About Your Root Certificates

OpenSSL doesn’t come with its own trusted root certificates; you have to tell it where to find them. This should be straightforward – and it is – but Apple have found a way to make it trickier.

Normal *nix Systems

On a normal unix system, openssl is pretty good at locating the root certificates, but it still doesn’t automatically reference them. For example running Ubuntu:

john@ubuntu:~$ openssl s_client -connect www.microsoft.com:443
CONNECTED(00000003)
depth=2 C = US, O = "VeriSign, Inc.", OU = VeriSign Trust Network,
  OU = "(c) 2006 VeriSign, Inc. - For authorized use only", CN =
  VeriSign Class 3 Public Primary Certification Authority - G5
verify error:num=20:unable to get local issuer certificate
verify return:0
[...removed for brevity...]
    PSK identity hint: None
    SRP username: None
    Start Time: 1425842365
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---

Openssl is unable to validate the Verisign certificate. So where are the trusted root certificates stored? Actually, Openssl will tell us:

john@ubuntu:~$ openssl version -d
OPENSSLDIR: "/usr/lib/ssl"

Add that into the command as the -CApath parameter, and:

john@ubuntu:~$ openssl s_client -CApath /usr/lib/ssl -connect 
  www.microsoft.com:443
CONNECTED(00000003)
depth=3 C = US, O =  Continue reading

Five Essential OpenSSL Troubleshooting Commands

 Troubleshooting SSL certificates and connections? Here are five handy openssl commands that every network engineer should be able to use. Bookmark this – you never know when it will come in handy!

1. Check the Connection

 openssl s_client -showcerts -connect www.microsoft.com:443

This command opens an SSL connection to the specified site and displays the entire certificate chain as well. Here’s an abridged version of the sample output:

MBP$ openssl s_client -showcerts -connect www.microsoft.com:443
CONNECTED(00000003)
depth=2 /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006
 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public
 Primary Certification Authority - G5
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/1.3.6.1.4.1.311.60.2.1.3=US/1.3.6.1.4.1.311.60.2.1.2=
     Washington/businessCategory=Private Organization/
     serialNumber=600413485/C=US/postalCode=98052/ST=Washington/
     L=Redmond/street=1 Microsoft Way/O=Microsoft Corporation/
     OU=MSCOM/CN=www.microsoft.com
   i:/C=US/O=Symantec Corporation/OU=Symantec Trust Network/
     CN=Symantec Class 3 EV SSL CA - G3
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
 1 s:/C=US/O=Symantec Corporation/OU=Symantec Trust Network/
     CN=Symantec Class 3 EV SSL CA - G3
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006
     VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3
     Public Primary Certification Authority - G5
-----BEGIN CERTIFICATE-----
[...]
 Continue reading

When SSL Certificates Go Wild

You’ve set up your website and secured it with an SSL certificate that you bought through your ISP. Everything works fine and the chain of trust is just fine in your browser, but when you try accessing your secured site using a command line tool, the connection fails. Why? There’s a good chance that you are not sending your intermediate certificate(s) along with the server certificate.

PKI Trust Review

As a quick reminder, the whole point of SSL certificates and the Public Key Infrastructure is to prove that the site you connected to is the one it says it is. How do we know? The server sends you a certificate with its name in it, digitally signed by an Issuer. If you choose to trust that Issuer’s honesty and believe that they made sure they issued to the right site, you implicitly trust that the end site is the right one; it’s a “Chain of Trust.”

In reality, we don’t typically trust many Issuers. Look in the Trusted Root certificates for your browser, or on a Mac, open Keychain Access and look in System Roots, and you’ll see that for Yosemite in this case, globally – to establish SSL Continue reading

Office Word 2011 – An Annoying Little Bug

Today I take a gentle swoop away from networking into the world of documentation. Specifically, I’m looking at a bug in Microsoft Office Word for Mac that drives me bonkers.

Custom Properties

The management of custom properties in Microsoft Word has never been the most elegant part of Word’s interface. The mechanism by which the property values are managed has always been a bit clunky and unfriendly, but at least in the Windows version of Word, it at least functioned as stated.

In Office for Mac however, there’s a bug when you select a property to edit. Since it’s easier to show than tell, here’s a quick video demonstrating the issue.

Now isn’t that a silly bit of coding? Yes, I thought so too! Microsoft just announced availability of the Office 2016 Mac Preview, but sadly Document Properties is not yet available:

If you find out before I do, please let me know!

If you liked this post, please do click through to the source at Office Word 2011 – An Annoying Little Bug and give me a share/like. Thank you!

This man pressed Print. What happened next left me speechless.

Although I attended HP Discover in Barcelona as a guest of the folks at HP Networking (via their Independent Bloggers program), I didn’t restrict myself to looking at etherstuff; HP makes way too broad a portfolio of products to get away with that. I ended up looking at printers, and I found something that pretty much blew me away.

Before I forget, please accept my apologies for the clickbait headline. I’ve always wanted to do one of those; but unlike so many others I’ve seen, I hope that this article won’t disappoint. You’ll see “the man” in the headline in a video later.

A Sign of the Times

It was hard to miss at HP Discover that HP believes in eating their own dog food. The signage at the event – many large, inspiring, multicultural images reminding us that HP’s mission is to provide “solutions for the New Style of IT” – was all printed on HP printers. The signs looked pretty amazing, I have to say:

Not only were they printed using HP products, but the poster tells you which printer was used, you know, in case you wanted to buy one for your spare bedroom or something:

Given the price Continue reading

Why Do TFD Delegates Hate Gartner?

A thought, since I’ve just returned form Networking Field Day 9: I’ve heard it said that delegates to Tech Field Day events hate Gartner; that this is stupid, or that it’s some other form of snobbery. It’s certainly true that on the whole, TFD delegates do not like to see Gartner slides in presentations.

That Gartner Slide

So why do Tech Field Day delegates hate Gartner?

We don’t.

We do.

We don’t, but we do. It’s complicated.

Hey, it’s complicated, alright?

Bonus points for obvious movie reference identification. So look, here’s the thing. You folks reading this blog are likely pretty intelligent. The fact that you read tech blogs means you are looking to get information that will help you make a decision or solve a problem.

In my opinion, the problem with a company at Tech Field Day putting up a Gartner slide and maybe boasting that “we’re in the top right quadrant” is that it’s not what we think we’re there to hear. We’re sitting in front of the actual people who make the product, and instead of convincing us of their product’s merits by telling and showing us how amazing it is, they’re saying “Here’s what Continue reading

Software Defined Reality – NFD9 Redux

I’ve just got back from Networking Field Day 9 (NFD9) and my head is buzzing after a busy week of presentations. I posted a preview of NFD9 so it seems only fair to give a quick wrap up of the week’s themes and presentations as I saw it.

My NFD9

After some time spent thinking on the flights back home, I came to the conclusion that there were two themes that were recurring this week.

The dominating theme for me was, at last, seeing the magic rainbow-expelling problem-solving unicorn that is Software Defined Networking – SDN – and all its inherent paradigm-shifting magic, turned into products that actually seem real, and are starting to deal with some of the issues that were flagged up when SDN was first being described. It’s relatively easy to SDN-wash a product, but making it something from which a user can actually benefit, well, that’s something else.

The second theme was that many of the products looked to the concept of detecting or fixing problems before the users were aware of them, whether as an alert from a monitoring system, or a network that automatically self-heals or otherwise avoids problem areas.

SDN == Programming

Don’t Continue reading

Networking Field Day 9 Is Almost Here!

Confession: I fly out to San Jose, CA for Networking Field Day 9 tomorrow morning and I have yet to pack a single thing.

This last minute preparation thing seems to have become a habit. I’d like to tell you that it’s because I’m a seasoned traveller and I can calmly pack for a two week vacation in under 15 minutes, but really it’s just procrastination. It’s not even that I dislike travel; I quite enjoy going places and I definitely enjoy Networking Field Day so I know there’s something amazing waiting for me when I get there. I conclude then that my lackadaisical approach to travel packing is laziness in its purest form.

Networking Field Day 9

Any idea of being lazy at NFD9 though is laughable. As ever, we have a packed schedule meeting some really interesting vendors. There are some old faces and some new ones both in the attendees and the vendors!

Wednesday 10th February

• VeloCloud – “Cloud-Delivered Software Defined WAN”. If they give prizes for buzz-word density, VeloCloud just won. The only thing missing is “as a Service”. Humor aside, VeloCloud seem to have an interesting product; it sounds almost like Cisco Continue reading