Three Months with Google Fiber

I'm one of the lucky few to benefit from Google Fiber's recent expansion into new regions (before they nixed the whole thing). I've had the service fire three months now and figured I should write up my experience with it thus far.

The Installation

Google Fiber announced that it would be expanding to the Raleigh-Durham metro area, known locally as "The Triangle", in January 2015. It's been a long game of hurry-up-and-wait since then, watching crews laying fiber all over town without hearing a peep from Google regarding availability. But in the fall of 2016, people were finally able to start signing up for service. Here's how my installation went.

September 3

Google Fiber registration opens! I sign up for service and pay a paltry $10 deposit, which gets credited toward my first bill. Over the next couple weeks, various utilities swing by to mark their lines in the ground. (Here's the color code for utility markings in the US, if you're curious.)

September 24

Google's contractor arrives on site to lay fiber from the curb to my house and to many of my neighbors' houses. Surprisingly, they cut my trench by hand, possibly due to the steep Continue reading

Legacy TLS cipher support in Firefox

After upgrading Firefox recently, I noticed that I could no longer access certain embedded devices via HTTPS. It seems that recent versions of Firefox and Chrome no longer support certain TLS ciphers due to recently discovered vulnerabilities. That's all well and good, except the error returned offers no recourse if you need to connect anyway.

Firefox returns the error SSL_ERROR_NO_CYPHER_OVERLAP with no option to temporarily allow connectivity. (Chrome reports a similar error named ERR_SSL_VERSION_OR_CIPHER_MISMATCH.) Presumably, this choice was made by the developers with the intention of forcing people to upgrade outdated devices. Unfortunately, in order to upgrade an out-of-date device, we typically must first be able to connect to it. I wasted a fair bit of time digging up a solution, so I figured I'd document the workaround here for when I inevitably run into this problem again a year from now and have forgotten what I did.

Continue reading · 2 comments

The Overlay Problem: Getting In and Out

I've been researching overlay network strategies recently. There are plenty of competing implementations available, employing various encapsulations and control plane designs. But every design I've encountered seems ultimately hampered by the same issue: scalability at the edge.

Why Build an Overlay?

Imagine a scenario where we've got 2,000 physical servers split across 50 racks. Each server functions as a hypervisor housing on average 100 virtual machines, resulting in a total of approximately 200,000 virtual hosts (~4,000 per rack).

In an ideal world, we could allocate a /20 of IPv4 space to each rack. The top-of-rack (ToR) L3 switches in each rack would advertise this /20 northbound toward the network core, resulting in a clean, efficient routing table in the core. This is, of course, how IP was intended to function.

Unfortunately, this approach isn't usually viable in the real world because we need to preserve the ability to move a virtual machine from one hypervisor to another (often residing in a different rack) without changing its assigned IP address. Establishing the L3 boundary at the ToR switch prevents us from doing this efficiently.

Continue reading · 4 comments

Taking the CCIE Lab in RTP

Cisco's campus in Research Triangle Park, North Carolina, is one of only two places in the United States where candidates can complete a CCIE lab exam (the other being in San Jose, California). People fly in from all over the eastern US and beyond to spend a day taking the exam. Lots of folks who've taken the exam have written up their experiences, but I haven't seen many talk at length about their time in RTP outside of Cisco's building 3.

I've lived just a few minutes away from the testing site for the past few years, and it occurred to me recently that visitors might benefit from some local knowledge.

Getting Here

Most people fly in via Raleigh-Durham International Airport (RDU). RDU is a medium-sized airport with two terminals. Most flights operate out of Terminal 2, except for Southwest Airlines, which is based in the newly-renovated Terminal 1.

As airports go, I'm a big fan of RDU. It's a very modern, clean, and well-organized facility. The interior of Terminal 2 is beautifully designed to resemble an early airplane wing and is flooded with natural light during the day. (It's also one of very few places where you can Continue reading

NetBox v1.1.0 Released

One year ago today, I made the first commit to a repository named "netbox" hosted internally at DigitalOcean. It was the first iteration of a tiny little app I scratched together using the Django Python framework to track IP prefix utilization. A year later, NetBox has grown into an extensive tool that we use to track IPs, racks, devices, connections, circuits, and even encrypted credentials. And I'm happy to say that it's now open source!

Continue reading · 2 comments

Announcing NetBox

Several years ago, I lamented the few options available for a provider-grade IPAM solution. Specifically, I explained why building a custom application would be undesirable:

Could I create a custom IPAM solution with everything we need? Sure! The problem is that I'm a network engineer, not a programmer (a natural division of labor which, it seems, is mostly to blame for the lack of robust IPAM solutions available). Even if I had the time to undertake such a project, I have little interest in providing long-term maintenance of it.

But I suppose time makes fools of us all.

Nearly one year ago, I started developing an IPAM application as part of my day job. Leveraging my experience with the Django Python framework, I had a working proof-of-concept in just a week. Over the next several months, the project grew more mature and began to take on additional roles: data center infrastructure management, circuit tracking, and credentials storage. Today, the tool functions as our "source of truth" for many aspects of our infrastructure. We call it NetBox.

Continue reading · 6 comments

Don’t be Discouraged by Plagiarists

Recently, a friend pointed out that an individual had taken one of my cheat sheets, superimposed his own logo and URL on it, and published it as his own work. This is certainly not the first time I've been plagiarized, nor will it be the last, I suspect. I called out the individual on Twitter, and I'm very gratefully for the many people who helped me compel him to remove the illegitimate content. Eventually.

I wanted to write a quick post sharing my thoughts on this incident for the benefit of everyone who has expressed interest in starting their own blog or web site. I've heard plenty of people comment over the years to the effect of, "Why bother starting a blog if someone's just going to harvest the RSS feed and re-publish it on their own site to make a few bucks?" Indeed, this has always been a concern among producers of both free and paid content.

I wish I could tell you that plagiarism isn't that big a deal, or that it won't happen to you. But the truth is plagiarism is a huge problem in our industry (and across the Internet in general), and if Continue reading

Can You Keep a Secret? (Part 2)

In part one, we saw how AES can be used to encrypt sensitive data so that it can be retrieved only by using an encryption key. The problem with this approach is that everyone who needs access to the data must have a copy of the key. If any one of these copies becomes compromised, the entire database must be re-encrypted using a new key, and the new key must be distributed securely to all parties involved. In this article, we'll see how symmetric encryption can be combined with asymmetric cryptography (namely RSA) to create a hybrid cryptosystem.

Let's begin by encrypting some data using AES as we did in part one. First we pad our plaintext's length to a multiple of 16 using null bytes, then generate a 256-bit encryption key and a 128-bit IV, and finally encrypt it with CFB-mode AES to generate a string of ciphertext.

>>> from Crypto.Cipher import AES
>>> import os
>>> plaintext = "Operation Neptune will launch on June 6th"
>>> plaintext += (16 - len(plaintext) % 16) * chr(0)
>>> encryption_key = os.urandom(32)
>>> iv = os.urandom(16)
>>> cipher = AES.new(encryption_key, AES.MODE_CFB, iv)
>>> ciphertext =  Continue reading

Can You Keep a Secret?

I've been developing an IPAM/DCIM tool for work over the past several months (more on that soon), and recently my focus has been on expanding it to store confidential data associated with network devices. Backup login credentials, TACACS+/RADIUS secrets, SNMP communities, and so on: Short strings that need to be stored securely.

Hashing

Storing a password or other small piece of sensitive data is different from merely authenticating against it. Most password storage mechanisms never actually store a user's actual password, but rather an irreversible hash of it. (That is if you're doing it correctly, at least.)

For example, the Django Python framework (which powers packetlife.net) by default employs salted SHA256 hashes to authenticate user passwords. When a password is saved, a random salt is generated and concatenated with the plaintext password. (A salt is used to prevent two identical passwords from producing the same hash.) The SHA256 algorithm is then run against the whole thing to produce a fixed-length hash. Here's an example in Python using Django's built-in make_password() function:

>>> from django.contrib.auth.hashers import make_password
>>> make_password("MyP@ssw0rd!")
u'pbkdf2_sha256$12000$x5E0yB2dh13m$ablUOER8qn4CxjmHZlJrUUA1Cb9MeLXvfggTnG56QpM='

Continue reading · No comments

Stretch’s Hierarchy of Network Needs

Remember Maslow's hierarchy of needs from school? The theory asserts that every human shares the same set of basic physical and psychological needs in order to be happy, with more primal needs like food and shelter taking precedence over emotional needs like love and companionship.

A while back, I was pondering what would be necessary to fully automate a network, and it occurred to me that a very similar hierarchy of needs can be laid out for a computer network to achieve its optimal state.

1. A Functioning Network

At the very bottom of the hierarchy is everything a network requires to function: Routers, switches, cabling, power, and so on, just as tier one of Maslow's hierarchy encompasses everything a human needs to stay alive. At this stage, a network can function, and can even function well, but it cannot adapt or grow.

Many small businesses operate their networks at this stage for years with no major problems. After all, when left alone, computers and networks tend to just keep chugging along. And if your entire network comprises a cable modem, a switch, and a few access points, it's entirely possible that it will run for years without needing Continue reading

Thoughts on Two Years of Working from Home

I've spent the past two years working from home as a network engineer for two different companies. At first, I wasn't sure how well the remote lifestyle would suit me, but after a short time I settled into a very comfortable routine. And to my surprise, I discovered that I was much more productive working from the serenity of my home office than I ever was in a cubicle. I'd like to share my observations with the hope of convincing others to try ditching the office as well.

Why Work Remote?

No More Commute

This is the most obvious benefit to working remote. No more sitting in rush hour traffic twice a day. Even if you take public transit and are able to play on your laptop for most of the trip, commuting is a major time sink. Most people will instantly gain back at least an hour of time by foregoing the daily drive to and from the office. What could you do with an extra hour each day?

And beyond time, there are ample corollary benefits. You (or your company) are no longer paying for as much fuel or fare. You're greatly reducing your risk of being injured Continue reading

Writing a Custom IPAM Application

Four years ago, I lamented the lackluster selection of IPAM applications available for service providers. Unfortunately, it seems not much has changed lately. I was back to exploring IPAM offerings again recently, this time with the needs of a cloud hosting provider in mind. I demoed a few tools, but none of them seemed to fit the bill (or they did, but were laughably overpriced).

So, I decided to write my own. In my rantings a few years back, I had considered this option:

Could I create a custom IPAM solution with everything we need? Sure! The problem is that I'm a network engineer, not a programmer (a natural division of labor which, it seems, is mostly to blame for the lack of robust IPAM solutions available). Even if I had the time to undertake such a project, I have little interest in providing long-term maintenance of it.

My opinion has not changed, but I've come to realize that if I want a tool that fits my requirements, I will need to build it. And after surprisingly little time, I'm happy to report that I have now have a kick-ass IPAM tool that does exactly what I want it to.

Continue reading

What the CCIE does not Prove

I came across an article today about a 19-year-old who earned his CCIE. It reminded me of a Reddit post from a few weeks ago. Someone asked why, when evaluating a CCIE, hiring managers still demand a number of years of practical experience in the field.

I'm in a situation where I'm a CCNP with 3 years of experience. I want to get my CCIE but I keep being told left and right I don't have enough experience and I'll never get a CCIE job without 7 years of experience. Am I supposed to just laze around and wait until I get more experience? It just doesn't make sense.

This is a fairly common misunderstanding among people new to our field, and is largely the result of vendor marketing. People want so badly to believe that a certification proves their worth as an individual, when in reality its value is much more narrowly defined.

Continue reading · 2 comments

Traceroute and Not-so-Equal ECMP

I came across an odd little issue recently involving equal-cost multipath (ECMP) routing and traceroute. Traceroutes from within our network to destinations out on the Internet were following two different paths, with one path being one hop longer than the other. This resulted in mangled traceroute output, impeding our ability to troubleshoot.

The relevant network topology comprises a mesh of two edge routers and two core switches. Each edge router has a number of transit circuits to different providers, and advertises a default route via OSPF to the two core switches below. The core switches each load-balance traffic across both default routes to either edge routers.

Because each edge router has different providers, some destinations are routed out via edge1 and others via edge2, which means sometimes a packet will be routed to edge2 via edge1, or vice versa.

Routers typically employ a hash function using layer three and four information from each packet to pseudo-randomly distribute traffic across equal links. Typically, all packets belonging to a flow (e.g. all packets with the same source and destination IP and port numbers) follow the same path.

However, in this case traceroute packets were being split across two path of unequal Continue reading

Networking FAQ 4: Fundamentals

Continue reading · 1 comment

Networking FAQ 3: Names and Addresses

Continue reading · No comments

Networking FAQ 2: Certifications

Continue reading · No comments

Networking FAQ 1: Breaking into the Field

This post is the first in a series I plan to publish over the next few months regarding frequently asked questions in networking. Each post will cover a different subject, roughly following the outline I shared last summer. I hope people find this useful!

Continue reading · 2 comments

MAC Address Aggregation and Translation as an Alternative to L2 Overlays

Not so long ago, if you wanted to build a data center network, it was perfectly feasible to place your layer three edge on the top-of-rack switches and address each rack as its own subnet. You could leverage ECMP for simple load-sharing across uplinks to the aggregation layer. This made for an extremely efficient, easily managed data center network.

Then, server virtualization took off. Which was great, except now we had this requirement that a virtual machine might need to move from one rack to another. With our L3 edge resting at the top of the rack, this meant we'd need to re-address each VM as it was moved (which is apparently a big problem on the application side). So, now we have two options: We can either retract the L3 edge up a layer and have a giant L2 network spanning dozens of racks, or we could build a layer two overlay on top of our existing layer three infrastructure.

Most people opt for some form of the L2 overlay approach, because no one wants to maintain a flat L2 network with dozens or hundreds of thousands of end hosts, right? But why is that?

Continue reading · 1 Continue reading

PDU-12C

"After all, what's the best part of Halloween?" Jimmy pleaded over the phone. He was trying yet again to convince Tom to skip work for the night and head over to the party he was throwing. Tom and Jimmy were good friends, but he already knew how the conversation was going to end.

"I dunno, the candy?" Tom played dumb.

"No, the eye candy! I'm telling you bro, you don't want to miss it. Rachel will be there." Jimmy sang the last bit tauntingly.

"I told you," Tom countered. "I've got work." It was around 6pm now, and he was just pulling into the parking lot outside the data center where he planned to spend the night recabling several racks of equipment. The scariest part of his Halloween would be picking through years' worth of undressed patch cabling.

"I don't get why you have to do that shit at night anyway. Why can't you do it during the day when you're stuck at work anyway?" Jimmy prodded.

Tom parked across from the building's entrance and turned off his car. Other than a couple vehicle belonging to the operations staff, the parking lot was deserted. He Continue reading