Rolling out Change
We all know that “Change is Hard.” But often we, as engineers, focus on the technical aspects of that change. How do I minimise customer impact while upgrading those routers? How can I migrate customer data safely to the new system? But we can forget about the wider implications of what we’re doing. If we do, we may struggle to get our changes implemented, or see poor take-up of new systems.
Why Can’t I Make That Change?
I was talking to an engineer who had planned a huge configuration management implementation. Everything had been manually configured in the past, but this was hitting scale issues. So he had worked for months on a fully automated process. It was going to be amazing. It would configure everything, across all systems and applications. Standards enforced, apps deployments done in a repeatable way, etc. It was going to be a thing of beauty. No-one would ever need to login to a server again. Total automation.
It was all tested, and was just waiting for approval to put it into production. But for some reason, no-one was willing to give the go-ahead to roll it out. Weeks were dragging by, and things were going Continue reading
Relevance of SDN in Cloud Networking
SDN (Software Defined Networking) is finally becoming clearer. It is not “Still Don’t Know” nor is it a specific overlay controller. Simply put, it is an open and programmable way to build networks for customers looking at utilizing hybrid combinations of public and private cloud access.
We are witnessing a shift from multi-tier oversubscribed legacy enterprise networks to two-tier leaf-spine or single-tier Spline™ cloud networks with east- west traffic patterns scaling across thousands of servers. Arista was the first to introduce this new architectural “leaf-spine” approach for cloud-based networks and five years later others are still attempting to mimic. Lets review some practical examples.
Facebook: Take an important and familiar social networking application, Facebook. Their public information shows that they deploy a memcache architecture, which allowed them to reduce the user access time to half a millisecond by using fewer network tiers, resulting in lower application latency. As we log into Facebook, the single login request triggers thousands of look-ups on databases and memcache servers. Legacy enterprise multi-tiered networks would result in delayed look-ups and would negatively impact the user experience and interest in a significant way.
Amazon: Shopping couldn’t be easier than online on your favorite site. Have you Continue reading
Scaling Overlay Networks: Distributed Data Plane
“Thou Shalt Have No Chokepoints” is one of those simple scalability rules that are pretty hard to implement in real-life products. In the Distributed Data Plane part of Scaling Overlay Networks webinar I listed data plane components that can be easily distributed (layer-2 and layer-3 switching), some that are harder to implement but still doable (firewalling) and a few that are close to mission-impossible (NAT and load balancing).
BGP Communities
BGP Communities has to be one of my favorite features added to the BGP protocol. As you should know by now, BGP passes several attributes between peers that help influence the BGP best path selection algorithm. One of these is the BGP Community attribute. Think of this as another tag placed on the route advertisement that can give us additional information.
This tag is formatted as 32bit value that is typically displayed in two 16 bit parts. The most typical, and most widely accepted best practices treat these as your Autonomous System Number (ASN), followed by a 16 bit attribute. For example, if your ASN was 65248 and you wanted to tag this route with the number 666, you would set something like 65248:666.
As for the second half of our tag, this number is arbitrary. BGP Communities are a transitive attribute that is completely optional. This means that BGP treats these tags as purely information, and that it is up to the network engineer to decide what these values mean, and what to do with them.
RFC 1997 defines some well known communities that have global significance. These values and their operations should be respected by any community Continue reading
My First Week at Cumulus Networks
From the first time I spoke with someone at Cumulus Networks, I realized I’d come across something spectacular. My interviews started with a “screening” call from Nat Morris, but it didn’t start with any of those awkward “tell me about yourself” ice-breaker questions. Nat immediately began the conversation with a detailed description of the Cumulus Workbench, his plans for where to take it, and what I thought about them. For someone like me, being able to talk candidly about a concrete project made it easy for me to see myself in the position, and I felt like I was being treated as if I were already a part of the company. This conversation left such a powerful impression that it literally carried me through the entire interview process.
I started on the day before the RDU team moved from the Apex office to the new office in Cary. All of the furniture was gone, everyone was huddled at folding card tables, and I ended up sitting on the floor in another room since we didn’t have enough chairs even after borrowing some from the restaurant next door! The team assured me that it wasn’t a bait-and-switch by showing me Continue reading
The Packet Flow Duality
Quantum physics is a funny thing. It seeks to solve all the problems in the physical world by breaking everything down into the most basic unit possible. That works for a lot of the observable universe. But when it comes to light, quantum physics has issues. Thanks to experiments and observations, most scientists understand that light isn’t just a wave and it’s not just a collection of particles either. It’s both. This concept is fundamental to understanding how light behaves. But can it also explain how data behaves?
Moving Things Around
We tend to think about data as a series of discrete data units being pushed along a path. While these units might be frames, packets, or datagrams depending on the layer of the OSI model that you are operating at, the result is still the same. A single unit is evaluated for transmission. A brilliant post from Greg Ferro (@EtherealMind) sums up the forwarding thusly:
- Frames being forwarded by MAC address lookup occur at layer 2 (switching)
- Packets being forwarded by IP address lookup occur at layer 3 (routing)
- Data being forwarded at higher levels is a stream of packets (flow forwarding)
It’s simple when you think Continue reading