Aldrin Isaac

Author Archives: Aldrin Isaac

Blending into software infrastructure


Electronic networks existed long before electronic compute and storage.   Early on, the network was simple wires and switch boards, and the endpoints were humans.  Telegraphs transcoded taps into on/off voltage on the wire and back to audible clicks.   Telephones transcoded voice into amplitude modulated voltage and back to voice. 

Since then, the network has existed as a unique entity apart from the things it connected.  Until now.  

Less than two decades ago most applications were built in vertical silos.  Each application got its own servers, storage, database and so on.  The only thing applications shared was the network.  The network was the closest thing to a shared resource pool — the original “cloud”.  With increasing digital transformation, other services were also pooled, such as storage and database.  However each application interfaced with these pooled resources and with other applications directly.  Applications had little in common other than Continue reading

Strong identity is strong security


I think of classical firewall-based security like a community with a common manned gate, but where the homes in the community don’t have locks on their doors.   Strong locks on strong doors is better if you ask me.

Traditional firewall security matches on patterns in the packet header to determine what action to take on the matching application flows.  I'd equate this to the security guard allowing people into the gated community based by how they look.  If a person looks like someone who belongs to the community, then let them in.   In the same way, if a bad person crafts packet headers to match permit rules in the firewall, then it is allowed through.

One might say that’s where app-based firewalls come in.  They go deeper than just the packet header to determine what application is being transported.  How do these firewalls know what application a transport session belongs to?  Well, they do Continue reading

Death by a thousand scripts


Early on in my automation journey, I learned some basic lessons from observing what happens when multiple scripting newbies independently unleash their own notions of automation on the network.  I was one of them. 

Our earliest automations were largely hacky shell scripts that spat out snippets of config which would then be shoved into routers by other scripts.  We’re talking mid 1990s.   It started out fun and exciting, but with more vendors, hardware, use cases, issues, and so on, things went sideways.  

We ended up with a large pile of scripts with different owners, often doing similar things to the network in different ways.  Each script tailored for some narrow task on a specific vendor hardware in a specific role.  As folks who wrote the scripts came and went, new scripts of different kinds showed up, while other scripts became orphaned.  Other scripts ran in the shadows, only known Continue reading

Network automation as network architecture

I spent a very large part of my professional life as a network engineer working on automation.  A journey that started back in 1996 when I and a few colleagues engineered Bloomberg’s first global IP WAN.  That WAN evolved into the most recognized (and agile) WAN in the financial services industry.  And that automation which started small, over the years evolved into a very lean and flexible model-based provisioning library, with the various programs (provisioning, health-checking, discovery, etc) that were built on top.  The automation library drove a high function multi-service network with less than 15K of OO code, and with support for 6+ different NOS and 100+ different unique packet forwarding FRUs.  It was quite unique in that I have yet to see some of its core concepts repeated elsewhere.  

Over the thousands of hours I spent evolving and fine tuning that network automation engine, I’ve learned quite a lot along the way.  I hope to write about some of my high level learnings over the next year.  In this blog entry, I want to share my perspective on the foundational basis of any proper network automation — this is network Continue reading

Intelligent Network, Intelligent Operator

Maybe I’m old school, but I’m just not into black box networking.  Especially if critical services are dependent on my network infrastructure.  I wore those shoes for 19 years, so this feeling has burned into my psyche through many real world experiences.  I don’t think I’m alone in this thinking.  

When bad things happen to network users, all eyes are on the physical network team, even when it’s not a physical network problem.  The physical network is guilty until it can be proven otherwise.  So it’s only fair that physical network operators are skeptical of technology whose inner workings are unknowable.  Waiting on your network vendor’s technical support team isn’t an option when the CIO is breathing down your neck.  Especially if a mitigation exists that can be acted on immediately.  

That said, there is indeed a limit to the human ability.  It is increasingly lossy as the number of data points grow.  Moreover, the levels of ability are inconsistent across individuals.  Box level operations leaves it up to the operator to conceptualize the physical network state as a whole in his/her head and this generally results in Continue reading

I’m back, again.


It’s been over 3 years since I last shared my thoughts here.  It’s been that long since I left an amazing 19 year journey at Bloomberg, at the helm of the team that developed the financial industry’s most prominent IP network.  A network that I took great pride in and gave so much of my personal life for.  I am grateful to have had the opportunity to build what I did there and learn many things along the way.

Three years ago I decided to go from building mission-critical global networks to building network technologies with the team at Juniper Networks.  Two very different worlds.  It wasn’t easy, but I have evolved.  For the record, my heart is still that of a network operator.  20+ years at the front lines doesn’t wash off Continue reading

Better than best effort — Rise of the IXP

In my last post I talked about how the incentive models on the Internet keep ISPs from managing peering and backbone capacity in a way that supports reliable communication in the face of ever growing volume of rich media content.  If you haven't done so, please read that post before you read this one.

It's clear that using an ISP for business communication comes with the perils associated to the "noisy neighbor" ebb and flow of consumer related high volume data movement.  Riding pipes that are intentionally run hot to keep costs down is a business model that works for ISP, but not for business users of the Internet.  Even with business Internet service, customers may get a better grade of service within a portion of an ISP's networks, but not when their data needs to traverse another ISP which they are not a customer of.  There is no consistent experience, for anyone.

However there is an evolving solution to avoid getting caught in the never ending battle between ISP and large consumer content.  As the title of this blog gives away, the solution is called an Internet eXchange Point (IXP).

IXPs are where the different networks that make up Continue reading

Better than best effort — reliability and the Internet.

Metcalfe’s law states that the value of a telecommunications network is proportional to the square of the number of connected users of the system.

Networks prior to the Internet were largely closed systems, and the cost of communicating was extraordinarily high.   In those days, the free exchange of ideas at all levels was held back by cost.  On the Internet, for a cost proportional to a desired amount of access bandwidth, one can communicate with a billion others.  This has propelled human achievement forward over the last 20 years.  By way of Metcalfe’s law, the Internet’s value is immeasurably larger than any private network ever will be.

So why do large private service delivery networks still exist?

The answer lies primarily in one word: reliability.  What Metcalfe’s law doesn’t cover is the reliability of communication of connected users, and the implications of a lack of reliability on the value of services delivered.  Although Internet reliability is improving, much like the highway system, it still faces certain challenges inherent with open and unbiased systems.

On a well run private network, bandwidth and communications are regulated to deliver an optimal experience, and network issues are addressed more rapidly as all components Continue reading

EVPN. The Essential Parts.

In a blog post back in October 2013 I said I would write about the essential parts of EVPN that make it a powerful foundation for data center network virtualization.  Well just when you thought I'd fallen off the map, I'm back.  :)

After several years as an Internet draft, EVPN has finally emerged as RFC7432.  To celebrate this I created a presentation, EVPN - The Essential Parts, that I hope will be helpful to people who are interested.

Use cases are intentionally left out of this presentation as I prefer the reader to creatively consider whether their own use cases can be supported with the basic features that I describe.

Let me know your thoughts and I will try to expand/improve this presentation or create other presentations to address them.

EVPN - The Essential Parts

Evolving the data center network core

As complex functions that were historically shoehorned into the network core move out to the edge where they belong, data center core network operators can now focus on improving the experience for the different types of applications that share the network.  Furthermore, with fewer vertically scaled systems giving way to many horizontally scaled systems, the economics of data center bandwidth and connectivity needs to change.

I’ve jotted down some thoughts for improving the data center core network along the lines of adding bandwidth, managing congestion and keeping costs down.

Solve bandwidth problems with more bandwidth


Adding bandwidth had been a challenge for the last several years owing to the Ethernet industry not being able to maintain the historical Ethernet uplink:downlink speedup of 10:1, and at the same time not bringing down the cost of Ethernet optics fast enough.  Big web companies started to solve the uplink bandwidth speed problem in the same way they had solved the application scaling problem -- scale uplinks horizontally.  In their approach, the role of traditional bridging is limited to the edge switch (if used at all), and load-balancing to the edge is done using simple IP ECMP across a “fabric” topology.  The number Continue reading

The real Slim Shady

Historically when an application team needed compute and storage resources they would kick off a workflow that pulled in several teams to design, procure and deploy the required infrastructure (compute, storage & network).  The whole process generally took a few months from request to delivery of the infrastructure.  

The reason for this onerous approach was really that application groups generally dictated their choice of compute technology.  Since most applications scaled vertically, the systems and storage scaled likewise.  When the application needed more horsepower, it was addressed with bigger more powerful computers and faster storage technology.  The hardware for the request was then staged followed by a less-than-optimal migration to the new hardware.  

The subtlety that gets lost regarding server virtualization is that a virtualization cluster is based on [near] identical hardware.  The first machines that were virtualized where the ones who’s computer and storage requirements could be met by the hardware that the cluster was based on.  These tended to be the applications that were not vertically scaled.  The business-critical vertically scaled applications continued to demand special treatment, driving the overall infrastructure deployment model used by the enterprise.

The data center Continue reading

The bumpy road to E-VPN

In 2004 we were in the planning phase of building a new data center to replace one we had outgrown.   The challenge was to build a network that continued to cater to a diverse range of data center applications and yet deliver significantly improved value.

Each operational domain tends to have one or more optimization problem whose solution is less than optimal for another domain.  In an environment where compute and storage equipment come in varying shapes and capabilities and with varying power and cooling demands, the data center space optimization problem does not line up with the power distribution and cooling problem, the switch and storage utilization problem, or the need to minimize shared risk for an application, to name a few.

The reality of the time was that the application, backed by it's business counterparts, generally had the last word -- good or bad.  If an application group felt they needed a server that was as large as a commercial refrigerator and emitted enough heat to keep a small town warm, that's what they got, if they could produce the dollars for it.  Application software and hardware infrastructure as a whole was the bastard of a hundred independent self-proclaimed Continue reading

Air as a service

Have you ever wondered about air?  We all share the same air.  We know it's vitally important to us.  If we safeguard it we all benefit and if we pollute it we all suffer.  But we don't want to have to think about it every time we take a breath.  That's the beauty of air.  Elegantly simple and always there for you.

Imagine air as a service (AaaS), one where you need to specify the volume of air, the quality of the air, etc before you could have some to breathe.  As much as some folk might be delighted in the possibility to capitalize on that, it would not be the right consumption model for such a fundamental resource.  If we had to spend time and resources worrying about the air we breathe we'd have less time and resources to do other things like make dinner.



Why does air as it is work so well for us?  I think it's for these reasons, (1) there is enough of it to go around and (2) reasonable people (the majority) take measures to ensure that the air supply is not jeopardized.

Network bandwidth and transport should be more like how we want Continue reading

Regarding scale-out network virtualization in the enterprise data center

There's been quite a lot of discussion regarding the benefits of scale-out network virtualization.   In this blog I present some additional thoughts to ponder regarding the value of network virtualization in the enterprise DC.  As with any technology options, the question that enterprise network operators need to ask themselves regarding scale-out network virtualization is whether it is the right solution to the problems they need to address.

To know whether scale-out network virtualization in the enterprise DC is the answer to the problem, we need to understand the problem in a holistic sense.  Let's set aside our desire to recreate the networks of the past (vlans, subnets, etc, etc) in a new virtual space, and with an open mind ask ourselves some basic questions.

Clearly at a high level, enterprises wish to reduce costs and increase business agility.  To reduce costs it's pretty obvious that enterprises need to maximize asset utilization.   But what specific changes should enterprise IT bring about to maximize asset utilization and bring about safe business agility?  This question ought to be answered in the context of the full sum of changes to the IT model necessary to gain all the benefits of the scale-out Continue reading

Angry SDN hipsters.

Some folks seem to get a little too hung up on one philosophy or another -- too blind to see good in any other form except the notions that have evolved in their mind.  I'm hoping I'm not one of them.  I do have opinions, but which I believe are rational.

The counter culture of networking waves the SDN banner.  That acronym seems to belong to them.  They don't know what it stands for yet, but one thing they seem to be sure of is that nothing good can come by allowing networking innovations to evolve or even to exist in their birthplace.

The way I see evolving the network fabric is through improving on the best of the past.  Every profession I know from medicine, finance, law, mathematics, physics, you name it -- all of them are building their tomorrow on a mountain of past knowledge and experience.  So I'm sure my feeling the same about the network doesn't make me outdated, just maybe not a fashionable SDN hipster.




Some angry SDN hipsters say that the core network needs to be dumbed down.  They must have had a "bad childhood," technically speaking.  One too many Cisco 6500's stuffed with Continue reading

Straight talk about the enterprise data center network.


Building a mission critical system requires the careful selection of constituent technologies and solutions based on their ability to support the goals of the overall system.   We do this regardless of whether we subscribe to one approach or another of building such a system.

It is well known that the network technologies commonly used today to build a data center network have not adequately met the needs of applications and operators.  However, what is being drowned out in the media storm around "SDN" is that many of the current day challenges in the data center network can be addressed within an existing and familiar toolkit.  The vision of SDN should be beyond where today’s data center networks could have been yesterday.

In this "treatise" I highlight what I believe has been lacking in todays data center core network toolkit as well as address certain misconceptions.  I'll begin by listing key aspects of a robust network, followed by perspectives on each.  

A robust network should be evolved in all of the following aspects:
  1. Modularity - freedom to choose component solutions based on factors, such as bandwidth, latency, ports, cost, serviceability, etc.,  This generally requires the network to Continue reading