The EIGRP SIA Incident: Positive Feedback Failure in the Wild
Reading a paper to build a research post from (yes, I’ll write about the paper in question in a later post!) jogged my memory about an old case that perfectly illustrated the concept of a positive feedback loop leading to a failure. We describe positive feedback loops in Computer Networking Problems and Solutions, and in Navigating Network Complexity, but clear cut examples are hard to find in the wild. Feedback loops almost always contribute to, rather than independently cause, failures.
Many years ago, in a network far away, I was called into a case because EIGRP was failing to converge. The immediate cause was neighbor flaps, in turn caused by Stuck-In-Active (SIA) events. To resolve the situation, someone in the past had set the SIA timers really high, as in around 30 minutes or so. This is a really bad idea. The SIA timer, in EIGRP, is essentially the amount of time you are willing to allow your network to go unconverged in some specific corner cases before the protocol “does something about it.” An SIA event always represents a situation where “someone didn’t answer my query, which means I cannot stay within the state machine, so I Continue reading
Reaction: The NRE as the new architect
Over at the Packet Pushers, Anthony Miloslavsky suggests that network architects have outlived their usefulness, so it is time to think of a new role. He describes a role called the “NRE” to replace the architect; the NRE would—
…spend no less than 50% of their time focusing on automation, while spending the other 50% deeply embedded in the operations/engineering/architecture realms of networking. They participate in an on-call rotation to stay in touch with the ops side of the house, with a focus on “treating operations as if it’s a software problem” in response. NREs would provide a expert big picture view of BOTH the development/automation and network operation/design sides of the house.
The author goes on to argue that we need someone who will do operations, engineering, architecture, and development because “pure architecture” folks tend to “lose touch” with the operations side of things. It is too easy to “throw a solution over the cubicle wall” without considering the implementation and operational problems. But, as a friend used to ask of everything when I was still in electronics, will it work? I suspect the answer is no for several reasons.
First, there is no such person as described, and Continue reading
Deconfusing the Static Route
Configuring a static route is just like installing an entry directly in the routing table (or the RIB).
I have been told this many times in my work as a network engineer by operations people, coders, designers, and many other folks. The problem is that it is, in some routing table implementations, too true. To understand, it is best to take a short tour through how a typical RIB interacts with a routing protocol. Assume BGP, or IS-IS, learns about a new route that needs to be installed in the RIB:
- The RIB into which the route needs to be installed is somehow determined. This might be through some sort of special tagging, or perhaps each routing process has a separate RIB into which it is installing routes, etc.. In any case, the routing process must determine which RIB the route should be installed in.
- Look the next hop up in the RIB, to determine if it is reachable. A route cannot be installed if there is no next hop through which to forward the traffic towards the described destination.
- Call the RIB interface to install the route.
The last step results in one of two possible reactions. The first Continue reading