Implementing ‘Undo’ Functionality in Network Automation

Kurt Wauters sent me an interesting challenge: how do we do rollbacks based on customer requests? Here’s a typical scenario:

You might have deployed a change that works perfectly fine from a network perspective but broke a customer application (for example, due to undocumented usage), so you must be able to return to the previous state even if everything works. Everybody says you need to “roll forward” (improve your change so it works), but you don’t always have that luxury and might need to take a step back. So, change tracking is essential.

He’s right: the undo functionality we take for granted in consumer software (for example, Microsoft Word) has totally spoiled us.

Implementing ‘Undo’ Functionality in Network Automation

Kurt Wauters sent me an interesting challenge: how do we do rollbacks based on customer requests? Here’s a typical scenario:

You might have deployed a change that works perfectly fine from a network perspective but broke a customer application (for example, due to undocumented usage), so you must be able to return to the previous state even if everything works. Everybody says you need to “roll forward” (improve your change so it works), but you don’t always have that luxury and might need to take a step back. So, change tracking is essential.

He’s right: the undo functionality we take for granted in consumer software (for example, Microsoft Word) has totally spoiled us.

Applying BGP Policy Templates

I got this question after publishing the BGP Session Templates lab exercise:

Would you apply BGP route maps with a peer/policy template or directly to a BGP neighbor? Of course, it depends; however, I believe in using a template for neighbors with the same general parameters and being more specific per neighbor when it comes to route manipulation.

As my reader already pointed out, the correct answer is It Depends, now let’s dig into the details ;)

Applying BGP Policy Templates

I got this question after publishing the BGP Session Templates lab exercise:

Would you apply BGP route maps with a peer/policy template or directly to a BGP neighbor? Of course, it depends; however, I believe in using a template for neighbors with the same general parameters and being more specific per neighbor when it comes to route manipulation.

As my reader already pointed out, the correct answer is It Depends, now let’s dig into the details ;)

Network Layer: Interface or Node Addresses

The fun question about network layer addresses is: are we addressing nodes or individual node interfaces? On the data link layer, we never had this issue because it was obvious that a data link layer endpoint is an interface, so each interface should have a unique data link layer address.

Interestingly, that’s not the case on transparent bridges. Even though they have multiple interfaces, the whole bridge has a single MAC address, so one could claim we’re addressing nodes connected to a single data link layer. The IEEE standard is unambiguous: in every relevant diagram, the MAC address sits on top of multiple interfaces because the MAC address belongs to the control plane.

Network Layer: Interface or Node Addresses

The fun question about network layer addresses is: are we addressing nodes or individual node interfaces? On the data link layer, we never had this issue because it was obvious that a data link layer endpoint is an interface, so each interface should have a unique data link layer address.

Interestingly, that’s not the case on transparent bridges. Even though they have multiple interfaces, the whole bridge has a single MAC address, so one could claim we’re addressing nodes connected to a single data link layer. The IEEE standard is unambiguous: in every relevant diagram, the MAC address sits on top of multiple interfaces because the MAC address belongs to the control plane.

Public Videos: 400 Gigabit Ethernet

The 400 Gigabit Ethernet videos with Mark Nowell and Lukas Krattiger are now public. Enjoy!

BGP Labs: Session Templates

Configuring an IBGP session on a route reflector takes a half-dozen parameters, starting with the remote BGP AS number (equal to the local one), remote IP address, and the source IP address or interface. You might have to specify the propagation of BGP communities and an MD5 password, and you will definitely have to specify that the BGP neighbor is a route reflector client.

Wouldn’t it be nice if you could group those parameters into a template and apply the template to a neighbor? Most BGP implementations have something along those lines. That feature could be called a session template or a peer group, and you can practice it in the next BGP lab exercise.

BGP Labs: Session Templates

Configuring an IBGP session on a route reflector takes a half-dozen parameters, starting with the remote BGP AS number (equal to the local one), remote IP address, and the source IP address or interface. You might have to specify the propagation of BGP communities and an MD5 password, and you will definitely have to specify that the BGP neighbor is a route reflector client.

Wouldn’t it be nice if you could group those parameters into a template and apply the template to a neighbor? Most BGP implementations have something along those lines. That feature could be called a session template or a peer group, and you can practice it in the next BGP lab exercise.

Can We Skip the Network Layer?

I mentioned that you don’t need node addresses when dealing with only two entities. Now and then, someone tries to extend this concept and suggests that the network layer addressing isn’t needed if the solution is local. For instance, if we have a solution that is supposed to run only on a single Ethernet segment, we don’t need network layer addressing because we already have data link layer addresses required for Ethernet to work (see also: ATAoE).

Too often in the past, an overly ingenious engineer or programmer got the idea to simplify everyone’s life and use the data link layer addresses as the ultimate addresses of individual nodes. They would then put the transport layer on top of that to get reliable packet transport. Finally, put whatever application on top of the transport layer. Problem solved.

Can We Skip the Network Layer?

I mentioned that you don’t need node addresses when dealing with only two entities. Now and then, someone tries to extend this concept and suggests that the network layer addressing isn’t needed if the solution is local. For instance, if we have a solution that is supposed to run only on a single Ethernet segment, we don’t need network layer addressing because we already have data link layer addresses required for Ethernet to work (see also: ATAoE).

Too often in the past, an overly ingenious engineer or programmer got the idea to simplify everyone’s life and use the data link layer addresses as the ultimate addresses of individual nodes. They would then put the transport layer on top of that to get reliable packet transport. Finally, put whatever application on top of the transport layer. Problem solved.

Removing FRRouting Configuration Is Not Idempotent

One of the few beauties of most “industry standard CLI” implementations¹ is that they’re idempotent: nothing changes (apart from ACLs) if you configure the same stuff a dozen times. Most of these implementations allow you to deconfigure the same stuff multiple times; FRRouting is one of the unfortunate exceptions.

What Am I Talking About?

Imagine you have a bunch of IP prefixes you want to advertise with BGP. You could use network statements within the router bgp configuration to get that done:

Removing FRRouting Configuration Is Not Idempotent

One of the few beauties of most “industry standard CLI” implementations¹ is that they’re idempotent: nothing changes (apart from ACLs) if you configure the same stuff a dozen times. Most of these implementations allow you to deconfigure the same stuff multiple times; FRRouting is one of the unfortunate exceptions.

What Am I Talking About?

Imagine you have a bunch of IP prefixes you want to advertise with BGP. You could use network statements within the router bgp configuration to get that done:

BGP Labs: Use Multiple AS Numbers on the Same Router

Can you use BGP to connect to the global Internet without having a public BGP AS number? Of course, assuming your Internet Service Provider is willing to run BGP with a network using a private AS number. But what happens if you want to connect to two ISPs? It’s ridiculous to expect you’ll be able to persuade them to use the same private AS number.

That’s one of the many use cases for the local-as functionality available in most BGP implementations. You can practice it in the Use Multiple AS Numbers on the Same Router lab exercise.

BGP Labs: Use Multiple AS Numbers on the Same Router

Can you use BGP to connect to the global Internet without having a public BGP AS number? Of course, assuming your Internet Service Provider is willing to run BGP with a network using a private AS number. But what happens if you want to connect to two ISPs? It’s ridiculous to expect you’ll be able to persuade them to use the same private AS number.

That’s one of the many use cases for the local-as functionality available in most BGP implementations. You can practice it in the Use Multiple AS Numbers on the Same Router lab exercise.

Precedence of Ansible Extra Variables

I stay as far away from Ansible as possible these days and use it only as a workflow engine to generate device configurations from Jinja2 templates and push them to lab devices. Still, I manage to trigger unexpected behavior even in these simple scenarios.

Ansible has a complex system of variable (fact) precedence, which mostly makes sense considering the dozen places where a variable value might be specified (or overwritten). Ansible documentation also clearly states that the extra variables (specified on the command line with the -e keyword) have the highest precedence.

Now consider these simple playbooks. In the first one, we’ll set a fact (variable) and then print it out:

Precedence of Ansible Extra Variables

I stay as far away from Ansible as possible these days and use it only as a workflow engine to generate device configurations from Jinja2 templates and push them to lab devices. Still, I manage to trigger unexpected behavior even in these simple scenarios.

Ansible has a complex system of variable (fact) precedence, which mostly makes sense considering the dozen places where a variable value might be specified (or overwritten). Ansible documentation also clearly states that the extra variables (specified on the command line with the -e keyword) have the highest precedence.

Now consider these simple playbooks. In the first one, we’ll set a fact (variable) and then print it out:

BGP Labs: Override Neighbor AS Number in AS Path

When I described the need to turn off the BGP AS-path loop prevention logic in scenarios where a Service Provider expects a customer to reuse the same AS number across multiple sites, someone quipped, “but that should be fixed by the Service Provider, not offloaded to the customer.”

Not surprisingly, there’s a nerd knob for that (AS override), and you can practice it in the next BGP lab exercise: Fix AS-Path in Environments Reusing BGP AS Numbers.

BGP Labs: Override Neighbor AS Number in AS Path

When I described the need to turn off the BGP AS-path loop prevention logic in scenarios where a Service Provider expects a customer to reuse the same AS number across multiple sites, someone quipped, “but that should be fixed by the Service Provider, not offloaded to the customer.”

Not surprisingly, there’s a nerd knob for that (AS override), and you can practice it in the next BGP lab exercise: Fix AS-Path in Environments Reusing BGP AS Numbers.

Public Videos: Kubernetes Container Network Interface (CNI)

All the Container Network Interface videos from the Kubernetes Networking Deep Dive webinar with Stuart Charlton are now public. Enjoy!