- L2 routing protocol (MAC in MAC Routing).

- Alternative to running STP, built-in loop prevention and mitigation.

  - Even with vPC, you still have STP.

  - vPC is a physical triangle and logical P2P link.

  - can have only 2 vPC peers.

  - thus distribution blocks can be only 2 upstream switches.

- FP builds arbitrary topologies.

  - full mesh

  - partial mesh

  - triangle

  - square

- Single CP for unknown ucast, ucast, bcast and mcast traffic.

- Enhances mobility and virtualization in FP network.

 - MAC mobility : physically move L2 node, but retain the same MAC address and VLAN association for the VM.

- FP retains config across an ISSU.

- FP is not TRILL.

- FP is not Ethernet (hence HW support is limited), it is not Ethernet in Ethernet, is Ethernet in FP tunneling.

 - F line card.

 - Enhanced L2 license (on every system that enables FP).

 - NX7K and NX55K.

 - Beginning NX-OS 6.1, FEX with vPC+ on F2 cards is supported.

CE : Classical Ethernet

- regular ethernet with regular flooding, STP

Leaf Continue reading

I was trying to configure a FEX in Rack 27, so I called it FEX 127 and was configuring Port-Channel 127 so I could vPC to the FEX. However, as soon as I entered the following, I got an error message:

NX5K-1(config-if)# channel-group 127

command failed: internally used, configuration not allowed

When you have a Nexus 5K with an L3 daughter card, the switch internally allocates Po127 and Po128 to bind the L3 interfaces to the ASICs. So if you have a 5K running L2 only, and have already allocated Po127 and Po128, and then in the future add an L3 daughter card, the system will use other free Port-Channel IDs. But if you have an L3 daughter card, you cannot use Po127 and Po128 when setting up a new Port-Channel.

In my previous blog - Data Center Design Constraints, I mentioned some of the design caveats involving M and F series line cards in the same VDC or 7K chassis. In this blog, I'll talk about design considerations with classical ethernet on a 7K. For a deep dive, refer Design Considerations for Classical Ethernet Integration if the Cisco Nexus 7000 M1 and F1 Modules.

What prompted me to write this blog was a design whiteboard session with a customer to order the right line cards for the 7K switch. We know the F1 series line cards are strictly for Layer 2 functionality only. They do not perform Layer 3 routing functions, they cannot. The M series line cards perform Layer 3 routing. Consider a Nexus 7009 switch populated with F1 and M1 series line cards. Say servers A and B connect to the 7K. Server A and B belong to VLANs A and B respectively. If Server A needs to talk to Server B, the switch requires inter VLAN routing to route between VLANs A and B. Without the M series, or a separate router on a stick device, this is not possible, right?

If this were a Catalyst Continue reading

I took notes while reading up on OTV documentation. If you don't want to read through the OTV IETF or related Cisco documentation, this can be a quick deep dive into what OTV is and how it functions. Hopefully I'll implement this in a real network sometime soon to share actual configs. 

OTV is a data center interconnect technology that routes mac based information by encapsulating mac addresses with IP addresses in an overlay transit network. As long as there is IP connectivity between two or more sites (DCs), OTV will function (ofcourse with the right hardware and configuration).

'MAC in IP' technique for supporting L2 VPNs over L2/L3 infrastructure. 

Routing and forwarding states are maintained at the network edge devices between sites, not within the site or core.

There is no need to extend STP across sites, and STP topologies can be local to each site.

As of March 2013, from here.

Cisco Nexus 7000

M1/M2 line cards

NX-OS 5.0(3) or later, 5.2(3) or 6.2(2) are Cisco recommendations

Transport Services License

Cisco ASR 1000

IOS XE 3.5 or later

Adv IP Srvcs / Adv Ent Srvcs license

In my previous blogs NX-OS vPC FEX config example and Configuring VSS Cisco 6500 and vPC, I've covered virtual Port-Channels in detail. For those new to this terminology, virtual port-channel or vPC makes a pair of Cisco Nexus 5K/7K switches (vPC peers) appear as a single logical switch to a downstream device. Arp tables, mac address tables and route tables can be synchronized across the vPC Peer-Link (link connecting the two peer switches). This provides great redundancy in typical Data Center networks. 

Every time I configure a Nexus environment at a Data Center, or a campus core, I have to explain to the end user that the configurations on the vPC peers are not synchronized. Say for example you configure Eth1/30 on Nexus-1, and forget to configure Eth1/30 on Nexus-2, and a device is dual homed to Eth1/30 on both Nexus switches, then when vPC failover occurs and vPC Primary/Secondary roles are switched over, the new primary switch will not have any configuration in store for Eth1/30, and this will cause a service disruption.

To avoid this, you have to be very meticulous and have matching configurations across both vPC peers. Or, you can use configuration synchronization available Continue reading

While assigning ports on a Cisco Nexus 5500 switch, it is a good practice to span out ports that belong to separate ASICs on board to uplink to upstream or downstream devices. For e.g. while connecting a 5500 to say a UCS FI, it is good practice to select 2 ports from one ASIC bay and 2 from another ASIC bay. The way you find out ASIC to port mapping is with the following CLI command : 'show hardware internal carmel asic <0-13>' where 0-13 are the ASIC numbers. 

Here, Ports 9-16 are mapped to ASIC 1. On Nexus 5500 switches, 8 ports are mapped to each ASIC.

I performed an eFSU (enhanced Fast Software Upgrade) of a pair of Cisco WS-6513-E switches in VSS. eFSU is the closest one can get to a true ISSU (in service software upgrade) of a VSS chassis. The entire process took 25 minutes. It is slower than a conventional FSU that involves changing the boot variables on the switch to point to the new software image file and then reloading the switches. This will cause a complete outage irrespective of whether devices are single or dual attached to the VSS. With the eFSU, we lost a single ping in the complete process from a workstation that was connected to an upstream 4500 switch that was dual homed to the VSS chassis. However, for devices that were single attached, there was a considerable outage (60-180 seconds).

I'll go over the process and hopefully this post should familiarize other network engineers who want to attempt an eFSU but haven't done it before. I've capture screenshots of the upgrade process, so we can look at some of the interesting logs on the switches.

Current IOS: 15.1(1)SY

New IOS: 15.1(1)SY1

Cisco's documentation is very precise and straight forward and can be found here.

Continue reading

Here's a no frills cheat sheet for a quick vPC and FEX configuration on a pair of Nexus 5000 switches. Refer visio for switch connectivity.

Note, my management network is a /29, just in case you want to configure config-sync between the two NX5K switches. You will need a cfs mcast-address which could be the same as the default gateway used in the management vrf.

int mgmt0
 ip add 1.1.1.2/29

vrf context management
 ip route 0.0.0.0/0 1.1.1.1

int eth1/1-2
 desc vPC Peer-Link members
 channel-group 1 mode active

int po1
 desc vPC Peer-Link
 vpc peer-link
 switchport mode trunk
 sw trunk allowed vlan list
 spanning-tree port type network

vpc domain 100
 peer-switch
 role priority 4096
 system priority 4096
 peer-keepalive destination 1.1.1.3 source 1.1.1.2 vrf management

fex 100

 desc "FEX 100"
 pinning max-links 1

fex 101
 desc "FEX 101"

 pinning max-links 1

int eth1/9-10
 channel-group 100

int eth1/11-12
 channel-group 101

int po100
 switchport mode fex-fabric
 fex associate 100
 vpc 100

int po101
 switchport mode fex-fabric
 fex associate 101
 vpc 101

int mgmt0
 ip add 1.1.1.3/29

vrf context management
 ip route 0. Continue reading

Don't you just love the option of configuring aliases on IOS? Call it being lazy or saving time, but I can't begin a network implementation without some awesome aliases!

Get creative!

On a related note, some MAC OS X aliases


alias flushdns='sudo killall -HUP mDNSResponder'
alias menubar='killall -Kill SystemUIServer'
alias p4='ping 4.2.2.2'

If you've upgraded your Cisco AnyConnect VPN Client to 3.1, you might run into the following warning pop up when you attempt to establish a VPN connectivity.

When this pops up, if you click on 'keep me safe', it will disconnect the VPN and you will not be prompted for a username/password.

Instead, if you click on 'change setting…', it will allow you to uncheck the box which blocks connections to untrusted server. By doing this, you trust the ip address or fqdn which is the VPN server (usually a public IP address or fqdn = vpn.company.com).

After the previous image, the above pop up shows up. 'Connect anyway' will complete the VPN connection.

Why do these security warnings show up in the first place?

Cisco made some changes to their Server Certificate Verification on 3.1. Following is a relevant excerpt from this Cisco document.

"–If a Subject Alternative Name extension is present with relevant attributes, name verification is performed solely against the Subject Alternative Name. Relevant attributes include DNS Name attributes for all certificates, and additionally include IP address attributes if the connection is being performed to an IP address."

How do you make these Continue reading

I spent some time recently building anyconnect VPN on an ASA 5525-X and integrating it with LDAP. It could be fairly quick and simple if you have the necessary information to build out an aaa-server for LDAP and match the attribute maps for LDAP service.

Integrating VPN with LDAP involves two phases:

1. Build an aaa-server for LDAP (multiple aaa-servers for load-balancing/redundancy) and then associate the LDAP server to the VPN tunnel-group.

2. Build LDAP attribute maps for determining what AD user gets associated to what VPN Group-Policy.

Before building your aaa-server, you need to grab some information from your AD server. This includes the following:

1. AD server IP address

2. AD server type (microsoft etc)

3. LDAP (389) or Secure LDAPS (636) port

Create aaa-server 'LDAP' that uses protocol ldap.

Assign an IP address / Server (dns resolved) name to that LDAP server.

Note: Here the ASA realizes the dc01 server has an IP address which is internal to the ASA, hence it puts it in the 'inside' interface.

Since we use LDAPS and require LDAP over SSL, and the AD server is a Microsoft Continue reading

Anyone who learns how to use Cisco IOS CLI knows how to erase a switch to delete the configuration. So this blog will not teach how to erase a switch. What I intend to do is share two screen shots that can help make network admins more cautious when deleting the switch configs, to adhere to security best practice policies. 

Consider a scenario when you need to decommission your switches. You don't want to keep the config on the switch, including the VLANs, so you do a write erase and reload the switch, thinking you've deleted everything. This is not always true. You must check the onboard nvram flash on the switch to look for configuration files. If you find a configuration file (a .text file), check that file content with a 'more flash:filename.text' command. 

When you do a 'wr er' on a switch you think you've deleted all configs. Look at the following screenshot.

Notice how the private-config.text.backup and config.text.backup files still exist. A 'wr er' will only delete the 'private-config.text' and 'config.text' files. If these files are renamed, the 'wr er' does not delete the renamed files. 

So, Continue reading

Designing a data center network with nexus switches involves virtual port channels (vPC), fabric extenders (FEX) and at times virtual device contexts (VDC). In this blog, we will take a look at some design constraints faced when dealing with the following hardware:

Cisco Nexus 7000 series switches

Cisco 2000 FEX

Cisco F2 series line cards

Cisco M1/2 series line cards

1. vPC/FEX

The biggest difference between a 7K switch and a 5K switch is the ability to configure enhanced vPC and dual home FEX to two parent 5K switches. The reason you need enhanced vPC is because you can dual home a end host (server/PC) to two separate FEX and dual home the FEX to two separate parent 5K switches. 

You CANNOT dual home FEX to two separate 7K switches YET. Cisco does not support this feature in NX-OS. So if you have a design where you want to connect a single attached device (server/PC) to a pair of 7K switches, you either connect the device to the primary vPC peer switch or connect a catalyst switch like a 3750 and vPC it to both 7K switches. But again, what are the chances of a 3750 failing over a Continue reading

The switch was upgraded from 6.0(4) to 6.1(3) because only 6.1(3) supports host vPC to two upstream 7K switches.

It is important to note, that in a HA pair of two 7K switches, if you're performing an ISSU on switch 1, you should not perform ISSU on switch 2. That defeats the purpose of HA because switch 1 assumes switch 2 is now the primary vPC peer, since switch 1 is in the process of upgrading and will require reloading its supervisor engines (one at a time).

In spite of this, if you still perform an ISSU on switch 2, while switch 1 is being upgraded, the upgrade check will complete on switch 2, but the system will prompt an error stating switch 1 is going through an upgrade so switch 2 cannot be upgraded. This little thing might go Continue reading

Some weird NX-OS stuff!

For enabling 'FEX' feature on the Nexus 7K switches, following steps are required:

Wonder why FEX cannot be enabled by entering feature fex or feature-set fex on the switch. 

When you do a 'show feature', FEX shows up as disabled. On a 'show feature-set', FEX shows up as enabled.

After configuring an interface as a fex-fabric mode, 

Cisco developed two commands on NX-OS to do the same thing. Define a hostname for the switch. 'switchname' and 'hostname'. 

No feedback really required here. Boring post. I could just delete it. Naah I'll keep it. Peace m/.

vPC configuration on a pair of NX7K switches

For a pair of Nexus 7009 switches running NX-OS 6.0(4), configuring vPC (virtual Port Channel) was an easy breezy task. In a few minutes and less than 15 commands, the two switches stood up as a vPC pair. There are 3 non default VDCs and each VDC has its own vPC domain, vpc peer-links and vpc peer-keepalive links. Cisco recommends each VDC has its own unique vPC domain ID. The port-channel ID need not be unique across the VDCs. Also, vPC will not work if port-channel members are interfaces allocated to separate VDCs. 

On the CLI, switch to the non default VDC.

Each feature needs to be enabled on individual VDCs. Simply enabling it on the admin VDC does not propagate the features to the non default VDCs. For vPC, we need link aggregation control protocol for port-channel load-balancing (LACP) and the vPC feature.

The two switches need to exchange heartbeats to maintain a vPC role over the vPC keep alive links. Cisco recommends this traffic must be isolated to a VRF. If we do not specify a VRF for the keep alive link, by default Continue reading

When you start an Xfinity (Comcast) Internet service, the activation process involves plugging in your service provider provided modem's cable port to a cable outlet, and the connecting the modem's ethernet port to your laptop. You see the green blinking lights for RX/TX and a steady green light for Internet. 

However, if you connect the modem's ethernet port to a wireless router (Linksys E2000 in my case), assuming you have correctly configured all LAN settings, wifi settings, you may still not be able to connect to the Internet. 

Log in to the router's admin page, typically http://192.168.1.1, with username admin and password admin (again, unless you have changed these default settings). Go to the status page and notice the IP address. What you'll see is a '0.0.0.0' IP Address for the router's Internet connection.

I tried everything to get a public IP (from Comcast's range) here. Then I read Comcast's documentation which says connect a laptop to the ethernet port on the modem. Well, what's so special about a laptop which a router can't do?

Not always, but some times Comcast's modem expects a laptop on the other end of the ethernet Continue reading

PortFast is a feature enabled on switchports, specifically 'edge' ports that reduces convergence time by transitioning to forwarding state quickly bypassing 30 seconds of listening and learning. When you connect an end device like a PC, server, phone etc, the switchport almost instantly starts forwarding frames.

PortFast also suppresses TCN BPDUs (Topology Change Notification Bridge Protocol Data Units). However, it sends BPDUs and actively participates in the STP topology mechanism. 

Why would one want an edge port to send BPDUs? Well, if you 'accidentally' connect a switch (managed or unmanaged) to an edge port configured with PortFast, that switch needs to know it is connected to a switch. 

If this port were to receive a BPDU, it would lose its PortFast status and transition to a normal STP port. This could be dangerous. You would want to block this port and prevent a loop from occurring. 

In order to accomplish the above, PortFast must be configured in conjunction with BPDUGuard. If a port configured with PortFast BPDUGuard receives a BPDU, it disables the port. The switchport moves into error-disabled state. And you need to manually shut and no shut the port to bring it back up, although Continue reading

In this lab, I tried to simulate an environment where there are two customers, each connected to their respective ISP. Now, in real world, this might not be the best way things are done, but this lab is for the sake of understanding how VPNs deal with networks behind a VPN peer. 

PE: Provider Edge equipment

CPE: Customer Premise equipment

Following is the network diagram. CPE1 and CPE2 are customer edge routers. PE1 and PE2 are respective ISP provider edge routers. Each router connects to another over a /30 point to point link. Each router has a loopback (Lo0) with an IP address in the 192.168.0.0/16 range as shown.

CPE1 has a site to site VPN tunnel to PE1.

PE1 has two site to site VPN tunnels, one to CPE1 and another to PE2.

PE2 has two site to site VPN tunnels, one to PE1 and another to CPE2.

CPE2 has a site to site VPN tunnel to PE2.

I had a problem with VPN Hairpinning and wanted to build a lab to find possible solutions. I started off building the lab and after bringing up VPNs, I realized I built the lab wrong. Notice how Continue reading