More and more Enterprises come to a decision to deploy Cisco Nexus switches in their corporate data centers. One of the main design considerations relates to Cisco 2200 Fabric Extenders (FEX) connectivity topology. To provide high availability Cisco Nexus 5500 series switches support different options to connect FEXes (all are based on a Virtual Port Channel feature):
- Straight-Through, where every FEX is connected to a single N5K (Active/Active and Active/Passive servers);
- Dual-Homed, where each FEX has one or more uplinks to two N5K switches (Active/Passive servers);
- Enhanced vPC, same as Dual-Homed but with Active/Active servers;
Each option has its own limitations and field of use, but this time we will concentrate our attention on the Dual-Homed FEX topology with Active/Standby Dual-Homed servers. Mainly because I recently deployed a pair of Cisco Nexus 5596UP switches with a number of Cisco 2248 TP-E Fabric Extenders using this approach.
Let’s take a look at the following diagram.
Dashed ellipse implies the use of Port-Channels. These links are logical and may consist of multiple physical connections. Well, they very likely do.
Each FEX is Dual-Homed to both Nexus 5596UP switches using vPC feature. Server A is using Active/Passive NIC configuration, so that NIC-B is only active when NIC-A, or FEX-01, is faulty. To make this topology work and provide High Availability, both N5K switches have to be configured using the following guidelines.
- Both switches must have Layer 2 connectivity via mgmt0 interfaces;
- Both mgmt0 interfaces must have IP addresses configured that belong to the same subnet;
interface mgmt0 description Out-of-Band Management Interface ip address 192.168.0.11/24
- Both switches must have a vPC domain configured using each peer’s mgmt0 interface’s IP address as keepalive destination;
vpc domain 10 role priority 100 peer-keepalive destination 192.168.0.12
- Both switches must be interconnected using a Port Channel (2n physical links) that is also configured to be a vPC peer-link;
interface port-channel69 description vPC Peer-Link switchport mode trunk switchport trunk native vlan 10 switchport trunk allowed vlan 10,150-155 spanning-tree port type network speed 10000 vpc peer-link ! interface ethernet1/47 description Link to N5K-02 (vPC peer-link) switchport mode trunk switchport trunk native vlan 10 switchport trunk allowed vlan 10,150-155 channel-group 69 ! interface ethernet1/48 description Link to N5K-02 (vPC peer-link) switchport mode trunk switchport trunk native vlan 10 switchport trunk allowed vlan 10,150-155 channel-group 69
- Fabric Extenders must be pre-provisioned on both Nexus switches, otherwise the topology will not provide High Availability in one particular case – when one switch loses its connectivity to the FEX, while the other switch keeps it. By default, Cisco Nexus 5500 switches dynamically reacts to any FEX connectivity changes – they remove any configuration commands for the non-existing interfaces (this happens when FEX is not connected). In turn, this brings the relevant interfaces on the other switch into vPC inconsistent state even though the FEX is still connected to it;
slot 101 provision model N2K-C2248TP-E-1GE slot 102 provision model N2K-C2248TP-E-1GE
- Each Dual-Homed FEX’s configuration must existon both N5K switches using identical numbering and vPC configuration;
fex 101 pinning max-links 1 fex 102 pinning max-links 1 ! interface port-channel101 description Link to FEX-01 switchport mode fex-fabric fex associate 101 vpc 101 ! interface port-channel102 description Link to FEX-02 switchport mode fex-fabric fex associate 102 vpc 102 ! interface ethernet1/1 description Link to FEX-101 switchport mode fex-fabric fex associate 101 channel-group 101 ! interface ethernet1/2 description Link to FEX-102 switchport mode fex-fabric fex associate 102 channel-group 102
- Dual-Homed FEX interfaces configuration must be identical, otherwise interface will be shutdown due to the configuration inconsistency (Remember about this requirement every time a new FEX interface is configured);
interface ethernet101/0/10 description Server-A-NIC-A switchport access vlan 150 spanning-tree port type edge ! interface ethernet102/0/10 description Server-A-NIC-B switchport access vlan 150 spanning-tree port type edge
That’s it. Considering all the requirements listed above were met, a vPC peer- and keepalive links will come up.