Configuring Cisco Nexus 5500 series switches with Dual-Homed FEXes

More and more Enterprises come to a decision to deploy Cisco Nexus switches in their corporate data centers. One of the main design considerations relates to Cisco 2200 Fabric Extenders (FEX) connectivity topology. To provide high availability Cisco Nexus 5500 series switches support different options to connect FEXes (all are based on a Virtual Port Channel feature):

  • Straight-Through, where every FEX is connected to a single N5K (Active/Active and Active/Passive servers);
  • Dual-Homed, where each FEX has one or more uplinks to two N5K switches (Active/Passive servers);
  • Enhanced vPC, same as Dual-Homed but with Active/Active servers;

Each option has its own limitations and field of use, but this time we will concentrate our attention on the Dual-Homed FEX topology with Active/Standby Dual-Homed servers. Mainly because I recently deployed a pair of Cisco Nexus 5596UP switches with a number of Cisco 2248 TP-E Fabric Extenders using this approach.

Let’s take a look at the following diagram.

Cisco Nexus Dual-Homed FEX Topology

Dashed ellipse implies the use of Port-Channels. These links are logical and may consist of multiple physical connections. Well, they very likely do.

Each FEX is Dual-Homed to both Nexus 5596UP switches using vPC feature. Server A is using Active/Passive NIC configuration, so that NIC-B is only active when NIC-A, or FEX-01, is faulty. To make this topology work and provide High Availability, both N5K switches have to be configured using the following guidelines.

  • Both switches must have Layer 2 connectivity via mgmt0 interfaces;
  • Both mgmt0 interfaces must have IP addresses configured that belong to the same subnet;
    interface mgmt0
     description Out-of-Band Management Interface
     ip address 192.168.0.11/24
  • Both switches must have a vPC domain configured using each peer’s mgmt0 interface’s IP address as keepalive destination;
    vpc domain 10
     role priority 100
     peer-keepalive destination 192.168.0.12
  • Both switches must be interconnected using a Port Channel (2n physical links) that is also configured to be a vPC peer-link;
    interface port-channel69
     description vPC Peer-Link
     switchport mode trunk
     switchport trunk native vlan 10
     switchport trunk allowed vlan 10,150-155
     spanning-tree port type network
     speed 10000
     vpc peer-link
    !
    interface ethernet1/47
     description Link to N5K-02 (vPC peer-link)
     switchport mode trunk
     switchport trunk native vlan 10
     switchport trunk allowed vlan 10,150-155
     channel-group 69
    !
    interface ethernet1/48
     description Link to N5K-02 (vPC peer-link)
     switchport mode trunk
     switchport trunk native vlan 10
     switchport trunk allowed vlan 10,150-155
     channel-group 69
  • Fabric Extenders must be pre-provisioned on both Nexus switches, otherwise the topology will not provide High Availability in one particular case – when one switch loses its connectivity to the FEX, while the other switch keeps it. By default, Cisco Nexus 5500 switches dynamically reacts to any FEX connectivity changes – they remove any configuration commands for the non-existing interfaces (this happens when FEX is not connected). In turn, this brings the relevant interfaces on the other switch into vPC inconsistent state even though the FEX is still connected to it;
    slot 101
     provision model N2K-C2248TP-E-1GE
    slot 102
     provision model N2K-C2248TP-E-1GE
  • Each Dual-Homed FEX’s configuration must existon both N5K switches using identical numbering and vPC configuration;
    fex 101
     pinning max-links 1
    fex 102
     pinning max-links 1
    !
    interface port-channel101
      description Link to FEX-01
      switchport mode fex-fabric
      fex associate 101
      vpc 101
    !
    interface port-channel102
      description Link to FEX-02
      switchport mode fex-fabric
      fex associate 102
      vpc 102
    !
    interface ethernet1/1
     description Link to FEX-101
     switchport mode fex-fabric
     fex associate 101
     channel-group 101
    !
    interface ethernet1/2
     description Link to FEX-102
     switchport mode fex-fabric
     fex associate 102
     channel-group 102
  • Dual-Homed FEX interfaces configuration must be identical, otherwise interface will be shutdown due to the configuration inconsistency (Remember about this requirement every time a new FEX interface is configured);
    interface ethernet101/0/10
     description Server-A-NIC-A
     switchport access vlan 150
     spanning-tree port type edge
    !
    interface ethernet102/0/10
     description Server-A-NIC-B
     switchport access vlan 150
     spanning-tree port type edge

That’s it. Considering all the requirements listed above were met, a vPC peer- and keepalive links will come up.

7 Comments

  1. Priba says:

    Hi Tim,

    Thank you for this great post.
    One thing I am not sure that is necessary is adding trunk commands on N5K FEX facing interfaces & port channels.
    I mean, you can add them, but they do not have any affect, because switchport mode of those interfaces is “fex-fabric”, not “trunk”

  2. Tim Dmitrenko says:

    Thanks for the input, Priba! Well spotted, indeed. I think, this piece of config exists since my lab games with Nexus… and it successfully migrated to the live env 🙂 I will amend my post now, and will certainly need to look into cleaning the live configs in DCs at some point. Cheers

  3. Christian says:

    Hello Tim! We are currently trying to connect a Server the same way as in your diagram. Does this mean that one Port of the Server will be down on the FEX (active/inactive)? We are using LACP and i don’t get the Servers MAC’s on the channel…the Ports are both up though.

    • Hi! I am sure you already solved your issue, but I will respond in case someone else going to ask the same. What you’ve tried to configure is actually called Enhanced vPC, because you wanted to have LACP on the server (and this is indeed Active/Active). This article covers Active/Passive scenario for Dual Homed FEX deployment, where servers have dual NICs, but one of those stays inactive all the time (and this is done from the server end). Interface is not DOWN, it’s just not forwarding any traffic until Primary path fails.

  4. chucky says:

    Hi Tim,

    at the end of your guide, it seems like you are mixing things up a bit. You’re writing that dual-homed ports must always be identical but after that you are posting the configuration of two different ports with an identical condfiguration (101/0/10 and 102/0/10). Those ports don’t need to be identical!

    Because your FEX-01 is dual-homed on N5K-01 AND N5K-02, you have a configuration for Eth101/0/10 on N5K-01 and N5K-02, like that:

    N5K-01#interface ethernet101/0/10
    description Server-A-NIC-A
    switchport access vlan 150
    spanning-tree port type edge

    N5K-02#interface ethernet101/0/10
    description Server-A-NIC-A
    switchport access vlan 150
    spanning-tree port type edge

    While you can have a cpompletely different config an Port 102/0/10

    N5K-01#interface ethernet102/0/10
    description Server-A-NIC-B
    switchport access vlan 261
    spanning-tree port type edge

    N5K-02#interface ethernet102/0/10
    description Server-A-NIC-B
    switchport access vlan 261
    spanning-tree port type edge

    • Hi! Thanks for feedback. What you say is correct (that I can have completely different config on FEX 101 and FEX 102). However, what you’re missing is that my last configuration example has to be applied to N5K-01 and N5K-02 for BOTH interfaces. I.e. you need identical config for 101 and 102 on both core switches. 101 is for NIC-A, 102 is for NIC-B on the same server. Does it make sense?

Leave a Reply

%d bloggers like this: