vPC Domain Configuration Synchronization Guidelines

Configuration synchronization, also known as Switch Profiles, is a new feature that has been introduced by Cisco to primarily support Nexus vPC Domain topologies in modern data centers, specifically the Dual-Homed FEX scenarios. One of the main requirements in Dual-Homed FEX topologies is configuration consistency across both Nexus 5K switches. Remember that vPC domain switches represent one logical switch. Thus, must be consistent from QoS-, VLAN-, Spanning-Tree- and, in some cases, FEX interfaces- configuration perspective. Switch Profiles, if well understood, can help to lessen administration overhead.

First of all, you have to understand that Configuration Synchronization feature is based on Atomic commit (verify/commit) logic. It has nothing similar to the legacy VTP that lacks tools to verify compatibility and consistency of the new configuration before it is applied. Secondly, Configuration Synchronization introduces new configuration mode, called Config Sync as opposed to Global Config mode. Thirdly, configuration commands, with few exceptions, cannot exist in both modes. If you decided to use Config Sync and, for example, synchronize QoS configuration (ACLs, classes and policy-maps) across vPC domain, system will reject any changes to the existing QoS configuration if those applied via traditional Global Config mode. Finally, if someone commits changes within the Switch Profile, all other configuration sessions will be locked for required amount of time. These are fundamental things and you must understand them to avoid frustrations while working with vPC topologies where Config Sync is in use.

Before configuring Switch Profile ensure that your vPC domain is healthy. The following steps show how to configure this neat feature.

  • Enable Cisco Fabric Services over IP (CFSoIP) on both vPC peers and, optionally, configure non-default CFSoIP multicast address;
  • Create Switch-Profile on both vPC peers with identical name;
  • Configure synchronization destination peer’s IP address
    Note: Config Sync feature only supports mgmt0 (Out of Band management) interface’s IP addresses as sync destination peer. You are not allowed to use mgmt SVI (In-Band Management) interface for this purposes.
cfs ipv4 distribute
configure sync
 switch-profile vPC
 sync-peer destination 192.168.0.12

In few moments your switches’ configuration will get synchronized. Frankly speaking, they have nothing to synchronize as yet. So, a simple check is performed to confirm that Switch Profiles are empty on both switches. Use the show switch-profile status command to confirm both switches got sync’d:

N5K-01(config)# show switch-profile status

switch-profile  : vPC
----------------------------------------------------------

Start-time: 236757 usecs after Mon Sep 30 10:13:06 2013
End-time: 672830 usecs after Mon Sep 30 10:13:30 2013

Profile-Revision: 1
Session-type: Commit
Session-subtype: -
Peer-triggered: No
Profile-status: Sync Success

Local information:
----------------
Status: Commit Success
Error(s):

Peer information:
----------------
IP-address: 192.168.0.12
Sync-status: In sync
Status: Commit Success
Error(s):

Remember this command, it will be your fav now. It displays the status of the latest sync operation, including any errors that were encountered. If you need to confirm status of any previous sync sessions, then use “show switch-profile session-history” command.

If you are working with a clean deployment, then nothing stops you from entering the Config Sync mode and configuring both switches using synchronization feature as shown below.

  • Enter into Switch Profile and apply configuration same way as you do in Global Config mode:
    N5K-01# configure sync
    N5K-01(config-sync)# switch-profile vPC
    Switch-Profile started, Profile ID is 1
    N5K-01(config-sync-sp)# interface ethernet 101/1/10
    N5K-01(config-sync-sp-if)# speed 1000
    N5K-01(config-sync-sp-if)# duplex full
    N5K-01(config-sync-sp-if)# switchport access vlan 150
    N5K-01(config-sync-sp-if)# description TEST COMMIT
  • Check and confirm the Switch Profile buffer using show switch-profile buffer command. Remember that Config Sync mode does not apply configuration as you enter it, switches have to agree it is consistent before committing the change.
    N5K-01(config-sync-sp)# show switch-profile buffer
    
    switch-profile : vPC
    ----------------------------------------------------------
    Seq-no Command
    ----------------------------------------------------------
    20 interface Ethernet101/1/10
    20.1 speed 1000
    20.2 duplex full
    20.3 switchport access vlan 150
    20.4 description TEST COMMIT
  • If required, you can delete (buffer-delete id-range|all) inaccurate and unwanted commands, or move (buffer-move from-id to-id) them within buffer to align any dependencies, i.e.
    N5K-01(config-sync-sp)# buffer-delete 20.2
    N5K-01(config-sync-sp)# show switch-profile buffer
    
    switch-profile : vPC
    ----------------------------------------------------------
    Seq-no Command
    ----------------------------------------------------------
    20 interface Ethernet101/1/10
    20.1 speed 1000
    20.3 switchport access vlan 150
    20.4 description TEST COMMIT
  • Before committing the changes, execute verify command to confirm they are consistent across the vPC domain.
    N5K-01(config-sync-sp-if)# verify
    Verification Successful
  • Finally, assuming verification was successful, commit all changes.
    N5K-01(config-sync-sp)# commit
    Verification successful...
    Proceeding to apply configuration. This might take a while depending on amount of configuration in buffer.
    Please avoid other configuration changes during this time.
    Commit Successful
  • Optionally, confirm if changes exist in the running-config
    N5K-01(config-sync)# show int description | inc 101/1/10
    Eth101/1/10 eth 1000 TEST COMMIT

Although commit operation implies an execution of background verify process, I strongly recommend to verify buffered commands explicitly. The main advantage of “verify, then commit” approach is that commit operation will throw you out of Switch Profile mode into Config Sync mode regardless if it was successful or not. In opposite, verify command will keep you within Switch Profile mode – as result, you know if commit was successful and, if not, you don’t need to enter into Switch Profile again to re-align configuration commands (if required)…

Easy!

Well, it becomes more tricky if Switch Profile was created after vPC domain switches were fully configured. In this case you will have to import configuration from the Global Config mode into the Switch Profile mode. Before I’ll demonstrate how to do this, you have to understand one important thing about import process. It is not allowed to import any configuration from Global Configuration into the Switch Profile configuration while vPC peer switches are in sync mode. Hmm… It doesn’t sound obvious, doesn’t it? Here’s an example.

  • Imagine you have a FEX interface, that was configured in Global Config mode
    N5K-01(config-sync-sp)# show run int eth101/1/10
    
    !Command: show running-config interface Ethernet101/1/10
    !Time: Thu Oct  3 11:06:30 2013
    
    version 6.0(2)N1(2)
    
    interface Ethernet101/1/10
      switchport access vlan 150
  • Now, let’s try to import this configuration
    N5K-01(config-sync-sp)# import int eth101/1/10
    N5K-01(config-sync-sp-import)# show switch-profile buffer
    
    switch-profile  : vPC
    ----------------------------------------------------------
    Seq-no  Command
    ----------------------------------------------------------
    1       interface Ethernet101/1/10
    1.1       switchport access vlan 150
  • And verify it, of course
    N5K-01(config-sync-sp-import)# verify
    Failed: Verify Failed
  • The switch thinks that mutual-exclusion check failed:
    N5K-01(config-sync-sp-import)# show switch-profile status
    
    switch-profile  : vPC
    ----------------------------------------------------------
    
    Start-time: 311896 usecs after Thu Oct  3 11:06:48 2013
    End-time: 318291 usecs after Thu Oct  3 11:06:49 2013
    
    Profile-Revision: 12
    Session-type: Import-Verify
    Session-subtype: -
    Peer-triggered: No
    Profile-status: Verify Failed
    
    Local information:
    ----------------
    Status: Verify Success
    Error(s):
    
    Peer information:
    ----------------
    IP-address: 192.168.0.12
    Sync-status: In sync
    Status: Verify Failure
    Error(s):
    Following commands failed mutual-exclusion checks: interface Ethernet101/1/10 switchport access vlan 150

This is absolutely normal behavior for the latest version of NX-OS – 6.x. The import command is locally significant. The config is copied from the Global Config mode into Switch Profile mode, then standard verification process starts which cannot be completed due to the inconsistency of the existing configuration in (a) local Global and Sync Config modes and (b) remote Global and Sync modes. To work around this (an official Cisco process), you have to

  • Temporarily disable synchronization between two switches (vPC peers) with no sync-peer destination command under Switch Profile configuration mode;
    N5K-01(config-sync-sp)# no sync-peer destination
  • Import configuration on both switches (separately);
    N5K-01(config-sync-sp)# import int eth104/1/10
    N5K-01(config-sync-sp-import)# verify
    Verification Successful
    N5K-01(config-sync-sp-import)# commit
    Verification successful...
    Proceeding to apply configuration. This might take a while depending on amount of configuration in buffer.
    Please avoid other configuration changes during this time.
    Commit Successful
    N5K-02(config-sync-sp)# import int eth104/1/10
    N5K-02(config-sync-sp-import)# verify
    Verification Successful
    N5K-02(config-sync-sp-import)# commit
    Verification successful...
    Proceeding to apply configuration. This might take a while depending on amount of configuration in buffer.
    Please avoid other configuration changes during this time.
    Commit Successful
  • Re-enable synchronization;
    N5K-01(config-sync)# switch-profile vPC
    N5K-01(config-sync-sp)# sync-peer destination 192.168.0.12
  • Confirm synchronization was successful with show switch-profile status command;
    N5K-01(config-sync)# show switch-profile status
    
    switch-profile  : vPC
    ----------------------------------------------------------
    
    Start-time: 435605 usecs after Thu Oct  3 13:10:46 2013
    End-time: 808603 usecs after Thu Oct  3 13:11:08 2013
    
    Profile-Revision: 13
    Session-type: Commit
    Session-subtype: -
    Peer-triggered: No
    Profile-status: Sync Success
    
    Local information:
    ----------------
    Status: Commit Success
    Error(s):
    
    Peer information:
    ----------------
    IP-address: 192.168.0.12
    Sync-status: In sync
    Status: Commit Success
    Error(s):

For the additional information about Switch Profiles please refer to the relevant section of the Cisco Nexus 5500 Series NX-OS System Management Configuration Guide, Release 6.x. Stay tuned, I am going to cover Config Sync issues and troubleshooting techniques in my next post.

Leave a Reply

%d bloggers like this: