Category Archives: NX-OS

Meet Ethanalyzer – Cisco NX-OS Control Plane Packet Capture

I was always under the impression that Cisco N5Ks deployed in our environment lack packet capture capabilities. Well, this is true for Data Plane, but apparently they have this feature to capture CPU-bound traffic, such as ARP, BGP, EIGRP, OSPF, ICMP, etc.

A bit of story. We’ve faced an issue with Silver-Peak SD-WAN appliances which (under the hood) use Unicast ARP frames as nexthop reachability check. Basically, they send Unicast ARP Request every 30-50 seconds and expect a response within 1 second. If ARP Response doesn’t come through, they repeat 3 times before marking nexthop unreachable and invalidating all relevant routes. Once this happens, Silver-Peak appliances fallback to discovery mode and use Broadcast ARP Request to find nexthop’s MAC address. It took us a while to understand it and of course we had to use packet capture feature on Silver-Peak appliances.

But then we engaged with Cisco TAC to see where the problem lies (Cisco Nexus, or Cisco FTD deployed in transparent mode). To my surprise Cisco TAC has used Ethanalyzer tool to capture traffic destined to N5K’s control plane (CPU). The command is

ethanalyzer local interface inbound-hi display-filter display-filter
ethanalyzer local interface inbound-low display-filter display-filter 

From N5K’s perspective CPU-bound traffic can be High Priority (inbound-hi), or Low Priority (inbound-low). High Priority includes, but is not limited to, ARP (unicast), STP, BGP, EIGRP, OSPF – anything that affects Layer 2 or Layer 3 convergence. Low Priority set includes ICMP, SSH, ARP (broadcast).

Display filters match logic available in Wireshark. If you’re familiar with the tool, you won’t have any issues filtering the output. Keep in mind, these filters control what’s sent to VTY, while in background everything is captured (some Nexus platforms support capture filters, which limit what’s captured). Cisco recommends to use display-filters on N5Ks. Here’s the amazing Wireshark Display Filters Cheatsheet created by Jeremy Stretch (packetlife). The following example will capture unicast arp traffic (inbound-hi) from endpoint with MAC address of AA:BB:CC:DD:EE:FF.

ethanalyzer local interface inbound-hi display-filter arp.src.hw_mac==aabb.ccdd.eeff

By default capture will run until it processes 10 packets, if you want to run it longer use limit-captured-frames X keyword-value pair. Set X to any number, or 0, if you want capture to run indefinitely (until CTRL+X is pressed).

Here are couple examples of real packet captures I did in our environment (IP addresses are obfuscated of course):

N5K-1# ethanalyzer local interface inbound-low display-filter eth.src==001B.BC11.5B5C limit-captured-frames 0
 Capturing on inband
 2020-12-04 00:46:11.454501 00:1b:bc:11:5b:5c -> ff:ff:ff:ff:ff:ff ARP Who has 10.180.10.113?  Tell 10.180.10.118
 2020-12-04 00:46:12.586073 10.180.10.118 -> 10.180.10.113 ICMP Echo (ping) request
 2020-12-04 00:46:22.586722 10.180.10.118 -> 10.180.10.113 ICMP Echo (ping) request
 2020-12-04 00:46:32.587417 10.180.10.118 -> 10.180.10.113 ICMP Echo (ping) request
 2020-12-04 00:46:41.493429 00:1b:bc:11:5b:5c -> ff:ff:ff:ff:ff:ff ARP Who has 10.180.10.113?  Tell 10.180.10.118

As you can see, the capture was done for Low Priority traffic, therefore only ICMP and Broadcast ARP are shown. The following capture shows High Priority control traffic.

N5K-2# ethanalyzer local interface inbound-hi display-filter eth.src==001B.BC11.5B6C limit-captured-frames 0
 Capturing on inband
 2020-12-03 09:12:29.476910 10.180.10.126 -> 10.180.10.121 BGP KEEPALIVE Message
 2020-12-03 09:12:31.053569 10.180.10.126 -> 10.180.10.121 TCP bgp > 28839 [ACK] Seq=20 Ack=20 Win=444 Len=0
 2020-12-03 09:12:33.476911 00:1b:bc:11:5b:6c -> 00:2a:6a:20:79:c1 ARP Who has 10.180.10.121?  Tell 10.180.10.126
 2020-12-03 09:12:38.680526 10.180.10.126 -> 10.180.10.121 BGP KEEPALIVE Message
 2020-12-03 09:12:41.071191 10.180.10.126 -> 10.180.10.121 TCP bgp > 28839 [ACK] Seq=39 Ack=39 Win=444 Len=0
 2020-12-03 09:12:48.199107 10.180.10.126 -> 10.180.10.121 BGP KEEPALIVE Message
 2020-12-03 09:12:51.090945 10.180.10.126 -> 10.180.10.121 TCP bgp > 28839 [ACK] Seq=58 Ack=58 Win=444 Len=0

Here you can see BGP packets (Keepalives and TCP ACK), as well as Unicast ARP Requests. Keep in mind, this capture only shows ingress traffic, egress traffic cannot be captured. It is possible to capture traffic into binary file and review it using Wireshark, such as

ethanalyzer local interface inbound-hi limit-captured-frames 1000 write bootflash:capture-file.pcap

As you can see, display-filters cannot be used if captured packets have to be writen down into file. You will have to use Wireshark to filter certain packets out.

More information can be found here: Nexus 3000/5000/7000 Use of the Ethanalyzer Tool

I hope it was useful!

Troubleshooting NX-OS Config Sync

I have spent last weeks configuring our new Cisco Nexus 5596UP switches in two data centers. The decision to use configuration synchronization feature (also known as Switch Profile) seemed logical as our new DC infrastructure design dictates to use Dual Homed FEXes with Active/Passive NIC teaming topology. This scenario (like any Dual Homed) requires almost all configuration to be identical on both switches that are part of vPC domain. Overall, I like this neat feature. In my humble opinion, Cisco had to come up with it years ago. It works like a charm if you are working on a clean deployment and follow Cisco guidelines. But… when it comes to the migration from Legacy configuration mode to the Switch Profile mode with both vPC domain switches already being pre-configured separately… well, you’ll definitely face some issues! I personally have spent few days trying to solve one puzzle that driven me nuts!  Continue reading

vPC Domain Configuration Synchronization Guidelines

Configuration synchronization, also known as Switch Profiles, is a new feature that has been introduced by Cisco to primarily support Nexus vPC Domain topologies in modern data centers, specifically the Dual-Homed FEX scenarios. One of the main requirements in Dual-Homed FEX topologies is configuration consistency across both Nexus 5K switches. Remember that vPC domain switches represent one logical switch. Thus, must be consistent from QoS-, VLAN-, Spanning-Tree- and, in some cases, FEX interfaces- configuration perspective. Switch Profiles, if well understood, can help to lessen administration overhead. Continue reading

FET-10G considerations

We are currently deploying new data centers using brand new Cisco Nexus 5596UP switches with Fabric Extenders (Cisco Nexus 2248TP-E). When you order a bundle from Cisco (switches and FEXes) it goes pre-packed with FET-10G modules (8 per FEX). Ok, not pre-packed, you actually need to include those modules into the order for free. The reason I am making notes on this is due to the additional configuration required for this module to be discovered. If you simply put it into the switch, it will show those modules as Invalidated. For them to work, it is required to manually change switchport mode to “fex-fabric” – switchport mode fex-fabric. Once you’ve done that, FET-10G will be recognized and any FEX behind it will be discovered.

This can be confusing if you’re deploying Nexus switches and FEXes using this modules for the first time. They have a limited scope of use – only switch to FEX connections. They go for free when you buy bundles, so make sure to include them into your order! You won’t be able to get them for free later (or even buy them). General SFP+ modules will discover any connected FEXes automatically.