Tag Archives: Cisco Nexus

Meet Ethanalyzer – Cisco NX-OS Control Plane Packet Capture

I was always under the impression that Cisco N5Ks deployed in our environment lack packet capture capabilities. Well, this is true for Data Plane, but apparently they have this feature to capture CPU-bound traffic, such as ARP, BGP, EIGRP, OSPF, ICMP, etc.

A bit of story. We’ve faced an issue with Silver-Peak SD-WAN appliances which (under the hood) use Unicast ARP frames as nexthop reachability check. Basically, they send Unicast ARP Request every 30-50 seconds and expect a response within 1 second. If ARP Response doesn’t come through, they repeat 3 times before marking nexthop unreachable and invalidating all relevant routes. Once this happens, Silver-Peak appliances fallback to discovery mode and use Broadcast ARP Request to find nexthop’s MAC address. It took us a while to understand it and of course we had to use packet capture feature on Silver-Peak appliances.

But then we engaged with Cisco TAC to see where the problem lies (Cisco Nexus, or Cisco FTD deployed in transparent mode). To my surprise Cisco TAC has used Ethanalyzer tool to capture traffic destined to N5K’s control plane (CPU). The command is

ethanalyzer local interface inbound-hi display-filter display-filter
ethanalyzer local interface inbound-low display-filter display-filter 

From N5K’s perspective CPU-bound traffic can be High Priority (inbound-hi), or Low Priority (inbound-low). High Priority includes, but is not limited to, ARP (unicast), STP, BGP, EIGRP, OSPF – anything that affects Layer 2 or Layer 3 convergence. Low Priority set includes ICMP, SSH, ARP (broadcast).

Display filters match logic available in Wireshark. If you’re familiar with the tool, you won’t have any issues filtering the output. Keep in mind, these filters control what’s sent to VTY, while in background everything is captured (some Nexus platforms support capture filters, which limit what’s captured). Cisco recommends to use display-filters on N5Ks. Here’s the amazing Wireshark Display Filters Cheatsheet created by Jeremy Stretch (packetlife). The following example will capture unicast arp traffic (inbound-hi) from endpoint with MAC address of AA:BB:CC:DD:EE:FF.

ethanalyzer local interface inbound-hi display-filter arp.src.hw_mac==aabb.ccdd.eeff

By default capture will run until it processes 10 packets, if you want to run it longer use limit-captured-frames X keyword-value pair. Set X to any number, or 0, if you want capture to run indefinitely (until CTRL+X is pressed).

Here are couple examples of real packet captures I did in our environment (IP addresses are obfuscated of course):

N5K-1# ethanalyzer local interface inbound-low display-filter eth.src==001B.BC11.5B5C limit-captured-frames 0
 Capturing on inband
 2020-12-04 00:46:11.454501 00:1b:bc:11:5b:5c -> ff:ff:ff:ff:ff:ff ARP Who has 10.180.10.113?  Tell 10.180.10.118
 2020-12-04 00:46:12.586073 10.180.10.118 -> 10.180.10.113 ICMP Echo (ping) request
 2020-12-04 00:46:22.586722 10.180.10.118 -> 10.180.10.113 ICMP Echo (ping) request
 2020-12-04 00:46:32.587417 10.180.10.118 -> 10.180.10.113 ICMP Echo (ping) request
 2020-12-04 00:46:41.493429 00:1b:bc:11:5b:5c -> ff:ff:ff:ff:ff:ff ARP Who has 10.180.10.113?  Tell 10.180.10.118

As you can see, the capture was done for Low Priority traffic, therefore only ICMP and Broadcast ARP are shown. The following capture shows High Priority control traffic.

N5K-2# ethanalyzer local interface inbound-hi display-filter eth.src==001B.BC11.5B6C limit-captured-frames 0
 Capturing on inband
 2020-12-03 09:12:29.476910 10.180.10.126 -> 10.180.10.121 BGP KEEPALIVE Message
 2020-12-03 09:12:31.053569 10.180.10.126 -> 10.180.10.121 TCP bgp > 28839 [ACK] Seq=20 Ack=20 Win=444 Len=0
 2020-12-03 09:12:33.476911 00:1b:bc:11:5b:6c -> 00:2a:6a:20:79:c1 ARP Who has 10.180.10.121?  Tell 10.180.10.126
 2020-12-03 09:12:38.680526 10.180.10.126 -> 10.180.10.121 BGP KEEPALIVE Message
 2020-12-03 09:12:41.071191 10.180.10.126 -> 10.180.10.121 TCP bgp > 28839 [ACK] Seq=39 Ack=39 Win=444 Len=0
 2020-12-03 09:12:48.199107 10.180.10.126 -> 10.180.10.121 BGP KEEPALIVE Message
 2020-12-03 09:12:51.090945 10.180.10.126 -> 10.180.10.121 TCP bgp > 28839 [ACK] Seq=58 Ack=58 Win=444 Len=0

Here you can see BGP packets (Keepalives and TCP ACK), as well as Unicast ARP Requests. Keep in mind, this capture only shows ingress traffic, egress traffic cannot be captured. It is possible to capture traffic into binary file and review it using Wireshark, such as

ethanalyzer local interface inbound-hi limit-captured-frames 1000 write bootflash:capture-file.pcap

As you can see, display-filters cannot be used if captured packets have to be writen down into file. You will have to use Wireshark to filter certain packets out.

More information can be found here: Nexus 3000/5000/7000 Use of the Ethanalyzer Tool

I hope it was useful!

Troubleshooting NX-OS Config Sync

I have spent last weeks configuring our new Cisco Nexus 5596UP switches in two data centers. The decision to use configuration synchronization feature (also known as Switch Profile) seemed logical as our new DC infrastructure design dictates to use Dual Homed FEXes with Active/Passive NIC teaming topology. This scenario (like any Dual Homed) requires almost all configuration to be identical on both switches that are part of vPC domain. Overall, I like this neat feature. In my humble opinion, Cisco had to come up with it years ago. It works like a charm if you are working on a clean deployment and follow Cisco guidelines. But… when it comes to the migration from Legacy configuration mode to the Switch Profile mode with both vPC domain switches already being pre-configured separately… well, you’ll definitely face some issues! I personally have spent few days trying to solve one puzzle that driven me nuts!  Continue reading

vPC Domain Configuration Synchronization Guidelines

Configuration synchronization, also known as Switch Profiles, is a new feature that has been introduced by Cisco to primarily support Nexus vPC Domain topologies in modern data centers, specifically the Dual-Homed FEX scenarios. One of the main requirements in Dual-Homed FEX topologies is configuration consistency across both Nexus 5K switches. Remember that vPC domain switches represent one logical switch. Thus, must be consistent from QoS-, VLAN-, Spanning-Tree- and, in some cases, FEX interfaces- configuration perspective. Switch Profiles, if well understood, can help to lessen administration overhead. Continue reading

Choosing between Dynamic and Static FEX interfaces pinning

Cisco Nexus 2200 Fabric Extenders can be connected to the parent switches using two different modes: Static and Dynamic interfaces pinning. Static pinning mode instructs the switch to virtually split FEX into few blocks of ports and statically associate each block of ports with its own physical uplink. In other words, if one particular uplink fails, a range of FEX ports, associated with this uplink, fail as well. Hence, the word Static. Dynamic pinning mode is based on a Port-Channel logic. In very basic scenario you would have all your physical uplinks associated with a single Port-Channel that will stay up as long as there is at least one working physical uplink. At first glance, difference seems to be obvious and not in the favor of Static mode. But let’s dive into the subject to understand when Static mode becomes handy. Continue reading

Configuring Cisco Nexus 5500 series switches with Dual-Homed FEXes

More and more Enterprises come to a decision to deploy Cisco Nexus switches in their corporate data centers. One of the main design considerations relates to Cisco 2200 Fabric Extenders (FEX) connectivity topology. To provide high availability Cisco Nexus 5500 series switches support different options to connect FEXes (all are based on a Virtual Port Channel feature):

  • Straight-Through, where every FEX is connected to a single N5K (Active/Active and Active/Passive servers);
  • Dual-Homed, where each FEX has one or more uplinks to two N5K switches (Active/Passive servers);
  • Enhanced vPC, same as Dual-Homed but with Active/Active servers;

Each option has its own limitations and field of use, but this time we will concentrate our attention on the Dual-Homed FEX topology with Active/Standby Dual-Homed servers. Mainly because I recently deployed a pair of Cisco Nexus 5596UP switches with a number of Cisco 2248 TP-E Fabric Extenders using this approach.

Continue reading