I was always under the impression that Cisco N5Ks deployed in our environment lack packet capture capabilities. Well, this is true for Data Plane, but apparently they have this feature to capture CPU-bound traffic, such as ARP, BGP, EIGRP, OSPF, ICMP, etc.
A bit of story. We’ve faced an issue with Silver-Peak SD-WAN appliances which (under the hood) use Unicast ARP frames as nexthop reachability check. Basically, they send Unicast ARP Request every 30-50 seconds and expect a response within 1 second. If ARP Response doesn’t come through, they repeat 3 times before marking nexthop unreachable and invalidating all relevant routes. Once this happens, Silver-Peak appliances fallback to discovery mode and use Broadcast ARP Request to find nexthop’s MAC address. It took us a while to understand it and of course we had to use packet capture feature on Silver-Peak appliances.
But then we engaged with Cisco TAC to see where the problem lies (Cisco Nexus, or Cisco FTD deployed in transparent mode). To my surprise Cisco TAC has used Ethanalyzer tool to capture traffic destined to N5K’s control plane (CPU). The command is
ethanalyzer local interface inbound-hi display-filter display-filter ethanalyzer local interface inbound-low display-filter display-filter
From N5K’s perspective CPU-bound traffic can be High Priority (inbound-hi), or Low Priority (inbound-low). High Priority includes, but is not limited to, ARP (unicast), STP, BGP, EIGRP, OSPF – anything that affects Layer 2 or Layer 3 convergence. Low Priority set includes ICMP, SSH, ARP (broadcast).
Display filters match logic available in Wireshark. If you’re familiar with the tool, you won’t have any issues filtering the output. Keep in mind, these filters control what’s sent to VTY, while in background everything is captured (some Nexus platforms support capture filters, which limit what’s captured). Cisco recommends to use display-filters on N5Ks. Here’s the amazing Wireshark Display Filters Cheatsheet created by Jeremy Stretch (packetlife). The following example will capture unicast arp traffic (inbound-hi) from endpoint with MAC address of AA:BB:CC:DD:EE:FF.
ethanalyzer local interface inbound-hi display-filter arp.src.hw_mac==aabb.ccdd.eeff
By default capture will run until it processes 10 packets, if you want to run it longer use
limit-captured-frames X keyword-value pair. Set X to any number, or 0, if you want capture to run indefinitely (until CTRL+X is pressed).
Here are couple examples of real packet captures I did in our environment (IP addresses are obfuscated of course):
N5K-1# ethanalyzer local interface inbound-low display-filter eth.src==001B.BC11.5B5C limit-captured-frames 0 Capturing on inband 2020-12-04 00:46:11.454501 00:1b:bc:11:5b:5c -> ff:ff:ff:ff:ff:ff ARP Who has 10.180.10.113? Tell 10.180.10.118 2020-12-04 00:46:12.586073 10.180.10.118 -> 10.180.10.113 ICMP Echo (ping) request 2020-12-04 00:46:22.586722 10.180.10.118 -> 10.180.10.113 ICMP Echo (ping) request 2020-12-04 00:46:32.587417 10.180.10.118 -> 10.180.10.113 ICMP Echo (ping) request 2020-12-04 00:46:41.493429 00:1b:bc:11:5b:5c -> ff:ff:ff:ff:ff:ff ARP Who has 10.180.10.113? Tell 10.180.10.118
As you can see, the capture was done for Low Priority traffic, therefore only ICMP and Broadcast ARP are shown. The following capture shows High Priority control traffic.
N5K-2# ethanalyzer local interface inbound-hi display-filter eth.src==001B.BC11.5B6C limit-captured-frames 0 Capturing on inband 2020-12-03 09:12:29.476910 10.180.10.126 -> 10.180.10.121 BGP KEEPALIVE Message 2020-12-03 09:12:31.053569 10.180.10.126 -> 10.180.10.121 TCP bgp > 28839 [ACK] Seq=20 Ack=20 Win=444 Len=0 2020-12-03 09:12:33.476911 00:1b:bc:11:5b:6c -> 00:2a:6a:20:79:c1 ARP Who has 10.180.10.121? Tell 10.180.10.126 2020-12-03 09:12:38.680526 10.180.10.126 -> 10.180.10.121 BGP KEEPALIVE Message 2020-12-03 09:12:41.071191 10.180.10.126 -> 10.180.10.121 TCP bgp > 28839 [ACK] Seq=39 Ack=39 Win=444 Len=0 2020-12-03 09:12:48.199107 10.180.10.126 -> 10.180.10.121 BGP KEEPALIVE Message 2020-12-03 09:12:51.090945 10.180.10.126 -> 10.180.10.121 TCP bgp > 28839 [ACK] Seq=58 Ack=58 Win=444 Len=0
Here you can see BGP packets (Keepalives and TCP ACK), as well as Unicast ARP Requests. Keep in mind, this capture only shows ingress traffic, egress traffic cannot be captured. It is possible to capture traffic into binary file and review it using Wireshark, such as
ethanalyzer local interface inbound-hi limit-captured-frames 1000 write bootflash:capture-file.pcap
As you can see, display-filters cannot be used if captured packets have to be writen down into file. You will have to use Wireshark to filter certain packets out.
More information can be found here: Nexus 3000/5000/7000 Use of the Ethanalyzer Tool
I hope it was useful!