Verify port-based QoS configuration in hardware of Cisco 6509

It’s a pleasure to work with Cisco TAC. I would say this is the only way to instantly improve your troubleshooting skills in any area. I was involved in Cisco 6509 troubleshooting that turned to be an IOS software-to-hardware QoS configuration bug. With this post I will slightly cover the bug logic and will show you the way to confirm QoS configuration in Cisco 6509’s module hardware.

So, everything started with some strange issue that was happening every time we executed a “default interface” command on the core switch. Each time a default settings were applied to any single interface on any module, we ended up with few unresponsive ports on the same module and not necessarily the port that got a default configuration. I tried to re-create this issue using different approaches and noticed it happens when port-based QoS configuration is being removed (or re-applied) from at least one interface (or range of ports).

Now, one interesting fact. All Cisco 6509 modules differ in how they treat QoS configuration. Mostly all old hardware revisions apply the same port-based QoS configuration to the whole ASIC, that is, all ports on this ASIC. No matter which port you change, the same configuration propagates to the rest of ports. Usually, one module consisted of two ASICs. So, let’s say, 48 ports module is represented as two ASICs from QoS perspective. If you change QoS configuration on the port 1/13, it is propagated to ports 1/1-/24, and, you can do nothing here. Newer modules, however, allow you to configure port-based QoS configuration on a single port. Having this in mind, let me continue my story…

Taking into account that switch consists of the modules of the same type (prone to this bug) which share server and user ports (yes, design is shitty but that’s a different story), you can imagine how risky it was when any QoS related configuration was about to change! Add here, it is a regional hub core switch. A nightmare! The one odd thing that made my troubleshooting efforts almost useless is the fact that ARP and MAC tables had the valid dynamic entries that matched devices behind the unresponsive ports (even after clearing both tables from those particular entries). In fact, switch had seen the devices behind those ports. Unfortunately, ping, nor any other traffic, was allowed through.

WS-X6548-GE-45AF is the module proved to be prone to this issue. Code name Revati.

That was the story. Now, I’ll tell you how Cisco TAC engineers found it is a software to hardware mapping bug.

The first things they checked was interface statistics. Although, being a predictable step, I never paid attention to the Output Drops counter. Here’s an example of the affected interface.

SWITCH#show int gi8/31
GigabitEthernet8/31 is up, line protocol is up (connected)
 Hardware is C6k 1000Mb 802.3, address is 0023.04ab.09ce (bia 0023.04ab.09ce)
 Description: PC/VoIP
 MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
 reliability 255/255, txload 1/255, rxload 1/255
 Encapsulation ARPA, loopback not set
 Keepalive set (10 sec)
 Full-duplex, 100Mb/s, media type is 10/100/1000BaseT
 input flow-control is off, output flow-control is off
 Clock mode is auto
 ARP type: ARPA, ARP Timeout 04:00:00
 Last input never, output 00:00:05, output hang never
 Last clearing of "show interface" counters never
 Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 351576
 Queueing strategy: fifo
 Output queue: 0/40 (size/max)
 5 minute input rate 13000 bits/sec, 9 packets/sec
 5 minute output rate 144000 bits/sec, 18 packets/sec
 2379911 packets input, 559779371 bytes, 0 no buffer
 Received 115592 broadcasts (89172 multicasts)
 0 runts, 0 giants, 0 throttles
 0 input errors, 0 CRC, 2 frame, 0 overrun, 0 ignored
 0 watchdog, 0 multicast, 0 pause input
 0 input packets with dribble condition detected
 14667436 packets output, 2962088579 bytes, 0 underruns
 0 output errors, 0 collisions, 4 interface resets
 0 babbles, 0 late collision, 0 deferred
 0 lost carrier, 0 no carrier, 0 PAUSE output
 0 output buffer failures, 0 output buffers swapped out

Check output drops value. This counter tells you how many packets were dropped from the interface’s TX queue. QoS configuration is one of the factors that influence the rate this counter increments. In normal conditions it is not necessarily equals to zero (as QoS policy, if applied, will certainly discard some output packets), but it shouldn’t increment with every single packet. Basically, if you see this counter increments quickly and this followed by connectivity issues you most likely has QoS configuration problems.

In our case everything was OK from the port-based QoS configuration perspective. Here’s the example.

interface GigabitEthernet8/31
 description PC/VoIP
 switchport
 switchport access vlan 10
 switchport mode access
 switchport voice vlan 11
 wrr-queue bandwidth 30 70
 wrr-queue queue-limit 40 30
 wrr-queue random-detect min-threshold 1 1 5
 wrr-queue random-detect min-threshold 2 1 80
 wrr-queue random-detect max-threshold 1 5 100
 wrr-queue random-detect max-threshold 2 80 100
 wrr-queue cos-map 1 1 1
 wrr-queue cos-map 1 2 0
 wrr-queue cos-map 2 1 2 3 4
 wrr-queue cos-map 2 2 6 7
 spanning-tree portfast edge
 spanning-tree bpduguard enable
 service-policy input DATA-AND-VOICE

That looks absolutely normal and, being our default QoS configuration for PC ports on Cisco 6509, works without any problems on other ports located on different modules of the same chassis. The queuing and COS-DSCP mapping can be confirmed by the “show queuing interface” command as shown below.

TEST#show queueing int gi8/31
Interface GigabitEthernet8/31 queueing strategy: Weighted Round-Robin
 Port QoS is enabled
Trust boundary disabled
Port is untrusted
 Extend trust state: not trusted [COS = 0]
 Default COS is 0
 Queueing Mode In Tx direction: mode-cos
 Transmit queues [type = 1p2q2t]:
 Queue Id Scheduling Num of thresholds
 -----------------------------------------
 1 WRR low 2
 2 WRR high 2
 3 Priority 1
WRR bandwidth ratios: 30[queue 1] 70[queue 2]
 queue-limit ratios: 40[queue 1] 30[queue 2] 30[Pri Queue]*same as Q2
queue random-detect-min-thresholds
 ----------------------------------
 1 1[1] 5[2]
 2 1[1] 80[2]
queue random-detect-max-thresholds
 ----------------------------------
 1 5[1] 100[2]
 2 80[1] 100[2]
queue thresh cos-map
 ---------------------------------------
 1 1 1
 1 2 0
 2 1 2 3 4
 2 2 6 7
 3 1 5
Queueing Mode In Rx direction: mode-cos
 Receive queues [type = 1q2t]:
 Queue Id Scheduling Num of thresholds
 -----------------------------------------
 1 Standard 2
 queue tail-drop-thresholds
 --------------------------
 1 100[1] 100[2]
queue thresh cos-map
 ---------------------------------------
 1 1 0 1 2 3 4 5 6 7
 1 2
 Packets dropped on Transmit:
 BPDU packets: 0
queue thresh dropped [cos-map]
 ---------------------------------------------------
1 1 403 [1 ]
 1 2 29 [0 ]
 2 1 0 [2 3 4 ]
 2 2 0* [6 7 ]
 3 1 0* [5 ]
 * - shared transmit counter
Packets dropped on Receive:
 BPDU packets: 0
queue thresh dropped [cos-map]
 ---------------------------------------------------
 1 1 0 [0 1 2 3 4 5 6 7 ]

After checking the interface statistics and confirming the port configuration is ok, the Cisco TAC engineers began to do magic. The rest of the actions were new to me. Ok, maybe this is because I haven’t attended Cisco QoS classes yet. Anyway…

By default when you telnet/ssh to the Cisco 6509 switch you get into Routing Processor (software), while QoS configuration is being kept in both – Software (user-friendly readable format) and Hardware – the Switching Processor. So, whenever you configure anything under port configuration it is converted into low level values and put into SP’s ASIC hardware registers. To get into Switching Processor use “remote login switch“.

Once in there, you can easily confirm the QoS configuration of any port in hardware! The magic command is “show qm-sp port-data module port-no“, where module is module number (check with “show module” under RP’s exec mode) and port-no reflects port number on that particular blade.

Here’s the example configuration for the same port Gi8/31, where the configuration in software confirmed to be Ok!

SWITCH-sp#sh qm port 8 31
-----------------------------------------------------------------------------
* Type: Tx[1p2q2t] Rx[1q2t] [0] Revati [1] PinnacleRxPortType differs[1p1q4t] 
* Per-Port: [Untrusted] Default COS[0] force[1] [port based] 
-----------------------------------------------------------------------------
* COSMAP(C[Q/T]) TX: 0[1/1] 1[1/1] 2[1/2] 3[1/2] 4[2/1] 5[3/1] 6[2/1] 7[2/2] 
 RX: 0[1/1] 1[1/1] 2[1/1] 3[1/1] 4[1/1] 5[1/1] 6[1/1] 7[1/1] 
-----------------------------------------------------------------------------
* WRR bandwidth: [7680 17920]
* TX queue limit(size): [425984 327680 327680]
* WRED queue[1]: wredLowThr[4259 21299] wredThr[-9792 425856] rndNumber[0]
 AveWeight[0] SampleCntr[255] PktCntr[0] QFreecnt[425984]
 queue[2]: wredLowThr[3276 262144] wredThr[265472 323584] rndNumber[0]
 AveWeight[0] SampleCntr[255] PktCntr[0] QFreecnt[327680]
-----------------------------------------------------------------------------
* TX drop thr queue[1]: type[2 QOS_SCP_2_THR] dropThr[22976 458624]
 queue[2]: type[2 QOS_SCP_2_THR] dropThr[249088 307200]
* RX drop threshold: type[2 QOS_SCP_2_THR] dropThr[16382 16384]

There are few strange things that are shown in red color. First, notice that COS value to TX Queue/Threshold mapping does not correspond to the one configured in software! In particular, COS0 (default) is mapped to Queue 1/Threshold 1 (while has to be Q1/T2)

wrr-queue cos-map 1 2 0

Moreover, Q1/T1 values in hardware seems to be invalid. It is configured with

wrr-queue random-detect min-threshold 1 1 5
wrr-queue random-detect max-threshold 1 5 100

That is read as Queue 1, Threshold 1 from 1% to 5% and Queue 1 Threshold 2 from 5% to 100%. In fact, in hardware Q1/T1 values are configured as follows: minimal value is 4259 (which seems to be correct, it is 1% of 425984) and maximum value is -9792 (an invalid, negative value!). So, whenever a packet is being sent to TX Queue 1, Threshold 1 it is ALWAYS DISCARDED! Period.

You may ask about ARP and MAC tables mentioned previously. How could it be that those two worked as expected? Easy. Both types of traffic are management and treated differently. Moreover, MAC table is being updated because RX queues were not affected – packets were accepted by the switch. In turn, ARP frames have no idea about COS, they lack this field.

If you have the same you are unlucky. Cisco TAC engineers promised to get this raised with Cisco Business Unit and fixed in the next software release. I am kind of sceptic to believe it will be solved – the module is EOS/EOL with software maintenance support ends July ’13. The only permanent fix available – replace those modules with the new one. Or, use the following workarounds.

  1. Never ever use default interface command. It will reset QoS settings on half of your blade ending up with few of them being unresponsive!
  2. Never ever remove QoS settings from the interface or it will got propagated to the rest of ports of the same ASIC and you will end up with the same result. If you have to change QoS, think twice, go for a major change – you will likely have interruptions!
  3. If you accidentally or intentionally ended up with unresponsive ports, save configs and do one of the following
    • Reload switch: reload
    • Reload affected module: hw-module module number reset
    • Or, the more preferable, disable/enable QoS globally: no mls qos (wait 60 seconds to apply in hardware), then mls qos. This is similar to switch restart from the QoS configuration perspective. First command rewrites hardware registers with default values, second command forces switch to take configuration from the RP (software) and put it into hardware (SP).

That’s it!

 

4 Comments

  1. 釣りビジョン 2ch,ダイワ スティーズ ベイトキャスティングモデル [651MLMRB-LM] 【BLIT…

    モーニング!私は単にFarmeramaのを楽しむが、誰doesntの?私はライブ検索を介…

  2. 即納!送料無料 ワルケラ walkera G400 用 メインフレームフィキシングセット (HM-G400-…

    ありがとう。私は、検索エンジンで、それに走りました。追加ariclesを投…

  3. 【送料無料】英語でおしゃべり ちょうだいなくんコンビ combi 知育玩具…

    は、より多くのあなたに同意します!私はこれをブックマークし、更…

  4. Salman says:

    Hi,
    Slightly off topic but…
    Recently I had elam captures on a 7600 DFC supported LC, to check if customer is sending us the correct markings or not, I can see in DBUS data that IP_TOS is set correctly, how can I verify that marking is correctly rewrite to mpls exp when the packet is sent out the mpls interface ?

Leave a Reply

%d bloggers like this: