Basic QoS part 2 – Catalyst 3560 QoS
In this post I will talk about Cisco Catalyst 3560 QoS. In Basic QoS part 1 I talked about policing and shaping on Cisco IOS routers.
To illustrate the different examples in this post I will use the following topology:
IGP: EIGRP 10
Platform/IOS: Catalyst 3560/ c3560-advipservicesk9-mz.122-44.SE6.bin
Catalyst 3560 QoS Port-Based Classification
As long as the QoS in not enabled with the command mls QoS, the QoS values in the packets are left intact when flowing through SW1 and SW2. That can be verified as follows:
As we can see from the output above, SW1 has QoS disabled (default) and is seeing packets from R1 marked with DSCP 48 (These are EIGRP updates, since routing traffic uses IP precedence of six which corresponds to DSCP 48). All incoming routing updates bear COS value of zero, since IOS does not set the COS field by default.
Let´s verify on R2 if packets with DSCP 48 are received by the control-plane:
Side note: In order to get the debug output above I just configured a policy-map matching DSCP 48 which is attached to the control-plane. Then under the class-map I configured the log keyword.
Now let´s just reset the counters on the policy-map on R2 (clear control-plane *) and let´s enable QoS on SW1.
So as soon as we enable QoS on SW1, the port connected to R1 is in an untrusted state which means that any markings will be rewritten by the switch to zero which in turns means that R2 will not see EIGRP packet marked with DSCP of 48 but with a DSCP of zero:
As a matter of fact SW1 is still receiving packet sent by R2 with a DSCP of 48 via the trunk link connected to SW2:
But as we can see in the output above SW1 is sending packets toward R2 from R1 out the trunk link with a DSCP of 0.
Side note: Just as a little reminder, CoS bits (three bits) is located in 802.1q/ISL header of a tagged frame. Those bits are known as 802.1p priority bits and are used for L2 QoS. ToS field (8 bits) located in the IP header is used for L3 QoS. 6 bits in the ToS field represent the DSCP value.
In order to keep the DSCP value sent by R1 intact we configure R1 to trust the port connected to R1 (F0/1). This step forms part of the classification stage. You may classify using either interface level setting or by applying a pre-configured policy-map. Here is how classification works on the catalyst 3560:
- Trust DSCP or IP precedence (higher 3 bits in ToS field): For IP packets we can either trust on IP precedence or on DSCP value. When you trust DSCP the switch resolve the CoS value using the DSCP to CoS mapping table. When trusting IP precedence (IPP) the switch resolve the DSCP value using the IP precedence to DSCP mapping
- Trust CoS: Works for both IP and non-IP packets (ARP for example). If no CoS present in the 802.1q/ISL header the default CoS value assigned on the interface is used. The IPP and DSCP value are not taking into account and the DSCP/IPP value will be rewrite in the packet using the CoS to DSCP mapping table
Side note: CoS marking is removed on non-trunk links and native VLAN.
To test the behavior described above let´s configure the following scenario:
- SW1 will trust CoS on the interface connected to R1 and set the default CoS to 5. Then using the CoS to DSCP map we will map the CoS of 5 to DSCP 50 so R2 should receive all packets from R1 with a DSCP of 50. So let´s configure the following:
Before making the change we can see that the CoS value of 5 correspond to a DSCP value of 40 by default:
After the change:
Side note: QoS is disabled on SW2 so all QoS marking will remain unchanged when traffic is flowing through SW2.
In the above configuration SW1 assign a CoS of 5 to all packets from R1 entering F0/1 as we are trusting on CoS and CoS is not present in the frames received by R1 because it is an access-port. So internally the switch will assign a CoS of 5 and a DSCP value of 50 due to the modification made in the CoS to DSCP mapping. Let´s verify that. To test we are pinging from R1 to R2 with a ToS value of 184 (which is equivalent to a DSCP value of 46) using an extended ping (ToS of 184 = 10111000 and the DSCP part = 101110 as we are using only the high 6 bits):
So from the output above we can clearly see that SW1 is receiving the ping packets from R1 with a DSCP value of 46 (ToS 184) and that it is assigning the CoS value of 5 to theses ping packets. You may notice in the previous output that the DSCP outgoing is zero although SW1 receives a DSCP of 50 in the packets coming from SW2. That is because we have left the trunk interface F0/13 on SW1 with the default QoS parameters, that is to say, this port is in untrusted mode and both DSCP and CoS are rewritten to zero by the switch internally before transmitting the packets to interface F0/1 towards R1. So if we were to use mls qos cos 5 and mls qos cos override on F0/13 we would see that SW1 will send packets to R1 with a DSCP of 50 and a CoS of 5 (which will be ignored as it is an access port). We have to use the cos override command because SW2 is sending packets back from R2 to SW1 with a CoS of zero so if we were just trusting the CoS value that will not work.
On the trunk link SW1 is rewriting the DSCP of 46 to a DSCP 50 using the mls qos cos to DSCP translation map. Also SW1 is sending packets to SW2 with a CoS of 5 as the link between SW1 and SW2 is a trunk and the 802.1q header is present.
Side note: We could disable the DSCP rewriting behavior when trusting CoS by using the following command: no mls qos rewrite ip DSCP which is on by default. If we were to do so the DSCP of 46 will not be rewritten to DSCP of 50.
So let´s check what SW2 is seeing on its trunk link connected to SW1:
Sure enough SW2 is seeing packets coming from SW1 with a DSCP of 50 and a CoS of 5. Returning packets from SW2 to SW1 have a DSCP of 50 and CoS of 0 as R2 is not sending packets with CoS value set. As a last step, let´s now verify that R2 is receiving ping packets with a DSCP value of 50:
Sure enough, R2 is receiving the ICMP packets with the correct DSCP value and is sending ICMP echo reply packets back to R1 with a DSCP of 50.
Side note: Nothing to do with the topic but as we are sending ICMP packets to the router itself with a high repeat count and we are also logging these packets in the control plane policy-map, this will cause a high CPU on R2 as all the packets must be process switched. We could have configured a policy-map on the control plane to rate limit ICMP packets in order to protect the CPU of R2.
Now let´s trust on DSCP instead of CoS by configuring the following:
We are going to send packets with a ToS of 200 (DSCP 50) from R1 to R2. Let´s check the QoS statistics on SW1 now:
So SW1 is receiving ping packets from R1 with a DSCP of 50 now and it is assigning a CoS of 5 to theses packets as we have configured it to do so in case the CoS value is not present which is the case here.
Let´s check the trunk link now:
As expected SW1 is sending the ping packets with the DSCP of 50 unchanged as we are trusting DSCP now and the CoS value has been rewritten to 6 according the QoS table DSCP-CoS map:
According to the table a DSCP of 50 corresponds to a CoS of 6. And R2 is still receiving ICMP packets with a DSCP of 50:
Perfect. That was an example to demonstrate how basic QoS port-based classification works on the Catalyst 3560. Now let´s talk about the other way of doing QoS classification using ACLs:
Catalyst 3560 QoS ACL Based Classification and marking
Using QoS ACL on Cisco Catalyst switches for classification and marking is the most flexible method. QoS ACL is applied using the MQC format.
Before we begin the configuration part let´s see some facts about Catalyst 3560 QoS ACL:
- Class-maps can match on MAC access-list (for non IP-packets) or IP access-list (for IPv4 and IPv6 packets)
- Class-maps can match DSCP or IP precedence values but not CoS values in Ethernet Headers
- Under a class in the policy-map, you can trust CoS, DSCP and IP precedence and set DSCP or IP precedence (but not both at the same time, the one configured later erases the previous one). Note that you cannot set CoS so you will have to set it at the interface level.
- If a class under a policy-map contains trust and set, the trust statement takes precedence over explicit set
- Applying service-policy to an interface will remove any interface-level QoS settings on the interface with the exception of the default CoS which is used by the policy-map when you trust CoS
- If the input traffic flowing through the interface does not match any class under the policy-map, the switch will set all markings to zero.
Let´s define the following QoS policy in order to demonstrate how Catalyst 3560 QoS ACL Classification and Marking works:
1) Ping packets sent from R1 are already marked with DSCP 46 (ToS 184) and this marking should be trusted by SW1
2) Telnet Packets should be marked inbound at SW1 with DSCP of 24
3) SSH Packets should be marked inbound at SW1 with IP precedence of 2
4) Default CoS should be set to 1 and trusted in the class-default
Side note: QoS is enabled on SW1 and disabled on SW2 to keep this example as simple as possible
So let´s configure the following on SW1 to reflect this policy:
So let´s test now. Let´s test the Telnet traffic first:
Let´s see how the Telnet traffic is seen by SW1 on inbound:
We can conclude by looking at the output above that Telnet traffic is already marked by R1 with DSCP 48 (IP precedence of 6) as control-plane originated traffic is always marked with DSCP 48 by Cisco IOS as we saw before. Regarding the CoS value, we have a value of 1 set by SW1 as there is no CoS value present in the Telnet packet because R1 is connected to an access-port to SW1. So SW1 uses the default CoS configured on the interface which is in this case CoS 1 specified by our QoS policy.
Now let´s see if SW1 is marking Telnet packet with DSCP of 24 (CS3) when sending them over the trunk to SW2:
From the output above we can see that SW1 is marking Telnet packet with DSCP of 24 and a CoS of 3 which came from the DSCP to CoS mapping table.
We can also confirm that R2 is receiving Telnet packet marked with DSCP of 24:
Perfect! Let´s test SSH now:
So SW1 is effectively marking Telnet packets with DSCP of 16 (IP precedence of 2 -> 010).
Let´s now check on R2:
Perfect! So let´s try the last test with Ping packets. We are going to send 148 packets from R1 with a ToS of 184 (DSCP 46).
From the above output we can see that SW1 is receiving Ping traffic already marked with DSCP 46.
As we are trusting DSCP value for Ping packets, this value is left intact in the IP packets when sent over the Trunk link from SW1. However the CoS value is translated from 1 to 5 which is done using the DSCP to CoS map:
We can verify on R2:
Alright! That was it for the QoS classification/marking using QoS ACL on the Catalyst 3560.
Catalyst 3560 Per VLAN Based Classification and marking
There is a third way to classify/mark traffic on the Catalyst 3560 which is done per-VLAN.
Before we begin the configuration part let´s see some facts about Catalyst 3560 per-VLAN QoS:
- The policy-map is applied on the SVI interface
- Ports are enabled for per-VLAN QoS with the command: mls qos vlan-based and inherit all QoS configuration from the respective SVIs
To demonstrate this feature we will use a simple QoS policy. We will just say that Telnet traffic should be marked with DSCP of 24 for VLAN 12 (where R1 and R2 are located). So let´s configure the following on SW1:
So let´s telnet to R2 from R1 and check the QoS statistics on SW1:
From the output above we can see that SW1 is marking Telnet traffic (originated from R1 with DSCP of 48) with DSCP of 24. As QoS is disabled on SW2, R2 will see Telnet traffic marked with DSCP of 24.
If we were to send the same ping packets as we did before (ToS of 184 -> DSCP of 46) we will see the following result (we are sending 150 ping packets with ToS of 184 from R1 to R2) on SW1:
So here what is happening is that we are trusting CoS in the class-default where ping packets will end in. As there is no CoS value in the ping packets (access port), SW1 will use the default value from the interface F0/1 which is set to 1 in our example. Then SW1 uses the CoS to DSCP QoS map table to compute the internal DSCP value and rewrite it (rewrite from DSCP 46 to DSCP 8) in the IP packet header:
In this case a CoS of 1 corresponds to a DSCP of 8 which is what we are seeing in the previous output (SW1 marked ping packets with a DSCP of 8 and a CoS of 1 on outbound).
Perfect! So we have covered QoS classification and marking on the Catalyst 3560. Let´s know see which methods we can use on this platform to achieve traffic policing.
Catalyst 3560 Port-Based Policing and marking
Along with classification in the policy-maps we saw in our example earlier, it is possible to set policer parameters in order to control the rate of input traffic.
Before we begin the configuration part let´s see some facts about Catalyst 3560 Port-Based Policing and marking:
- Policer applies only to traffic matching the respective class
- 2 options in the policer: mark/remark or drop
- Markdown DSCP is based on the global mapping table
- Per-port policy applied on a Trunk will applies to all VLANs at the same time
- Policing on the Catalyst switches uses ASICs which allow policing at high rates but results in some limitations imposed on rate and burst sizes
- Remarking is only possible on DSCP values
- Markdown settings are configured globally using the policed-dscp mapping table
- The original DSCP value used for markdown is the value from the QoS label and not the packet itself which means that the switch will pick up the DSCP value based on the set or trust statement
- Policing only apply to transit traffic (hardware switched) so traffic destined to the switch itself going up to the CPU will not be affected
So let´s configure the following policy to demonstrate this feature:
1) Ping traffic should be marked with a DSCP of 16 for conforming traffic and if this traffic exceed 256 Kbit/s it should be remarked to CS1 (DSCP 8)
2) All other traffic should have DSCP value set to DSCP 16 and police to 256 Kbit/s. Exceeding traffic should be dropped
Let´s configure the following to reflect this QoS policy:
Let ´s test by generating Ping traffic from R1 to R2 with ToS of 184 (DSCP 46) at a high repeat count:
So let´s check first the QoS statistics on SW1 for traffic leaving the Trunk link:
So from the output above we can see that 18940 packets of size 1000 Bytes and being marked with DSCP 16 are conforming and 101324 packets of size 1000 Bytes are exceeding and remarked with DSCP 8.
On R2 we can see that almost 256 Kbit/s is conforming and that 812 Kbit/s is exceeding:
To test the second requirement, we just generate HTTP traffic between R1 and R2 by using the following method:
Let´s check QoS statistics on SW1:
So SW1 is marking HTTP traffic with DSCP of 16 as expected. Let´s see if R2 is receiving almost 256 Kbit/s of HTTP traffic as exceeding traffic is dropped by SW1:
So we are almost approaching the desired rate but as it’s averaged over 5 min we should wait longer to get closer to 256 Kbit/s
Perfect! Let´s see another method for doing traffic policing on the Catalyst 3560.
Catalyst 3560 Per-Port Per-VLAN Policing and marking
As we saw before we can do classification and marking per-VLAN instead of per-port. Well we can also add policing to the Per-VLAN classification.
Before we begin the configuration part let´s see some facts about Catalyst 3560 Per-Port Per-VLAN Policing and marking:
- You can use single-level or two-level policy maps
- You cannot do policing in the first level policy map so VLAN-wide aggregate policing is not possible in the 3560 but it is possible to limit the amount of traffic entering the VLAN on per-port basis
- First level policy-map can only mark
- Second level policy can only police and cannot be applied under the class-default
To demonstrate how this feature works let´s use the same QoS policy as before:
1) Ping traffic should be marked with a DSCP of 16 for conforming traffic and if this traffic exceed 256 Kbit/s it should be remarked to CS1 (DSCP 8)
This time we are only focusing on ICMP traffic as it is not possible to do policing in the first level policy-map. Let´s configure the following to reflect the policy above:
Let´s test this as we did before with Ping traffic generated from R1 to R2 with ToS of 184 (DSCP 46).
So let´s look at SW1 QoS statistics for DSCP values:
Basically we get the same result as before when we were doing per-port policing but by using per-port per-VLAN QoS policing we decide which port in a specific VLAN is going to be affected by the policy with the command mls qos vlan-based and by matching on specific interface (up to six ports) with the command match input-interface specified under the class-map.
There is a last feature used by Catalyst switches for QoS classification/marking and policing which is called aggregate policers. I will not demonstrate how this feature works here as the concept is really similar to the configuration examples we have seen so far. Basically you define an aggregate policer with the command: mls qos aggregate-policer <NAME> <POLICER_RATE> <BURST_SIZE> | exceed-action policed-dscp-transmit
Then you can call this aggregate policer in the different class-maps defined under your policy-map. The policy-map is applied directly to the port of your choice. All classes configured with the same aggregate policer share the configured rate.
Alright! In the next section we will see how the different queuing mechanisms work on the Catalyst 3560 platform.
Catalyst 3560 Ingress queuing
There is no easy way to verify the actual working of ingress queues unless you really congest the switch so for this part we will just do some show commands.
Let´s see some important facts about ingress queuing on the Catalyst 3560:
- The Catalyst 3560 has 2 ingress queues
- Queuing is occurring after classification and marking if the internal backplane experiences congestion.
- Ingress queuing uses the QoS label assigned to a packet in order to decide in which queue the packet should be queued.
- Ingress queuing uses 2 mapping tables (dscp-input and CoS-input) in order to resolve the QoS label to queue mapping
- Configuration of ingress queuing is global
- Queue weights define the logic of input packet scheduler which is called Shared Round Robin or SRR
- One of the two queues can be configured as priority queue. Priority queue is defined with a bandwidth threshold which correspond to how much percent of the internal ring bandwidth is available to the priority queue. Priority queue is served in exclusive mode up to the max threshold defined
- Exceeding packet in the priority queue (that is to say, going over the threshold) are not rate limited but are subject to SRR using the respective weight configured on both queues
- If any queue does not use its minimum guarantee bandwidth, the other queue may use the remaining bandwidth
To get on overview of the input queue configuration we use the following show command:
So per default the input queues on the Catalyst C3560 are configured as shown in the output above.
Priority queue: In this case queue 2 is configured as priority queue and will get 10 % of the internal ring bandwidth.
SRR weights: The SRR weights are 90 and 10 which means that traffic is shared in a 90:10 ratio between the exceeding packets from queue 2 (Configured as priority queue) and all packets from queue 1
Buffer space: The buffer value means how much buffer space is allocated to each queue. Per default queue 1 gets 90 % and queue 2 gets 10 %. Both queues share the same buffer space.
Drop thresholds: The Catalyst 3560 has 3 drop thresholds per queue. The third one cannot be configured and it is set to 100 % of the queue size. In order to map DSCP and CoS values to a threshold use the following command:
Here for example we are mapping CoS 6 (IP routing) to queue 2 threshold 1. To change a drop threshold value for a queue use the following command:
Here we are changing the threshold 1 and 2 for queue 2 to be 80 % and 100 %.
In the above output we can see that the threshold have been changed for queue 2. That means that when queue 2 will be at 80 % of its size capacity routing packets will start to be dropped as we configure routing traffic to be mapped to threshold 1. Finally we would need to map routing traffic (CoS 6) to queue 2:
We can then verify that CoS 6 is mapped to queue 2 threshold 1:
Alright, let´s move on to egress queuing.
Catalyst 3560 egress queuing
Before we begin the configuration part let´s see some important facts about Catalyst 3560 egress queuing:
- Catalyst 3560 has 4 egress queues which use the SRR scheduler.
- 2 modes of operation: Shared Round Robin (same as ingress queue) and Shaped Round Robin.
- Each of the 4 queues can be configured in either mode
- When using shaped mode you assign an absolute weight to a queue and the SRR scheduler limits the queue sending rate to 1/(weight)*interface speed where interface speed corresponds to the speed sets with the speed command. The rate is guaranteed up to the limit configured and the SRR scheduler delays the exceeding traffic
- Shaped traffic does not use more than the allocated bandwidth even if the link is idle
- All queues not configured for shape mode operate in share mode. If the shaped weight for a queue is 0, this queue operate in shared mode
- Shape mode takes precedence over share mode. That is to say, if a queue is configured with a weight in shaped and shared mode, shape mode will be used with the weight configured and the shared weight will be ignored
- Shape queues are served up to their limit and then shared queues share the remaining bandwidth proportional to their configured weight
- One queue can be configured as priority queue. Per default is queue 1
- Priority queue has unlimited bandwidth and may starve other queues. Shared and shaped weights set for the priority queue are ignored
- Global egress sending rate limit can be configured per port to a value between 10% and 100% of its physical rate
Alright! So let´s configure the following scenario in order to test egress queuing on the Catalyst 3560. We will test the shared mode first:
- The port connected to R2 on SW2 should limit outbound bandwidth to 2 Mbit/s
- Only shared mode should be used
- Bandwidth between queue 3 and 4 should be shared in the ratio 50:50 respectively
- DSCP 0 should be mapped to queue 4
- Priority queuing should be enabled
To simulate this scenario we are going to generate 3 flows towards R2 with the following DSCP values:
- Flow 1: DSCP of EF. This flow will simulate voice traffic
- Flow 2 : DSCP of AF21
- Flow 3: DSCP of 0
So let´s configure the following on SW2 to reflect the above scenario:
The speed of the port connected to R2 is changed to 10 Mbit in order to be able to limit to 2 Mbit/s the outbound traffic (20% of 10). All queues are disabled for shape mode (all 0) and queue 1 is the priority queue (default). Queue 3 and 4 get allocated the share ratio according the QoS policy. Finally DSCP 0 is mapped to queue 4 (default is mapped to queue 2). EF and AF21 are mapped to queue 1 and queue 3 per default:
Side note: QoS has been enabled on SW2 and has been disabled on SW1. F0/13 on SW2 is configured to trust DSCP for all traffic coming from R1 and the CoS values will by calculated accordingly by the switch using DSCP to CoS map.
So let´s generate the 3 flows and see if we achieve the desired results:
So the 3 flows are seen by R2 with the respective DSCP values. Note that even if there is congestion, voice traffic is getting 256 Kbps and DSCP 0 and DSCP 18 share the remaining bandwidth in a 50:50 ratio which is seen in the output 615Kbps vs 636 Kbps. Note that we are not achieving the total sum of 2Mbit/s and I guess that is due to the lack of precision of hardware QoS done on the Catalyst 3560.
To illustrate how shaping mode is working we could configure the following on SW2:
That means that traffic getting in queue 4 will be shaped to 400 Kbps approximately (1/25 * 10 =0,4 Mbit/s). In our example packets with DSCP 0 are mapped to queue 4. So let´s see the result on R2:
The result is really approximate but DSCP 0 traffic is shaped down to 253 Kbps.
Catalyst 3560 egress queuing tuning
The last part of this post will cover buffer allocation for each egress queue as well as drop thresholds.
Each interface has its own buffer space which is partitioned between the different queues using the following command:
By default all four queues get an equal buffer space. Here note that in order to configure the buffer space for each queue the Catalyst uses templates called queue-sets. There are 2 templates (2 queue-sets) and queue set 1 by default is applies to all interfaces.
Queue-sets are also used to configure the different drop thresholds for each queue. There are 3 drop thresholds per queue like with the input queues and the third threshold cannot be configured and corresponds to queue-full threshold. Drop thresholds are set with the following command:
The first value (50) corresponds to the first drop threshold and the second value (100) corresponds to drop threshold number 2. Here we set the drop thresholds for queue-set 1 and for queue 1. Drops thresholds are specified in percent.
The last thresholds that you can configure in a queue-set are the reserved threshold and the maximum threshold. The reserved threshold defines the amount of buffer space the system guarantee to a queue. For example if the buffer space allocated to a queue is 25 % of the reservable pool of 1000 buffers, the queue will have 250 buffers allocated. Now if you define the reserved threshold to be 50 % the system only guarantee 125 buffers to the queue. The remaining 125 buffers are allocated to a common pool shared by the other queues.
The reservable threshold may vary between 0 and 100 %.
The maximum threshold specifies the maximum queue size in buffers taking in account that the queue can grow above its initial allocated buffer space using the concept of the common pool. So it is possible to allocate to a queue a maximum threshold between 0% and 400 %. When above 100 % the queue will borrow extra buffers from the common pool if any available. So a queue could grow to 4 times its size.
To configure the reserved threshold and maximum threshold we use the same command as before. We define first the drop thresholds and then the reserved thresholds:
So here we are defining the following for queue-set ,1 queue 1:
- Drop threshold 1: 50 % of the allocated buffer space to the queue
- Drop threshold 2: 100 % of the allocated buffer space to the queue
- Reserved threshold: 100 % of the allocated buffer space to the queue
- Maximum threshold: 300 % of the allocated buffer space to the queue
The last step of egress queue tuning is to allocate the DSCP/CoS values to the different drop thresholds. So if we were to allocate DSCP 0 traffic to drop threshold 1, DSCP 24 to drop threshold 2 and voice traffic to threshold 3 we will be using the following commands:
Then, for the final verification we first check queue-set 1 that we have modified for queue 1:
And finally we check the DSCP to drop threshold table:
Regarding Catalyst QoS on 3560 there is a last feature which I will not develop here and it is called QoS DSCP mutation. You basically map ingress DSCP value received on a port to other DSCP values. The only requirement is that you trust DSCP on the inbound interface. The command as the following format:
Thanks for reading!