MPLS Layer 3 VPNs Troubleshooting

I would like to dedicate this post to MPLS L3 VPNs troubleshooting and more particularly using the Traceroute command. It can be sometimes difficult to find out where is the issue when testing connectivity between sites attaches to a MPLS VPN backbone. I will explain the behavior of Traceroute in MPLS VPN environment which is quite different than in “classical” IP environment. Finally I will talk about the MPLS LSP Ping feature and how to use it to detect break in MPLS LSP.

Topology:

 

I am using almost the same topology as the one used in this post: Basic MPLS. I have just added a P router.

 Platform/IOS:

 Cisco 2691/12.4(15)T11 Adv IP services

 First of all if you want to get a basic understanding of MPLS and MPLS Layer 3 VPNs I suggest to read my previous posts: Basic MPLS and MPLS Layer 3 VPNs

 The following is already configured:

• MP-iBGP between R2 and R5
• EBGP between R2 and R1, R5 and R6
• OSPF area 0 between R2,R3,R4,R5
• LDP between R2,R3,R4,R5
• R3 and R4 are P routers and R2 and R5 are PE routers
• R1 is representing the CE device at Site A announcing its loopback address 1.1.1.1/32
• R2 is representing the CE device at Site B announcing its loopback address 6.6.6.6/32

 In this troubleshooting scenario we want to focus on the loopbacks reachability between Site A and Site B.

 Let´s start with some basic verification of the control plane.

Control plane in the MPLS VPN core

• EBGP between R2 and R1:

So the EBGP session between R2 and R1 is established and R2 is receiving the following prefix from R1:

• MP-iBGP session between R2 and R5:

The MP-iBGP session is established between the two PEs and R2 is receiving a VPNv4 prefix to R2 corresponding to the loopback IP of R6.

• LDP session on R2

The LDP session between R2 and R3 is established and the output is telling that F0/1 on R2 is transmitting and Receiving LDP Hellos. Let´s check if R2 is receiving a label for 5.5.5.5 which will be the BGP next-hop of all the prefixes announced by R5 to R2 (Both PEs are peering using their loopbacks):

The output above is showing the LFIB (Label Forwarding Table) of R2 used to switch labeled packets. In this case R2 has received a label of 16 for 5.5.5.5/32 from its upstream LDP neighbor R3.

Let´s now verify the EBGP session between R5 and R6.

• EBGP between R5 and R6:

We can also check which prefix R5 is receiving from R6:

Data plane in the MPLS VPN core

Let´s verify now the data plane in the MPLS VPN core

• LSP between R2 and R5 for LDP FEC 5.5.5.5/32

We can now verify if the LSP (Label-Switched Path) is end to end and not broken between R2 and R5 somewhere in the core for the FEC (Forwarding Equivalent Class) 5.5.5.5/32. A FEC is represented in MPLS by a label. In an MPLS network the FEC is determined only once, at the ingress to an LSP, rather than at every router hop along the path as it will be the case with generic IP forwarding. A FEC is a set of packets that a single router forwards to the same next-hop, have same interface and same treatment (queuing, QoS, etc). Let´s test the LSP for 5.5.5.5/32:

Good, we know that the LSP is not broken for 5.5.5.5/32 (which will be the next-hop used by R2 to reach the loopback of R6).

Site note: We can tell that the LSP is not broken as we can see the label being used. Actually we will see later on in this post that the LSP could be broken even with the output above showing the labels. If the LSP was broken somewhere in the LSP path, the routers will use the FIB to make a lookup on the destination IP. This will result in a successful Traceroute but the labels will not be present in the output. That is find if we are using Traceroute in the MPLS VPN core but it gets more complicated to troubleshoot MPLS VPN when using Traceroute from a CE site as we will see later in this topic.

Let´s see what is happening when we are doing the Traceroute from R2 to R5:

1. R2 send a first UDP packet with destination port 33434, destination IP 5.5.5.5 TTL of 1. R2 make a FIB lookup for 5.5.5.5 and sees that the packets must be labeled with 16 and send towards 10.0.23.3 out interface F0/1. The TTL value of 1 is copied into the MPLS header:
2. When this UDP packet reach R3, R3 decrement the TTL found in the MPLS header from 1 to 0, R3 drops the packet and generate a an ICMP type 11 (Time-To-Live exceed), code 0 destined to the originating router, in this case R2 with IP 10.0.23.2. The ICMP packet sends by R3 includes the header of the original packet in the response as well as the MPLS extensions (MPLS extensions for ICMP in order to track the label used through the LSP) as we can see inthe following Wireshark output :

3. Next R2 sends another probe (by default 3 probes per TTL, however here we have configured a probe of 1 for simplicity) with a TTL of 2 and an UDP port incremented by 1 (33435).

4. R3 receives the new UDP probe with a TTL of 2, decrement it to a TTL of 1 and swap the MPLS packet using its LFIB:

5. Then R4 receive the UDP packet with a TTL 1. The packet is dropped and an ICMP Time-To-Live exceed is generated towards R2. As before  the original header as well as the MPLS ICMP extension are encapsulated in the ICMP packet. Note that this packet has a default TTL.

6. Finally, R2 sends a probe with a TTL 3 which reach R5 and this time, no ICMP Time-To-Live exceed are generated but R5 generate an ICMP Type 3 (Destination Unreachable) code 3 (Port unreachable) as it has no UDP service listening on this port (33436):

• LSP between R5 and R2 for LDP FEC 2.2.2.2/32

So the LSP is not broken as we can see the labels being used.

 Side note: step back! Actually the LSP could be broken on R3 if for example OSPF and LDP were out of sync with each other (loopback of R2 configured as /24 but advertised by R2 OSPF process as /32 and advertised as /24 by R2 LDP process) which will result in a broken LSP on R3. R3 will not be able to find a label for R2´s loopback and will mark the FEC 2.2.2.2/32 as “untagged”.

 Let´s now troubleshooth the control plane at the CE sites.

 Control plane at CE sites

 Let´s verify if Site A receive the loopback prefix of R6:

Perfect!

Let´s do the same for Site B:

 It looks good at both sites. Let´s verify now the final step which is the end-to-end connectivity between the CE sites.

Data plane at CE sites

 Until now, everything looks fine, there is no LSP broken in the MPLS VPN core for the respective VPNv4 next-hops and each CE site has received the correct prefixes. Let´s try to Traceroute from Site B to Site A so:

 Not looking good! The Traceroute is dying. Now image you are in the real TS lab; you have been through all the verification steps as we did and you were almost sure that it would work! Then you can already feel the pressure as you have used 10 min already making these verification steps.

That is why with classic Traceroute used in MPLS environement it can be hard to troubleshoot. When the traceroute operation fails, you cannot tell whether it’s due to breaking of LSP path in the core or breaking of the classic layer 3 path within the customer site. In our case, however we know that there is no break in the LSP as we have verified that before but it can be quite hard to determine the cause of the issue if you didn’t verify the LSP in the core in the first place.

So now how can we proceed and troubleshoot this issue? Well we could try to Traceroute from R5 instead:

Not good either! What can we do? Well we could debug on mpls packets on the P routers to see if there are receiving the labeled packets while the Traceroute is running on R5:

It looks good on R4. We can see that the transport label (18) and the VPN label (21) are present in the received traffic.

Same case for R3 apart from the fact his “popping” the transport label before transmitting the MPLS packet to R2. Let´s see what R2 is receiving:

R2 is receiving the labeled packet, does a VRF lookup based on the VPN label and transmit the packet out interface F0/0.

Let´s see now what R1 is receiving:

So R1 is receiving the UDP packet from R5 and send an ICMP port unreachable to R5. Let´s do a debug ip packet:

Here we can clearly see what the issue is. R1 and R6 only knows each other loopbacks IP and nothing else. The problem was that we didn´t source the Traceroute on R6 from its loopback so you have to be really careful when troubleshooting especially under pressure.

Now it is working when using the correct source IP address.

 The problem also in this kind of environment (MPLS VPN) is to know exactly how Traceroute is working. In a generic IP environment we would have detect the failure straight away when using Traceroute, where the Traceroute fails we know that the next-hop after the successful probe as an issue with routing. But here in MPLS VPN it is harder as the Traceroute behaves differently. So I would like to explain in detail the Traceroute behavior when doing a Traceroute from Site B to Site A for example.

Traceroute explained from a CE site to another CE site

 Let´s illustrate the explanation by using the following output:

When facing this kind of output it can be quite hard to see what is happening behind the scene. So let´s detailed what is happening:

1. First R6 sends an UDP packet with TTL of 1 toward R5

2. R5 decrement the TTL, drop the packet and send an ÌCMP Time-To-Live exceed back to R6. R6 deduce then that R5 is one hop away due to the UDP port number (33434) encapsulated in the ICMP packet sent by R5.

3. R6 sends a second probe with TTL of 2, R5 receives it, does a FIB lookup in the VRF and sees that the packet should be labeled with 2 labels (transport+VPN) and send it out interface F0/0 towards R4. The TTL is decrement to 1 and copied in the topmost label header:

4. R4 receives the MPLS packet, decrement the TTL and drop the packet which trigger the following Traceroute process special to MPLS environment:

a)    R4 has no idea about the source IP which is 10.2.56.6 (R6). This IP is not known by the P routers as it is a Customer prefix. Traceroute implementation has been modified for MPLS (not real MPLS Traceroute, we will see it later) and when R4 generates the ICMP packet it sets the destination IP = source IP found in original packet and sets the source IP=IP of interface where the probe was received (in this case R4 F0/1). Then as we can see in the Wireshark output bellow R4 encapsulates the original UDP packet with the MPLS extensions for ICMP containing the label stack. Finally R4 swaps the MPLS packet based on the LFIB and sends it to R3 with a TTL of 255. This process of setting the destination IP to the source IP is done in order for the last PE in the LSP to reflect the probe back to R6.

b)    So the ICMP packet is moving away from the source, going downstream towards R3. This is the opposite of the classical Traceroute behavior and this because only the PE routers have routing knowledge about the CE prefixes. So now the ICMP packet encapsulated in MPLS is reaching R3. But as this packet as a TTL of 255 (sets by R4), R3 just decrement the TTL by 1, and swaps the MPLS packet. When R3 makes the LFIB lookup, it removes the transport label 18 and forwards the MPLS packet with one label only (21, VPN label) to R2.

c)    R2, receives the MPLS packet, make a LFIB lookup on the VPN label 21 which in turn triggers a FIB lookup for the VRF of SITEA. So R2 makes the following FIB lookup on the destination IP:

d)    The issue is that R2 does not know the 10.2.56.0/24 prefix so it cannot reply back to R6:

Site note: R2 is using an IP address in the VRF of Site A when sourcing the ICMP Time-To-Live exceed destined to R6 but it is not able to find an outgoing interface for the prefix and the routing process fails.

 5. R6 sends another probe with a TTL of 3 and so on. Then the probes will finally reach R1. Like R2, R1 doesn´t know 10.2.56.0/24 so the packet is dropped and R1 generate an ICMP type 3 (Destination unreachable) code 3 (port unreachable) towards R6. But the same problem occurs again as R1 does not have any IP reachability to R6.

Now if we were to source the Traceroute on R6 from its loopback address the result will be different:

So now that the Traceroute is successful let´s move on to point 4,C when R2 is receiving the MPLS packet encapsulating the ICMP Time-To-Live exceed generated by R4. So R2 is receiving the following MPLS packet from R4:

As before, R2 does a FIB lookup into the VRF associated with Site A for the destination prefix 6.6.6.6/32 and sends back the packet towards R5 (PE).

Side note: This Traceroute behavior in MPLS VPN is like a “boomerang effect” where the last PE of the LSP is catching the ICMP packets generated from the P routers in the MPLS VPN core and reflects them back to the source of the Traceroute located at the CE site.

So now R2 is reflecting the ICMP Time-To-Live exceeded packet from R4 back towards R5 as we can see in the following Wireshark output:

The output above is almost the same as the previous one (when R2 received the packet from R4). The difference here is that the packet is now going upstream towards R5 (endpoint of the LSP). Also we can tell that this packet is going upstream (away from R2) as it is encapsulated with a transport and VPN label.

The packet will be swapped by the P routers and finally reach R5 which will forward it to R6. When R6 receives the packet it will see that the probe 33435  associated with R4 IP 10.0.45.4 and the MPLS label stack {18,21} attached (MPLS ICMP extension) to the original UDP packet sent by R6. Then the Traceroute application on R6 will deduce the following output (highlight med yellow):

The same process occurs for the other probes.

Side note: Note that the label stack {18,21} reported by R4 in this case in the one received by R4 from R5 and not the one used by R4 when swapping the packet towards R3. That means that R5 is encapsulating the UDP packet with a label stack of {18, 21}, R4 with {16,21} and R3 with {21}, popping the transport label to avoid multiple lookup on R2.

Broken LSP

 Let´s now imagine that the LSP is broken somewhere in the MPLS network. We first check the connectivity to the LSP endpoint (R2) from R5 with a ping:

Good, there is IP connectivity to R2. Now we try with a Traceroute:

Good, it looks as before and we can also see the labels used along the LSP. We can now check on R5 if it as a VPNv4 prefix for Site A:

And it does! We can also see the VPN label (21) which will be used by R5 when encapsulating IP packets into MPLS destined to Site A. So everything looks good. Let´s try if you have IP connectivity from Site B to Site A now:

The ping and the Traceroute die after R5. If it was a big MPLS network (not just a small MPLS network of 4 routers) it could be hard to troubleshoot and find out where is the issue. So it is not possible to detect an LSP breaking in the MPLS network with this a “classic” Traceroute.

 Side note: Of course, one could tell that there is a break in the LSP if looking carefully at the Traceroute output and check if all routers are using the MPLS label stack to forward packets. But in our case we cannot see any issue in the Traceroute output as all the routers in the LSP are reporting labels.

 That is when MPLS LSP Ping could be useful when troubleshooting MPLS.

MPLS LSP Ping 

I will not get into the details on how MPLS LSP Ping works and if you want you can read more about it here in the RFC. Basically it is an improvement to usual ping and Traceroute adapted to MPLS environment in order to detect failure in the data plane.

 With this utility you are testing connectivity to a FEC so you can only use this feature on a LSR. Cisco only supports testing of FEC that have been learned via LDP so you cannot test it on VPNv4 prefixes. The Traceroute feature for MPLS (MPLS LSP Ping) is quite different from the “classical” Traceroute as it has much more information into the payload and the destination IP address is a reserved loopback in the range 127/8 with a UDP port of 3503. Let´s try it:

So what we are trying to find out with this command is where the LSP is broken. In the output above we can clearly see that the LSP is breaking on R3 as the result output says that R3 has no label for this FEC. Let´s have a look at a Wireshark capture:

So we can see the relevant information that are included in the MPLS LSP Ping payload, including the targeted FEC we are trying to test (2.2.2.2/32), the downstream LDP neighbor of R5 and also which label is R5 using for this FEC (Label 20).

 Side note: The goal of MPLS Traceroute (MPLS LSP ping) is to detect failure in the data plane in an MPLS environment. So the packets are only routed based on the label stack (using the LFIB) and never using the FIB which prevents the case we saw with “classic” Traceroute” before where the packets where routed using the FIB (based on IP) although there was a break in the LSP. Also the destination IP of 127/8 prevents the LSR in the LSP to forward the MPLS echo packet further downstream. The result of MPLS Traceroute is that if there is a break in the LSP the LSR who detects the break will report it to the originate LSR of the MPLS echo packet and the packet will not be sent further downstream in the MPLS network.

 Back to R5. We can see in the MPLS Traceroute output that R3 is reporting to R5 that it hasn´t any label for this FEC by reporting in the MPLS Echo Reply an error code of 9 (see RFC) meaning that is no label for the given FEC and that is because R3 LIB is out of sync with OSPF as we have configured the loopback IP of R2 as /24:

Let´s correct this and MPLS Traceroute again:

Now we see that the MPLS echo request is successful for all the LSR in the LSP and that there is no break in the LSP. The MPLS LSP Ping can be a really powerful troubleshooting tool when troubleshooting data plane failure in MPLS networks.

Side note: Make sure that MPLS LSP Ping is supported on the IOS you are using.

That was pretty much it for this post. The goal of this post was to get a better understanding on how to troubleshoot MPLS VPN Layer 3 with the “classic” Traceroute command but also to understand how this command is working. Finally I wanted to explain the MPLS LSP Ping feature and how it could be useful for finding break in MPLS LSP.

Thanks for reading.

/Laurent