Category Archives: route

Multiple next-hops in policy routing

Imagine the following topology with routers 1 to 5:

         / (2) \
(4) - (1) (5)
         \ (3) /

There is Policy-based routing active on router 1, and the possible end-to-end paths are two:
4-1-2-5
4-1-3-5

Let us play around with the next-hop settings of PBR. Imagine we have a rudimentary PBR setup with the following:

route-map test permit 10
match ip address 100
set ip next-hop 10.0.12.2 10.0.13.3

10.0.12.2 is router 2 and 10.0.12.3 is router 3. We really don’t care about the access lists or anything else, let’s see what happens when we have multiple hops defined:

R1#debug ip policy
Policy routing debugging is on
R1#
*Jul 17 12:28:42.387: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, len 44, FIB policy match
*Jul 17 12:28:42.387: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, g=10.0.12.2, len 44, FIB policy routed
*Jul 17 12:28:42.447: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, len 40, FIB policy match
*Jul 17 12:28:42.447: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, g=10.0.12.2, len 40, FIB policy routed
*Jul 17 12:28:42.459: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, len 49, FIB policy match
*Jul 17 12:28:42.459: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, g=10.0.12.2, len 49, FIB policy routed
*Jul 17 12:28:42.467: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, len 40, FIB policy match
*Jul 17 12:28:42.467: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, g=10.0.12.2, len 40, FIB policy routed

There, so it keeps using the first next-hop no matter what. I guess the only way to nudge it to use the other one is to shutdown the interface connected to R2. Look what happens after I shut down that interface:

 *Jul 17 12:32:09.551: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, len 44, FIB policy match
*Jul 17 12:32:09.551: CEF-IP-POLICY: fib for address 10.0.12.2 is with flag 257
*Jul 17 12:32:09.551: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, g=10.0.13.3, len 44, FIB policy routed
*Jul 17 12:32:09.611: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, len 40, FIB policy match
*Jul 17 12:32:09.611: CEF-IP-POLICY: fib for address 10.0.12.2 is with flag 257
*Jul 17 12:32:09.611: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, g=10.0.13.3, len 40, FIB policy routed
*Jul 17 12:32:09.619: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, len 49, FIB policy match
*Jul 17 12:32:09.619: CEF-IP-POLICY: fib for address 10.0.12.2 is with flag 257
*Jul 17 12:32:09.619: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, g=10.0.13.3, len 49, FIB policy routed

The process sees that the route towards its next-hop is marked as down (flag 257) in the CEF tables (or non-existent), and goes on towards the next one.

Conclusion: Multiple next-hops in PBR are used for redundancy, not load-sharing/balancing.

Advertisements

Unequal Load-Balancing on Cisco IOS

I just wanted to share a neat trick that a fellow CCIE colleague showed me.

In case of being connected to two ISPs, there is a way of doing unequal load-balancing with the help of static routes. For example, ISP X provides you with 25Mbps, and ISP Y with 50Mbps – a 2:1 ratio.

In order to achieve any kind of load-balancing on the Cisco IOS, we need multiple entries in the routing table, pointing to the same specific destination. As we would like to load-balance our uplink traffic towards the internet, we would need multiple entries towards our default gateways.

We are all familiar with the concept that there can be only one default route for a specific gateway – you can’t have multiple routing entries pointing to the same default gateway. That means that if we have multiple ISPs and multiple default gateways, our load-balance ratio will always be 1:1, as there is just a single entry in the routing table for each default gateway.

However, we can install multiple routing entries for seemingly different default gateways. That way, we can fool the device and have the same default gateway listed multiple times in the routing table. Ok, it sounds confusing, but just take a look at the configuration and it’ll become clear.

ip route 10.0.1.1 255.255.255.255 192.168.1.2  #(ISP X)
ip route 10.0.2.1 255.255.255.255 172.16.1.2  #(ISP Y)
ip route 10.0.2.2 255.255.255.255 172.16.1.2  #(ISP Y)

ip route 0.0.0.0 0.0.0.0 10.0.1.1
ip route 0.0.0.0 0.0.0.0 10.0.2.1
ip route 0.0.0.0 0.0.0.0 10.0.2.2

First, we define static routes for a couple of fake default gateways. Those IPs do not exist, and will only be used for the current load-ballancing trick, so be careful when setting up those and try not to assign some IPs in use.

After that, we define these fake IPs as default gateways. Having in mind that the ratio of the link bandwidth is 2:1, we created two routes towards the faster ISP and a single route towards the slower ISP.

What happens is the IOS uses all three of these default gateways, because the destination is seemingly different during the first look up in the routing table. The second look up will reveal that the fake default gateway’s IP is reachable only by either ISP X or ISP Y’s next-hop router. This is quite the ingenious way of tricking the device into installing multiple entries in its routing table.

Dynamic WAN failover with IP SLA ICMP test on HP Networking

Recently I had to make use of the SLA service in order for a border router to choose between two identical service providers’ uplinks for backup reasons. The goal was to ping a specific IP, and should the ping fail (N amount of times), switch to the other provider.

The setup consisted of the HP MSR920 Router, and two uplinks to the ISPs. In case any of you are wondering, this device is a rebranded H3C MSR920.

For the sake of simplicity, let’s say that we’re connected to ISP1 via gateway 192.168.1.1 and interface eth0/0, and to ISP2 via gateway 192.168.2.1 and interface eth0/1.

First, we have to create the specific Network Quality Analyzer (NQA) entries, which resemble profiles of test and actions to be taken. You can read more about it here.

nqa entry admin one
type icmp-echo
destination ip 192.168.1.1
frequency 1500
history-record enable
reaction 1 checked-element probe-fail threshold-type consecutive 1 action-type trigger-only
source interface Ethernet0/0
ttl 255

nqa entry admin two
type icmp-echo
destination ip 192.168.2.1
frequency 1500
history-record enable
probe timeout 500
reaction 1 checked-element probe-fail threshold-type consecutive 1 action-type trigger-only
source interface Ethernet0/1
ttl 255

While it seems pretty self-explanatory, there are some pitfalls in this configuration. First, the entries are categorized by 2 strings, with the first being the admin entry, and the second the name entry. This way, admin John may have test called ISP1Ping, and admin Jack may also have a test called ISP1Ping. It seems to me that it’s a bit of overkill to use a two-dimensional array for the tests, but anyway. In my case, there is a single admin, so we’ll name the first parameter just “admin”, and the second – “one” and “two”.

There can be multiple types of tests: UDP echo, UDP jitter, UDP, VoIP, TCP, HTTP, FTP (some of them need a special NQA server at the other end). We will just use the ICMP-Echo test. We set up the destination IP, the frequency between pings in milliseconds, and we enable keeping history of the probes’ success. It is a good idea to measure the average time pings take to obtain a reply from your test verification destination, and then make the test’s icmp timeout value to be 1.3 or 1.5 times the needed trip time. There is no need to wait 2 seconds for an icmp timeout, if we know the trip takes only 20msec. On the second ISP, we set the timeout to 500msec, as we know that the icmp echo test takes 100msec for a reply.

(WARNING) The reaction command was stripped in the current version of the device, so you could only trigger the reaction, or not (effectively disabling the whole test scenario). This is important, as if there is no trigger-only statement, your tests will not alert the so-called tracker, that I’ll be explaining about in a minute.

We source the test from the specific interface for the current ISP.
(WARNING) We also set the TTL to maximum value. The default value on the current device was 20. As you notice, sometimes that might just not be enough to reach your testing destination.

track 1 nqa entry admin one reaction 1
track 2 nqa entry admin two reaction 1

Now we piece together the tests with the respective reaction. Notice that there can be multiple reactions defined for a test, and you can choose which one to associate with the tracker. It’s time to associate the trackers with something.

ip route-static 0.0.0.0 0.0.0.0 192.168.1.1 track 1 preference 150
ip route-static 0.0.0.0 0.0.0.0 192.168.2.1 track 2 preference 100

As you can see, we define two static default routes to both ISPs. We associate each route with its respective track entry. The preference value is used to prefer one router over the other – for example, if ISP1 is a fast fiber optic, while ISP2 runs on a backup ADSL line. This way, if ISP1 goes down, when it comes back up – it’ll override the default route to ISP2. Just remember that the lower the preference metric – the better, with 60 being the default metric for static routes.

nqa schedule admin one start-time now lifetime forever
nqa schedule admin two start-time now lifetime forever

We schedule our test to run as soon as we enter the commands (or as soon as the device starts) and to last forever. If you need to use the fail-over on specific times of day – you can define them here.

What have we achieved so far:

  • We have two default routes that are tracked by the NQA. One of them has a higher preference, and gets installed in the routing table by default.
  • As soon as the ICMP test fail, the NQA alerts the specific tracker, which brings down the failing default route.
  • As soon as the ICMP test succeeds, the NQA alerts the specific tracker, which brings up the failed default route.
  • If both routes are up, the one with the bigger preference is used
  • However, what if both routes are down?

    • None of the static default routes is present in the routing table
    • The device keeps on testing the links as we have explicitly set the output interfaces

  • Whichever ping test succeeds first, it will bring up a static route with a higher preference that any automatic default route. We have restored preferred connectivity.

That’s about all for the ICMP Echo part of HP’s Network Quality Analyser (or should we say H3C’s NQA?). Some complex scenarios can be created, but they do require a special NQA responder on the other end of the testing. Hopefully, I’ll cover them in the future. Thanks for reading, and have fun with networking!

Edit (Apr/2013): Corrected static route preference metrics thanks to Alisson Curunzi

/24 bit masks for loopback interfaces in OSPF

Staying true to its name, ones of the first posts here in LoopPacket are about, well, loopback interfaces 😀

Does it bother you that when you advertise a loopback interface into OSPF, it gets distributed with a /32 bit mask?

Let’s have a look at it:

R1#show ip interface loopback1
Loopback1 is up, line protocol is up
Internet address is 10.10.10.10/24
[output omitted]

We have a loopback1 interface with a /24 bit mask, yet on the neighbor router we get:

R2#show ip route
[output omitted]

10.0.0.0/32 is subnetted, 1 subnets
O       10.10.10.10 [110/2] via 192.168.0.1, 00:01:16, FastEthernet0/0

So, how do we change the advertised mask in OSPF to a /24 bit one? There are 3 ways:

  • Change the OSPF interface type to Point to Point
  • Summarize at ABR level (we need the loopback in a separate area)
  • Redistribute the connected subnets into OSPF, (preferably with a route-map, but not required for our mini lab)

Let’s try the first method – using the interface type:

R1(config)#interface loopback1
R1(config-if)#ip ospf network point-to-point

R2#show ip route
[output omitted]
10.0.0.0/24 is subnetted, 1 subnets
O       10.10.10.0 [110/2] via 192.168.0.1, 00:00:09, FastEthernet0/0

Cool, that worked! Now let’s summarize the loopback route. First – we need to get it to another area, because summarization in OSPF is only possible on ABR/ASBR devices.

R1(config)#interface loopback1
R1(config-if)#ip ospf 1 area 10
R1(config-if)#exit
R1(config)#router ospf 1
R1(config-router)#area 10 range 10.10.10.0 255.255.255.0

R2#show ip route
[output omitted]
10.0.0.0/24 is subnetted, 1 subnets
O IA    10.10.10.0 [110/2] via 192.168.0.1, 00:00:04, FastEthernet0/0

Great, it works. Now, let’s try the redistribution way:

R1(config)#router ospf
R1(config-router)#redistribute connected subnets

R2#show ip route
[output omitted]
10.0.0.0/24 is subnetted, 1 subnets
O E2    10.10.10.0 [110/20] via 192.168.0.1, 00:00:07, FastEthernet0/0

We can see how each implementation changes the route type, from an intra-area route, to an inter-area route, to an external route. Phew! You should consider which method to choose, as there are different ways of computing the metric for each method. If not sure, check out the article on advertising loopback interfaces in OSPF.

These are the three ways of “fixing” the subnet mask for the loopback routes in OSPF. If you know any other method, please do share it in the comments. Have fun with loopbacks!

Advertising loopback interfaces in OSPF

Loopback interfaces are a very convenient method to experiment with connectivity on routing labs. We use them often, but how should we advertise them? Let’s take a look on the possible ways:

    • redistributing connected interfaces, preferably with a specific route map, permitting the loopbacks only

 

  • using the ip ospf instance-id area area-id command in the loopback interface view

 

 

  • using the net x.x.x.x y.y.y.y area z command in the router ospf view

 

 

There are some subtle differences in implementing each method. Let’s experiment and see what they are!

First and foremost, unless you run IOS 12.4 or newer, you won’t be able to use the interface view command “ip ospf instance-id area area-id“. However, I think this is the cleanest and best way of applying areas on interfaces, and do recommend using it, so you should consider updating your software.

Let’s look at the topology we’ll be using for this lab:

area0

As you can see, we have a simple topology with 3 routers, the router’s loopback interfaces, and the method for loopback inclusion each router will be using.

The following is the initial config for the devices, specifically for router 1. The others are equivalent.

interface Loopback0
 ip address 1.1.1.1 255.255.255.0
!
interface FastEthernet0/0
 ip address 192.168.0.1 255.255.255.0
 ip ospf 1 area 0
 duplex auto
 speed auto
!
router ospf 1
 log-adjacency-changes

 

Now let’s configure router 1 – create a route map that permits only the loopback0 interface and apply it to the redistribution of the connected interfaces in OSPF:

R1(config)#route-map Lo0 permit 10
R1(config-route-map)#match interface lo0
R1(config-route-map)#exit
R1(config)#

R1(config)#router ospf 1
R1(config-router)#redistribute connected subnets route-map Lo0

 

Let’s see the effect this had on the devices. Let’s look at R1’s topology, and then at a neighbor device.

R1#sh ip ospf database

            OSPF Router with ID (1.1.1.1) (Process ID 1)

                Router Link States (Area 0)

Link ID         ADV Router      Age         Seq#       Checksum Link count
1.1.1.1         1.1.1.1         138         0x80000003 0x0074DD 1
2.2.2.2         2.2.2.2         454         0x80000003 0x00301B 1
3.3.3.3         3.3.3.3         422         0x80000002 0x00F34F 1

                Net Link States (Area 0)

Link ID         ADV Router      Age         Seq#       Checksum
192.168.0.1     1.1.1.1         421         0x80000002 0x001995

                Type-5 AS External Link States

Link ID         ADV Router      Age         Seq#       Checksum Tag
1.1.1.0         1.1.1.1         138         0x80000001 0x00A5F3 0

R2#sh ip route
Gateway of last resort is not set

     1.0.0.0/24 is subnetted, 1 subnets
O E2    1.1.1.0 [110/20] via 192.168.0.1, 00:00:27, FastEthernet0/0
     2.0.0.0/24 is subnetted, 1 subnets
C       2.2.2.0 is directly connected, Loopback0
C    192.168.0.0/24 is directly connected, FastEthernet0/0

 

The route towards 1.1.1.1 has appeared as an external OSPF route of type 2. Redistributed routes become OSPF external type 2 routes by default. The default cost or metric of a redistributed route is 1 for BGP and 20 for all other protocols. This may, or may not be what you’re expecting as a cost for that route. Be careful – routes are redistributed in OSPF as either type 1 (E1) routes or type 2 (E2) routes, with type 2 being the default. A type 1 route has a metric that is the sum of the internal OSPF cost and the external redistributed cost. A type 2 route has a metric equal only to the redistributed cost. If routes are redistributed into OSPF as type 2 then every router in the OSPF domain will see the same cost to reach the external networks. If routes are redistributed into OSPF as type 1, then the cost to reach the external networks could vary from router to router. Let’s change the type to Type 1 and see how the cost changes:

R1(config)#router ospf 1
R1(config-router)#redistribute connected subnets route-map Lo0 metric-type 1

R3#sh ip route ! some of the following output omitted
     1.0.0.0/24 is subnetted, 1 subnets
O E1    1.1.1.0 [110/21] via 192.168.0.1, 00:01:12, FastEthernet0/0

 

So the cost has changed from 20 to 21, because of the default OSPF cost for a FastEthernet interface being 1, and the route is 1 hop away.

This is great and all, but if for some reason you need this route to be present in a stub area, for example you need a specific /32 route to this specific host, you need to use another method. This is because stub areas block Type5 external routes, remember?

Let’s see what else we can get by not using redistribution:

R2(config)#int lo0
R2(config-if)#ip ospf 1 area 0

R2#sh ip ospf database

            OSPF Router with ID (2.2.2.2) (Process ID 1)

                Router Link States (Area 0)

Link ID         ADV Router      Age         Seq#       Checksum Link count
1.1.1.1         1.1.1.1         219         0x80000003 0x0074DD 1
2.2.2.2         2.2.2.2         41          0x80000004 0x00F937 2
3.3.3.3         3.3.3.3         503         0x80000002 0x00F34F 1

                Net Link States (Area 0)

Link ID         ADV Router      Age         Seq#       Checksum
192.168.0.1     1.1.1.1         502         0x80000002 0x001995

                Type-5 AS External Link States

Link ID         ADV Router      Age         Seq#       Checksum Tag
1.1.1.0         1.1.1.1         219         0x80000001 0x00A5F3 0

R1#sh ip route
Gateway of last resort is not set

     1.0.0.0/24 is subnetted, 1 subnets
C       1.1.1.0 is directly connected, Loopback0
     2.0.0.0/32 is subnetted, 1 subnets
O       2.2.2.2 [110/2] via 192.168.0.2, 00:00:50, FastEthernet0/0
C    192.168.0.0/24 is directly connected, FastEthernet0/0

 

As we can see, this time there is no type 5 router for the route we advertised. Also, the other router receives it as an intra-area route. Let’s see the routes we’ve define up until now, on the device we’re about to advertise on:

R3#sh ip route
Gateway of last resort is not set

     1.0.0.0/24 is subnetted, 1 subnets
O E1    1.1.1.0 [110/21] via 192.168.0.1, 00:01:55, FastEthernet0/0
     2.0.0.0/32 is subnetted, 1 subnets
O       2.2.2.2 [110/2] via 192.168.0.2, 00:01:55, FastEthernet0/0
     3.0.0.0/24 is subnetted, 1 subnets
C       3.3.3.0 is directly connected, Loopback0
C    192.168.0.0/24 is directly connected, FastEthernet0/0

 

R1’s loopback is an external type 1 route, and R2’s loopback is an intra-area route. Nice. Let’s configure the third router:

R3(config-router)#net 3.3.3.3 0.0.0.0 area 0

R1#sh ip route
Gateway of last resort is not set

     1.0.0.0/24 is subnetted, 1 subnets
C       1.1.1.0 is directly connected, Loopback0
     2.0.0.0/32 is subnetted, 1 subnets
O       2.2.2.2 [110/2] via 192.168.0.2, 01:05:17, FastEthernet0/0
     3.0.0.0/32 is subnetted, 1 subnets
O       3.3.3.3 [110/2] via 192.168.0.3, 01:05:17, FastEthernet0/0
C    192.168.0.0/24 is directly connected, FastEthernet0/0

 

So, using the net statement, we get the same result as using the ip ospf x area y statement – an intra-area route. So, to sum up all our work, here are the final routing tables of all three devices:

R1#sh ip route
Gateway of last resort is not set

     1.0.0.0/24 is subnetted, 1 subnets
C       1.1.1.0 is directly connected, Loopback0
     2.0.0.0/32 is subnetted, 1 subnets
O       2.2.2.2 [110/2] via 192.168.0.2, 01:05:17, FastEthernet0/0
     3.0.0.0/32 is subnetted, 1 subnets
O       3.3.3.3 [110/2] via 192.168.0.3, 01:05:17, FastEthernet0/0
C    192.168.0.0/24 is directly connected, FastEthernet0/0

R2#sh ip route
Gateway of last resort is not set

     1.0.0.0/24 is subnetted, 1 subnets
O E2    1.1.1.0 [110/20] via 192.168.0.1, 01:06:09, FastEthernet0/0
     2.0.0.0/24 is subnetted, 1 subnets
C       2.2.2.0 is directly connected, Loopback0
     3.0.0.0/32 is subnetted, 1 subnets
O       3.3.3.3 [110/2] via 192.168.0.3, 01:06:09, FastEthernet0/0
C    192.168.0.0/24 is directly connected, FastEthernet0/0

R3#sh ip route
Gateway of last resort is not set

     1.0.0.0/24 is subnetted, 1 subnets
O E2    1.1.1.0 [110/20] via 192.168.0.1, 01:06:12, FastEthernet0/0
     2.0.0.0/32 is subnetted, 1 subnets
O       2.2.2.2 [110/2] via 192.168.0.2, 01:06:12, FastEthernet0/0
     3.0.0.0/24 is subnetted, 1 subnets
C       3.3.3.0 is directly connected, Loopback0
C    192.168.0.0/24 is directly connected, FastEthernet0/0

 

One single scenario remains – what is we don’t advertise the loopbacks into area 0, but rather each loopback in its own area? Simple – we’d get the loopback routes as inter-area routes (O IA in the routing table).

I hope this lab helps in better understanding the inclusion of loopbacks in your labs.