Category Archives: Cisco Networking

Multiple next-hops in policy routing

Imagine the following topology with routers 1 to 5:

         / (2) \
(4) - (1) (5)
         \ (3) /

There is Policy-based routing active on router 1, and the possible end-to-end paths are two:
4-1-2-5
4-1-3-5

Let us play around with the next-hop settings of PBR. Imagine we have a rudimentary PBR setup with the following:

route-map test permit 10
match ip address 100
set ip next-hop 10.0.12.2 10.0.13.3

10.0.12.2 is router 2 and 10.0.12.3 is router 3. We really don’t care about the access lists or anything else, let’s see what happens when we have multiple hops defined:

R1#debug ip policy
Policy routing debugging is on
R1#
*Jul 17 12:28:42.387: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, len 44, FIB policy match
*Jul 17 12:28:42.387: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, g=10.0.12.2, len 44, FIB policy routed
*Jul 17 12:28:42.447: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, len 40, FIB policy match
*Jul 17 12:28:42.447: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, g=10.0.12.2, len 40, FIB policy routed
*Jul 17 12:28:42.459: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, len 49, FIB policy match
*Jul 17 12:28:42.459: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, g=10.0.12.2, len 49, FIB policy routed
*Jul 17 12:28:42.467: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, len 40, FIB policy match
*Jul 17 12:28:42.467: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, g=10.0.12.2, len 40, FIB policy routed

There, so it keeps using the first next-hop no matter what. I guess the only way to nudge it to use the other one is to shutdown the interface connected to R2. Look what happens after I shut down that interface:

 *Jul 17 12:32:09.551: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, len 44, FIB policy match
*Jul 17 12:32:09.551: CEF-IP-POLICY: fib for address 10.0.12.2 is with flag 257
*Jul 17 12:32:09.551: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, g=10.0.13.3, len 44, FIB policy routed
*Jul 17 12:32:09.611: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, len 40, FIB policy match
*Jul 17 12:32:09.611: CEF-IP-POLICY: fib for address 10.0.12.2 is with flag 257
*Jul 17 12:32:09.611: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, g=10.0.13.3, len 40, FIB policy routed
*Jul 17 12:32:09.619: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, len 49, FIB policy match
*Jul 17 12:32:09.619: CEF-IP-POLICY: fib for address 10.0.12.2 is with flag 257
*Jul 17 12:32:09.619: IP: s=10.0.14.4 (GigabitEthernet1/0), d=5.5.5.5, g=10.0.13.3, len 49, FIB policy routed

The process sees that the route towards its next-hop is marked as down (flag 257) in the CEF tables (or non-existent), and goes on towards the next one.

Conclusion: Multiple next-hops in PBR are used for redundancy, not load-sharing/balancing.

Advertisements

Playing around with OSPF

I did some fooling around with OSPF priorities and here’s what I learned:

If you have multiple routers on an Ethernet segment,  and all routers have priority 0 – there will be no DR, no BDR, and only 2way connectivity between all of them.

If you have just one router has with priority >0, then that router will be the DR, but there will still be no bdr, and the only full connection would be, obviously, between the DR and the others.

With P2P links, OSPF doesn’t care about priority, but does care about area mismatches:

An area mismatch, for example (r1:normal)-(r2:stub) would tear down neighborship due to the mismatch, and it’s possible it won’t even show up on the error logs. Fun fun 🙂

Cisco Command Prompt Tricks and Gotchas part 2

Shall you ever be unfortunate enough to come across a device with a setup similar to

exec-timeout 0 1

or maybe you’re just doing a CCIE exam, then you’ll need a little bit of practical trickstery to overcome the one second timeout – just use notepad or whatever text editor you have to type something like

enable
conf t
line vty 0 4/line con 0
exec-timeout 10

and then quickly paste it in the window of 1 second. Done! 🙂

Cisco Command Prompt Tricks and Gotchas

I guess most of you are familiar with the usual CLI prompt, be it on a Linux system, Cisco device, or whatever. On a standard *nix machine, you can modify your prompt appearance, and its configuration is specific to the shell you’re using – BASH, KSH, ZSH, etc.

Recently, I was surprised to figure out that you can also modify the standard Cisco prompt. I owe this knowledge to my friend and mentor Vladi – thanks! 🙂 Interestingly, the only place I could find more info on the matter was the Cisco IOS in a Nutshell book.

Back on topic. A regular prompt would read

[hostname]>
or
[hostname]#

You can modify the prompt directly with prompt command, and use any of the following escaped variables with it:

%% - the percent character itself
%h - hostname
%n - tty command counter number
%p - prompt character (> or #)
%s - white space character
%t - tab character

For example:

Router#config t
Router (config)# prompt %h:%n%p
Router:1# show ver
[output omitted]
Router:2#

So now you can either modify your prompt, or play a trick on a fellow colleague 😀

Unequal Load-Balancing on Cisco IOS

I just wanted to share a neat trick that a fellow CCIE colleague showed me.

In case of being connected to two ISPs, there is a way of doing unequal load-balancing with the help of static routes. For example, ISP X provides you with 25Mbps, and ISP Y with 50Mbps – a 2:1 ratio.

In order to achieve any kind of load-balancing on the Cisco IOS, we need multiple entries in the routing table, pointing to the same specific destination. As we would like to load-balance our uplink traffic towards the internet, we would need multiple entries towards our default gateways.

We are all familiar with the concept that there can be only one default route for a specific gateway – you can’t have multiple routing entries pointing to the same default gateway. That means that if we have multiple ISPs and multiple default gateways, our load-balance ratio will always be 1:1, as there is just a single entry in the routing table for each default gateway.

However, we can install multiple routing entries for seemingly different default gateways. That way, we can fool the device and have the same default gateway listed multiple times in the routing table. Ok, it sounds confusing, but just take a look at the configuration and it’ll become clear.

ip route 10.0.1.1 255.255.255.255 192.168.1.2  #(ISP X)
ip route 10.0.2.1 255.255.255.255 172.16.1.2  #(ISP Y)
ip route 10.0.2.2 255.255.255.255 172.16.1.2  #(ISP Y)

ip route 0.0.0.0 0.0.0.0 10.0.1.1
ip route 0.0.0.0 0.0.0.0 10.0.2.1
ip route 0.0.0.0 0.0.0.0 10.0.2.2

First, we define static routes for a couple of fake default gateways. Those IPs do not exist, and will only be used for the current load-ballancing trick, so be careful when setting up those and try not to assign some IPs in use.

After that, we define these fake IPs as default gateways. Having in mind that the ratio of the link bandwidth is 2:1, we created two routes towards the faster ISP and a single route towards the slower ISP.

What happens is the IOS uses all three of these default gateways, because the destination is seemingly different during the first look up in the routing table. The second look up will reveal that the fake default gateway’s IP is reachable only by either ISP X or ISP Y’s next-hop router. This is quite the ingenious way of tricking the device into installing multiple entries in its routing table.

Default gateway on Cisco IOS

On Cisco hardware, or at least on most of the IOS family, there are three ways of specifying a default gateway. Let us look into that:

ip default-gateway
ip default-network
and ip route 0.0.0.0 0.0.0.0

The ip default-gateway command differs from the other two commands, as it should only be used when ip routing is disabled on the Cisco router, which means most probably never, unless in boot mode. In such case, you can use it to define a gateway to use TFTP to transfer a Cisco IOS image to the router. Apparently, the router does not have ip routing enabled in boot mode.

***

ip default-network

So, unless in boot mode, you should probably be using the ip default-network command. When you configure ip default-network the router considers routes to that network for installation as the gateway of last resort on the router.

For every network configured with ip default-network, if a router has a route to that network, that route is flagged as a candidate default route.

show ip route
Gateway of last resort is not set
161.44.0.0/24 is subnetted, 1 subnets
C 161.44.192.0 is directly connected, Ethernet0
131.108.0.0/24 is subnetted, 1 subnets
C 131.108.99.0 is directly connected, Serial0
S 198.10.1.0/24 [1/0] via 161.44.192.2

show ip route
ip default-network 198.10.1.0

 Gateway of last resort is 161.44.192.2 to network 198.10.1.0
161.44.0.0/24 is subnetted, 1 subnets
C 161.44.192.0 is directly connected, Ethernet0
131.108.0.0/24 is subnetted, 1 subnets
C 131.108.99.0 is directly connected, Serial0
S* 198.10.1.0/24 [1/0] via 161.44.192.2

The gateway of last resort is now set as 161.44.192.2. This result is independent of any routing protocol, as shown by the show ip protocols command at the bottom of the output. This can help you solve some tricky connectivity scenarios, as well as utilize two (or ever more) routing protocols for default gateway redundancy.

You can add another candidate default route by configuring another instance of ip default-network:

ip route 171.70.24.0 255.255.255.0 131.108.99.2
ip default-network 171.70.24.0
show ip route

Gateway of last resort is 161.44.192.2 to network 198.10.1.0
171.70.0.0/16 is variably subnetted, 2 subnets, 2 masks
S 171.70.0.0/16 [1/0] via 171.70.24.0
S 171.70.24.0/24 [1/0] via 131.108.99.2
161.44.0.0/24 is subnetted, 1 subnets
C 161.44.192.0 is directly connected, Ethernet0
131.108.0.0/24 is subnetted, 1 subnets
C 131.108.99.0 is directly connected, Serial0
S* 198.10.1.0/24 [1/0] via 161.44.192.2

However, changes did not take effect. That is because there is a potential pitfall with the ip default-network command – it is classful. Due to this, the command must be issued again, using the major net, in order to flag the candidate default route. Kind of like a recursive default-network path.

ip default-network 171.70.0.0
show ip route

Gateway of last resort is 171.70.24.0 to network 171.70.0.0
* 171.70.0.0/16 is variably subnetted, 2 subnets, 2 masks
S* 171.70.0.0/16 [1/0] via 171.70.24.0
S 171.70.24.0/24 [1/0] via 131.108.99.2
161.44.0.0/24 is subnetted, 1 subnets
C 161.44.192.0 is directly connected, Ethernet0
131.108.0.0/24 is subnetted, 1 subnets
C 131.108.99.0 is directly connected, Serial0
S* 198.10.1.0/24 [1/0] via 161.44.192.2

As the Cisco documentations describes this interesting ‘hack’, if the original static route had been to the major network, the extra step of configuring the default network twice would not have been necessary. As you can see, this may create some implications if your dynamic routing protocol is advertising networks with a subnet mask higher than the classless one. Do proceed with care, and lab and test any configuration change before deploying it in a production network. It is always easy to roll-back with a quick ip route 0.0.0.0, but if you’re troubleshooting from afar, never take the risk of cutting yourself off. Shall you have no other choice, scheduled reloading may help you in such case.

Let us test the fallback mechanism of the ip default-network command. If we are to remove a route to the particular default network, the router selects the other candidate default. Let’s try it:

no ip route 171.70.24.0 255.255.255.0 131.108.99.2
show ip route

Gateway of last resort is 161.44.192.2 to network 198.10.1.0
161.44.0.0/24 is subnetted, 1 subnets
C 161.44.192.0 is directly connected, Ethernet0
131.108.0.0/24 is subnetted, 1 subnets
C 131.108.99.0 is directly connected, Serial0
S* 198.10.1.0/24 [1/0] via 161.44.192.2

 

Of couse, there’s always the good old ip route command.

Creating a static route to network 0.0.0.0 0.0.0.0 is another way to set the default gateway on almost any layer 3 device.

Most network engineers think that a static route using the ip route command takes precedence over any other route, but there is an exception.

As stated by the Cisco documentation:
If you use both the ip default-network and ip route 0.0.0.0 0.0.0.0 commands to configure candidate default networks, and the network used by the ip default-network command is known statically, the network defined with the ip default-network command takes precedence and is chosen for the gateway of last resort. Otherwise if the network used by the ip default-network command is derived by a routing protocol, the ip route 0.0.0.0 0.0.0.0 command, which has a lower administrative distance, takes precedence and is chosen for the gateway of last resort. If you use multiple ip route 0.0.0.0 0.0.0.0 commands to configure a default route, traffic is load-balanced over the multiple routes.

So, to sum it up, if there is a default-network statement, and it points to a statically defined network, it overrides the ip route 0.0.0.0 0.0.0.0 command. Plain and simple.

Summary

When in doubt, always check the documentation first 🙂
The ip default-gateway command should only be used when ip routing is disabled on a Cisco router. In any other case, use either the ip default-network or ip route 0.0.0.0 0.0.0.0 commands to set the gateway. Just take care with the classfull behavior of the former command.