Category Archives: Services

Routallica – WAN / Metallica – One

Original song courtesy of Metallica
No copyright infringement intended, just having fun

Routallica – WAN

I can’t connect to anything
Can’t tell if this is routing or bridging
Deep down inside I want to ping
This terrible access-list stops me

Now that the routes are passed to me
Network is convergin, but I cannot see
And there are not many neighbors here
Nothing is up but loopbacks

Hold my frames as I wait for STP
Oh please, broadcasts, don’t storm me

Back in the WAN it’s much too serial
Terrible speeds that I must feel
But I look forward to police
Police those excess TCP bursts

Weighted RED is throttling me
Just like a greedy TCP stream
Class-based shaping’s buffering me
Dropping tokens from the bucket

Hold my updates as I split horizons
Oh please, RIP, route me

Now the network’s gone, I’m just one
Oh redundant link, help me
Split my brain as my peer faces death
Oh please, VSS, help me

VLANs imprisoning me
No adjacency
Asymmetric routing
I cannot ping
I cannot trace
Trapped in my shell
Subnet my holding cell

Black holes have poisoned my routes
Taken my ping
Taken my peering
Taken my ARPs
Taken my V(e)RFs
Taken my pools
Left me with Frame Relaaay

Upgrade firmware and bootrom on HP A5120

A simple software upgrade of an HP A5120 EI switch is explained in the following post.

The device software includes the Boot ROM program and the system boot file. After powered on, the device runs the Boot ROM program, initializes the hardware, and displays the hardware information. Then the device runs the boot file. The boot file provides drivers and adaption for hardware, and implements service features. The Boot ROM program and system boot file are required for the startup and running of a device.

NOTE: Regarding commands on the device, the BootROM is called bootrom, while the boot file is called boot-loader. So boot-loader and boot file are interchangeable in context, but not in syntax.

The Boot ROM program and system boot file can both be upgraded at the Boot ROM menu or at the command line interface (CLI). We will perform this upgrade by the command line this time.

dis ver
HP Comware Platform Software
Comware Software, Version 5.20, Release 2208
Copyright (c) 2010-2011 Hewlett-Packard Development Company, L.P.
HP A5120-48G EI Switch with 2 Interface Slots uptime is 0 week, 0 day, 17 hours, 56 minutes
HP A5120-48G EI Switch with 2 Interface Slots with 1 Processor
128M bytes SDRAM
16384K bytes Flash Memory
Hardware Version is REV.B
CPLD Version is 007
Bootrom Version is 607
[SubSlot 0] 48GE+4SFP Hardware Version is REV.B

This is the output of the “display version” command before the updates take place. Now, on to the real update – first, enable the bootrom security check. This should help you in case you try to update your device with a wrong boot file, but do not rely too much on it. After all, we should know what we’re doing in the first place 🙂

system-view
[HP]bootrom-update security check enable
[HP]quit

tftp [tftp server IP] get A5120EI-BTM-610.btm
 ...
File will be transferred in binary mode
Downloading file from remote TFTP server, please wait...\
TFTP: 0 bytes received in 0 second(s)
File downloaded successfully.

bootrom update file flash:/a5120ei-btm-610.btm slot 1
This command will update bootrom file on the specified board(s), Continue? [Y/N]:y
Now updating bootrom, please wait...
Succeeded to update bootrom of Board 1.

We have successfully updated the bootrom, by downloading the new file from a TFTP server. I will cover more on TFTP servers in a future blogpost.

Due to the insufficient space on the device, the current boot loader file needs to be deleted before the new one is uploaded. That is an interesting situation, where the device is left running with its boot loader in the RAM. Do not reboot the device before setting up the new boot loader or recovery steps will need to be taken.

The /unreserved parameter deletes the file from memory, as opposed to only moving it to the “Recycle Bin”. While in the Bin, the file will still take up space, hence the need for the complete removal.

delete /unreserved flash:/a5120ei-cmw520-r2208-s168.bin
The contents cannot be restored!!! Delete flash:/a5120ei-cmw520-r2208-s168.bin?[Y/N]:y
Deleting a file permanently will take a long time. Please wait...
.................................................................................................
%Delete file flash:/a5120ei-cmw520-r2208-s168.bin...Done.

tftp 192.168.15.39 get A5120EI-CMW520-R2215.bin
..
File will be transferred in binary mode
Downloading file from remote TFTP server, please wait......................................................................................................................................................................................................
TFTP: 12625865 bytes received in 198 second(s)
File downloaded successfully.

We are successful so far. Now, instruct the device to select the new boot-loader file. After that, verify that the new boot-loader will get loaded on the next reboot with the command “display boot-loader”. Do not forget to save the configuration before reloading, as missing that may make your device unbootable, and you may have to manually point to the new boot-loader again, from the bootrom (which means that you will incur downtime and would need physical access to the device – a nasty situation if you’re doing this from afar).

boot-loader file flash:/a5120ei-cmw520-r2215.bin slot 1 main
This command will set the boot file of the specified board. Continue? [Y/N]:y
The specified file will be used as the main boot file at the next reboot on slot 1!
display boot-loader
Slot 1
The current boot app is: flash:/a5120ei-cmw520-r2208-s168.bin
The main boot app is: flash:/a5120ei-cmw520-r2215.bin
The backup boot app is: flash:/
save main force
Validating file. Please wait......................
Saved the current configuration to mainboard device successfully.
Configuration is saved to device successfully.
reboot
Start to check configuration with next startup configuration file, please wait.........DONE!
This command will reboot the device. Continue? [Y/N]:y

After the reboot, check out the new version of both the bootrom and the boot-loader.

dis ver
HP Comware Platform Software
Comware Software, Version 5.20.99, Release 2215
Copyright (c) 2010-2012 Hewlett-Packard Development Company, L.P.
HP A5120-48G EI Switch with 2 Interface Slots uptime is 0 week, 0 day, 0 hour, 2 minutes
HP A5120-48G EI Switch with 2 Interface Slots with 1 Processor
128M bytes SDRAM
16384K bytes Flash Memory
Hardware Version is REV.B
CPLD Version is 007
Bootrom Version is 610
[SubSlot 0] 48GE+4SFP Hardware Version is REV.B

Always be very careful if doing this procedure remotely, backup both bootroms and bootloaders, as well as configuration files.
Never update the device during non-maintenance windows, and always be ready for the worst – which may very well be the need to physically access the device.

If the update takes place on an IRF system stack, you may speed up the procedure by enabling automatic boot-loader update during the stack formation, then updating only the master of the stack, and then rebooting the slave members.

By having the auto-update enabled, the slave members will download the new boot-loader from the master right after they have formed their neighborship. This way, you will only have to update a single device.

Traffic Shaping and Policing

Crash course in QoS

What is traffic shaping/policing? In a nutshell, policing is dropping packets when the traffic exceeds a certain speed threshold, while shaping is queuing the incoming traffic in order to send it at a lower rate. Naturally, shaping is applied to outbound traffic, while policing can be applied on both directions, although it is usually applied to the inbound traffic. The following is the general QoS terminology:

  • Tc – Time interval, over which the commited burst (Bc) can be sent
  • Bc – Commited burst, measured in bits. This is the amount of traffic to be sent each Tc.
  • Be – Excess burst in bits. This is the traffic sent above your Bc, and most of the time risks being dropped, due to being in excess
  • CIR – Commited information rate, in bits per second. This is you allowed speed from your Internet contract.
  • Shaping rate – This is the rate at which your device will be sending traffic, which may be equal to the CIR, or even a little bit greater (more on that – later)
  • Policing rate – The rate after which your ISP starts to drop your traffic, in order to control your speed (this may be bigger that the actual CIR)

The deal with Bc, Be and Tc is that if you have a 128kbps line, and the intervals are 10 in a second (that can be configured), each 10th of a second you send 12.8kb. If you don’t have anything to send one interval, you’ve wasted 12.8kb. So, to reclaim it, you could send 25.6kb the next interval, but now you’ve overused your allowance. That means that your Tc is 0.1, your Bc is 12.8, and when you reclaimed your lost bandwidth, your Be was 12.8kb.

As I mentioned, the time interval Tc can be configured. The time interval directly impacts your Be burst. Why would you modify your time interval, when you can burst all the traffic up as fast as you can, then just wait ‘in silence’ for the current time interval to end?

Consider you have a 32kbps serial line from your ISP. Which means that you can transfer 32 kilobits per second. However, what if the clock rate on your router is running with clock rate 64000? That means that the router is transmitting at the hardware speed of 64 kilobits per second. Does mean that we get twice the bandwidth allotted for free? No. Our device, as the DTE end of the line, cannot change the physical speed of transmission. Then how do we maintain the 32kbps speed? Simple – we transmit the most we can, and then wait. Since we can transmit 64kilobits per second, then we can transmit 32kilobits per half a second, and then wait another half of the second.

The VoIP guys now scream in terror “500ms latency?”. Yeah, it’s no good – we need the use of a shaper in such case.

Shapers

Using a traffic shaper (usually) means transmitting at a lower rate that receiving. There are a couple of gotchas to traffic shaping, mainly which traffic should you send first, and which one should wait in line, as well as the speed you are transmitting with. The first problem is resolved through queuing strategies.  The second – using careful planning of your shaping configuration. So let’s dive in!

We already established that if no shaping is used, our router will transmit at the physical clock rate as much as possible, and when your limit is reached (in our former case – at the half of a second), the ISP will drop police any other traffic for the rest of the interval (again, in our former case – for the rest of the half second). This 500ms latency is most of the time unacceptable, so we employ shaping. To assume a safe figure of many intervals in a single second (in order to minimize delay), Cisco routers have a predefined limit of the Bc value. How does the Bc affect your Tc? To calculate your Tc time interval, use the following formula

Tc = (Bc / CIR) x 1000

By default, Cisco routers will use a value of 8000 bits for Bc if the interface bandwidth rate <= 320kbps; and calculates the Tc using the upper formula (that’s why it is important to set up your bandwidth [speed] in the interface view). If your line is > 320kbps, your Tc will be 25ms fixed, and your Bc will equal = ( shaping rate * Tc ).

This setup ensures that delays are kept to a minimum, even with the default settings. Of course, you can tune your Bc, Be, CIR(using the bandwidth command), and by extension of the former values – the Tc (which cannot be directly modified).

Policing

Depending on the traffic contract with your ISP, your ISP may police your traffic with a Single rate or Dual rate policer. Single rate traffic contract usually defines an average speed, which the contractor guarantees. However, taking into account the “burstyness” of the traffic on packet-switched networks, and oversubscriptions, the contractor may decide to offer a Dual rate scenario. With the latter option, the contractor guarantees a minimum speed, and also provides a higher one, which is non-guaranteed.

One should also know that the policers are categorized into two groups: two-color and three-color. What that means is that the two-colored policer distinguishes traffic within the CIR, and traffic above it; while the three-color policer has the notions of two kinds of exceeding traffic – regular exceeding traffic, and extremely exceeding traffic (violating traffic). We can draw parallels with the Dual rate policer here – the CIR is the minimum speed guaranteed, which is the regular traffic speed for the policer. The non-guaranteed traffic is the regular exceeding traffic to the policer, and when traffic exceeds the average non-guaranteed limit, it is considered the violating traffic. Thus the policer has the notions of Conforming traffic, Exceeding traffic, and if three-colored, Violating traffic. As you can probably guess, the Dual-Rate policer can only be three-colored, as we have clearly defined minimum and average speeds. The way I like to differentiate between the single-rate and double-rate three-color policers is the following: it depends on the way we reserve bandwidth for the exceeding speeds. Let’s visualise it with some ASCII art! Here the higher rate “sits” on top of the minimum guaranteed rate.

Maximum "unsafe" utilization
|Be rate>===============
|Bc rate>

Maximum "safe" utilization
|Be rate>
|Bc rate>===============

No utilization
|Be rate>
|Bc rate>_______________

A peak in the midst of no utilization
|Be rate> /****\
|Bc rate>___/ *****\_____

A peak of violating traffic in the midst of no utilization
|Bv rate> /*\
|Be rate> /*** \
|Bc rate>___/ \_____

With a single-rate policer, whenever your traffic passes over the Bc rate, it will either get dropped, or get marked “eligible for discard”, which most of the time means that it will get dropped somewhere along the way if congestion occurs.

With each Tc interval, you gain the right to transmit Bc amount of traffic. If you transmit more than the Bc traffic, you get sanctioned.

With a double rate policer, whenever your traffic passes over the Bc rate, the Be rate is ‘utilized’. Each Tc interval you gain the right to transmit Bc+Be amounts of traffic. If your traffic is within the Bc rate, only the Bc ‘bucket’ is depleted. If you have stood a couple of intervals in silence, and then need to transmit, you could compensate for the previous ‘wasted’ time intervals, by filling in the Be bucket, thus getting a Bc+Be speed.

Summary

As one can see, the ISP limitations on speed are often rounded up to an interval of a second, which tends to get interesting to configure. Depending on what your goals are and which technology needs to be supported, an uplink traffic can be shaped in many ways. Fine tuning the shaping variables can either make wonders with your network, or make it as unresponsive as a dead cloud unicorn.

So, have you got anything peculiar to share on the QoS matter?