logo CCIE Blog

Helping you become a Cisco Certified Internetwork Expert


rss Entries (RSS) | rss Comments (RSS)
Welcome to Internetwork Expert's CCIE Blog

Welcome to Internetwork Expert’s CCIE Blog! This site is dedicated to helping you in your pursuit of becoming a Cisco Certified Internetwork Expert in Routing & Switching, Voice, Security, Service Provider, and Storage. Through this blog you can submit questions to our expert instructors, Brian Dennis - Quad-CCIE #2210, Brian McGahan – Triple CCIE #8593, and Petr Lapukhov - Quad-CCIE #16379. Check back daily as this blog will be updated frequently.

Click here to submit a question.

April 24th, 2008

GLBP Explained

GLBP, an acronym for Gateway Load Balancing Protocol, is a virtual gateway protocol similar to HSRP and VRRP. However, unlike it’s little brothers, GLBP is capable of utilizing multiple physical gateways at the same time. As we know, a single HSRP or VRRP group represents once virtual gateway, with single virtual IP/MAC addresses. Only one physical gateway in a standby/redundancy group is responsible for packet forwarding, others remain inactive in standby/backup state. Say if you have R1, R2, R3 sharing the segment 174.X.123.0/24 with the physical IP addresses 174.X.123.1, 174.X.123.2 and 174.X.123.3 you may configure them to represent one single virtual gateway with an IP address 174.X.123.254. The physical gateway priority settings will determine which physical gateway takes the role of packet forwarder. The hosts on the segment will set their default gateway to 174.X.123.254, staying out of the physical gateway failure issues.

GLBP further develops this idea, allowing multiple gateways to participate in packet forwarding simultaneously. Considering the example above, imagine you want the hosts on the segments to fully utilize all existing physical gateways, yet provide gateway failure recovery. For instance, you may want 50% of outgoing packets to be sent up to R1, 30% to R2 and 20% to R3. At the same time, you want to ensure, that hosts using either of the gateways will automatically switch to another if their gateway fails. On top of that, all hosts in the segment should reference to the virtual gateway using the same IP address 174.X.123.254. This is a complicated task, which is being addressed by GLBP protocol design.

To begin with, we should recall that each host on the segment would need to resolve the virtual gateway IP address 174.X.123.254 to a MAC address using ARP protocol. When we use HSRP or VRRP, the ARP response will be the virtual MAC addresses, which is assigned to the active physical gateway. At this point, GLBP differs in that it may respond with different virtual MAC addresses, belonging to various physical gateways in the GLBP group. So the key idea with GLBP is that load balancing is accomplished by responding to ARP requests with different virtual MAC addresses.

Here is how GLBP actually implements the above idea. One of the routers in a GLBP group is elected as AVG – Active Virtual Gateway. There is only one active AVG in a group, and its task is to respond to ARP requests sent to the virtual gateway IP address (e.g. 174.X.123.254) replying different virtual MAC addresses in response packets. The AVG will also implement load-sharing algorithm, e.g. by sending the replies in proportion to weights configured for physical gateways. Aside from AVG, the other logical component of GLBP implementation is AVF – Active Virtual Forwarder. Any physical gateway in a GLBP group may act as AVF – in fact all physical gateways are usually AVFs. Every AVF has a virtual MAC address assigned by an AVG and a weight value configured by an operator.

Now let’s discuss redundancy – the primary goal of any virtual router protocol. There are two logical entities used to build a GLBP group: AVGs and AVFs, and each of them needs a backup scheme. Since there is just one AVG per a GLBP group, the procedure is pretty simple: each candidate AVG has a priority value assigned; the highest priority router becomes an active AVG, the next by priority becomes a standby AVG. You may configure AVG preemption, so that a newly configured router with highest priority value becomes AVG, preemption the old one.

What about AVF redundancy? First, we need to understand that AVFs are always “active” in the sense that they are always used by a load-balancing algorithm. (However, by setting an AVG weight value below threshold, we may effectively take the AVF out of service. The weight value could be combined with object tracking to bring powerful traffic manipulation options). Next, with respect to redundancy, all AVFs backup each other. For instance, take any AVF: with respect to the other AVFs it is “Active”, and the remaining AVFs are in “Listen” state. If the AVF would fail, other gatewyas will detect the event using Hold timer expiration, and immediately try to take over the failed AVF virtual MAC address. Among the competitors, the AVF with highest weight value would win, and the remaining AVFs will switch back to “Listen” state. At this point, the “winner” will start accepting packets for two virtual MAC addresses: it’s own, and the one it has obtained from the failed AVF. At the same moment, two timers would start: Redirect and Secondary Hold. The Redirect timer determines how long will AVG continue to respond to ARP requests with the virtual MAC of the failed AVF. The Secondary Hold timer sets the amount of time the backup AVF will continue to accept packet for the virtual MAC address taken from the failed AVF.

This is basically how GLBP works. Different load-balancing algorithms are supported – the default one is round robin, with options for weighted load balancing and source-MAC based. The last one will always respond with the same vMAC to the same source MAC address, thereby defining sort of host-gateway “stickiness”. Now for a sample GLBP configuration, for the above mentioned R1, R2 and R3:

!
!  We set load-balancing to weighted only on R1
!  So if R2 will become the AVG, it will use round-robin
!  load-balancing technique
!
R1:
interface FastEthernet0/0
 ip address 174.1.123.1 255.255.255.0
 glbp 123 ip 174.1.123.254
 glbp 123 preempt
 glbp 123 weighting 50
 glbp 123 load-balancing weighted
!
!
!
R2:
interface FastEthernet0/0
 ip address 174.1.123.2 255.255.255.0
 glbp 123 ip 174.1.123.254
 glbp 123 priority 50
 glbp 123 preempt
 glbp 123 weighting 30
!
!
!
R3:
interface Ethernet0/0
 ip address 174.1.123.3 255.255.255.0
 glbp 123 ip 174.1.123.254
 glbp 123 priority 25
 glbp 123 preempt
 glbp 123 weighting 20

Some show output:

Rack1R1#show glbp
FastEthernet0/0 - Group 123
  State is Active
    2 state changes, last state change 03:12:05
  Virtual IP address is 174.1.123.254
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 0.916 secs
  Redirect time 600 sec, forwarder time-out 14400 sec
  Preemption enabled, min delay 0 sec
  Active is local
  Standby is 174.1.123.2, priority 50 (expires in 8.936 sec) <-- Standby AVG
  Priority 100 (default)
  Weighting 50 (configured 50), thresholds: lower 1, upper 50 <--
<-- Should the weight go below thresh, AVF is taken offline
  Load balancing: weighted
  Group members:
    ca00.0156.0000 (174.1.123.1) local <--   Hardware MACs
    ca01.0156.0000 (174.1.123.2)
    cc02.0156.0000 (174.1.123.3)
  There are 3 forwarders (1 active)
  Forwarder 1
    State is Listen <--  All other AVFs Listen to us
    MAC address is 0007.b400.7b01 (learnt) <--  Virtual MAC 
    Owner ID is ca01.0156.0000 <--  This is R2
    Redirection enabled, 598.928 sec remaining (maximum 600 sec) <--
<-- ARP replies with this vMAC are being sent by AVG
    Time to live: 14398.376 sec (maximum 14400 sec)
    Preemption enabled, min delay 30 sec
    Active is 174.1.123.2 (primary), weighting 30 (expires in 8.368 sec) <--
   <--  The AVF reports it’s own IP as active
    Arp replies sent: 1
  Forwarder 2
    State is Active <--  Active mean it’s us
      1 state change, last state change 03:12:45
    MAC address is 0007.b400.7b02 (default)
    Owner ID is ca00.0156.0000 <--  R1 MAC address
    Redirection enabled
    Preemption enabled, min delay 30 sec
    Active is local, weighting 50
    Arp replies sent: 1
  Forwarder 3
    State is Listen <--  All other AVFs Listen to us
    MAC address is 0007.b400.7b03 (learnt)
    Owner ID is cc02.0156.0000 <--  This is R3
    Redirection enabled, 597.916 sec remaining (maximum 600 sec)
    Time to live: 14397.916 sec (maximum 14400 sec)
    Preemption enabled, min delay 30 sec
    Active is 174.1.123.3 (primary), weighting 20 (expires in 7.916 sec)

Rack1R2#show glbp
FastEthernet0/0 - Group 123
  State is Standby
    4 state changes, last state change 03:16:56
  Virtual IP address is 174.1.123.254
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 0.236 secs
  Redirect time 600 sec, forwarder time-out 14400 sec
  Preemption enabled, min delay 0 sec
  Active is 174.1.123.1, priority 100 (expires in 9.148 sec)
  Standby is local <-- We are the standby AVG
  Priority 50 (configured)
  Weighting 30 (configured 30), thresholds: lower 1, upper 30
  Load balancing: round-robin
  Group members:
    ca00.0156.0000 (174.1.123.1)
    ca01.0156.0000 (174.1.123.2) local
    cc02.0156.0000 (174.1.123.3)
  There are 3 forwarders (1 active)
  Forwarder 1
    State is Active
      1 state change, last state change 03:18:06
    MAC address is 0007.b400.7b01 (default)
    Owner ID is ca01.0156.0000 <-- This is R2
    Preemption enabled, min delay 30 sec
    Active is local, weighting 30
  Forwarder 2
    State is Listen
    MAC address is 0007.b400.7b02 (learnt)
    Owner ID is ca00.0156.0000
    Time to live: 14398.644 sec (maximum 14400 sec)
    Preemption enabled, min delay 30 sec
    Active is 174.1.123.1 (primary), weighting 50 (expires in 8.636 sec)
  Forwarder 3
    State is Listen
    MAC address is 0007.b400.7b03 (learnt)
    Owner ID is cc02.0156.0000
    Time to live: 14399.260 sec (maximum 14400 sec)
    Preemption enabled, min delay 30 sec
    Active is 174.1.123.3 (primary), weighting 20 (expires in 9.260 sec)

Now let’s check how ARP redirection works:

Rack1SW1#ping 174.1.123.254

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 174.1.123.254, timeout is 2 seconds:
..!!!
Success rate is 60 percent (3/5), round-trip min/avg/max = 8/12/16 ms

Rack1SW1#sh ip arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  174.1.123.254           0   0007.b400.7b01  ARPA   Vlan1
Internet  174.1.123.7             -   cc06.0156.0000  ARPA   Vlan1
Internet  174.1.123.2             0   ca01.0156.0000  ARPA   Vlan1
Rack1SW1#clear arp-cache 

Rack1SW1#ping 174.1.123.254

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 174.1.123.254, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 4/13/32 ms

Rack1SW1#sh ip arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  174.1.123.254           0   0007.b400.7b02  ARPA   Vlan1
Internet  174.1.123.7             -   cc06.0156.0000  ARPA   Vlan1
Internet  174.1.123.2             0   ca01.0156.0000  ARPA   Vlan1

Repeat the above actions a few more times

Rack1SW1#sh ip arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  174.1.123.254           0   0007.b400.7b03  ARPA   Vlan1
Internet  174.1.123.7             -   cc06.0156.0000  ARPA   Vlan1
Internet  174.1.123.2             0   ca01.0156.0000  ARPA   Vlan1
Internet  174.1.123.3             0   cc02.0156.0000  ARPA   Vlan1

To summarize, GLBP is a virtual gateway protocol, with built-in load-balancing capabilities. Load balancing is based on manipulating ARP responses to the requests sent to the virtual gateway IP address. AVG role is used to load-balance and respond to ARP requests. AVF role manages one or more virtual MACs and is responsible for packet forwarding. AVG redundancy is controlled by GLBP priority and AVF redundancy is implemented using AVF weight value and two additional timers.

Further reading:

GLBP Overview

February 15th, 2008

The Inside and Outside of NAT

Quite many people don’t pay attention to the difference in handling packets on interfaces configured for NAT inside and outside. Here is an example to demonstrate how NAT “domains” interact with routing. Consider three routers connected in the following manner:

nat-inside-outside

For this scenario we have no routing configured. Let’s use static NAT to provide connectivity between R1 and R2. R2 would see R1 as a host on local connected segment with the IP address 155.1.23.1 and R1 would see R2 as a host on it’s local segment with the IP address 155.1.13.2. This goal could be achieved with the following configuration:


R3:
!
interface Serial 1/0.301 point-to-point
 ip address 155.1.13.3 255.255.255.0
 ip nat inside
 no ip route-cache
!
interface Serial 1/0.302 multipoint
 ip address 155.1.23.3 255.255.255.0
 frame-relay map ip 155.1.23.2 302
 ip nat outside
 no ip route-cache

!
! Static NAT: translations are effectively bi-directional
!
ip nat inside source static 155.1.13.1 155.1.23.1
ip nat outside source static 155.1.23.2 155.1.13.2

R2:
!
! Add a Frame-Relay mapping for the new IP (representing R1)
! so that R2 would know how to reach the address over multipoint FR interface
!
interface Serial 1/0.203 multipoint
 ip address 155.1.23.2 255.255.255.0
 frame-relay map ip 155.1.23.3 203
 frame-relay map ip 155.1.23.2 203

Let’s see how it’s working. Note that we disabled route-cache on both interfaces to intercept packets via CPU.


Rack1R3#debug ip nat detailed
IP NAT detailed debugging is on

Rack1R3#debug ip packet detail
IP packet debugging is on (detailed)

Rack1R2#ping 155.1.23.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 155.1.23.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

Hmm…it fails. Look at the debugging output on R3:


Rack1R3#
!
! Packet on NAT outside (o - for outside) hits the interface
!
NAT*: o: icmp (155.1.23.2, 16) -> (155.1.23.1, 16) [84]

!
! Source and destination for the packet rewritten according to NAT rules
!
NAT*: s=155.1.23.2->155.1.13.2, d=155.1.23.1 [84]
NAT*: s=155.1.13.2, d=155.1.23.1->155.1.13.1 [84]

!
! The packet is routed after translation (with new source and destination IPs). Note that routing decision
! and the actual forwarding take place only after translation rules were triggered by NAT tables
!
P: tableid=0, s=155.1.13.2 (Serial1/0.302), d=155.1.13.1 (Serial1/0.301), routed via RIB
IP: s=155.1.13.2 (Serial1/0.302), d=155.1.13.1 (Serial1/0.301), g=155.1.13.1, len 100, forward
    ICMP type=8, code=0
!
! The response packet from R1 comes in - to destination 155.1.13.2 -  routed via RIB (to the same interface)
! But no NAT rules were triggered since the destination interface is the same as input interface!
!
IP: tableid=0, s=155.1.13.1 (Serial1/0.301), d=155.1.13.2 (Serial1/0.301), routed via RIB
IP: s=155.1.13.1 (Serial1/0.301), d=155.1.13.2 (Serial1/0.301), len 100, rcvd 3
    ICMP type=0, code=0

OK hold here for a second.. Now we recall that for inside NAT routing is tried first, and only then the packet is translated according to the NAT rules. This is how the NAT order of operations works on the inside. So now it’s clear: IOS first tries to route packet to 155.1.13.2 - which is the same interface as it came in.. therefore the inside->outside translation never occurs! To fix this, let’s add a static route on R3:


R3:
ip route 155.1.13.2 255.255.255.255 155.1.23.2

Verification:


Rack1R2#ping 155.1.23.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 155.1.23.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/33/52 ms

Rack1R3#
!
! Outside: translate & route
!
NAT*: o: icmp (155.1.23.2, 17) -> (155.1.23.1, 17) [89]
NAT*: s=155.1.23.2->155.1.13.2, d=155.1.23.1 [89]
NAT*: s=155.1.13.2, d=155.1.23.1->155.1.13.1 [89]

!
! Routing decision and forwarding
!
IP: tableid=0, s=155.1.13.2 (Serial1/0.302), d=155.1.13.1 (Serial1/0.301), routed via RIB
IP: s=155.1.13.2 (Serial1/0.302), d=155.1.13.1 (Serial1/0.301), g=155.1.13.1, len 100, forward
    ICMP type=8, code=0
!
! Inside: Routing decision - the packet is routed using our fixup static route
!
IP: tableid=0, s=155.1.13.1 (Serial1/0.301), d=155.1.13.2 (Serial1/0.302), routed via RIB

!
! NAT rule (i - for inside) is triggered by the packet
!
NAT: i: icmp (155.1.13.1, 17) -> (155.1.13.2, 17) [89]     

!
! Source and destination addresses rewritten in the "opposite" direction
!
NAT: s=155.1.13.1->155.1.23.1, d=155.1.13.2 [89]
NAT: s=155.1.23.1, d=155.1.13.2->155.1.23.2 [89]

!
! Packet is sent to R2 (with the new source and destination) - forwarding takes place
!
IP: s=155.1.23.1 (Serial1/0.301), d=155.1.23.2 (Serial1/0.302), g=155.1.23.2, len 100, forward
    ICMP type=0, code=0

Nice. So now we know the difference for sure: packets on the NAT outside are first translated and then routed. On the inside interface routing decision kicks in first and only then translation rules get applied followed by forwarding. Before we finish with that, recall new 12.3T feature called NAT Virtual Interface. With this feature we can now configure any interface as “NAT enabled” an get rid of those “inside” and “outside” domains . All NAT traffic passed through new virtual interface called NVI, in symmetric manner. Let’s reconfigure out task using this new concepts.


R3:
interface Serial 1/0.301 point-to-point
 no ip nat inside
 ip nat enable
!
interface Serial 1/0.302 multipoint
 no ip nat outside
 ip nat enable

!
!  Remove old rules
!
no ip nat inside source static 155.1.13.1 155.1.23.1
no ip nat outside source static 155.1.23.2 155.1.13.2

!
! Add "domainless" rules
!
ip nat source static 155.1.13.1 155.1.23.1
ip nat source static 155.1.23.2 155.1.13.2

no ip route 155.1.13.2 255.255.255.255 155.1.23.2

Verification:


Rack1R2#ping 155.1.23.1                 

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 155.1.23.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/40/60 ms

Rack1R3#
!
! Routing decision it taken: packet classified for NAT, since destination is in NAT table
! Note that no actual forwarding occurs, just routing decision to send packet
!
IP: tableid=0, s=155.1.23.2 (Serial1/0.302), d=155.1.23.1 (Serial1/0.302), routed via RIB

!
! Packet translated according to NAT rules (note "i" for inside NAT)
!
NAT: i: icmp (155.1.23.2, 19) -> (155.1.23.1, 19) [95]
NAT: s=155.1.23.2->155.1.13.2, d=155.1.23.1 [95]
NAT: s=155.1.13.2, d=155.1.23.1->155.1.13.1 [95]

!
! Another routing decision, for translated packet - now actual forwarding occurs
!
IP: tableid=0, s=155.1.13.2 (Serial1/0.302), d=155.1.13.1 (Serial1/0.301), routed via RIB
IP: s=155.1.13.2 (Serial1/0.302), d=155.1.13.1 (Serial1/0.301), g=155.1.13.1, len 100, forward
    ICMP type=8, code=0

!
! Response comes in, first routing decision - NAT table entry matched
!
IP: tableid=0, s=155.1.13.1 (Serial1/0.301), d=155.1.13.2 (Serial1/0.301), routed via RIB

!
! Packet translated ("i" - inside NAT)
!
NAT: i: icmp (155.1.13.1, 19) -> (155.1.13.2, 19) [95]
NAT: s=155.1.13.1->155.1.23.1, d=155.1.13.2 [95]
NAT: s=155.1.23.1, d=155.1.13.2->155.1.23.2 [95]

!
! Another routing decision, for post-translated packet, followed by forwarding
!
IP: tableid=0, s=155.1.23.1 (Serial1/0.301), d=155.1.23.2 (Serial1/0.302), routed via RIB
IP: s=155.1.23.1 (Serial1/0.301), d=155.1.23.2 (Serial1/0.302), g=155.1.23.2, len 100, forward
    ICMP type=0, code=0

So what’s the difference with NVI? First, we see that now NAT behaves symmetrically. Next, we see that NAT translation tables are used to take a “routing decision” to send packet to virtual interface. Packet is translated there and then another routing decision takes place, followed by packet forwarding. So the difference from the old model is that now routing decision is taken twice: before and after translation. This allows to get rid of any static routes needed by “legacy” NAT, since lookup is performed after translation.

To summarize: Domain-based NAT uses different orders of operations for inside and outside domain. NVI based NAT is symmetrical and performs routing lookup twice: first to send packet to NVI, second to route packet using the post-translated addresses.

Links:

NAT Order of Operation

February 13th, 2008

Tricks with Local Policy Routing

Cisco IOS has a special feature called local policy routing, which permits to apply a route-map to local (router-generated) traffic. The first way we can use this feature is to re-circulate local traffic (and force it re-enter the router). Here’s an example. By default, locally-generated packets are not inspected by outgoing access-lists. This may cause issues when local traffic is not being reflected under relfexive access-list entries. Say with configuration like that:


!
! Reflect all "session-oriented" traffic
!
ip access-list extended EGRESS
 permit tcp any any reflect MIRROR
 permit icmp any any reflect MIRROR
 permit udp any any reflect MIRROR
!
! Evalute the reflected entries
!
ip access-list extended INGRESS
 evaluate MIRROR
 permit ospf any any
!
interface Serial 0/0
 ip address 54.1.1.6 255.255.255.0
 ip access-group INGRESS in
 ip access-group EGRESS out

You would not be able to telnet out of a router to destinations behind the Serial interface, even though TCP sessions are reflected in access-list. To fix the issue, we may use local-policy to force the local traffic re-enter the router and be inspected by outgoing access-list:


!
! Redirect local telnet traffic via the Loopback interface
!
ip access-list extended LOCAL_TRAFFIC
 permit tcp any any eq 23
!
route-map LOCAL_POLICY 10
 match ip address LOCAL_TRAFFIC
 set interface Loopback0
!
! Traffic sent to Loopback interface re-enters the router
!
interface Loopback0
 ip address 150.1.6.6 255.255.255.50

!
! Apply the local-policy
!
ip local policy route-map LOCAL_POLICY

With this configuration, local telnet session will re-enter the router and hit the outgoing access-list, thereby triggering a reflected entry. This same idea may be utilized to force CBAC inspection of locally-generated traffic, by since 12.3T there has been a special IOS feature to do this natively.

The other useful application of local policy routing is using it for traffic filtering. For example you may want to prohibit outgoing telnet sessions from local router to a certain destination:


ip access-list extended BLOCK_TELNET
 permit tcp any host 150.1.1.1 eq 23
!
route-map LOCAL_POLICY 10
 match ip address BLOCK_TELNET
 set interface Null 0

!
! Apply the local-policy
!
ip local policy route-map LOCAL_POLICY

The syntax is somewhat similar to the vlan access-maps used on Catalyst switches, and similarly the route-map is applied “globally”, i.e. to all router traffic, going out on any interface. Note that you may use the same idea to block incoming session, simply by reversing entries in access-list. (e.g. “permit tcp any eq 23 host 150.1.1.1″). Best of all, with PBR you may apply additional criteria to incoming traffic, e.g. match packet sizes.

The last example is the use of local PBR to apply special treatment to management/control plane traffic - e.g. use different output interfaces for out-of-band management. With local PBR you may also apply special marking for control traffic, e.g. selectively assign IP precedence values.


ip access-list extended MANAGEMENT_TRAFFIC
 permit tcp any eq 23 any
 permit tcp any eq 22 any
!
route-map LOCAL_POLICY 10
 match ip address MANAGEMENT_TRAFFIC
 set interface Serial 0/1
 set ip precedence 7
!
ip local policy route-map LOCAL_POLICY

Keep these simple features in mind, while considering options for you CCIE lab task solution.

-->