CCIE Blog

This is a good example of fragmentation and interleaving, applied in a complex context. To begin with, why whould anyone need to run Multilink PPP (MLPPP or MLP) with Interleaving over Frame-Relay? Well, back in days, when Frame-Relay and ATM were really popular, there was a need to interwork the two technologies: that is, transparently pass encapsulated packets between FR and ATM PVCs. (This is similar in concept with modern L2 VPN interworking, however it was specific to ATM and Frame-Relay). Let’s imagine a situation where we have slow ATM and Frame-Relay links, used to transport a mix of VoIP and data traffic. As we know, some sort of fragmentation and interleaving scheme should be implemented, in order to keep voice quality under control. Since there was no fragmentation scheme common to both ATM and Frame-Relay, people came with idea to run PPP (yet another L2 tech) over Frame-Relay and ATM PVCs and use PPP multilink and interleave feature to implement fragmentation. (Actually there was no good scheme for native fragmentation and interleaving with VoIP over ATM - the cell mode technology - how ironic!)

Before coming up with a configuration example, let’s discuss briefly how PPP Multilink and Interleave works. MLPPP is defined under RFC 1990, and it’s purpose is to group a number of physical links into one logical channel with larger “effective” bandwidth. As we discussed before, MLPPP uses a fragmentation algorithm, where one large frame is being split at Layer2 and replaced with a bunch of sequenced (by the use of additional MLPPP header) smaller frames which are then being sent over multiple physical links in parallel. The receiving side will then accept fragments, reorder some of them if needed, and assemble the pieces into complete frame using the sequence numbers.

So here comes the interleave feature: small voice packets are not fragmented by MLPPP (no MLPPP header and sequence number added) and are simply inserted (intermixed) among the fragments of large data packet. Of course, a special interleaving priority queue is used for this purpose, as we have discussed before.

To summarize:

1) MLPPP uses fragmentation scheme where large packets are sliced in pieces and sequence numbers are added using special MLPPP headers
2) Small voice packets are interleaved with fragments of large packets using a special priority queue

We see that MLPPP was originally designed to work with multiple physical links at the same time. However, PPP Multilink Interleave only works with one physical link. The reason is that voice (small) packets are being sent without sequence numbers. If we were using multiple physical links, the receiving side may start accepting voice packets out of their original order (due to different physical link latencies). And since voice packets bear no fragmentation headers, there is no way to reorder them. In effect, packets may arrive to their final destination out of order, degrading voice quality.

To overcome this obstacle, Multiclass Multilink PPP (MCMLPPP or MCMLP) has been introduced in RFC 2886. Under this RFC, different “fragment streams” or classes are supported at sending and receiving sides, using independent sequence numbers. Therefore, with MCMLPPP voice packets may be sent using MLPPP header with separate sequence numbers space. In result, MCMPPP permits the use of fragmentation and interleaving over multiple physical links at time.

Now back to our MLPPPoFR example. Let’s imagine the situation where we have two routers (R1 and R2) connected via FR cloud, with physical ports clocked at 512Kpbs and PVC CIR values equal to 384Kbps (There is no ATM interworking in this example). We need to provide priority treatment to voice packets and enable PPP Multilink and Interleave to decrease serialization delays.

[R1]---[DLCI 112]---[Frame-Relay]---[DLCI 211]---[R2]

Start by defining MQC policy. We need to make sure that software queue gives voice packets priority treatmet, or else interleaving will be useless

R1 & R2:

!
! Voice bearer
!
class-map VOICE
 match ip dscp ef

!
! Voice signaling
!
class-map SIGNALING
 match ip dscp cs3

!
! CBWFQ: priority treatment for voice packets
!
policy-map CBWFQ
 class VOICE
  priority 48
 class SIGNALING
  bandwidth 8
 class class-default
  fair-queue

Next create a Virtual-Template interface for PPPoFR. We need to calculate the fragment size for MLPPP. Since physical port speed is 512Kpbs, and required serialization delay should not exceed 10ms (remember, fragment size is based on physical port speed!), the fragment size must be set to 512000/8*0,01=640 bytes. How is the fragment size configured with MLPPP? By using command ppp multilink fragment delay - however, IOS CLI takes this delay value (in milliseconds) and multiplies it by configured interface (virtual-template) bandwidth (in our case 384Kbps). We can actually change the virtual-template bandwidth to match the physical interface speed, but this would affect the CBWFQ weights! Therefore, we take the virtual-template bandwidth (384Kpbs) and adjust the delay to make sure the fragment size matches the physical interace rate is 512Kpbs. This way, the “effective” delay value would be set to “640*8/384 = 13ms” (Fragment_Size/CIR*8) to accomodate the physical and logical bandwidth discrepancy. (This may be unimportant if our physical port speed does not differ much from PVC CIR. However, if you have say PVC CIR=384Kbps and port speed 768Kbps you may want to pay attention to this issue)

R1:
interface Loopback0
 ip address 177.1.101.1 255.255.255.255
!
interface Virtual-Template 1
 encapsulation ppp
 ip unnumbered Loopback 0
 bandwidth 384
 ppp multilink
 ppp multilink interleave
 ppp multilink fragment delay 13
 service-policy output CBWFQ

R2:
interface Loopback0
 ip address 177.1.102.1 255.255.255.255
!
interface Virtual-Template 1
 encapsulation ppp
 ip unnumbered Loopback 0
 bandwidth 384
 ppp multilink
 ppp multilink interleave
 ppp multilink fragment delay 13
 service-policy output CBWFQ

Next we configure PVC shaping settings by using legacy FRTS configuration. Note that Bc is set to CIR*10ms.

R1 & R2:
map-class frame-relay SHAPE_384K
frame-relay cir 384000
frame-relay mincir 384000
frame-relay bc 3840
frame-relay be 0

Finally we apply all the settings to the Frame-Relay interfaces:

R1:
interface Serial 0/0/0:0
 encapsulation frame-relay
 frame-relay traffic-shaping
!
! Virtual Template bound to PVC
!
interface Serial 0/0/0:0.1 point-to-point
 no ip address
 frame-relay interface-dlci 112 ppp virtual-template 1
  class SHAPE_384K

R2:
interface Serial 0/0/1:0
 encapsulation frame-relay
 frame-relay traffic-shaping
!
! Virtual Template bound to PVC
!
interface Serial 0/0/1:0.1  point-to-point
 no ip address
 no frame-relay interface-dlci 221
 frame-relay interface-dlci 211 ppp virtual-Template 1
  class SHAPE_384K

Verification

Two virtual-access interfaces have been cloned. First for the member link:

R1#show interfaces virtual-access 2
Virtual-Access2 is up, line protocol is up
  Hardware is Virtual Access interface
  Interface is unnumbered. Using address of Loopback0 (177.1.101.1)
  MTU 1500 bytes, BW 384 Kbit, DLY 100000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation PPP, LCP Open, multilink Open
  Link is a member of Multilink bundle Virtual-Access3   <---- MLP bundle member
  PPPoFR vaccess, cloned from Virtual-Template1
  Vaccess status 0x44
  Bound to Serial0/0/0:0.1 DLCI 112, Cloned from Virtual-Template1, loopback not set
  Keepalive set (10 sec)
  DTR is pulsed for 5 seconds on reset
  Last input 00:00:52, output never, output hang never
  Last clearing of "show interface" counters 00:04:17
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo       <---------- FIFO is the member link queue
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     75 packets input, 16472 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     86 packets output, 16601 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions

Second for the MLPPP bundle itself:

R1#show interfaces virtual-access 3
Virtual-Access3 is up, line protocol is up
  Hardware is Virtual Access interface
  Interface is unnumbered. Using address of Loopback0 (177.1.101.1)
  MTU 1500 bytes, BW 384 Kbit, DLY 100000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation PPP, LCP Open, multilink Open
  Open: IPCP
  MLP Bundle vaccess, cloned from Virtual-Template1   <---------- MLP Bundle
  Vaccess status 0x40, loopback not set
  Keepalive set (10 sec)
  DTR is pulsed for 5 seconds on reset
  Last input 00:01:29, output never, output hang never
  Last clearing of "show interface" counters 00:03:40
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: Class-based queueing    <--------- CBWFQ is the bundle queue
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/1/128 (active/max active/max total)
     Reserved Conversations 1/1 (allocated/max allocated)
     Available Bandwidth 232 kilobits/sec
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     17 packets input, 15588 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     17 packets output, 15924 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions

Verify the CBWFQ policy-map:

R1#show policy-map interface
 Virtual-Template1 

  Service-policy output: CBWFQ

    Service policy content is displayed for cloned interfaces only such as vaccess and sessions
 Virtual-Access3 

  Service-policy output: CBWFQ

    Class-map: VOICE (match-all)
      0 packets, 0 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: ip dscp ef (46)
      Queueing
        Strict Priority
        Output Queue: Conversation 136
        Bandwidth 48 (kbps) Burst 1200 (Bytes)
        (pkts matched/bytes matched) 0/0
        (total drops/bytes drops) 0/0

    Class-map: SIGNALING (match-all)
      0 packets, 0 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: ip dscp cs3 (24)
      Queueing
        Output Queue: Conversation 137
        Bandwidth 8 (kbps) Max Threshold 64 (packets)
        (pkts matched/bytes matched) 0/0
        (depth/total drops/no-buffer drops) 0/0/0

    Class-map: class-default (match-any)
      17 packets, 15554 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any
      Queueing
        Flow Based Fair Queueing
        Maximum Number of Hashed Queues 128
        (total queued/total drops/no-buffer drops) 0/0/0

Check PPP multilink status:

R1#ping 177.1.102.1 source loopback 0 size 1500

Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 177.1.102.1, timeout is 2 seconds:
Packet sent with a source address of 177.1.101.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/64/64 ms

R1#show ppp multilink

Virtual-Access3, bundle name is R2
  Endpoint discriminator is R2
  Bundle up for 00:07:49, total bandwidth 384, load 1/255
  Receive buffer limit 12192 bytes, frag timeout 1000 ms
  Interleaving enabled            <------- Interleaving enabled
    0/0 fragments/bytes in reassembly list
    0 lost fragments, 0 reordered
    0/0 discarded fragments/bytes, 0 lost received
    0x34 received sequence, 0x34 sent sequence   <---- MLP sequence numbers for fragmented packets
  Member links: 1 (max not set, min not set)
    Vi2, since 00:07:49, 624 weight, 614 frag size <------- Fragment Size
No inactive multilink interfaces

Verify the interleaving queue:

R1#show interfaces serial 0/0/0:0
Serial0/0/0:0 is up, line protocol is up
  Hardware is GT96K Serial
  MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation FRAME-RELAY, loopback not set
  Keepalive set (10 sec)
  LMI enq sent  10, LMI stat recvd 11, LMI upd recvd 0, DTE LMI up
  LMI enq recvd 0, LMI stat sent  0, LMI upd sent  0
  LMI DLCI 1023  LMI type is CISCO  frame relay DTE
  FR SVC disabled, LAPF state down
  Broadcast queue 0/64, broadcasts sent/dropped 4/0, interface broadcasts 0
  Last input 00:00:05, output 00:00:02, output hang never
  Last clearing of "show interface" counters 00:01:53
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: dual fifo                        <--------- Dual FIFO
  Output queue: high size/max/dropped 0/256/0         <--------- High Queue
  Output queue: 0/128 (size/max)                      <--------- Low (fragments) queue
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     47 packets input, 3914 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     1 input errors, 1 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     47 packets output, 2149 bytes, 0 underruns
     0 output errors, 0 collisions, 4 interface resets
     0 output buffer failures, 0 output buffers swapped out
     1 carrier transitions
  Timeslot(s) Used:1-24, SCC: 0, Transmitter delay is 0 flags

Further Reading

Reducing Latency and Jitter for Real-Time Traffic Using Multilink PPP
Multiclass Multilink PPP
Using Multilink PPP over Frame Relay

This is a “modern” way to configure FRTS, using MQC commands only to accomplish the task. With MQC approach, an unified interface has been introduced to configure all QoS settings, irrelevant of underlying technology.

In summary:

- Legacy command frame-relay traffic-shaping is incompatible with MQC-based FRTS (you can’t mix them)
- Fancy queueing could not be used as a PVC-queueing strategy: CBWFQ is the only option available
- Per-VC CBWFQ is configured via hierarchical policy-maps configuration: Parent policy sets shaping values, while child policy implements CBWFQ
- You may apply policy-map per-interface (subinterface) or per-VC, using match fr-dlci under class-map submode

Example: Shape PVC to 384Kbps and provide LLQ treatment for voice bearer packets on PVC queue

class-map VOICE
 match ip dscp ef
!
class-map DATA
 match ip dscp cs1

!
! "Child" policy-map, used to implement CBWFQ
!

policy-map CBWFQ
 class VOICE
  priority 64
 class DATA
  bandwidth 128
 class class-default
  fair-queue

!
! "Parent" policy map, used for PVC shaping
!

policy-map SHAPE_384K
 class class-default
  shape average 384000
  shape adaptive 192000
  service-policy CBWFQ

interface Serial 0/0/0:0.1
 ip address 177.0.112.1 255.255.255.0
 service-policy output SHAPE_384K
 frame-relay interface-dlci 112

Verification: check out policy map settings

Rack1R1#show policy-map interface serial 0/0/0:0.1

 Serial0/0/0:0.1 

  Service-policy output: SHAPE_384K

    Class-map: class-default (match-any)
      1942 packets, 1590741 bytes
      5 minute offered rate 48000 bps, drop rate 0 bps
      Match: any
      Traffic Shaping
           Target/Average   Byte   Sustain   Excess    Interval  Increment
             Rate           Limit  bits/int  bits/int  (ms)      (bytes)
           384000/384000    2400   9600      9600      25        1200     

        Adapt  Queue     Packets   Bytes     Packets   Bytes     Shaping
        Active Depth                         Delayed   Delayed   Active
        -      0         1936      1581717   0         0         no

      Service-policy : CBWFQ

        Class-map: VOICE (match-all)
          0 packets, 0 bytes
          5 minute offered rate 0 bps, drop rate 0 bps
          Match: protocol rtp
          Match: ip dscp ef (46)
          Queueing
            Strict Priority
            Output Queue: Conversation 40
            Bandwidth 64 (kbps) Burst 1600 (Bytes)
            (pkts matched/bytes matched) 0/0
            (total drops/bytes drops) 0/0

        Class-map: DATA (match-all)
          0 packets, 0 bytes
          5 minute offered rate 0 bps, drop rate 0 bps
          Match: ip dscp cs1 (8)
          Queueing
            Output Queue: Conversation 41
            Bandwidth 128 (kbps) Max Threshold 64 (packets)
            (pkts matched/bytes matched) 0/0
        (depth/total drops/no-buffer drops) 0/0/0

        Class-map: class-default (match-any)
          1942 packets, 1590741 bytes
          5 minute offered rate 48000 bps, drop rate 0 bps
          Match: any
          Queueing
            Flow Based Fair Queueing
            Maximum Number of Hashed Queues 32
        (total queued/total drops/no-buffer drops) 0/0/0

The amount of bandwidth, available for allocation to CBWFQ classes, is taken from shape adaptive value. If the latter is not configured, shape average
value is used instead. Note, that as you configure bandwidth settings for classes, their values are not subtracted from remaining bandwidth. This is in contraty with
“classic” CBWFQ, applied to a physical interface (not subinterface or PVC)

Verification (with the example above):

Rack1R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
Rack1R1(config)#policy-map CBWFQ
Rack1R1(config-pmap)#class class-default
Rack1R1(config-pmap-c)#no fair-queue
Rack1R1(config-pmap-c)#bandwidth 256
I/f  shape  class class-default requested bandwidth 256 (kbps), available only 192 (kbps)

Note that available bandwidth is set to shape adaptive value, even though we have priority configured under class VOICE and bandwidth
settings under class DATA

- You can’t apply FRF.12 fragmentation with MQC commands - it should be applied at physical interface level. By doing so, FRF.12 is effectively enabled for all PVCs
- Physical interface queue could be set to any of WFQ/CQ/PQ or CBWFQ (not restricted to FIFO as with FRTS legacy) - though this is rarely needed

Example: Shape PVC DLCI 112 to 384Kpbs and enable FRF.12 fragmentation for all PVCs

class-map VOICE
 match ip dscp ef
!
class-map DATA
 match ip dscp cs1

!
! Match the specific DLCI
!
class-map DLCI_112
 match fr-dlci 112

!
! "Child" policy-map, used to implement CBWFQ
!

policy-map CBWFQ
 class VOICE
  priority 64
 class DATA
  bandwidth 128
 class class-default
  fair-queue

!
! "Parent" policy map, used for PVC shaping
!  With multiple classes, we can match different DLCIs
!  all at the same physical interface (where they belongs)
!

policy-map INTERFACE_POLICY
 class DLCI_112
  shape average 384000
  shape adaptive 192000
   service-policy CBWFQ

!
! Apply the parent policy map at physical interface level
! Also, configure FRF.12 "global" settings here
!

interface Serial 0/0/0:0
 service-policy output INTERFACE_POLICY
 frame-relay fragment 640 end-to-end

Verification:

Rack1R1#show policy-map interface serial 0/0/0:0

 Serial0/0/0:0 

  Service-policy output: INTERFACE_POLICY

    Class-map: DLCI_112 (match-all)
      1040 packets, 95287 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: fr-dlci 112
      Traffic Shaping
           Target/Average   Byte   Sustain   Excess    Interval  Increment
             Rate           Limit  bits/int  bits/int  (ms)      (bytes)
           384000/384000    2400   9600      9600      25        1200     

        Adapt  Queue     Packets   Bytes     Packets   Bytes     Shaping
        Active Depth                         Delayed   Delayed   Active
        -      0         1040      95287     4         1373      no

      Service-policy : CBWFQ

        Class-map: VOICE (match-all)
          0 packets, 0 bytes
          5 minute offered rate 0 bps, drop rate 0 bps
          Match: protocol rtp
          Match: ip dscp ef (46)
          Queueing
            Strict Priority
            Output Queue: Conversation 40
            Bandwidth 64 (kbps) Burst 1600 (Bytes)
            (pkts matched/bytes matched) 0/0
            (total drops/bytes drops) 0/0

        Class-map: DATA (match-all)
          0 packets, 0 bytes
          5 minute offered rate 0 bps, drop rate 0 bps
          Match: ip dscp cs1 (8)
          Match: fr-dlci 112
          Queueing
            Output Queue: Conversation 41
            Bandwidth 128 (kbps) Max Threshold 64 (packets)
            (pkts matched/bytes matched) 0/0
        (depth/total drops/no-buffer drops) 0/0/0

        Class-map: class-default (match-any)
          1040 packets, 95287 bytes
          5 minute offered rate 0 bps, drop rate 0 bps
          Match: any
          Queueing
            Flow Based Fair Queueing
            Maximum Number of Hashed Queues 32
        (total queued/total drops/no-buffer drops) 0/0/0

    Class-map: class-default (match-any)
      2594 packets, 153695 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any

Verify fragmentation settings:

Rack1R1#show interface serial 0/0/0:0
Serial0/0/0:0 is up, line protocol is up
  Hardware is GT96K Serial
  MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation FRAME-RELAY, loopback not set
  Keepalive set (10 sec)
  LMI enq sent  21224, LMI stat recvd 21224, LMI upd recvd 0, DTE LMI up
  LMI enq recvd 0, LMI stat sent  0, LMI upd sent  0
  LMI DLCI 1023  LMI type is CISCO  frame relay DTE
  FR SVC disabled, LAPF state down
  Fragmentation type: end-to-end, size 640, PQ interleaves 0   <--------- Fragment Size
  Broadcast queue 0/64, broadcasts sent/dropped 63160/0, interface broadcasts 56080
  Last input 00:00:03, output 00:00:02, output hang never
  Last clearing of "show interface" counters 2d10h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 6
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/1/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1152 kilobits/sec
  5 minute input rate 0 bits/sec, 1 packets/sec
  5 minute output rate 0 bits/sec, 1 packets/sec
     272202 packets input, 27557680 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     15 input errors, 15 CRC, 8 frame, 0 overrun, 0 ignored, 5 abort
     333676 packets output, 42152431 bytes, 0 underruns
     0 output errors, 0 collisions, 16 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
  Timeslot(s) Used:1-24, SCC: 0, Transmitter delay is 0 flags

This is the most well-known FRTS method, which has been available for quite a while on Cisco routers. It is now being outdated by MQC configurations.
The key characteristic is that all settings are configured under map-class command mode, and later are applied to a particular set PVCs. The
same configuration concept was used for legacy ATM configuration mode (map-class atm).

Legacy FRTS has the following characteristics:

- Enabled with frame-relay traffic-shaping command at physical interface level
- Incompatible with GTS or MQC commands at subinterfaces or physical interface levels
- With FRTS you can enforce bitrate per-VC (VC-granular, unlike GTS), by applying a map-class to PVC
- When no map-class is explicitly applied to PVC, it’s CIR and Tc are set to 56K/125ms by default
- Shaping parameters are configured under map-class frame-relay configuration submode
- Allows to configure fancy-queueing (WFQ/PQ/CQ) or simple FIFO per-VC
- No option to configure fancy-queueing at interface level: interface queue is forced to FIFO (if no FRF.12 is configured)
- Allows for adaptive shaping (throttling down to minCIR) on BECN reception (just as GTS) and option to reflect incoming FECNs as BECNs
- Option to enable adaptive shaping which responds to interface congestion (non-empty interface queue)

Example: Shape PVC to 384Kbps with minimal Tc (10ms) and WFQ as interface queue

map-class frame-relay SHAPE_384K
 frame-relay cir 384000
 frame-relay bc 3840
 frame-relay be 0
 !
 ! Adaptive shaping: respond to BECNs and interface congestion
 !
 frame-relay adaptive-shaping becn
 frame-relay adaptive-shaping interface-congestion
 !
 ! Per-VC fancy-queueing
 !
 frame-relay fair-queue
!
interface Serial 0/0/0:0
 frame-relay traffic-shaping
!
interface Serial 0/0/0:0.1
 ip address 177.0.112.1 255.255.255.0
 frame-relay interface-dlci 112
  class SHAPE_384K

Verification: Check FRTS settings for the configured PVC

Rack1R1#show frame-relay pvc 112

PVC Statistics for interface Serial0/0/0:0 (Frame Relay DTE)

DLCI = 112, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial0/0/0:0

  cir 384000    bc 3840      be 0         byte limit 480    interval 10   <------ Shaping parameters
  mincir 192000    byte increment 480   Adaptive Shaping BECN and IF_CONG <---- Adaptive Shaping enabled
  pkts 0         bytes 0         pkts delayed 0         bytes delayed 0
  shaping inactive
  traffic shaping drops 0
  Queueing strategy: weighted fair  <---- WFQ is the per-VC queue
  Current fair queue configuration:
   Discard     Dynamic      Reserved
   threshold   queue count  queue count
    64          16           0
  Output queue size 0/max total 600/drops 0

The other PVC, with no class configured, has CIR set to 56Kbps and uses FIFO as per-VC queue:

Rack1R1#show frame-relay pvc 113

PVC Statistics for interface Serial0/0/0:0 (Frame Relay DTE)

DLCI = 113, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial0/0/0:0.2

  cir 56000     bc 7000      be 0         byte limit 875    interval 125 <---- CIR=56K
  mincir 28000     byte increment 875   Adaptive Shaping none
  pkts 74        bytes 5157      pkts delayed 0         bytes delayed 0
  shaping inactive
  traffic shaping drops 0
  Queueing strategy: fifo <------------------ FIFO
  Output queue 0/40, 0 drop, 0 dequeued

Check the physical interface queue:

Rack1R1#show interfaces serial 0/0/0:0 | inc Queue
  Queueing strategy: fifo

- Interface queue could be changed to PIPQ (PVCs are assigned to 4 pririty groups, with PQ being physical interface queue)
- PIPQ stands for PVC Interface Priority queueing

Example: Map PVC 112 traffic to high interface queue, and PVC 113 to low interface queue, with WFQ being per-VC queueing

!
! Shape to 384K an assign to high interface level queue
!
map-class frame-relay SHAPE_384K_HIGH
 frame-relay cir 384000
 frame-relay bc 3840
 frame-relay be 0
 !
 ! Per-VC fancy-queueing
 !
 frame-relay fair-queue
 frame-relay interface-queue priority high

!
! Shape to 256k an assign to low interface level queue
!
map-class frame-relay SHAPE_256K_LOW
 frame-relay cir 256000
 frame-relay bc 2560
 frame-relay be 0
 !
 ! Per-VC fancy-queueing
 !
 frame-relay fair-queue
 frame-relay interface-queue priority low

!
! Enable PIPQ as interface queueing strategy
!
interface Serial 0/0/0:0
 frame-relay traffic-shaping
 frame-relay interface-queue priority
!
interface Serial 0/0/0:0.1
 ip address 177.0.112.1 255.255.255.0
 frame-relay interface-dlci 112
  class SHAPE_384K_HIGH
!
interface Serial 0/0/0:0.2
 ip address 177.0.113.1 255.255.255.0
 frame-relay interface-dlci 113
  class SHAPE_256K_LOW

Verfication: Check PVC interface-level priorities

Rack1R1#show frame-relay pvc 112

PVC Statistics for interface Serial0/0/0:0 (Frame Relay DTE)

DLCI = 112, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial0/0/0:0.1

  cir 384000    bc 3840      be 0         byte limit 480    interval 10
  mincir 192000    byte increment 480   Adaptive Shaping none
  pkts 1687      bytes 113543    pkts delayed 0         bytes delayed 0
  shaping inactive
  traffic shaping drops 0
  Queueing strategy: weighted fair
  Current fair queue configuration:
   Discard     Dynamic      Reserved
   threshold   queue count  queue count
    64          16           0
  Output queue size 0/max total 600/drops 0
  priority high
  ^^^^^^^^^^^^^

Rack1R1#show frame-relay pvc 113

PVC Statistics for interface Serial0/0/0:0 (Frame Relay DTE)

DLCI = 113, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial0/0/0:0.2

  cir 256000    bc 2560      be 0         byte limit 320    interval 10
  mincir 128000    byte increment 320   Adaptive Shaping none
  pkts 1137      bytes 79691     pkts delayed 0         bytes delayed 0
  shaping inactive
  traffic shaping drops 0
  Queueing strategy: weighted fair
  Current fair queue configuration:
   Discard     Dynamic      Reserved
   threshold   queue count  queue count
    64          16           0
  Output queue size 0/max total 600/drops 0
  priority low
  ^^^^^^^^^^^^

Verify interface-level queue:

Rack1R1#show interfaces serial 0/0/0:0 | inc (Output|high)
  Output queue (queue priority: size/max/drops):
     high: 0/20/0, medium: 0/40/0, normal: 0/60/0, low: 0/80/0

- With FRF.12 fragmentation configured per any PVC, physical interface queue is changed to dual-FIFO
- This is due to the fact that fragmentation is ineffective without interleaving
- Fragment size is calculated based on physical interface speed to allow minimum serialization delay

Example: Enable FRF.12 fragmentation for PVC DLCI 112 and physical interface speed 512Kbps

!
! PVC shaped to 384Kbps, with physical interface speed 512Kbps
!
map-class frame-relay SHAPE_384K_FRF12
 frame-relay cir 384000
 frame-relay bc 3840
 frame-relay be 0
 !
 ! Per-VC fancy-queueing
 !
 frame-relay fair-queue

 !
 ! Enable FRF.12 per VC. Fragment size = 512Kbps*0,01/8 = 640 bytes
 !
 frame-relay fragment 640
!
!
!
interface Serial 0/0/0:0
 frame-relay traffic-shaping
!
interface Serial 0/0/0:0.1
 ip address 177.0.112.1 255.255.255.0
 frame-relay interface-dlci 112
  class SHAPE_384K_FRF12

Verfication: Check PVC settings

Rack1R1#show frame-relay pvc 112

PVC Statistics for interface Serial0/0/0:0 (Frame Relay DTE)

DLCI = 112, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial0/0/0:0.1

  fragment type end-to-end fragment size 640
  cir 384000    bc   3840      be 0         limit 480    interval 10
  mincir 192000    byte increment 480   BECN response no  IF_CONG no
  frags 1999      bytes 135126    frags delayed 0         bytes delayed 0
  shaping inactive
  traffic shaping drops 0
  Queueing strategy: weighted fair
  Current fair queue configuration:
   Discard     Dynamic      Reserved
   threshold   queue count  queue count
    64          16           0
  Output queue size 0/max total 600/drops 0

Look at physical interface queue:

Rack1R1#show interfaces serial 0/0/0:0 | inc Queu|Output q
  Queueing strategy: dual fifo
  Output queue: high size/max/dropped 0/256/0
  Output queue: 0/128 (size/max)

- You can map up to 4 priority queues to 4 different VCs (inverse PIPQ)
- This scenario usually implies multiple PVCs running between two sites (e.g. PVC for voice and PVC for data)

Example: Map voice packets to high interface-level priority queue and send them over PVC 112

!
! Voice bearer
!
access-list 101 permit udp any any range 16384 32767

!
! Simple priority list to classify voice bearer to high queue
!
priority-list 1 protocol ip high list 101

interface Serial 0/0/0:0
 ip address 177.1.0.1 255.255.255.0
 !
 ! We apply the priority group twice: first to implement queueing
 !
 priority-group 1
 !
 ! Next to map priority levels to DLCIs
 !
 frame-relay priority-dlci-group 1 112 112 113 113

Verfication:

Rack1R1#show queueing interface serial 0/0/0:0
Interface Serial0/0/0:0 queueing strategy: priority

Output queue utilization (queue/count)
	high/217 medium/0 normal/1104 low/55 

Rack1R1#show frame-relay pvc 112

PVC Statistics for interface Serial0/0/0:0 (Frame Relay DTE)

DLCI = 112, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial0/0/0:0

  pvc create time 3d01h, last time pvc status changed 3d01h
  Priority DLCI Group 1, DLCI 112 (HIGH), DLCI 112 (MEDIUM)
  DLCI 113 (NORMAL), DLCI 113 (LOW)

- You can change per-VC queue to CBWFQ/LLQ, and still shape with FRTS
- Note that available bandwidth will be calculated from minCIR value, which is CIR/2 by default

Example: Implement CBWFQ Per-VC

!
! Classify voice using NBAR
!
class-map VOICE
 match protocol rtp

!
! Simple LLQ
!
policy-map CBWFQ
 class VOICE
  priority 64
 class class-default
  bandwidth 64

!
! Use CBWFQ as queueing strategy
! Note that MinCIR = 384/2=192Kbps
!
map-class frame-relay SHAPE_384K_CBWFQ
 frame-relay cir 384000
 frame-relay bc 3840
 frame-relay be 0
 service-policy output CBWFQ
!
!
!
interface Serial 0/0/0:0
 frame-relay traffic-shaping
!
interface Serial 0/0/0:0.1
 ip address 177.0.112.1 255.255.255.0
 frame-relay interface-dlci 112
  class SHAPE_384K_CBWFQ

Verfication: Check PVC queueing strategy

Rack1R1#show frame-relay pvc 112

PVC Statistics for interface Serial0/0/0:0 (Frame Relay DTE)

DLCI = 112, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial0/0/0:0

  cir 384000    bc 3840      be 0         byte limit 480    interval 10
  mincir 192000    byte increment 480   Adaptive Shaping none
  pkts 0         bytes 0         pkts delayed 0         bytes delayed 0
  shaping inactive
  traffic shaping drops 0
  service policy CBWFQ
 Serial0/0/0:0: DLCI 112 -

  Service-policy output: CBWFQ

    Class-map: VOICE (match-all)
      0 packets, 0 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: protocol rtp
      Queueing
        Strict Priority
        Output Queue: Conversation 40
        Bandwidth 64 (kbps) Burst 1600 (Bytes)
        (pkts matched/bytes matched) 0/0
        (total drops/bytes drops) 0/0

    Class-map: class-default (match-any)
      32 packets, 2560 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any
      Queueing
        Output Queue: Conversation 41
        Bandwidth 128 (kbps) Max Threshold 64 (packets)
        (pkts matched/bytes matched) 0/0
        (depth/total drops/no-buffer drops) 0/0/0
  Output queue size 0/max total 600/drops 0

To verify that only MinCIR of bandwidth is allocated to CBWFQ under map-class do the following:

Rack1R1(config)#policy-map CBWFQ
Rack1R1(config-pmap)# class VOICE
Rack1R1(config-pmap-c)#  priority 64
Rack1R1(config-pmap-c)# class class-default
Rack1R1(config-pmap-c)#  bandwidth 192
I/f Serial0/0/0:0 DLCI 112 Class class-default requested bandwidth 192 (kbps) Only 128 (kbps) available

As first and very basic option, you may use Generic Traffic Shaping to implement FRTS. This is a common technique, not unique to Frame-Relay, with the following properties:

- Configured by using traffic-shape interface command
- As with standard GTS, internal shaper queue is basic WFQ
- Configured per inteface/subinteface (no PVC granularity)
- GTS may adapt to BECNs & reflect FECNs (BECN received at any interface/subinteface PVC will cause shaper rate to throttle back to minCIR)
- FECN reflection activates sending BECNs in responce to incoming FECN frames. Note, that FECN/BECN responce requires provider to mark frames with FECN/BECN bits
- You can configure Fancy-Queueing (WFQ/PQ/CQ) at physical interface level with GTS.

Example:

!
! Physical Interface, fancy-queueing may apply here
!

interface Serial 0/0/0:0
 no ip address
 encapsulation frame-relay
 fair-queue

!
! Subinterface, apply GTS here
!

interface Serial0/0/0:0.1 point-to-point
 ip address 177.0.112.1 255.255.255.0
 !
 ! Shaping rate
 !
 traffic-shape rate 512000
 !
 ! "MinCIR", adapt to BECNs
 !
 traffic-shape adaptive 256000
 !
 ! Reflect FECNs as BECNs
 !
 traffic-shape fecn-adapt
 frame-relay interface-dlci 112

Verification:

Rack1R1#show traffic-shape 

Interface   Se0/0/0:0.1
       Access Target    Byte   Sustain   Excess    Interval  Increment Adapt
VC     List   Rate      Limit  bits/int  bits/int  (ms)      (bytes)   Active
-             512000    3200   12800     12800     25        1600      BECN

Rack1R1#show traffic-shape statistics
                  Acc. Queue Packets   Bytes     Packets   Bytes     Shaping
I/F               List Depth                     Delayed   Delayed   Active
Se0/0/0:0.1             0     157       10500     0         0         no

Rack1R1#show traffic-shape queue
Traffic queued in shaping queue on Serial0/0/0:0.1
  Queueing strategy: weighted fair
  Queueing Stats: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/0/32 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 512 kilobits/sec

When configuring a Frame Relay switch layer 1 DCE/DTE is independent of layer 2 DCE/DTE. The “clock rate” command can only be applied on the layer 1 DCE side of the cable. This can be determined by looking at the cable for a DTE/DCE labeling, using the “show controllers serial X/X” command or by just issuing the “clock rate” command on both sides. The side that accepts the command is the layer 1 DCE.

In regards to Frame Relay layer 2 DCE is independent of the layer 1 DCE. Commonly the layer 1 DCE end of the cable is connected to the FRS and the layer 2 DCE is also configured on the FRS side. The configuration of the Frame Relay DCE can be done by using the “frame-relay intf-type dce” command. By default Frame Relay interfaces are DTE. Also as a point of interest the Frame Relay DCE side is the side that commonly generates LMI. I used the word commonly because although unusual you can have a Frame Relay connection without LMI.

Hello Brian,Can you explain how PPP over Frame Relay works? Also what are the advantages and disadvantages of using it over normal Frame Relay configuration?Thanks and regards,

Yaser

Hi Yaser,

Frame Relay does not natively support features such as authentication, link quality monitoring, and reliable transmission. Based on this it is sometimes advantageous to encapsulate an additional PPP header between the normal layer 2 Frame Relay encapsulation and the layer 3 protocol. By running PPP over Frame Relay (PPPoFR) we can then implement authentication of Frame Relay PVCs, or even bind multiple PVCs together using PPP Multilink.

PPPoFR is configure in Cisco IOS through the usage of a Virtual-Template interface. A Virtual-Template is a PPP encapsulated interface that is designed to spawn a “template” of configuration down to multiple member interfaces. The traditional usage of this interface has been on dial-in access servers, such as the AS5200, to support multiple PPP dialin clients terminating their connection on a single interface running IP.

The first step in configuring PPPoFR is to create the Virtual-Template interface. This interface is where all logical options, such as IP address and PPP authentication will be configured. The syntax is as follows:

interface Virtual-Template1
 ip address 54.1.7.6 255.255.255.0
 ppp chap hostname ROUTER6
 ppp chap password 0 CISCO

Note the lack of the “encapsulation ppp” command on the Virtual-Template. This command is not needed as a Virtual-Template is always running PPP. This can be seen by looking at the “show interface virtual-template1” output in the IOS. Additionally in this particular case the remote end of this connection will be challenging the router to authenticate via PPP CHAP. The “ppp chap” subcommands have instructed the router to reply with the username ROUTER6 and an MD5 hash value of the PPP magic number and the password CISCO.

Our next step is to configure the physical Frame Relay interface, and to bind the Virtual-Template to the Frame Relay PVC. This is accomplished as follows:

interface Serial0/0
 encapsulation frame-relay
 frame-relay interface-dlci 201 ppp Virtual-Template1

Note that the “no frame-relay inverse-arp” command is not used on this interface. Since our IP address is located on the Virtual-Template interface the Frame Relay process doesn’t actually see IP running over the link. Instead it simply sees a PPP header being encapsulated on the link, while the IPCP protocol of PPP takes care of all the IP negotiation for us. Note that the order that these steps are performed in is significant. If a Virtual-Template interface is applied to a Frame Relay PVC before it is actually created you may see difficulties with getting the link to become active.

Also when using a Virtual-Template interface it’s important to understand that a Virtual-Access “member” interface is cloned from the Virtual-Template interface when the PPP connection comes up. Therefore the Virtual-Template interface itself will always be in the down/down state. This can affect certain network designs such as using the backup interface command on a Virtual-Template. In our particular case we can see from the below output this effect:

R6#show ip interface brief | include 54.1.7.6
Virtual-Access1 54.1.7.6 YES TFTP up up
Virtual-Template1 54.1.7.6 YES manual down down

Aside from this there is no other configuration that directly relates to Frame Relay for PPP. Other options such as authentication, reliability, and multilink would be configured under the Virtual-Template interface.