Loading ...Generally, flow-control is a mechanics which allows the receiving party of a connection to control the rate of the sending party. You may see many different implementations of flow-control technologies at different levels of OSI model (e.g. XON/XOFF for RS232, TCP sliding window, B2B credits for Fibre Channel, FECN/BECN for Frame-Relay, ICMP source-quench message, etc).
For the original Ethernet technology on half-duplex connections there was no possibility to implement explicit (signaled) flow control, since only one side could send frames at time. (However, some may still remember the Cisco’s so-called “back-pressure” feature on some of the older switches (e.g. Cisco Catalyst 1924. The idea was that switch may send barrage of dummy frames on a half-duplex link, effectively preventing the station from transmitting information at given moments of time).
Things changed with full-duplex point-to-point connections. The idea of IEEE 802.3X flow-control was to allow a station on a point-to-point link to send a special “PAUSE” frame (sent to a reserved multicast MAC address 01:80:C2:00:00:01) to signal the other end of the connection to pause transmission for a certain amount of time - the amount is specified in the frame. (Note that the PAUSE frame uses MAC LLC encapsulation). This simple scheme allows for a switch to signal a station to stop overwhelming the connection and vice-versa: station is able to tell switch to slow the sending rate. What should a station or a switch do with the delayed frames left to a specific implementation.
As switches evolved, the simple idea ceased to work in scalable and effecient manner. Nowdays applications need more advanced QoS features other than simple flow-control. Modern Layer3 switches are capable of performing policing, queueing and scheduling to implement complicated Diff-Serv based QoS policies. Messing all that complex functionality with simple flow control may ruin the whole thing. Imagine your switch trying to do it best to provide expedited forwarding for CoS 5 voice frames and suddenly recieving PAUSE frame from a station, telling it to shut up for say 30 milliseconds.
This is why it’s always advised to turn off 802.3X flow control whether you enable MLS QoS on a Cisco switch be it 3550 or 3560. By default flowcontrol is disabled and you can only enable a Cisco switch to receive PAUSE frames, but not to send them.
Use the following commands to set up and verify your current settings:
SW1:
interface GigabitEthernet0/1
flowcontrol receive on
giga1-sw-ats64#show interfaces gigabitEthernet 0/1 flowcontrol
Port Send FlowControl Receive FlowControl RxPause TxPause
admin oper admin oper
——— ——– ——– ——– ——– ——- ——-
Gi0/1 Unsupp. Unsupp. on on 0 0
Nice article Petr! Recently with all the FCoE and datacenter ethernet fuzz and the demand for a lossless fabric, PAUSE frames are back. The 802.1Qbb workinggroup is working on a per-priority PAUSE mechanism, overcoming the problems you’ve described.
Hi Petr,
Thanks for the great post.
I once ‘broke’ a network by including “flowcontrol receive desired” in a macro applied to the configuration of all ‘uplinks’.
Please can you clarify something for me –> Can ethernet flowcontrol ‘break’ TCP’s mechanisms for flow control in any way?
Somebody once mentioned it to me and it had me thinking…
Richard
We ran into an issue with our 6500 (which was configured by default for send flow control to be “desired”). When interfacing with a Nortel Passport, the link wouldn’t pass traffic. Once we disabled flow control, things worked fine.
To: Richard Bannister
TCP uses somewhat sophisticated “end-to-end” flow-control model based on sliding-window concept. This model implicitly accounts for the RTT between the endpoints, available end-to-end bandwidth and buffer space utilization at each endpoint. Note that TCP interprets packet loss as a sign of congestion in the transmission path.
802.3x Flow-Control is based on a “hop-by-hop” local concept, e.g. it only takes in account the loading of the link between a stations and a switch. Moreover, it’s up to NIC driver to decide whether to send the PAUSE frame and how long to hold the locally connected sender, and this process probably does not involve consulting with TCP/IP stack.
Therefore, as you can see, 802.3x flow-control may affect TCP performance in the following manner:
1) Changing the effective RTT between the endpoints by introducing artificial delay
2) Causig excessive packet drops and resulting in TCP slowing the sending rate
3) Possibly introducing TCP flows “synchronization” issue due to tail-end dropping behavior
However, while this surely affects TCP end-to-end flow control, it’s not necessarily affecting it in a bad way. On contrary, it may even improve TCP performance - but this should be a subject to a separate study.
The following IEEE abstract demonstrates that people have been concerned with this interaction and even made some studies already.
http://ieeexplore.ieee.org/iel5/9789/30874/01431675.pdf
Hi Petr,
Thanks very much for going to the effort it must have taken to put your answer together.
Your explanation is very easy to understand (and ‘picture’), I will have a look at the pdf (url) this evening whilst I’m studying.
Many thanks again,
Richard
[...] Internetwork Expert Blog - Petr discusses the history of 802.3x flow control. [...]
[...] of Internetwork Expert posted a good article about why 802.3X pause frame should be disabled here Filed Under: Blog, DesignTagged: data centre, Design, Ethernet, [...]