Performance issues when using LACP across multiple NPMs
performance degradation when using LACP bundle across multiple NPMs
high value of Early NFI reinjection counter in show interface detail
problem disappears when using only single interface or when all links in LACP bundle are attached to a single NPM
tcpdump on NPMs shows that sending switch is transmitting frames for a specific uni-directional flow inconsistently over two different LACP ports
Since each packet for a single flow may reach another NPM, the NPMs have to re-classify the flow and this is causing significant performance impact. The IEEE specification for LACP requires that all frames for a given stream must be transmitted out of a single interface.
X-series is transmitting frames consistently out of the same interface for a specific uni-directional flow. Note that the port selection might be different for the reverse flow, i.e. each LACP station might select another link from the LACP bundle to transmit frames for that particular bi-directional flow.
Contact the vendor of the switch to provide a solution.
We cannot correct this problem if the sending switch is not complying with the IEEE standard. The following workarounds may be used to mitigate the issue on LACP bundle with two links:
- temporarily disable one of the two links in LACP or do not use LACP
- attach both links in the LACP bundle to the same NPM on the X-series side
The LACP operation is defined by the
IEEE 802.1ax Link Aggregation standard (or the previous IEEE 802.3ad). Here are the parts of the IEEE 802.1ax specification relevant for the link selection.
3. Definitions ...
3.8 conversation: A set of frames transmitted from one end station to another, where all of the frames form
an ordered sequence, and where the communicating end stations require the ordering to be maintained
among the set of frames exchanged. (See IEEE Std 802.1AX, Clause 5.)
3.16 Link Aggregation Group: A group of links that appear to a MAC Client as if they were a single link.
All links in a Link Aggregation Group connect between the same pair of Aggregation Systems. One or more
conversations may be associated with each link that is part of a Link Aggregation Group. (See IEEE Std
802.1AX, Clause 5.)
5.2 Link Aggregation operation
5.2.1 Principles of Link Aggregation
f) Frame ordering must be maintained for certain sequences of frame exchanges between MAC Clients
(known as conversations, see Clause 3).
The Distributor ensures that all frames of a given
conversation are passed to a single port. For any given port, the Collector is required to pass frames
to the MAC Client in the order that they are received from that port. The Collector is otherwise free
to select frames received from the aggregated ports in any order. Since there are no means for frames
to be misordered on a single link, this guarantees that frame ordering is maintained for any
5.2.4 Frame Distributor
This standard does not mandate any particular distribution algorithm(s); however, any distribution algorithm
shall ensure that, when frames are received by a Frame Collector as specified in 5.2.3, the algorithm shall
a) Misordering of frames that are part of any given conversation, or
b) Duplication of frames.
The above requirement to maintain frame ordering is met by ensuring that
all frames that compose a given
conversation are transmitted on a single link in the order that they are generated by the MAC Client; hence,
this requirement does not involve the addition (or modification) of any information to the MAC frame, nor
any buffering or processing on the part of the corresponding Frame Collector in order to reorder frames.
This approach to the operation of the distribution function permits a wide variety of distribution and load
balancing algorithms to be used, while also ensuring interoperability between devices that adopt differing
Imported Document ID: 000019503
Subscribing will provide email updates when this Article is updated. Login is required.