The symptom of the problem is when the first ProxySG is enabled for WCCP, it is added into the router, and works.  When an additional ProxySG is added, one or both of the ProxySGs are not kept in the WCCP service group.  The Cisco will show the status of the ProxySG as “Usable”, and then switch to “Not Usable” and back and forth.  The hash tables are never built, meaning that the redirection will not work.

show ip wccp 90 detail

WCCP Cache-Engine information:

        Web Cache ID:          10.50.60.70

        Protocol Version:      2.0

        State:                 Usable

        Redirection:           L2

        Packet Return:         L2

        Packets Redirected:    0

        Connect Time:          00:00:00

        Assignment:            MASK

        Web Cache ID:          10.50.60.71

        Protocol Version:      2.0

        State:                 NOT Usable

        Redirection:           L2

        Packet Return:         L2

        Packets Redirected:    0

        Connect Time:          00:00:00

        Assignment:            MASK

Check the IP-MTU setting on the VLAN or router interface on the Cisco.  In the Cisco command “show running-config” check the interface that the ProxySG is in and look for command that says “mtu XXXX”. 

interface Vlan20

description SG510 Subnet

mtu 1550

ip address 10.50.60.1 255.255.255.0

If that number is larger than 1500, then that is the problem.  The fix is to either reset the MTU back to 1500, or if there are other devices in that subnet that require a larger MTU, then move the ProxySGs into a different subnet that has a MTU of 1500.  WCCP may need to be re-initialized on the router and/or the ProxySGs.

Further details:

The IP-MTU is the Maximum Transmission Unit for a particular network segment.  This is the maximum packet size that is allowed to pass between any devices on a local network.  The MTU should be configured the same across all systems in the local network to ensure that there are no incompatibilities between any devices.  The lowest common denominator should be the MTU that is set on all devices in the local network. 

The reason the problem occurs has to do with the hash tables that the Cisco router returns in it’s “I See You” packets.  When more caches are added into WCCP, the “I See You” response packets contain the hash tables for each cache, which results in a larger packet being sent.  When a single ProxySG is running in WCCP, the size of the hash table is small enough to fit within a standard 1500 byte IP packet.  When two or more ProxySGs are added into WCCP, the combined hash tables become larger than 1500 bytes, so the “I See You” packet must be fragmented.  When the router fragments the packet, it uses the configured MTU for the interface to determine at what point in the datagram to fragment the packet.  The largest MTU setting that ProxySG will support is 1500, so if the router is configured to use an MTU that is larger than 1500, then the WCCP “I See You” packets are fragmented at a size larger than the ProxySG is able to read.  The packet gets dropped by the ProxySG, and the router eventually will remove it from the WCCP group.  The reason this happens is because the ProxySG is unable to see the Receive ID contained in the router’s packet.  With each “I See You” packet sent, the router increases the Receive ID, and it expects to see that same ID returned in the next “Here I Am” packet from the ProxySG. Since the “I See You” packets cannot be read, the ProxySG returns its last known Receive ID, and the Cisco sees the mismatch, and attempts to remove the ProxySG from the WCCP group.

Changing the IP-MTU on the router interface that the ProxySGs are in back to 1500 (default) will resolve this issue.  This issue may also occur if the router has a MTU value larger than the ProxySG, even if they are both below 1500 (for example, router is 1450, and ProxySG is 1400).  WCCP may need to be re-initialized on the router and/or the ProxySGs before they will start working again. 

show ip wccp 90 detail

WCCP Cache-Engine information:

        Web Cache ID:          10.50.60.70

        Protocol Version:      2.0

        State:                 Usable

        Redirection:           L2

        Packet Return:         L2

        Packets Redirected:    0

        Connect Time:          00:00:35

        Assignment:            MASK

        Mask  SrcAddr    DstAddr    SrcPort DstPort

        ----  -------    -------    ------- -------

        0000: 0x0000003F 0x00000000 0x0000  0x0000

        Value SrcAddr    DstAddr    SrcPort DstPort CE-IP

        ----- -------    -------    ------- ------- -----

        0032: 0x00000020 0x00000000 0x0000  0x0000  0x0A323C46 (10.50.60.70)

        0033: 0x00000021 0x00000000 0x0000  0x0000  0x0A323C46 (10.50.60.70)

       ....

       ....

        0062: 0x0000003E 0x00000000 0x0000  0x0000  0x0A323C46 (10.50.60.70)

        0063: 0x0000003F 0x00000000 0x0000  0x0000  0x0A323C46 (10.50.60.70)

        Web Cache ID:          10.50.60.71

        Protocol Version:      2.0

        State:                 Usable

        Redirection:           L2

        Packet Return:         L2

        Packets Redirected:    0

        Connect Time:          00:00:33

        Assignment:            MASK

        Mask  SrcAddr    DstAddr    SrcPort DstPort

        ----  -------    -------    ------- -------

        0000: 0x0000003F 0x00000000 0x0000  0x0000

        Value SrcAddr    DstAddr    SrcPort DstPort CE-IP

        ----- -------    -------    ------- ------- -----

        0000: 0x00000000 0x00000000 0x0000  0x0000  0x0A323C47 (10.50.60.71)

        0001: 0x00000001 0x00000000 0x0000  0x0000  0x0A323C47 (10.50.60.71)

              ....

       ....

        0030: 0x0000001E 0x00000000 0x0000  0x0000  0x0A323C47 (10.50.60.71)

        0031: 0x0000001F 0x00000000 0x0000  0x0000  0x0A323C47 (10.50.60.71)