You have been alerted that ICAP queueing is occurring on Edge SWG (formerly ProxySG). Users are experiencing delays in web pages loading.

You see these kinds of messages in your Event Log:

2020-03-06 21:29:23-00:00UTC "ICAP long scanning reqmod transaction for http://example.com/policy using dlp1 after 60 seconds and 1684703331 bytes"

2020-03-06 22:19:26-00:00UTC "ICAP scanning deferred for http://example.com/stream using cas1 after 126 seconds and 4544730464 bytes"

Edge SWG performing ICAP request scanning or response scanning.

Request and response scanning can take a long time to complete. Some requests, such as streams and stock tickers, are considered never-ending and run indefinitely.

If too many long-running requests take up all of the ICAP connections, requests will be queued. Clients will experience long delays if their requests are queued and are waiting for a stream or stock ticker to "finish".

Neither the Proxy nor the scanning device can determine if a request will take a long time or run indefinitely.

How to Troubleshoot ICAP Queued Connections on Edge SWG

Verify that the Edge SWG is queueing connections

• From the SGAC Management Console GUI: 

• •  Under Reports > Content Analysis > Active Requests, check for queued connections highlighted in orange.

• For the legacy Java Management Console GUI: 

• • Under Statistics > Content Analysis > Duration : Last Hour / Last Day (or choose the time frame of the occurrence from the pull down menu), check for queued connections highlighted in pink.

Check Active Sessions to see which sites may be causing the the queued sessions

If you are experiencing ICAP queued connections, then Active Sessions will give you a list of clients and destination hosts.

• From the SGAC Management Console GUI: 

• • Under Reports > Active Sessions > Proxied Sessions. This will give you a list of current proxied connections and the status of the connections.

• For the legacy Java management console: 

• • Under Statistics > Sessions > Active Sessions > Proxied Sessions

Use the following filter: ICAP, REQMOD (for DLP and other request modification ICAP service) or ICAP, RESPMOD (for CAS or other response modification ICAP service) then Click on Show.

Sort based on "Duration" and check for requests which are stuck in "Transferring" or "Scanning" when hovering the mouse pointer on the field containing the "I" (for ICAP). These are most likely the requests which are causing queued connections. See also Parsing and analyzing current Active Sessions for troubleshooting.

To find the URLs associated with the hosts, utilize the access logs from the Edge SWG.  Search on the hosts found above and see what URLs are being requested.

How to prevent "ICAP Queued Connections" on Edge SWG

•  Implement the latest ICAP Best Practice Policy for Response Modification, based on the threat risk score of websites provided by Intelligence Services, and Request Modification (attached to the end of this article). Instructions are in each text file as comments.

•  Bypass sites which are known streams if the Best Practices policy does not bypass them

For example, Microsoft Azure sites that contain "servicebus/webstream" are streams not bypassed by the Best Practices Policy and must be manually configured for exclusion of ICAP scanning like this:

<cache>
url.host.is_numeric=yes url.path.substring="servicebus/webstream" request.icap_service(no) response.icap_service(no)

The policy "url.host.is_numeric=yes" is used because of the dynamic nature of Azure IP addresses. We want to limit the matching of this policy to only numeric destinations.

Another site that circumvents the best practices is stream.launchdarkly.com. To exclude streams from/to stream.launchdarkly.com use this policy:

<proxy>
url.domain=stream.launchdarkly.com request.icap_service(no) response.icap_service(no)
url.domain=clientstream.launchdarkly.com request.icap_service(no) response.icap_service(no)

• If you don't have Intelligence Services, then bypass categories Radio/Audio Streams and TV/Video Streams from ICAP

See the ICAP best practices for Response Modification document for more information regarding the legacy method of handling these requests.