ICAP Queued Connections (DLP) with Microsoft Azure urls
Last Updated June 13, 2019
If you observe "ICAP Queued Connections" on Proxy where queued connections were observed in DLP (Request Mode) and if queued connections increases to high numbers, the proxy becomes unresponsive and will not take any new connections request from clients.
Proxy performing ICAP request scanning using DLP or Proxy performing ICAP response scanning using CAS/AV.
It has been observed that if incoming request is something "infinite stream" (i.e. client uploading streaming data to OCS), those requests gets stuck between Proxy and DLP, where those requests could cause "Queued" connections on Proxy. This is expected behavior as Proxy and DLP/CAS device can not determine end of the data stream.
For Example, we have recently observed that, Request to/from "Microsoft Azure" (one example given below) causes the request to get stuck in "transferring state" and causes high amount of queued connections on proxy:
Above URL looks simple, but it has embedded "webstream" in it.
To avoid Queued Connections on Proxy, it is recommended to not send requests with "infinite stream" to the DLP/CAS/AV [Where request of this type does not know where to Terminate (infinite stream)] and causes Loop between Proxy & DLP.
All ICAP Connections configured on ProxySG gets used up and no more further connections are available to serve new requests.
How to Troubleshoot ICAP Queued Connections on ProxySG Caused
If you are sending "HTTP/Main" logs from proxy to Reporter, then you can generate a report on Reporter base on Custom Date, Time and Method [PUT, POST].
Look for reports based on "Request" Field and Check for the "Highest Requests", based on that you can determine what "website" or "ServerIP" were requested the most (during queued connections ). Based on type of request if it is something "webstream" , you can make decision and should be able to determine the request causing problems.
How to prevent "ICAP Queued Connections" on ProxySG [Request Mode]
If you notice end users are complaining with slowness or degradation in performance while accessing websites:
Login to proxy management console, confirm if you are seeing "ICAP Queued Connections", under Statistics ---> Content Analysis --> Duration : Last Hour / Last Day (As per issue timing). See if there are any Queued Requests [Queued Requests will be notified with color pink]
if the Queued connection counts are still increasing , then you can determine which requests might be causing this queued, by going to
Statistics ---> Sessions --> Active Sessions ---> Proxies Sessions
Filter : ICAP, REQMOD and Click on Show
Search based on "Duration", if you notice any requests which are stuck in "Transferring" or "Scanning" when you hover mouse to "I" (internet content adoptation protocol). Those are the requests , which are causing queued connections and for some reason are not able to free up. [Due to type of request something like infinite streaming or infinite connection]
To Avoid Queued Connections caused due to Below Request which is related to Microsoft Azure, as we know its issue with request, kindly add below CPL code on ProxySG to avoid queueing: