Environment:  Proxy forwarding to Cloud SWG (formerly WSS)
Symptom:  Increased latency to websites
Symptom:  Page timeouts occur
Symptom:  The forwarding proxy's health checks will periodically timeout and fail
Symptom:  It seems to occur more frequently when there is more load on the local proxies.  When the load is low (early morning, late evening/night, or weekends and holidays), then no issues are reported.
Symptom:  There aren't any performance degradation announcements found on the Cloud SWG Status page for the data center you are connecting to.

The pool of TCP connections is exhausted because the ProxySG appliance is forwarding traffic to a single destination (Cloud SWG) rather than dispersing traffic to multiple public-content providers. Additionally, the pool of connections might also be exhausted by:

• A large number of users using the Cloud SWG
• Applications that consume a large number of TCP ports, such as Microsoft Office 365 applications

This article provides instructions on how to detect and resolve this situation. 

Determine If All Connections Are Being Consumed

To determine if all connections are being consumed, view the statistic for the ProxySG appliance that increases when the appliance was unable to find a source port:

1. In a web browser, go to:
   https://<ProxySG_IP_Address>:<Port_Number>/TCP/Statistics?stats_mode=3
2. Locate the row for TCP2.214. Ensure the number for this protocol remains static and does not increase as peak load occurs. A number that increases means that the appliance was not able to find a source port. 

Resolution: Increase the Number of Available Connections

To increase the number of available connections, do the following until the TCP2.214 statistic remains static, and the latency and timeout issues are resolved:

• Increase the maximum number of source ports for the ProxySG appliance
• Reduce the amount of time a TCP connection is in the TIME_WAIT state
• Increased Latency and Page Timeouts Occur in Proxy Forwarding to Cloud SWG Deployment OR configure the ProxySG appliance to connect to multiple Cloud SWG IP addresses

If the issues persist, contact Symantec support for further investigation.

Increase Maximum Number of Source Ports for the ProxySG Appliance

By default, the maximum number of source ports is 16,384. To ensure enough unique connections are available, use the following CLI command to increase the number of ports to the maximum possible for the appliance:

#(config) tcp-ip inet-lowport 16384

Note: The inet-lowport can be set as low as 1024. Setting the port lower than a listening port on the proxy can had adverse effects to regular proxy operations.

For further information, see the KB article: https://knowledge.broadcom.com/external/article?articleId=167384

Reduce the Amount of Time a TCP Connection is in the TIME_WAIT State

To reduce the amount of time a TCP connection is in the TIME_WAIT state, use the following CLI command:

#(config) tcp-ip tcp-2msl 30

Reducing the TCP TIME_WAIT state value ensures that the ProxySG source ports become reusable more quickly.

Configure Additional Egress IP Addresses from Your ProxySG Appliance to the Cloud SWG

To configure additional egress IP addresses from your appliance to Cloud SWG:

1. Configure additional IP addresses by using the following CLI command:
   #(config interface <interface_number>) ip-address <ip-address> <subnet-mask>
2. Create policy to divide outgoing connections between the additional IP addresses. To do this, configure client subnets that are approximately the same size to use different addresses. The following is an example of CPL to configure the subnets:
   <forward>
client.address=<ip_address_of_client_subnet> reflect_ip(<sg_ip_one>)
client.address=<ip_address_of_client_subnet> reflect_ip(<sg_ip_two>)
   and so on for as many client subnets as necessary.
3. Save and deploy policy.

Configure the ProxySG Appliance to Connect to Multiple Web Security Service IP Addresses (Limited Availability)

Use this option if you want to avoid adding public IP addresses to your ProxySG appliance.

To obtain additional IP addresses for Cloud SWG, contact your Symantec point of contact for assistance. When you have the additional IP addresses, do the following:

Set Up Forwarding Hosts for Each IP Address

To set up forwarding hosts:

1. In the Management Console, select the Configuration > Forwarding > Forwarding Hosts tab.
2. Click New.
3. Configure the host options appropriately for each IP address. For information on configuring host options for each Cloud SWG port, see steps 3, 4, and 5 from "Procedure—Configure the Appliance" of the following document:
   https://techdocs.broadcom.com/us/en/symantec-security-software/web-and-network-security/web-security-service/help/conn-matrix/conn-about-proxyforward/conn-prxyfwd-symapp.html
   Note: When creating aliases, ensure you create a unique alias for each forwarding host you create.
4. Repeat these steps for each IP address.
   Note: If you are configuring multiple ports per IP address, you will need to repeat these steps multiple times per IP address. For example, if you want to configure a host for ports 8080 and 8443 for an IP address, create two hosts for that address; one host for port 8080 for that address and another host for port 8443 for the same address.

Create a Forwarding Group

Create forwarding groups for each Cloud SWG port you configured a forward host for. For example, if you created forwarding hosts for ports 8080 and 8443, then you will create two forwarding groups, one for each port.

To create a forwarding group:

1. In the Management Console, select the Configuration > Forwarding > Forwarding Groups tab.
2. Click New.
3. In the Alias field, enter a unique name for the forwarding group.
   Note: Because the forwarding group alias is used in policy, the alias cannot be a CPL keyword, such as no, default, or forward.
4. In the Alias name field, select the hosts you previously created and click Add.
5. In the Load Balancing and Host Affinity section, select the following:
   • From the Load balancing method list, select Least Connections.
   • From the Host affinity methods list, select Client IP Address for all applicable host affinities.
6. Click OK.
7. Click Apply.
8. Repeat these steps for each Cloud SWG port you created a forwarding host for.

(Optional) Edit the Health Check for the Forwarding Group

To edit the health check for your forwarding group:

1. In the Management Console, select the Configuration > Health Checks > General.
2. Select the health check for the forwarding group you want to edit.
3. Click Edit.
4. In the Minimum number of members that must be healthy for the group to be healthy dropdown, select either All or Any.

Configure Policy to Use the Forwarding Groups

Using either the CPL or the VPM, configure policy to reference the appropriate forwarding group names. For information on proxy forwarding policy, see: https://techdocs.broadcom.com/us/en/symantec-security-software/web-and-network-security/web-security-service/help/conn-matrix/conn-about-proxyforward/conn-fwdpolicy.html

Verify Load Balancing is Functioning

Verify that your ProxySG appliance balances traffic equally among the Cloud SWG IP addresses.

To verify if load balancing is functioning:

1. In a web browser, go to:
   https://<ProxySG_IP_Address>:<Port_Number>/Forwarding/StatsIP
2. View the Connect Active (Total) statistic for each IP address in the forwarding groups you created to resolve the connection issues. If load balancing is functioning correctly, the number of connections for each address should be similar.