After the implementation of Application Control, the NS experiences 100% CPU utilization for large periods of time, which greatly affects the usage of NS. As more clients implemented the solution, the degrading of the NS increased. During the time of the CPU spike, the CPU utilization was primarily from w3wp.exe and sqlservr.exe processes. The Altiris Profiler showed a lot of spItemClassGet and spClientItemCacheSave being executed.
The cause for the CPU spike occurs during the client item synchronization between the ACS agent and the ACS web service layer on the Notification Server. This synchronization happens the first time the agent is install and then repeats on an interval. The cost of an initial sync for a client is dependent on the number of filters and/or the size of the Inventory filters that are used within ACS policies.
On a single agent the cost of a full synchronization is very negligible; however every agent that must sync with the Notification Server concurrently multiples this cost which can eventually overwhelm the Notification Server.
Both the initial rollout to a large number of systems and high agent settings for
‘Send Application Action Events’ and ‘Refresh Client Item Cache’ can cause a high number of ACS agent to server synchronization.
By design this synchronization cost is supposed to be minimized by a cache of all filters and actions used by ACS where the item is represented by a calculated Hash; however due to a Known Issue with the spItemClassGet and spClientItemCacheSave stored procedures only inventory collection filters are able to maintain there caculated hash. All other fitlers and actions keep getting there hash re-calculated and re-saved to the cache each and every time that client item is requested thus removing the benefit of having these client items cached.
First run the attached SQL (spClientItemCache.SQL) against the Altiris database where Application Control is installed. This SQL will updated spItemClassGet and spClientItemCacheSave to correct the hash calculation issue. These new stored procedure versions will allow all calculated hashes to remain valid until either the filter/action is updated or 24 hour period has passed. These stored procedure fixes will be included in the next service pack. Please subscribe to this KB article to receive updates to this issue.
Additionally make sure that the ACS agent configuration does not have the ‘Send Application Action Events’ and ‘Refresh Client Item Cache’ set to every one hours as this is too aggressive when deploying to a large number of systems and should be Modified to just every one day.
Go through active ACS policies to check the filters that are being. Check for any of the file inventory collection filter that has over 5,000 file resources as a member. Typically indicates that the inventory filter collection contains more than just executable type (.exe or dll) files. This can occur if the file scan policies where updated to collect a broader set of file types. Generally only executable type files need to be inventoried into a collection as only these type of files can be executed. Other file types (like txt) are opened by an application and so will never need to be matched in a file collection filter.
Another option is to create a separate Application Pool within IIS that has a request queue limit of a 100. Then this new Application Pool can be assigned to the Agent virtual folder within the Default Web Site\Altiris\MSoft folder. This will ensure that only 100 requests will be allowed concurrently. This number can then be adjusted based on the servers available resources and how the server is handling the load.
Application Control 6.1
Logged in OEM (Altiris - Partner Product) database