Resynchronizing Package servers with invalid or stale codebases for Management Platform 8.x
search cancel

Resynchronizing Package servers with invalid or stale codebases for Management Platform 8.x

book

Article ID: 181405

calendar_today

Updated On:

Products

Deployment Solution IT Management Suite Inventory Solution Software Management Solution Client Management Suite

Issue/Introduction

There are several conditions that this article attempts to resolve with a re-synchronization of the Package Servers but generally it is an issue where the Package Server agent shows that all packages are ready when within the Notification Server, its package status shows that one or more packages are not ready on one or more Package Servers.

Symptoms could include the following:
  • Stale Codebases
  • Invalid Packages
  • Packages won't download to clients
  • Packages Not Ready
  • Packages Invalid - Error while downloading package: Server is busy.  Package sources request is backing off (-214746725)

If the number of available packages on the Package Server is not changing, the package server is reporting 'Server is busy', and the trace data in the Altiris agent logs reports MaxConcurrentPackageInfoRequest, then you may need to increase the value in the NSConfigurator/core settings. 

Environment

Notification Server 8.x
Package Servers

 

Cause

There may be several causes for the problem, including:

  1. Package version mismatch between the Notification Server and the Package Server
  2. The Package Server may not have sent the status for the package in question back to the Notification Server.
    Examples could include the following:
    • A SQL Transaction rollback during the process of building the snapshot can cause a version mismatch.
    • A large number of NSEs in the queue and the package status is still waiting to be processed.
    • The NSE containing the status update from the Package Server fails to be delivered to the Notification Server.
    • An error occurred while processing the package status NSE.
    • The package server legitimately has not downloaded the package yet.
    • The Package Server is having client/server communication problems.
    • Permissions on the NSCap or Event Queue subfolders prevents NSE file processing.

Resolution

Be aware that improper Site Maintenance can cause similar behavior. If sites and subnets are not correctly created with the Package Server assigned to that site, then invalid code bases can be given to the clients within that site, causing the situation where clients won't download packages.

As mentioned in the Problem/Symptoms section there are several situations that this article tries to remedy:
  • Stale Codebases
  • Invalid Packages
  • Packages won't download to clients
  • Packages showing Not Ready (in the Notification Server console)
  • Packages showing Invalid (in the Notification Server console and on the Package Server tab of the Package Server's Altiris Agent properties)
  • Permissions on the NSCap and NSCap Event Queue subfolders on the Notification Server.

At a fairly high level, these issues can be attributed to synchronization problems between the Notification Server and client (Package Server), specifically dealing with the snapshot.xml. This ultimately causes the client an inability to download a package. The purpose of this information is to give several aspects of synchronization to assist in making a decision on how best to handle the current situation. This will be done in several steps.

There are six basic sections in this article. They are purposefully ordered in this fashion from the most likely to the least likely causes. It is unnecessary to follow all sections; start with Section 1 first, then if it fails to help, proceed to Section 2, and so on.

  1. Verifying Package Status
  2. Dealing with the Database
  3. Manual Synchronization
  4. Share Permissions
  5. Agent Version
  6. NSCap Permissions

Section 1 (Verifying Package Status)

To find out if the problem is because a package is not ready run the following SQL query and then open some of the corresponding snapshot.xml files that are located in:
Management Platform 7.0.x:   %ProgramFiles%\Altiris\Notification Server\Snapshots\
Management Platform 7.1.x and later:   C:\ProgramData\Symantec\SMP\Snapshots\
 
SELECT
vi.[Name] AS 'Package Server'
,pack.[Name] AS 'Package Name'
,fs.Name as [Orgination Server]
,osns.Id
,ps.[Status] AS 'Status'
,pack.[Package Version] as [Package Version on NS]
,ps.Version as [Version Last Reported by PS]      
,ps.[PackageId] AS 'Package Guid'
      
 FROM SWDPackageServer ps
 JOIN ItemNSSource ns on ns.ItemGuid = ps.PackageId
 JOIN OriginNSSourceNS osns on osns.Id = ns.OriginNSSourceNSId
 JOIN ForwardServer fs on fs.Guid = osns.OriginNSGuid
 JOIN vItem vi
    ON vi.[guid] = ps.[PkgSvrId]
 JOIN SWDPackage pack
    ON pack.[PackageId] = ps.[PackageId]
   AND pack.[_Latest] = 1
  
 WHERE ps.[Status] != 'Ready'
 ORDER BY vi.[Name], ps.[Status]

If there is no snapshot.xml for a specific package found in the “.\Snapshots” folder then it means that the NS did not build or maintain the package. Usually this is the result of two different possibilities:

  1. The path to the package could not be verified. You would normally see errors in the NS logs indicating a failure to refresh the package snapshot for packages who’s source path could not be verified. If so then correct the problem by editing the package and pointing it to the correct path.
  2. The origination server of the package may be different than that of the local NS. In a hierarchy packages can be replicated. When this occurs the server from which the package was replicated becomes the ultimate authority and the NS that has a replicated package will not try building snapshots or codebases for that package. It will instead redirect requests from its package server for information about the package up to the parent NS.

However, sometimes a server is no longer in a hierarchy, or it never was, but the package was imported from some other source. If you look at the results of the previous query above it will show, in the “Origination Server” column, which server it thinks the package came from. If the Name and ID are different than that of the local NS then this would confirm the problem. 

If the package shows a different source then you could plug the Package GUID into the following modification query and change its origination server:

First run the following query to find out what ID the local server is operating under. Usually it is ID 1, but not always.

select osns.Id as [Server ID]
from OriginNSSourceNS osns
 join vThisNS vtns on vtns.Guid = osns.OriginNSGuid
   and vtns.Guid = osns.SourceNSGuid

It is very important that the correct Server ID, returned by the previous query, be referenced in the following SQL that will modify source and origination properties of all or specific objects.

update ItemNSSource set OriginNSSourceNSId = <server ID> where ItemGuid = '<package GUID>'

Example:    update ItemNSSource set OriginNSSourceNSId = 1 where ItemGuid = '0562094A-ECBE-4617-9253-5E7FFB5E00C4'

Or, if there are many packages that are seen to have originated from another server then they could all be converted to originating from the local NS via the following SQL modification script:

update ItemNSSource set OriginNSSourceNSId = <server ID> where OriginNSSourceNSId <> 1 and ItemGuid in (select Guid from vRM_Package_Item)

NOTE: The default ID of a server that is not in a hierarchy is 1. However it may be a different number. To verify which ID the server is operating under do the following:

  1. Obtain the operating GUID of the server by opening “C:\ProgramData\Symantec\SMP\Settings\CoreSettings.config” and search for and copy the value in OwnerNSGuid
  2. Plug the OwnerNSGuid into the second line ( @Guid ) of the following query:
declare @Guid uniqueidentifier
set @Guid = '8b53108b-f9d6-491e-86e6-6473ed5eb57e'
 
select distinct OriginNSSourceNSId as ID, fs1.Name as [Source Server], osns.SourceNSGuid as [Source Guid], osns.OriginNSGuid as [Origination Server]
from ItemNSSource ns
       join OriginNSSourceNS osns on osns.Id = ns.OriginNSSourceNSId
       join ForwardServer fs1 on fs1.Guid = osns.SourceNSGuid
       join ForwardServer fs2 on fs2.Guid = osns.OriginNSGuid
 
where @Guid = osns.SourceNSGuid and @Guid = osns.OriginNSGuid

Section 2 (Dealing with the Database)

NOTE: Sections 2 and 4 include steps involving changes to the data stored in the database. Backup the database before proceeding.

VERIFY THE PACKAGE FILE SETTINGS BEFORE DOING THIS STEP! Ensure the global package server setting for deleting unused packages is set to “1 Week”.
Settings > All Settings > Notification Server >  Site Server Settings > Package Service > Package Service Settings
Global Package Service Settings
Check the box next to ‘Delete package files if they are unused for” and set the value to “1 Week”.
Recreate the Package Codebase on the Notification Server:
The following SQL is used to truncate the SWDPackageCodebase table. While it does copy the information into a temporary table, this generally isn't necessary.
-- Select each item distinctly into a temporary location
SELECT * INTO #tmp_SWDPackageCodebase1
FROM SWDPackageCodebase
TRUNCATE TABLE SWDPackageCodebase

-- Finally display the newly built table to verify the data
SELECT * FROM SWDPackageCodebase
 
Run the following two (2) Windows Scheduled Tasks manually or wait for them to run on schedule overnight. ( Control Panel > Scheduled Tasks)
  • NS.Package Distribution Point Update Schedule
  • NS.Package Refresh
The Package Codebases are now rebuilt on the Notification Server. To complete the codebase refresh, the Package Servers themselves must either send an “Update Configuration” request or "Resend Package Status". Once this completes, the SWDPackageCodebase table will be completely rebuilt. The NS.Package Refresh schedule normally runs once a day at 3:30 a.m.  Verify that it runs correctly (Check the date it last ran). If it does not, see article TECH26028, "Scheduled Tasks on the Notification Server show a status of 'Could not start'."
NOTE: Once the data has been recreated and verified, you can safely remove the temporary table created earlier:

DROP TABLE #tmp_SWDPackageCodebase1

In addition to truncating the SWDPackageCodebase table, a similar approach can be applied to the SWDPackageServer table. Although this is normally not needed for the synchronization process, it can be done to assure that the Package Server table is clean when the information is regenerated.

/* Step 1:
Copy the current contents into a temporary table for backup purposes if
needed */
SELECT * INTO #tmp_SWDPackageServer
FROM SWDPackageServer
/* Step 2:
Remove the contents from the table completely */
TRUNCATE TABLE SWDPackageServer
/* Step 3: */
Run the following tasks. 
NS.Package Distribution Point Update Schedule
NS.Package Refresh
NS.Package Server Status Event Capture Item
/* Step 4:
Display the contents of the table. */
SELECT * FROM SWDPackageServer
Verify the contents of the SWDPackageServer table. You are looking to ensure that each package is listed only once for each Package Server.
/* Step 5:
Remove  the Temporary table once you have verified that the data is intact */
DROP TABLE #tmp_SWDPackageServer

Section 3 (Manual Synchronization)

If Section 2 does not resolve the issue, then it may be necessary to manually synchronize the snapshots. The following steps illustrate how to do this.
From the Notification Server:
  1. Delete the snapshots ({GUID.EN_US}.xml) found here:
    • (In SMP 7.0) %ProgramFiles%\Altiris\Notification Server\Snapshots
    • (In SMP 7.1 and later) C:\ProgramData\Symantec\SMP\Snapshots
      *Important*: There are implications when taking this approach. When deleting the snapshots on the Notification Server, it does create a new version for that snapshot, therefore all the clients within the Notification Server infrastructure will download the new version after their next configuration update check. This will potentially generating a lot of IIS and Network traffic. If there are a lot of package servers out there you would only want to do this as a last resort (deleting all snapshots). It would be much better to delete the snapshot.xml of only the packages who’s status is in a “stale” state.
  2. Then go to Control Panel > Scheduled Tasks and run the task called NS.Package Refresh.{GUID.EN_US}. This will re-create the package snapshots on the server. Make sure that the scheduled task does in fact start. If it does not, see article, "Scheduled Tasks on the Notification Server show a status of 'Could not start" 

From the Package Servers:

  1. Delete the snapshots (snapshot.xml) here: %ProgramFiles%\Altiris\Altiris Agent\Package Delivery\{GUID.EN_US} (see Notes below).
  2. Delete the .xml file (<NS name>.xml) here: %ProgramFiles%\Altiris\Altiris Agent\Client Policies\
  3. Right-click on the Symantec Management Platform Agent icon > Symantec Management Agent Settings in the system tray and click on Update button in the Configuration section of the window that will open. This will cause the newly created snapshots to be downloaded from the Notification Server.
  4. Right-click on the Symantec Management Platform Agent icon > Symantec Management Agent in the system tray. Open the Package Server tab and click on the Resend Package Status button. This will cause the Package Delivery snapshots to be re-created and update the Notification Server of the Package Server status.  
From the Client (see Notes below, but deleting the snapshots from the clients should not normally be done):
  1. Delete the .xml file (<NS name>.xml) here: %ProgramFiles%\Altiris\Altiris Agent\Client Policies.
  2. Delete the snapshots (snapshot.xml) found under the %ProgramFiles%\Altiris\Altiris Agent\Package Delivery\{GUID.EN_US} folders
  3. Right-click on the Symantec Management Platform Agent icon > Symantec Management Agent Settings in the system tray and click on Update button in the Configuration section of the window that will open. This will cause the newly created snapshots to be downloaded from the Package Server.
Notes: How to delete snapshots quickly on the Client and/or Package Server: 
  • Open %ProgramFiles%\Altiris\Altiris Agent\
  • Search for "snapshot.xml".
  • Once you have the results, select all and delete. After this is done, there is potential for the clients within the infrastructure to download all the snapshots. This would occur if the snapshots on all the clients were out of sync with the information contained on the Notification Server. As the intent for this article is primarily to synchronize the Package Servers with the Notification Server, there generally shouldn't be a problem with the rest of the clients. To alleviate some of the potential for this, rather than removing all snapshots on the Notification Server, only remove the snapshots for the packages for which you are experiencing difficulties.

Section 4 (Share permissions)

NOTE: Sections 2 and 4 include steps involving changes to the data stored in the database. Backup the database before proceeding.

A possible file or folder permissions issue, especially if the packages are located on a share, could cause packages to be unavailable. The default permissions for manually created shares on a Windows 2003 Server is “Everyone -> Read”. The addition of “SYSTEM -> Full Control” is most often the solution.

Section 5  (Agent versions)

Verify that the Symantec Management Platform Server, Symantec Management Agent and the Package Server sub-agent versions are current or close to current and equal.  Version mismatches can cause communication errors such as: "Invalid packages - Error while downloading package: Server is busy.  Package sources request is backing off (-214746725)"

Section 6 (Permissions on the NSCap and NSCap Event Queue subfolders) (relevant only to versions prior to 7.1SP2)

When the package server tries to send a package status, it does this using an NSE file.  This NSE file is by default compressed at 200 KB.  Therefore, when this file goes to the Notification Server it will be placed into the NSCap\Temp directory for decompression.  Then it would be moved into the proper EvtQueue folder.  However, if the permissions are not correct on the NSCap\EventQueue folders, this process will not complete.  Unfortunately no errors are displayed in the Notification Server log file, or in IIS.  Using a network trace tool like Wireshark will show an error when this event is posted.  For further information on this and for information on the access rights needed for the NSCap and subdirectories, please view "What are the minimum account permissions required for the NSCap folder and subfolders for Notification to properly function?" 

Additional Queries

The following SQL cleans up duplicates in the Notification Server console under the Package Server status
SELECT DISTINCT * INTO #swdpackageserver
FROM swdpackageserver
 
TRUNCATE TABLE swdpackageserver
 
INSERT INTO swdpackageserver
SELECT * FROM #swdpackageserver
 
DROP TABLE #swdpackageserver
 
 
--
 
If the steps above couldn't resolve the issue, probably the package is stuck in the secure storage folder. in this case, follow the KB251232
Aslo check the UrgentList.xml on the PackageServer. It is possible that there will be no policy but still we have it in the UrgentList. If so, Delete the UrgentList.XML located in  'C:\Program Files\Altiris\Altiris Agent\Package Server Agent\PackageStatus\UrgentList.xml'. 
Powered by Wolken Software