The storage locations are filling up with BLOB files. None of the space is ever being reclaimed.
- or -
BLOBs are not shrinking after computer accounts are being marked for deletion.
There can be many causes for this.
1. Server Space Management job was not scheduled to run.
2. Server Space Management job is set to run, but there is little to no free space for the job to work with in order to complete.
3. There are old accounts that need to be cleaned up in “Manage Lost Recovery Solution Clients”.
4. Unnecessary files are being backed up because the exclude list has not been customized to best fit the end users needs.
Here are some storage space optimization practices for reclaiming used space and helping to prevent the storage volume from filling up. Please note that all paths mentioned are for the 6.2 build of Recovery Solution so if you are on 7.0 or 7.1 they will be slightly different.
1. Enable the SSM and IC job to run once a week. It is best to have the IC job run first to clean up any corruptions that may cause the SSM to fail. As the SSM job does run a delete job when it finishes there is no real need for the Delete Marked Items job to be scheduled. If the SSM job is not completing properly or you need to free up space in a pinch you can manually kick that job off by right-clicking on your Cluster and choosing Recovery Solution Tasks > Start Server Job > Delete Marked Items.
Jobs can be enabled in the Altiris Console by choosing “Configuration > Solution Settings > Incident Management > Recovery Solution > Recovery Solution Clusters > Recovery Solution Cluster Configuration” and then clicking on your Cluster. The settings are on the “Server Jobs Schedule” sub tab. If you are using RS 6.2.2332 or later, also make sure that the “Force deletion of the recently excluded files” option is checked off under the Server Space Management job.
Note: If you see the SSM and/or IC job is failing check to see how much free space you have available in your storage locations and make sure at least 10% free space is available. For example if your total Used Space is 1 TB then your Free Space needs to be at least 100 GB which would be 10%. If it is below 100 GB then you may begin to see issues with these jobs not completing properly. The reason for this is that these jobs create temp files while they run and need the space to move things around while running compaction as well.
2. Make sure the Storage Space’s default rules are enabled and have an appropriate time frame in the “Delete snapshots older than” and “Delete files from the Recovery Server” settings. This is listed under “Configuration > Solution Settings > Incident Management > Recovery Solution > Recovery Agent Settings > Default Recovery Agent Settings” under the “Space Management” sub tab. Here is a brief explanation of how these settings work.
"Delete Snapshots older than" - This option deletes all files from the snapshots of the protected computers you have selected, after the amount of time you specify has elapsed.
"Delete files from the Recovery Server" - This option only deletes files from the snapshots after the originals have been removed from the computers, and then only after the amount of time specified here has elapsed. For example if you have a file backed up as TEST.txt and that file still resides on your machine it will keep the latest revision of that file backed up until it gets deleted from the client. Once that is happened it will keep the file for the time you specified in this settings and then delete it from your snapshots. This is meant for recovering files accidently deleted or if a machine crashes.
3. Use the “File Extensions Backed Up by Cluster” report under “Reports > Incident Management > Recovery Solution > Server Reports > File Extensions Backed Up by Cluster” to streamline the exclude list. You can add the extensions that use the most space to the excludes to maximize space return.
Note: Some common top ten space users are *.log and *.pst files. If *.log is added to the excludes to reclaim space, add WinFAL.log to the exceptions to this exclude. The data in this log file is used by RS in the creation of some FSR data (see HOWTO8765).
4. Use the “Cluster Disk Space Currently Used” report under “Reports > Incident Management > Recovery Solution > Server Reports > Cluster Disk Space Currently Used” to discover computer that are using the most space. There may be computers in this list that are no longer in the network or are not a priority to back up. To mark these accounts for deletion, right-click on the computer and choose “Recovery Solution tasks > Administration > Mark Computer Account for Deletion…”
Note: This report takes a great deal of SQL cycles and may timeout in large database or BLOB environments. Also these machines will not be removed until an SSM or Delete Marked Items job is ran. The Delete Marked Items job would be the quickest way of purging these accounts if you are getting low on space.
5. Discover why machines are in the “Manage Lost Recovery Solution Clients” and clean up the list. Go to “Configuration > Solution Settings > Incident Management > Recovery Solution > Recovery Agent Settings > Manage Lost Recovery Solution Clients.” Using TECH28903 discusses some common causes as to why machines are placed here. For those that are in here because they are retired machines or ones that no longer need to be backed up mark the account for deletion by right-clicking on the computer(s) and choosing “Recovery Solution tasks > Administration > Mark Computer Account for Deletion…” If you have many in there that you want to delete you can select more than one by holding down the Ctrl key and click on them. Once all of the ones want to delete you can mark them all for deletion. As previously stated in the steps above an SSM or Delete Marked Items job must be ran in order for these accounts and their data to be removed.
6. Set up the NS Purging Maintenance to delete inactive machines after 60 days instead of retiring them. This will also help to reclaim licenses that are no longer in use as well as free up the space those machines are consuming. “Purging Maintenance” is found under “Configuration > Server Settings > Notification Server Settings > Purging Maintenance”.
7. Make sure there are not duplicate machines in the NS or RS. There are a few articles about finding and merging duplicate computers in the NS: TECH11704, HOWTO3592, HOWTO6365 and more.
8. Before you run out of storage space, set a fault limits that are less than the size of the storage. Don't run out of space if you can help it. Snapshots and Server Jobs take space and are less efficient if the storage space is near or at 100%. If you have a fault limit, you get a heads up that you have a storage problem before you are completely out of space. If you can reserve 5% or 10% of space, you will be able to give the RS some of the reserved room while you are working on freeing up space.
9. If there are machines which do not need to be backed up as often as others, it is a good idea to prioritize machines by deleting the less essential machine accounts. It is better to continue the backups of mission critical machines rather than running out of space by machines that an OS can simply be reinstalled on and the user can continue with work as usual. Also setting up different Recovery Solution settings for different groups can help too. If you have a group of VPs that need everything on the machine backed up daily create a settings group just for them. If you have some end users that may only need to be backed up once or twice a week and the only critical info that they need backed up is in their My Documents folder create a settings policy just for them to take Partial Snapshots of just that data on the days specified.
10. Make sure you have the latest Hotfixes installed for the build you are currently on especially ones that effect the Server jobs and their performance. You should be able to find them with a quick search of our Knowledge Base.
11. If there are specific files that need to be deleted from the server, they can be marked for deletion at the cluster level using the cluster's right click menu option “Recovery Solution Tasks > Administration > Mark Files for Deletion...”. The cluster is found in “Configuration > Solution Settings > Incident Management > Recovery Solution > Recovery Solution Clusters > Recovery Solution Cluster Configuration”. Remember that if the file still exists on the client, it will again be backed up during the next snapshot. This option is similar in function to the “Force deletion of the recently excluded files” that was added to RS.
12. It is a recommendation to have the SQL Database stored on a different IDE channel than the BLOBS. Size of the database files can cause Server Jobs to run for longer then they need to. There are related articles about keeping the database size under control like TECH24662. The transaction log of the database can also be a size and performance problem which is discussed in TECH26008 and TECH1026.
13. Check for unreferenced BLOB files by comparing the count of blobs in the storage volume to the results of this query:
Select count(*) from DiskFile
If the number of files in the folder is greater than the SQL returns then the database does not reference all of the files in your directory. If the number is less than the SQL returns, the storage is missing BLOB files which may also cause other problems. An IC w/CRC job may resolve the unattached BLOBs.
14. If the previous steps are unable to free up space, the space is being legitimately used and more storage will need to be added.
Subscribing will provide email updates when this Article is updated. Login is required.
Thanks for your feedback. Let us know if you have additional comments below. (requires login)
This will clear the history and restart the chat.