ProxySG Virtual Appliances (SG-VA) loses config upon restart

Products

ProxySG Software - SGOS

Issue/Introduction

ProxySG Virtual Appliances (SG-VA) will crash and upon reboot the config has been lost

After this auto restart , Virtual Appliance's core header will show below Minicontext output. This will be located inside sysinfo file under /CM/Core_image/Details

Hardware exception code: 0x0
Software exception code: 0x40013
Page fault linear address: 0x0
Process group: "PG_OBJECT_STORE"
Process: "CEA Cache Administrator" in "" at .text+0x0

Also following errors will be observed under eventlog. This also confirms that disk was running out of space

"Disk 1 contains 113746 web object, 4076 system objects, 100124 small objects, 52 newly-created objects, 0 objects with IO errors and 17698 large objects, some of which are:(0) SEVERE_ERROR ced.cpp 1091"

Also eventlog will report objects which are comparatively larger in size

"48025 url which is 4007899996 bytes long. (Flags 10000c, 109001)(0) SEVERE_ERROR ced.cpp 1126"

and similar events like this

Note - when this has happened the config and the event logs will be lost so you will need to export the logs via syslog so in the event the SG-VA loses config you still have the event logs off the SG-VA.

Environment

This issue affects the VA-Gen2 models with single disk. There are 7 models in total which are affected:

SG-VA-C1XS, SG-VA-C1S, SG-VA-C1M, SG-VA-C1L
SG-VA-C2S, SG-VA-C2M, SG-VA-C2L

VSWG-100 model (VA-Gen1) is not affected since it is configured with 2 virtual disks.

Cause

The issue happens when the SG-VA runs out of disk space and when this happens the SG-VA will be reinitialized and hence subsequent loss of disk space.

This can happen due to ICAP scanning infinite objects. For example ICAP scanning video/audio streams or large objects being uploaded/downloaded using the FTP proxy.

Resolution

The issue affects single disk systems. In multiple disk systems the drive running out of space would be reinitialized but you would not lose the config since the config is kept on both disks. We support the addition of a second virtual drive to the VA appliance. To do that you will need to turn off the virtual appliance. Then edit the settings of the virtual appliance and then

add a second disk. The disk should be 100GB and thick provisioned.

Upon reboot you will see this error message but it can be ignored.

When the SG-VA is rebooted you will find that the new disk has been recognized under SG-VA's web UI

You will also notice the "unsupported configuration" message in the sysinfo. This can be ignored. We are updating the licensing and the ovf templates to take the addition of a second drive into account. In future it will be supported configuration.

Model: VSWG-SE (unsupported configuration)
RAM: 2048 MB
RAM slot #0: 2048 MB, pn:Unspecified
Number of physical CPUs: 1
Number of cores: 1
CPU frequency: 2400 MHz
Storage: 2 drives (expecting 1 drive and the total formatted storage capacity must be between 50 GB - 200 GB)
Disk in slot 1: 107 GB VMware Virtual disk , rev:1.0 serial: status:present
Disk in slot 2: 107 GB VMware Virtual disk , rev:1.0 serial: status:present

For AWS environment:
Recent deployment method for AWS requires using Market Place. Once it is deployed, default disk layouts are here.

This looks two disks but actually only one data disk(disk 2). This probably will happen the issue above, hence It's needed to add one more data disk.

1. Stop SG-VA instance in AWS.