Cloud Detector Incident Queue maxed out at 1000 - no persisted incidents for several days
search cancel

Cloud Detector Incident Queue maxed out at 1000 - no persisted incidents for several days

book

Article ID: 170288

calendar_today

Updated On:

Products

Data Loss Prevention Cloud Prevent for Microsoft Office 365 Data Loss Prevention Cloud Service for Email Data Loss Prevention Cloud Detection Service Data Loss Prevention Data Loss Prevention Enforce Data Loss Prevention Cloud Package

Issue/Introduction

You have issues with your Cloud Detector(s) and have the following issues:

  • Several days with no new incidents from a Cloud Detector.
  • Incident queue, as it relates to the Cloud Detector has maxed out at 1000.
  • No incidents reported under 'Last 24 hours'.
     

Error seen in the SymantecDLPDetectionServerController.log:

Exception in thread "Incidents_application_updaterWorker_1" java.lang.OutOfMemoryError: Java heap space

Cause

Incidents are queued on the Cloud Detector(s) and cannot be processed by Enforce after hitting 1000 queued incidents because of the Java memory heap filling up.

 

Note: If you have more than 1 Cloud Detector, the total number of queued incidents between all of your Cloud Detectors combined will be 1000 queued incidents.

Resolution


Increase Java Heap memory in the SymantecDLPDetectionServerController.conf file located here:

■ Windows:
<drive>:\Program Files\Symantec\DataLossPrevention\EnforceServer\Services
■ Linux:
/opt/Symantec/DataLossPrevention/EnforceServer/Services


NOTE: Before increasing JVM memory, ensure the system has ample free memory or usable standby memory.

Example: Original Values

# Initial Java Heap Size (in MB)
wrapper.java.initmemory = 128
wrapper.java.maxmemory = 2048

Example: Sample Values - These new values will depend on how much memory is available in the server. The max memory value should not go beyond 8192.

# Initial Java Heap Size (in MB)
wrapper.java.initmemory = 1024
wrapper.java.maxmemory = 4096


After increasing the Java Heap init and max values, restart the SymantecDLPDetectionServerController service and wait a few minutes. You should see the queue numbers start dropping and the number of processed incidents go up as the incidents are sent to the Enforce Server to be written to the database.

Depending on the amount of time that the queue was backed up, it may take hours or days to completely parse through the backed up incident queue.

Do not increase the memory beyond 31GB.
At 32GB you lose memory compression and it becomes counter-productive.
In most circumstances there are better ways to handle out of memory errors than increasing the memory beyond 31GB.

Powered by Wolken Software