Q: What does SCSP's "Bulk Logging" feature do?
A: Bulk Logging allows you to distinguish between:
- Events that you want to see right away in the Console because you need to take action (real-time events)
- Events that you need to record and save, but may not look at for days, weeks, or months, e.g. data required for regulatory compliance (bulk logs)
For customers who have a large amount of regulatory data, this could flood the database and cause several problems: high network usage, high resource usage at the management server, and constant database maintenance to keep the database from filling up, to name a few.
In SCSP you can configure the agent to send a small number of critical events in real-time to the database for immediate display in the Management Console. If there's a large amount of data that's being recorded for future analysis or regulatory compliance, Bulk Logging records that data to .csv files on the agent file system. When the files fill up, they can be compressed and transferred to the management server. This bulk log transfer is more efficient than sending each record over the network individually; plus, the bulk log data isn't entered into the database at all, reducing database maintenance cost. If the data in the bulk log file requires analysis, SCSP contains a command line tool that can load a bulk log file into the database (i.e., if a regulatory audit requires access to the data, etc.).
Note: When doing bulk logging, any event that was sent in real-time will be loaded again when a bulk log file data is inserted into the database.
For detailed information about bulk log processing on an SCSP agent, please refer to Appendix A in the Symantec Critical System Protection Administration Guide. Please note the following important points:
- SISIDSService writes events to SISIDSEvents.csv. When this file reaches the size limit set in the common configuration, it rolls over. Only the rolled-over SISIDSEvent.csv files get compressed and uploaded as a bulk log transfer.
- When the agent communicates with the manager, the bulk logging thread in the IPS service wakes up at regular intervals to look for rolled-over files that require compression. Compressed files are then moved to the "upload" folder where they wait to be transferred to the manager. When the agent is offline, the IPS service doesn't process rolled-over files.
- The IPS service launches bulklogger.exe to upload zip files in the "upload" folder to the manager.