Questions about the EDM indexing process, such as:

What files are created during the EDM indexing process? 

Where are they created? 

How much space do they take up? 

How much memory is used in creating them?

What files are created during the EDM indexing process?

Two kinds of files are created during the EDM indexing process:  one .idx file and one or more .rdx files.

• The .idx file is temporary. It's removed once the indexing completes.

Where are they created?

They are stored in the index directory.

Windows: \ProgramData\Symantec\DataLossPrevention\ServerPlatformCommon\15.8.00000\index

Linux: /var/Symantec/DataLossPrevention/ServerPlatformCommon/15.8.00000/index

The original data remains in the datafiles directory during the indexing process.

Once the indexing completes the original data file is removed.

• • It is recommended to retain the originals as a roll-back/DR strategy, where feasible and applicable.

How much space do they take up?

.idx

The calculation for the size of the .idx file is:  

.idx file size = # rows * # columns * 25

This file takes up disk space only for the duration of the indexing process.

• The .idx file is temporary. It's removed once the indexing completes.

.rdx

The .rdx files can get very large but in general they are smaller than the .idx file. 

Symantec DLP may need to create several of these files due to the 32-bit platform limitations. 

The file size limit is defined by the max_loaded_index_memory setting in the Indexer.properties on Enforce.

The theoretical maximum is 2GB for Windows and 3GB for Linux. The difference is due to how 4GB of the 32-bit process virtual address space is split between kernel and user mode.  The Symantec DLP default install does not take advantage of this. 

However, the indexer.properties file can be changed to take advantage of an additional 1 GB available on Linux. 

When the EDM profile index doesn't fit in a single .rdx file, the .rdx files are enumerated with 0-n values.

For example: DataSource.#.#.rdx.0, DataSource.#.#.rdx.1, DataSource.#.#.rdx.2, etc. 

How much memory do they use?

EDMs are created in two phases. 

Phase 1 (.idx file produced) does not use much memory. 

Phase 2 (.rdx files produced) is more memory intensive. 

This second phase of indexing needs about 1.9 GB of RAM in addition to what Enforce normally requires.

Additional information:

Phase 1 reads in the input file sequentially, finds the patterns, and normalizes the data. 

The hashed files (*.rdx ) are also stored in the same index directory then later replicated to a similar path on all Symantec Data Loss Prevention Detection servers.