As of SGOS 6.7.2, if an access log has Kafka client and gzip file type selected, you can configure the ProxySG appliance to add a MessageSet header to the compressed log files so that the Kafka broker processes the data correctly as gzip-compressed data.

Use the following command to enable/disable the header (by default, the setting is disabled):

#(config log log_name)kafka-client [no] message-set-codec

Refer to the Command Line Interface Reference for details on this command.

When all of the following conditions are true, the MessageSet header is added to log files:

• Kafka is the upload client for the log
• gzip is selected as the log file type
• the Kafka codec is enabled in the CLI

When one or more of the following conditions are true, the MessageSet header is not added to log files:

• Kafka is not the upload client for the log
• Kafka is the upload client for the log but gzip is not selected as the file type
• Kafka is the upload client but the codec setting is disabled in the CLI (or does not exist, as in pre-6.7.2 versions) 

Making any change to an access log's upload client configuration that reverses the previous MessageSet header state (that is, the header's presence or absence in the log files) can cause future log uploads to fail. For example, if you change a log’s upload client from FTP to Kafka with gzip and codec enabled, the header did not exist in previous log files but it is added to the files after the upload client change. If you then change the Kafka file type from gzip to text, the header is no longer added to the log files. In both of these scenarios, you must take additional steps to ensure that logs are processed correctly.

Clearing the write buffer and removing any stored logs from the appliance will help prevent upload failures when you make any changes that affect the MessageSet header state. Symantec recommends that you perform these steps in the specified order: manually upload the logs, make your desired log changes, and then delete the access logs from the appliance.

Refer to the following procedure for the recommended sequence of steps and an example of how to change the Kafka codec:

1. Upload the access logs. In the Management Console, select Configuration > Access Logging > Logs > Upload Schedule and click Upload Now for logs using the Kafka client.
   
   Example: Your dns log uses the Kafka client and the gzip file format. On the Upload Schedule tab, select the dns log and click Upload Now.
    
2. Change the codec setting as appropriate. In the CLI, issue one of the following commands:
   #(config log log_name)kafka-client message-set-codec
   The header is added to the start of compressed log entries. 
   #(config log log_name)kafka-client no message-set-codec
   Headers are not added to the start of any log entries. 
   
   Example: You are enabling the header for the first time. Issue the following command for your dns log, named dns:
   #(config log dns)kafka-client message-set-codec
    
3. Delete the access logs. In the CLI, issue the following command for each log using the Kafka client: 
   #(config log log_name)commands delete-logs
   
   Example: Issue the following command for the log named dns:
   #(config log dns)commands delete-logs

The dns logs stored on the appliance are deleted. Subsequent uploads of the dns logs, whether system-initiated or user-initiated, will have the MessageSet header unless you do any of the following:

• select a different upload client
• change the log file format to text
• disable the codec setting

Whenever you intend to make any of these modifications, follow the previous procedure to ensure that logs continue to be processed correctly.