Configuring access logging on the ProxySG to an FTP server then to Reporter
How to configure the ProxySG to send data to a Reporter server?
Is there a step by step guide that will help in setting up access logging?
This document will help in configuring access logging on a ProxySG, upload the files to an FTP server, and then have Reporter process the logs. The document is meant to help in getting access logging and reporting up and running in a relatively short amount of time. For full details on setting up access logging to Reporter, including other options, please see the Symantec Reporter 9.x Initial Configuration Guide
PREAMBLE: WHY SHOULD FTP BE USED?
The ProxySG is able to upload the access logs using various protocols. This document will focus on one specific protocol, and that is the FTP protocol. Why is Symantec recommending the FTP protocol for access log uploads? Because it offers the most/best options in case there is ever the need to restore/re-import the access log data. The direct connection configuration does not keep the access log data in raw format and is used for POC only, as per the following Article on Reporter upload Client. The data is imported into the Symantec Reporter database and the access log file is discarded. With FTP, you can easily create new profiles, recreate profiles, or send data into Symantec Technical Support if need be.
Sizing: Make sure that sizing on Symantec Reporter deployments are appropriate. Please see the Symantec Reporter 9 sizing guide. Because Reporter is resource (disk, CPU, and memory) intensive, for the best performance please consider using real hardware and not virtualized hardware. If the dataset is large, you may also want to consider a 64-bit version of Windows or Linux to install a 64-bit version of Reporter on.
FTP server: Any proprietary or opensource FTP server will do. For simplicity sake this document will use a free opensource FTP server named FileZilla Server. Symantec does not implicitly or explicitly promote this free FTP server software. It is merely using it as an example in the configuration of access log to Reporter server setup. Please use with discretion when selecting an FTP server. NOTE: If interested in connecting to an external FTP server, or using the direct connect method please see the Symantec Reporter 9.x Initial Configuration Guide.
Base Reporter OS: The base operating system used for Reporter in the setup article will be a 32-bit Windows system. Other OSes (32-bit Linux or 64-bit Windows/Linux) may be more appropriate, depending on the results of the sizing guide. Again, this base OS was selected for simplicity sake.
The easiest way to set this up is to install the FTP server on the Reporter server. Make sure there is lots of free disk space. Then the FTP server will be setup and configured. Once the FTP server is up and running, the ProxySG will be configured to upload the access logs to the FTP server. The connectivity between the FTP server and the ProxySG will be tested. Lastly, Reporter 9 will be installed on the Windows server.
STEP I - Setting up Filezilla FTP server.
STEP II - Setting up access logging on the ProxySG to upload files
Troubleshooting: If testing from the ProxySG was unsuccessful, there are several things that you can try to troubleshoot the problem. They are as follows:
* Check/validate the username and password entered in Step 5 above.
* Double check the IP address of the FTP server.
* Make sure the Filezilla server is not blocking FTP traffic from an IP subnet.
* Use the Filezilla server interface to view what is happening. The interface can also be configured to show the passwords being sent in clear text so it can verify/validate what is being sent to the FTP server.
* Go to a DOS prompt and open an FTP session from a DOS window to the FTP server. Make sure login using the credentials work and that you can uploading a file to your FTP server works. If login fails, check the FTP user credentials on the FTP server. If login succeeds but upload fails, check the file system permissions and make sure all file and directory permissions have been given. On the Filezilla server, look at the Filezilla server interface.
* From the workstation if you get a long delay (30 - 60 seconds) before receiving an error and are never able to reach the Filezilla server, then there may have a network problem. A network problem can be as simple as a firewall blocking FTP traffic. Or there isn't any route between the workstation and the FTP server.
* If there is a short delay (1 or 2 seconds) before failure, that indicates that the server is reachable, but the port is not open. Make sure Filezilla is running and something like Windows Firewall is not blocking the port.
* Take a packet capture on the ProxySG (Management Console > Maintenance tab > Service Information > Packet Captures) for a minute or so while forceing an access log upload. This should allow visability to see if the ProxySG is communicating with the FTP server or not. If seeing multiple (three) SYN requests that have no response, then there is probably some sort of networking issue. If seeing SYN > RST three times, then the FTP port is not opened on the remote FTP server, or the wrong FTP port was entered into the access log configuration on the ProxySG.
9.Repeat steps 4 through 8 above for any other log files that is needed to be uploaded. Make sure that when setuping up the log files that the selection of the appropriate log, such as main, or SSL, or P2P, etc...is chosen
BEST PRACTISES FOR FTP UPLOADS:
* Be vigilant in ensuring your access logs are never left on the SG. Monitor your FTP upload/connectivitly to ensure the access logs aren't left on your SG for days, as this will create a backlog of access logs needing to be uploaded to your FTP server.
* Install a syslog tool that monitors the proxy FTP server, possibly using the second interface so you can alert yourself if the main interface goes down.
* Ensure you upload your acess logs at regular intervals. Uploading acess logs of greater than 12 gigs in size is considered burdensome, but uploading acess logs that are 12 K in size is too small. Find an intervall that, on average, uploads a size that is a good fit for your network.
STEP III - Setting up Reporter.
1) Go to https://support.symantec.com to download the latest version of Reporter.
2) Run the Reporter install. Install Reporter onto the hard drive that has the most room (LOTS of free disk space). In our example, that was on the D:\ drive. The install will also ask for an Admin user name, password, and license file (not mandatory).
3) From the Reporter server, open the following URL in Internet Explorer or Firefox Browser: http://127.0.0.1:8081/ . Or if remote you can login using the IP address of the Reporter server: http://<ip.address.of.reporter.server>:8081/ Login to Reporter using the Admin user that was created in the previous step. A message that states, "Welcome to Reporter" should be seen. In order to view reports in Reporter, a loaded database is required. After created (and/or loaded) a database, click on "View Reports" in the top right corner to see the data in the database." Click the OK button to remove the message.
4) Within Reporter under the Reporter Settings > Data Settings > Databases, click on the "New" button. There will be a prompted for a database name. In this example, use "proxysg" (without quotes). Click on the "Next" button.
5) Now you will add the source of your log files.
a.) Click on the "New Log Source" button. A new box will appear. It will ask if desired to pull data from a local file source or an FTP server source. Since the Filezilla FTP server was installed on the Reporter server, select "Local File Source" and click on the "Next" button.
b.) Give the log source a name. Call it proxysg in this example and then click on the "Next" button.
c.) For "Directory Path", browse to the FTP directory. In this example, this is D:\ftp\proxysg\ . Browse to your source directory, click on the "OK" button.
d.) For file pattern, leave it a wild card by using * as the wildcard marker. Click on the "Next" button.
e.) Now Reporter will ask what to do with the file after processing the log file. Rename, move, or delete it. In this example, use of the default of "Rename: Append '.done' to filename" will be used. Click on the "Done" button.
6) Now back to the Log Sources box. The default polling time is every 10 minutes. Increase or decrease this interval if necessary. Once the polling interval has been selected, click on the "Next" button.
7) By default the Reporter server will expire any data that is older than 30 days in the Reporter server. Increase or decrease the expiration date as desired, and also select when to run the database expiration command. (IMPORTANT LICENSING NOTE: Reporter 9 is licensed based on the number of lines in the Reporter database. Having more data in Reporter server may cause licensing issues. Additionally Reporter may reach its limit and no longer import access logs into the Reporter database. So if Reporter runs, but all the reports contain old data, check the number of lines in Reporter database and compare that with the Reporter licensing model to ensure the that upper limit is not reached . If reaching the limit, expire some data so Reporter can restart the data import. Please see TECH242386 for full details regarding Reporter licensing.) Leave the database at defaults and click on the "Next" button.
8) There will be a prompt for the location of the database files. If the database location is on the hard drive with the most room, then click on the "Done" button. The Reporter server will go out and start processing any uploaded access logs if there are any logs to be processed.
9) All done. You can click on "View Reports" in the top right hand corner to start viewing log data.
POST INSTALLATION THINGS TO CONSIDER:
In STEP II, number 7, it discusses the frequency of uploads to the FTP server. If the ProxySG is configured for frequent uploads, such as every five minutes, then the FTP server will end up with a lot of small files in that incoming FTP server directory. If the proxy is used in a 24x7 environment, there will be 288 files uploaded to the FTP server on a daily basis. Over a month's time, that will result in approximately 8,600 files, and over a year's time, that will result in about 100,000 files uploaded. File system performance and backup performance can suffer greatly with that many files stored in a single directory. If a Reporter database rebuild needs to occur, all those files will need to be renamed, which can be a time consuming process.
Because of the size and number of files that are uploaded to the FTP user's incoming directory, some sort of periodic movement of files from the FTP user's home directory to a separate storage location may be warranted. For example, a job can be scheduled to kick off a batch file that will move the files from the FTP directory where Reporter looks for new files to another directory. That way a minimal number of files will be maintained. Please see KB article TECH240942 for futher details.
Subscribing will provide email updates when this Article is updated. Login is required.
Thanks for your feedback. Let us know if you have additional comments below. (requires login)
This will clear the history and restart the chat.