Best practices for Citrix Netscaler AAA logging and retention

By default the Netscaler is set to certain log levels for certain modules on the device, including AAA (authentication, authorization and accounting) logging. Sometimes you may want to change the AAA log retention temporarily for easier troubleshooting.

I’ve posted several articles around Netscaler AAA already but if you’re new to it, AAA logging is saved to the /var/log/ns.log file (and all subsequent rollover files i.e ns.log.0.gz, ns.log.1.gz, ns.log.2.gz, etc.) and you can view these via GUI by going to System > Auditing > then clicking on “Syslog messages”. Once the window loads, in the Module drop down choose “AAA” or whatever you are trying to filter for and hit “Find Now”. Choose the logs in the bottom left (just doubleclick).

The default setting for AAA logs is set to save the last 25 log files (circular logging where it will overwrite the oldest) and the size is set to 100 Kilobytes. Not a whole lot. The best practice is not to utilize the device itself for historical logs. It’s really only meant to be used for realtime/neartime logs for troubleshooting purposes. Historical Netscaler AAA logs should be offloaded from the appliance using a syslog server. It’s very easy to configure. Just go to Auditing > Syslog and add your syslog server and policy. Make sure you select “ALL” events. It’s always best to have more and let your syslog server do it’s thing than try and filter on the device and end up with something you needed missing. Forward it all. Your syslog server should be sized to take it and retain the data for whatever period your information security/compliance policy dictates.

If you don’t have a syslog server available, there’s no excuse. Stand up a free edition of Solarwinds Kiwi or Splunk: http://www.solarwinds.com/products/freetools/free-kiwi-syslog-server.aspx or http://www.splunk.com/view/SP-CAAAE8W. If you have multiple Netscalers you might already be running Citrix Command Center (which is also free) and has syslog ability: http://download.citrix.com/downloads/command-center.html.

Now, to actually view the settings for each log, WinSCP in and go to:

/etc

and copy over the newsyslog.conf. If you open it in Notepad it’s kind of hard to make out. Only use WinSCP to make a backup. Don’t try and edit it with Notepad and push it back.

To view and edit this file easily, use Putty instead and SSH in. Then go to:

1. shell
2. cd /etc
3. cat newsyslog.conf

Right along the top, there is a little warning. Any changes to this file are lost after rebooting. Whatever changes you make, you need to copy into the /nsconfig folder. By default there isn’t one in that folder until you copy it over.

This will show you the contents of the configuration file. See the line that says “/var/log/ns.log“? Those numbers to right are the settings for the ns.log. As I mentioned before, by default is set to save the last 25 log files and the size is set 100 Kilobytes. That’s not a whole lot.

4. Now before you decide to increase the size or number of logs files that are kept on the appliance, check to see how much disk space you have available. Type

df -h

What you’re interested is in the /var partition which you can see on my test device has plenty of room left in my screenshot.

5. Now let’s edit the newsyslog.conf. I don’t want to go overboard here. In this example I’ve calculated that if I bump the log from 100 K to 3 MB I should be okay. That’s 3 MB x 25 logs = 75 MB. That’s a little bit of extra logging I need for troubleshooting users who can’t login with out having to pull logs from my syslog server. Again, don’t max it out because the appliance itself is not where you want to be storing logs. To edit, type:

vi newsyslog.conf

and arrow your mouse down to the 100 and type “i” to edit/insert. Now type 3000 to make the size 3 MB. Once done, press the ESC key and type:

:x!

which will save and exit. If you screwed up and want to exit without saving your changes, type:

:q!

6. After you’ve made your changes, you need to copy the file to /nsconfig so it persists after device reboots.

7. Finally, you may need to restart the syslog process for the changes to go into effect (or just reboot the device if you have a maintenance window). In my experience I usually don’t have to but it’s always best to test to see if your changes persist through a reboot. Type the following to find the syslog process ID:

cat /var/run/syslog.pid

then to kill the process type:

kill -9

and finally to restart the process:

/usr/sbin/syslogd -b 127.0.0.1 -n -v -v &

This is all FreeBSD so if you’re no familiar with the switches:

-b will bind the syslog to the localhost
-n run in the foreground (as in wait until the process completes)
-v -v (yes that’s two of them) maxes out the verbosity of the logged messages

You can see below in my screenshot my process ID changes after the process is restarted. This is a good way to verify the process actually restarted:

8. Go back to the GUI syslog and check the latest log (hopefully it has turned over by now). You’ll notice the first entry in the logs will begin say:

logfile turned over due to size>3000k

or whatever size you had specified.