Logger: Taking control of system logs

Logger is a gpl licensed LINUX utility to take control of system logs. Features include:

Abile to completely replace sys/klogd.
Abile to handle logs over 2 gigs.
Automatic nightly log rotation without effecting any services dependent on it.
Perforamce increase in software logging to Logger. High log usage software like Apache, Exim, and Squid show dramatic improvement.
Any application that writes a a text log can be maintained be Logger.
Light, small, and fast.
Emphasizes security.
TCP log data transfers. Log data is scrambled over TCP connections.
Reduces drive usage through memory queueing. On my system, my log drive temperature dropped from 125 deg. (F) to 95 deg (F). This also makes available the option of spinning down the hard drive to conserve power (a must for laptops).

Please visit Logger's Freshmeat page and rate it.
Logger requiremes a full developement system (gcc compiler/libraries)
File available for download:

Logger vers. 08361.0419 can be downloaded by clicking here.
Last updated on Friday, December 26, 2008 at 04:22:14 AM.

Setting up Logger is as follows:
Be sure to check the COMPILE script and modify the CPU variable as needed. Below is the Compilation script used to build Logger (included in the download):
#!/bin/bash clear # Modify for your CPU using GCC switches export CPU="-march=i686 -mtune=i686" # Required defines export DEF="-D_USE_LARGEFILE64 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE" export SRC="-D_REENTRANT -D_BSD_SOURCE -D_SVID_SOURCE" # Required Libraries # Required compilier options export OPT="-w -O3 -funroll-loops -ffast-math -fomit-frame-pointer -fno-exceptions" # Get down to business and compile it out. gcc -o Logger Logger.c $DEF $CPU $SRC $OPT strip Logger

After adjusting the CPU parameters to match your hardware and selecting the compile options, simply type:
./COMPILE

If all went well, the prompt will return and no errors will be displayed.
For easy administration, I recommend coping the Logger executable into your /sbin folder:
cp Logger /sbin/

Now on to the Logger configuration file. Below is an example of a basic configuration. This is in no means complete or accurate to any system. It is a basic blueprint to tailor to your system's specific needs. An example is included in the download.
### ### Logger configuration file ### ### ALL ENTRIES PER LINE ARE REQUIRED. ### Type: Queue ### Queue Name or Idenity (must be unique) ### Owner ### Group ### Permissions (leading ZERO (0) *must* be present) ### Keep this many lines in memory before saving to disk (min 1) ### Prefix to entry (* = no prefix) ### Output File(s)/TCP Connections (Logs can be sent via TCP only as well, just omit the file) ### ### Output Queues - MUST be listed first Queue Klog 0 0 0600 1 /var/log/Klog.log @10.100.0.107:14007 @10.100.0.103:14007 Queue Slog 0 0 0600 25 /var/log/Syslog.log @10.100.0.107:14000 @10.100.0.103:14000 Queue ApacheLog 0 0 0644 1000 /var/log/Apache.log @10.100.0.107:14001 @10.100.0.103:14001 Queue ApacheErr 0 0 0644 1 /var/log/Apache.err @10.100.0.107:14002 @10.100.0.103:14002 Queue EximLog 0 0 0640 500 /var/log/Exim.log @10.100.0.107:14003 @10.100.0.103:14003 Queue DynaStopLog 0 0 0644 25 /var/log/DynaStop @10.100.0.107:14004 @10.100.0.103:14004 Queue FtpLog 0 0 0640 25 /var/log/ftp.log ### Sys/KLog Entries ### ### Type: Kernel/Syslog ### Prefix (* = no prefix, I perfer to use hostname here) ### Queue Name ### ### If you use Logger in place of Sys/Klogd, be sure you disable them in your startup scripts. Kernel Sys1 Klog SysLog Sys1 Slog ### Type: Pipe/TCP ### Input pipe or TCP port ### Prefix (* = no prefix) ### Queue Name ### TCP relay for Sys/Klog ### ### Be sure to watch out for traffic loops TCP 14007 * Klog TCP 14000 * Slog ### Apache Entries Pipe /tmp/Mounts/tmpfs/ApacheLog * ApacheLog Pipe /tmp/Mounts/tmpfs/ApacheErr * ApacheErr ### Exim Entries ### ### Notice that I consolidate all Exim logs into one. Pipe /tmp/Mounts/tmpfs/Eximmain Main EximLog Pipe /tmp/Mounts/tmpfs/Eximpanic Panic EximLog Pipe /tmp/Mounts/tmpfs/Eximreject Reject EximLog ### DynaStop Pipe /tmp/Mounts/tmpfs/DynaStop * DynaStopLog ### FTP Log Pipe /var/log/xferlog * FtpLog

Any line with a number sign (#) at the befinning is a comment and ignored by Logger.
### FTP Log Queue FtpLog 0 0 0640 25 /var/log/ftp.log Pipe /var/log/xferlog * /var/log/ftp.log

The above is an example where the log name can not easily be changed, but you still want the benefits of Logger.
The owner, group and permissions can be obtained with the following command applied to each original log file you want Logger to handle:
# stat /var/log/xferlog File: `/var/log/xferlog' Size: 0 Blocks: 0 IO Block: 1024 regular file Device: 808h/2056d Inode: 70164528 Links: 1 Access: (0640/-rw-r-----) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2007-09-24 15:48:18.000000000 -0500 Modify: 2007-09-24 15:48:18.000000000 -0500 Change: 2007-09-24 15:48:18.000000000 -0500

Remember that the permission (the access field) must have the lead zero (0) for Logger to set them correctly.
This file needs to be saved in the /etc directory as Logger.conf.

Now we need to start Logger,
/sbin/Logger
Be sure to put the above line in your start up scripts before any application redirected to Logger.
Once Logger is up and running, we have three signals to control its operations:

USR1 This signal forces all logs in Logger's memory queue to be written to disk. This is automatically done each night at midnight.

HUP This signal tells Logger to rescan the Logger.conf file and adjust the lines stored in the memory queues. Due to security concerns, Logger will NOT open or close any logs.. Logs are automatically forced to disk.

QUIT
TERM
EXIT Any of these signals tell Logger to exit after force writing the logs to disk. You should plan for a 2 or 3 second delay in your shut down scripts just for added safety.

This will demonstrate in a step-by-step illustration on how to redirect an existing log to Logger (using Squid):
First we need information about the existing Squid log file:
# stat /var/log/Squid.log File: `/var/log/Squid.log' Size: 7671 Blocks: 16 IO Block: 1024 regular file Device: 808h/2056d Inode: 70164539 Links: 1 Access: (0640/-rw-r-----) Uid: ( 99/ nobody) Gid: ( 99/ nogroup) Access: 2007-09-25 14:39:56.000000000 -0500 Modify: 2007-09-25 14:40:19.000000000 -0500 Change: 2007-09-25 14:40:19.000000000 -0500

With this information, the Logger.conf entry might lokk like:
### Squid Cache Queue SquidLog 99 99 0640 100 /var/log/Squid.log Pipe /tmp/Mounts/tmpfs/SquidLog * SquidLog

Be sure you replace the log file name in Squid's config file:
access_log /tmp/Mounts/tmpfs/SquidLog

Now Logger and Squid are ready to work together.
If you have Logger running, be sure to restart it and all software now dependent on it (using a simple script for convience, or issue a reboot):
#!/bin/bash # Restart Logger and all dependent software. killall -QUIT Logger sleep 3 /sbin/Logger /usr/local/apache/bin/apachectl restart /usr/local/squid/bin/squid -k reconfigure -f /usr/local/squid/etc/squid.conf killall -HUP exim

The above script is for demonstration purposes only and will need to be adjusted to fit your system. As mention earlier, I use tmpfs for additional performance, be sure your file names match your system.
After Logger (et all) and Squid are restarted, Squid will now gain the benefits of Logger's abilities (automatic nightly rotations, log files greater then 2 gigs if needed, lower log drive usage, and improved performance).

It should be clearly noted that any performance gains are solely based upon the frequency of writes to the disk and the over all activity of the disk in general. A heavy use system will show better performance then a light use one. Each system is different and your milage will vary accordingly.

All advertisements at the bottom of this page do not reflect the views of the author.
They are only to help offset the costs of maintaining this site.

This page was last updated on Friday, 01-Feb-2008 15:22:22 CST


	Advertise throughout tanaya.net

USR1	This signal forces all logs in Logger's memory queue to be written to disk. This is automatically done each night at midnight.
HUP	This signal tells Logger to rescan the Logger.conf file and adjust the lines stored in the memory queues. Due to security concerns, Logger will NOT open or close any logs.. Logs are automatically forced to disk.
QUIT TERM EXIT	Any of these signals tell Logger to exit after force writing the logs to disk. You should plan for a 2 or 3 second delay in your shut down scripts just for added safety.