RE: naming multiple output files with syslog-ng

Hi Christian

I also thought about using the pipe to an executable in Apache log
configuration. But I personally do not like the solution, because the
logger gets invoked by the Apache process and this most likely produces
overhead in the Apache process.

On the net I found an interesting solution by using FIFO files under
Unix. This allows seperating the Apache log writing process from the
process which handles post-processing of the log file. The FIFO files
behave like a queue where a process can read from the top of the queue
while the writing process adds stuff to the end of the queue. FIFOs are
created using the Unix command mkfifo.

This concept allows you to have more complex post-processing without
interfering with the Apache web server. I use this to convert the log
files to CSV format which can then be efficently imported into a
database using the database loader.

Merry Chistmas
Jan


-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Christian
Folini
Sent: Montag, 17. Dezember 2007 15:21
To: [email protected]
Subject: [logs] naming multiple output files with syslog-ng

Hello everyone,

I am new to this list, after having visited the loganalysis website many
times in the last few weeks. I am working on a fairly big logfile
centralisation project. We are evaluating syslog-ng and I am trying to
configure it to meet our present standards. It basically works, but I
have not yet found an elegant solution to get the files to the right
destination. So I thought maybe you guys have a hint or two for me.

We have a central loghost, but so far the centralisation is not
concurrent. The logfiles I am talking about are apache error logs for a
start (and then ModSecurity audit logs, guardian logs, apache access
logs, cgi-logs, you name it.)

Now there are n hosts with p apache servers serving p virtual domains.
Most apaches serve http and https. We are used to keeping seperate error
logfiles for apache-server, virtualhost-port80 and virtualhost-port 443.

On the central host, they reside in a hierarchy as follows:
.../apache/<virtualhost>/<physicalhost>/port<port>-error.log

->

...
.../apache/www.example.com/h00341/port80-error.log
../apache/www.example.com/h00341/port443-error.log
.../apache/www.example.com/h00445/port80-error.log
.../apache/www.example.com/h00445/port443-error.log
...


I managed to get quite close by configuring ErrorLog in apache as
follows:
ErrorLog "| /usr/bin/logger -t www.example.com_port80 -u
/tmp/logger.socket" 
ErrorLog "| /usr/bin/logger -t www.example.com_port443 -u
/tmp/logger.socket" 

And then configure syslog-ng along the lines of destination df_file {
file(".../$PROGRAM/$HOST/apache-error.log"); };

This brings me files along the lines of:
.../apache/www.example.com_port80/h00341/apache-error.log
.../apache/www.example.com_port443/h00341/apache-error.log

That's close, but not quite.

I could use a program(regex) filter, but this is performance relevant
and highly unwelcome, and it would still not help me to get rid of the
"_port<portnum>" in the program variable.

I found one or two alternatives, but they are equally inelegant and I
would rather not use them.

To sum it up: I do have a working solution, but it's not good looking.

Any thoughts?

regs,

Christian Folini




_______________________________________________
LogAnalysis mailing list
[email protected]
http://www.loganalysis.org/mailman/listinfo/loganalysis