RE: naming multiple output files with syslog-ng

Hi

When I was looking at syslog-ng I got stuck at a similar issue: It is
not possible to further parse a $-parameters using the available
functionality. So this left me with the situation to execute further
code, which does the additional parsing. Now the question is where to do
it. Either the addional processing is directly invoked through the
Apache pipe or you execute through the syslog program() driver. 

Then I had a look how Apache executes the program. Quote from the Apache
documentation http://httpd.apache.org/docs/2.2/logs.html#piped: "Piped
log processes are spawned by the parent Apache httpd process, and
inherit the userid of that process. This means that piped log programs
usually run as root." I did not like the idea of executing stuff as
root.

This got me to the point where I started working with the FIFO files.
The application writes to the FIFO a second process which executes with
minimal privileges does the required parsing and post-processing. This
also fulfills the recommendation layed out in the Apache documentation
http://httpd.apache.org/docs/2.2/logs.html#piped: "As with conditional
logging, piped logs are a very powerful tool, but they should not be
used where a simpler solution like off-line post-processing is
available."

As alternative I see a modified syslog-ng daemon for Christian's issue,
which parses the logger tag and makes additional $-parameters available.
What way to go depends on what goal must be achieved. If the goal is to
just store the files in the structure as Christian wants it and
post-processing is not an issue then I think the syslog-ng modification
is the simplest way to go.

MRJ raised the question when Apache logging happens. Does anybody on the
list know more about this?

Kind regards
Jan


-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Christian
Folini
Sent: Freitag, 21. Dezember 2007 07:56
To: [email protected]
Subject: Re: [logs] naming multiple output files with syslog-ng

Hey guys,

I thought my configuration question was almost innocent and now we are
talking about the fundamentals.

Well, I do not like the additional logger binary running next to apache
in my setup. Apache supports writing the error log to syslog directly,
but that's not possible with the 15 other kinds of logfiles my project
is dealing with. For reasons of consistency, we will therefore stick
with a pipe.

Now as for a performance overhead, this reminds me of a University
lecture. The professor talked about storage and performance.
He grouped storage systems in 5 categories, grouped by their relative
distance to the CPU: cache, ram, disk, net, tape. 
(Maybe there were more, but that's the ones I remember.) The main
difference between them is the access time, according to his lecture. 
He recommended everyone should try to use the storage close to the CPU
for performance reasons. Unless you really have to use a different
storage. But once you are accessing storage further away from the CPU,
all access to storage closer to the CPU becomes practically irrelevant.

For a webserver this means, that the latency of network traffic makes
disk access for logging almost non-existing. Besides Marcus pointed out,
that Apache is logging after having dealt with the client's request.

To be more blunt: The bottleneck here are the majority of users, that
access our webservers with slow modems. It's not the additional pipe
that sits between the server and the logging facility.

I did some tests to be on safe grounds, when I started our project. The
actual numbers back my assumptions laid out above. This is especially
true for SSL, as the handshake makes this "storage access" even more
expensive. A couple of thousand times more expensive than the logging
process, no matter how much logfiles I would write.

Binary formats and csv logging. A great deal faster and a lot more
storage efficient. However, this is not always a priority.
My priority is, that the logfiles I am writing today be readable
tomorrow. Possibly with standard tools. I am quite sure that any binary
format would be the wrong format tomorrow. If these guys are smart, they
will be able to get the data out of the DB again, tomorrow.
But it's likely to cause headaches and meetings and stuff. 
Ascii logging is the safe path; and we can always import the data into a
db later on or transform them into csv format. Actually I did some work
on that lately. All that is bugging me is the time it takes to parse the
ascii logfiles. It takes too long for my taste, but then it's also a lot
of logfiles. ;)

In case you are still following my long post: I am still looking for an
answer for a syslog-ng configuration.

On the central host, I want to get a filetree as follows:
../apache/<virtualhost>/<physicalhost>/port<port>-error.log

And just can't make up my mind how to achieve this in an elegant way. 
ErrorLog "| /usr/bin/logger -t www.example.com_port443 -u \
/tmp/logger.socket"
and
destination df_file {
  file(".../$PROGRAM/$HOST/apache-error.log"); }; get me the following:
.../apache/www.example.com_port80/h00341/apache-error.log

That's close. But really what I wanted. More thoughts?

regs,

Christian

P.S. Marcus: Cool to read your message in this thread. I just wrote
about artificial ignorance in one of the conceptual papers for this
project. :) _______________________________________________
LogAnalysis mailing list
[email protected]
http://www.loganalysis.org/mailman/listinfo/loganalysis