Re: General questions, analog, summary, large log sets, incremental

On 10/4/2009 4:47 PM, Scott Haneda wrote:
>>> Every Apache server we have logs all hits to one log, which is rolled
>>> nightly.  Summary used built in ftp to pull down only the new log
>>> files.  Is there a provision to get logs from remote machines, or will
>>> I need to look at something like rsync to make this happen?
>>
>> Analog just analyzes the logfiles, it doesn't do any logfile 
>> "management", so you'd have to handle that yourself.
>
> What would be the best way to manage this then.

How long is a piece of string? Different people will set it up in 
different ways.

> Consider a system where 
> there acre apache access_logs from 10 machines.  There is an 11th 
> machine that will do analog.  I have a log rolling on 24 hours, which 
> means, I could rsync the remote logs directories of the 10 machines and 
> keep all 24 hour log files up to date.  However, the like log, 
> access_log, that is in progress, needs to come over just before analog 
> runs.  This, with incremental, gives the client, what appears to be near 
> real time stats.

It really depends on the size of the logfiles. When the client is 
looking for "real time" stats, are they just interested in the last 
hours worth of activity? Rather than having a machine churning away 24 
hours a day generating "real time" charts that get over-written every 5 
minutes, I'd be more inclined to use something like the Analog Form 
interface to allow the user to generate the report "on demand".

> I could run rsync every 4 minutes, and have analog run very 5, but this 
> is a poor method, as times get out of sync, some logs are larger than 
> others etc.  I am going to assume analog is triggered by scheduler?

You can trigger it by scheduler, or manually (though a cgi-type form, in 
this case - http://analog.cx/docs/form.html)

> I understand analog is one of the most popular, though if it is not a 
> good suit for a large shared hosting environment, please let me know.  I 
> have seen where logs are dropped into the virtual hosts client 
> directory, and analog is set as an option to point to just that users 
> files.  I however, prefer to parse out my entire facilities worth o f logs.

Analog can generate reports for the whole facility from a set of 
"combined" logs, or from a bunch of "per host" logs - it's simply a 
matter of configuration. If you're going to allow the user to customize 
their own reports, there's less chance of inadvertently giving them 
access to someone else's log data if you generate separate logfiles, but 
it's really just a matter of preference. (Your preference, and your 
customers - some customers pay scant attention, or only sporadic 
attention to their logs, others may spend a lot of time delving in to them).

Analog is extremely flexible, and is often used in large hosted 
environments. But there isn't one "right" way to deploy it - it really 
does depend on what you want to achieve.

Aengus
+------------------------------------------------------------------------
|  TO UNSUBSCRIBE from this list:
|    http://lists.meer.net/mailman/listinfo/analog-help
|
|  Analog Documentation: http://analog.cx/docs/Readme.html
|  List archives:  http://www.analog.cx/docs/mailing.html#listarchives
|  Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------