Re: traffic analysis

On Tue, Oct 07, 2003 at 06:14:03PM +0200, Miha Rozic @ domenca.si wrote:
> However, it's always better to merge as much data as possible and write a
> summary of days, weeks, months to a different table, rather than calculating
> everything all the time.

I've been doing a lot of thinking on this front: its a tradeoff
between flexibility, correctness vs. performance and space
efficiency.

I'm going to do some experimenting, but just to give you guys an
idea, my schema took 300 megs of logs (1 day) and compressed them into
under 15 megs without losing any granularity. Queries on this
data was very fast, but i'm not sure how it will scale when
there is a year of data in there.

I'm very determined not to fall into a model like Webtrends
Enterprise Reporting server where if you come up with some
missing data for a chunk of time you have already analyzed you
ahve to redo the whole thing.

My goal is to have the flexibility to add data in any order and
have it fill in without too much trouble.

A related issue is using the database for permanent storage. I
konw in environments I've worked on, we keep all the logs around
for 3 months before moving to tape so that we never lose any
data. I would like to avoid keeping logs around at all, but
there is this issue of the 2 Gig file size limit on a database
table that could be a problem.

Rhett
____________________________________________________________________
Reminder: to unsubscribe, send email to <[email protected]>
with the words "unsubscribe mod_log_sql" in the body (w/o quotes).
The module homepage is http://www.grubbybaby.com/mod_log_sql/
The list archives are here:
http://news.gmane.org/thread.php?group=gmane.comp.apache.mod-log-sql