Re: LogFormat in analog.cfg broken

Quoting "Ulf Hofemeier" <ulf-vqempv/qbUL2fBVCVOL8/[email protected]>:

> Quoting Aengus <[email protected]>:
>
>> Ulf Hofemeier <ulf-vqempv/qbUL2fBVCVOL8/[email protected]> wrote:
>>
>>> I have to admit that the analog configuration is getting quite
>>> confusing to me. Unfortunately the APACHELOGFORMAT line doesn't solve
>>> my problem, so please allow me to provide you with a little more
>>> information regarding the purpose of the updated analog.cfg, as well
>>> as what I'm doing before the problem occurs.
>>>
>>> 1. Copy the previous month Apache log to a temporary location
>>> 2. Run a script to extract page visitor data from the general Apache
>>> log file and store it in a separate file
>>> 3. Run a bash 'for i' loop on the new log files and store the data in
>>> page visitor sub directores
>>>
>>> Unfortunately I decided that Apache has to write more information to
>>> its access_log log file, which is finally the reason why there is
>>> issues with analog now. According to the analog documentation there is
>>> a way to set up a hierarchy so that it will understand a log file
>>> syntax even if it changes from old to new over time, but I haven't
>>> been able to figure out how to make it work.
>>
>> If you have multiple LOGFORMAT statements, Analog will try them  
>> each in turn until it finds one that matches the entries in each of  
>> your logiles. That means that if you have multiple logfiles, and  
>> they aren't all the same format, Analog can still create a single  
>> report from these different logfiles. Obviously the report may  
>> understate this items that weren't recorded in some of the logfiles  
>> - for example, you might have a million requests, but only only  
>> 200,000 Browser strings if you only added that field in leater log  
>> files.
>>
>> LOGFORMAT commands apply to LOGFILEs that are specified after the  
>> LOGFORMAT in the .cfg file. DEFAULTLOGFORMAT commands apply to  
>> logfiles that are specified on the command line.
>>
>> It's not clear from your description whether your script calls  
>> Analog and passes it the name of the logfile as a paramter, or  
>> whether Analog picks up the logfile from the LOGFILE log-????-??.gz  
>> statement in your .cfg file.
>>
>> If you're speciying the LOGFILES in the .cfg file, then these lines  
>> should do the job:
>> APACHELOGFORMAT (%h %l \"%u\" %t \"%r\" %>s %b)
>> APACHELOGFORMAT (%h %l \"%u\" %t \"%r\" %>s %b \"%{Referer}i\"  
>> \"%{User-Agent}i\")
>
> My script calls analog like this:
>
> cd $datadir/$domain
>
> # Determine the range of months from the list of log files that were  
> not empty.
> first=`ls log*|sort|head -1|cut -b5-`
> first=`echo $first |rev |cut -b4- |rev` ;          # YEAR-MO
> last=`ls access_log*|sort|tail -1|cut -b12-`
> last=`echo $last |rev |cut -b4- |rev` ;            # YEAR-MO
> range="$first--$last";                             # YEAR-MO--YEAR-MO
>
> # Collect summary information from all the log files.
> /data/stats/analog/analog access_log.????-??.gz >  
> $analogdir/$domain/$range.html
>
> # Collect information by month in seperate files.
> for i in access_log.????-??.gz ;
> do
>     file=`echo $i |cut -b12-`                       # YEAR-MO.gz
>     file=`echo $file |rev |cut -b4- |rev` ;        # YEAR-MO
>     /data/stats/analog/analog $i > $analogdir/$domain/$file.html
> done
>
> So I pass the log file to analog as a parameter on the command line  
> rather than using analog.cfg.
>
>> If you're calling Analog with the logfiles specified on the command  
>> line, then these lines should work:
>> DEFAULTAPACHELOGFORMAT (%h %l \"%u\" %t \"%r\" %>s %b)
>> DEFAULTAPACHELOGFORMAT (%h %l \"%u\" %t \"%r\" %>s %b  
>> \"%{Referer}i\" \"%{User-Agent}i\")
>>
>
> I will give these two lines a try in my analog.cfg.
>
> Thank you.

Adding DEFAULTAPACHELOGFORMAT didn't work (or my analog.cfg is still broken)

analog.cfg looks like this:

# If you need a LOGFORMAT command (most people don't -- try it without  
first!),
# it must go here, above the LOGFILE commands.
APACHELOGFORMAT (%S %l \"%u\" %t \"%r\" %>s %b \"%{Referer}i\"  
\"%{User-Agent}i\")

DEFAULTLOGFORMAT (%S %j "%u" [%d/%M/%Y:%h:%n:%j] "%j %r %j" %c %b)
DEFAULTLOGFORMAT (%S %j "%u" [%d/%M/%Y:%h:%n:%j] "%j %r %j" %c %b)
DEFAULTLOGFORMAT (%S %j "%u" [%d/%M/%Y:%h:%n:%j] "%j %r" %c %b)
DEFAULTLOGFORMAT (%S %j "%u" [%d/%M/%Y:%h:%n:%j] "%r" %c %b)

DEFAULTAPACHELOGFORMAT (%h %l \"%u\" %t \"%r\" %>s %b)
DEFAULTAPACHELOGFORMAT (%h %l \"%u\" %t \"%r\" %>s %b \"%{Referer}i\"  
\"%{User-Agent}i\")

LOGFORMAT (%S %j "%u" [%d/%M/%Y:%h:%n:%j] "%j %r %j" %c %b)
LOGFORMAT (%S %j "%u" [%d/%M/%Y:%h:%n:%j] "%j %r" %c %b)
LOGFORMAT (%S %j "%u" [%d/%M/%Y:%h:%n:%j] "%r" %c %b)
LOGFORMAT (%S %l \"%u\" %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\")

DEBUG ON

LOGFILE log-????-??.gz

The output after running analog on the latest apache log file looks like this:

/data/stats/analog/analog -G +g../../analog.cfg access_log.2008-09.gz  
 >test.html

C: 1.2.3.4 - "-" [17/Sep/2008:13:12:52 -0600] "GET  
/images/sandiamountains.jpg HTTP/1.1" 200 61671 "http://ladb.unm.edu/"  
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13  
(KHTML, like Gecko) Chrome/0.2.149.29 Safari/525.13"
C:                                                                      
                                   *
C: 1.2.3.4 - "-" [17/Sep/2008:13:12:53 -0600] "GET /favicon.ico  
HTTP/1.1" 200 437 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US)  
AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.2.149.29 Safari/525.13"
C:                                                                      
                  *
C: 1.2.3.4 - "-" [27/Sep/2008:17:55:25 -0600] "GET / HTTP/1.1" 200  
6576  
"http://www.google.com/search?q=latin+america+databases&ie=utf-8&oe=utf-8&aq=t&rls=com.yahoo:en-US:official&client=firefox" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 YFF3  
Firefox/3.0.1"
C:                                                                      
          *
F: Closing logfile access_log.2008-09.gz
S: Successful requests: 0
S: Redirected requests: 0
S: Failed requests: 0
S: Requests returning informational status code: 0
S: Status code not given: 0
S: Unwanted lines: 0
S: Corrupt lines: 23
/data/stats/analog/analog: Warning L: Large number of corrupt lines in logfile
   access_log.2008-09.gz: turn debugging on or try different LOGFORMAT
     Current logfile format:
       %S %j "%j" [%d/%M/%Y:%h:%n:%j] "%j %r %j" %c %b\n
       %S %j "%j" [%d/%M/%Y:%h:%n:%j] "%j %r %j" %c %b\n
       %S %j "%j" [%d/%M/%Y:%h:%n:%j] "%j %r" %c %b\n
       %S %j "%j" [%d/%M/%Y:%h:%n:%j] "%r" %c %b\n
F: Opening stdout as output file
F: Opening /data/stats/analog/requireanalogheader as header file
F: Closing header file /data/stats/analog/requireanalogheader
F: Opening /data/stats/analog/requireanalogfooter as footer file
F: Closing footer file /data/stats/analog/requireanalogfooter


+------------------------------------------------------------------------
|  TO UNSUBSCRIBE from this list:
|    http://lists.meer.net/mailman/listinfo/analog-help
|
|  Analog Documentation: http://analog.cx/docs/Readme.html
|  List archives:  http://www.analog.cx/docs/mailing.html#listarchives
|  Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------