Re: bug report: modlogan fails when given logfiles with very long urls

--=-rBEN8efCm3tRUz4tBEY/
Content-Type: text/plain
Content-Transfer-Encoding: 7bit

ons, 27 07 2005 kl. 17:15 +0200, skrev Christian:
> Henrik Christian Grove schrieb:
> > If I unpack it, and run `modlogan -c config/modlogan.conf <
> > prestat/httpd/access-02.log`, modlogan processes 5 records.
> >
> > If I unpack it, remove the first line of prestat/httpd/access-02.log
> > and the run the same command, modlogan processes 383 records.
> >
> > If I unpack it, and run `head -1 prestat/httpd/access-02.log| modlogan
> > -c config/modlogan.conf`, modlogan processes the single record it
> > gets.
> 
> thanks for your files; i could reproduce it now. i thought about
> something yesterday and now i tried it:
> 
> i trimmed your first line to *exactly* 4096 chars (from 83.108.160.88 to
> 54  [nb]" with the ' " ', the whole record) and running modlogan will
> only count 6 records. trimming the line to 4095 chars, all records are
> processed.
> 
> but that does not mean, modlogan does not *try* to process the following
> records. it does, because it complains when the following record is
> corrupted (e.g. remove one "-" from the next record), it just does not
> count it.

Yes, but it only processes the next 5 (or was it 6).

> but i don't know in which part of modlogan the limit exists
> (/src/processor/web/process.c?), maybe the author can comment on that..?

It's not a limit but the EOF handling that's wrong.

The problem only occurs when it's the first line that's longer than 4095
characters.

The error is in the mgets function found in src/mio.c.

The first time it's called the file has a buffer of 4096 bytes (it has
to have some size, so this isn't the error) and it fills that.

After that there's nothing special about the first invocation (there
shouldn't be).
It then looks for a newline character in the buffer, when it doesn't
find a newline it tries to refill the buffer and it does so by reading
(f->ibuf->size - f->ibuf->used) bytes, this makes sense in the general
case, as the most likely reason for not finding a newline is that buffer
is almost empty. But when the first line was longer than 4095 characters
(f->ibuf->size - f->ibuf->used) is 0, and when you ask for 0 bytes, you
get zero bytes (nothing surprising here). Now comes the problem: The
code then assumes that if we read 0 bytes, it's because we reached
end-of-file (but we've not even read the first line yet) and sets the
eof flag for the file. -- It then proceeds by searching for a newline,
not finding any, expands the buffer, reads data into it (this time it
works - effectively demonstrating that we hadn't reached end-of-file),
searches for a newline ... When it finds a newline it returns the line
found. The next time few times it's called it finds a newline in the
buffer before checking the eof flag, and therefore it returns those
lines. After a few iterations, no more newlines can be found in the
buffer, and it checks the eof flag and stops (instead of reading more
data into the buffer).

The attached patch solves it by not setting the eof flag if we asked for
0 bytes.

Comments and better fixes are welcome.

.Henrik

PS: Not having a function to read a line is one of the (many) reasons I
hate C.

-- 
Henrik Christian Grove <[email protected]>
B-one

--=-rBEN8efCm3tRUz4tBEY/
Content-Disposition: attachment; filename=modlogan.patch-1
Content-Type: text/plain; name=modlogan.patch-1; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

--- mio.c	2004-03-18 03:31:50.000000000 +0100
+++ mio.c-fixed	2005-08-11 17:01:58.090083896 +0200
@@ -344,7 +344,7 @@
 						f->ibuf->used += len;
 					} while (f->ibuf->used < f->ibuf->size && len != 0);
 					/* the read was 'empty' -> EOF */
-					if (len == 0) {
+					if ((len == 0) && (f->ibuf->size - f->ibuf->used)) {
 						f->eof = 1;
 					}
 					


--=-rBEN8efCm3tRUz4tBEY/
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: Quoted-printable

--
To unsubscribe, e-mail:   [email protected]
For additional commands, e-mail: [email protected]
--=-rBEN8efCm3tRUz4tBEY/--