Re: multilog filtering issues

On Sun, 07 Dec 2008 10:30:34 +1030
Daryl Tester <[email protected]> wrote:

Sorry Daryl for the late reply to your reply, but I sometinmes happen to
be a very busy man :)

Well, after all, I have abandonded the multilog filtering for the
moment. I have used this filtering for quite a long time now and damn,
I've been lucky that it worked...

E.g. this has worked for several years:

 '-*' '+*mark=384*'

And it worked because there just was no letter "m" in the logs.

R.

> richard lucassen wrote:
> 
> > Yep, that's it. And I miss completly the essence of this filtering
> > method. I'm convinced that Dan has good reasons for programming
> > multilog like this, but for me there is no _human_ logic in it.
> 
> I think your issue here is putting "Dan" and "human" in the same
> sentence. :-) (I kid because I love :-P ).
> 
> For instance, something that continually "surprises" me is that the
> timestamp gets included in the pattern, so adding 't' to the multilog
> script means you have to adjust the patterns to take this into
> account. I'm not sure under what circumstances I would want to include
> the timestamp in the pattern matching, but without the design
> rationale behind the program we can only guess the intent from the
> source code, which is fraught with peril and can lead to wailing and
> gnashing of teeth.
> 
> > I understand that the '*a' means that the *whole line* 'f abcde'
> > matches, because there is _no_ 'a' somewhere else in the line. Bit
> > weird IMHO ;-)
> 
> (* Honking big alert - the following is pure conjecture on my part *)
> 
> Well, yes, but bear in mind that multilog was used to log the output
> of programs that Dan had written, so he may have adjusted the
> producing program's log format accordingly.  And I suspect that he
> tokenises the patterns somewhat by using white space to jump to the
> appropriate position to match the strings he's after, ergo you'll see
> patterns of the form '+* foo'.
> 
> > But as I have already said: what is the use of it? This looks more
> > like odd behaviour of a bug. The way this filter works might be
> > logical to a computer, but not for me as a human being.
> 
> I thought I had read somewhere about match() being written to be
> linear in its run time, but all I can currently find is the current
> caveat on the multilog page about its memory constraint.
> 
> > So, if I use it as a simple filter (as explained in the docs) and if
> > I take all your explanations in consideration (like using the * with
> > a space or a puntuation), I fear there is still a big chance that
> > the filter will not always behave as expected. This is of course due
> > to my unability to oversee all consequences of writing a correct
> > filter statement.
> 
> I know, what you really want is regular expression matching.  See the
> attached patch, which adds actions '~' and '^' to select and deselect
> lines based on regular expressions, in addition to '-' and '+'.
> 
> Simple usage:
> 
> $ echo 'foo1234: hello' | command/multilog '-*' '~foo[0-9]*:' /tmp/log
> 
> will select, where as:
> 
> $ echo 'foo1234: hello' | command/multilog '+*' '^foo[0-9]*:' /tmp/log
> 
> will deselect.  Intermingling +, -, ~, ^ should behave as expected
> (well, as I expected anyway).
> 
> My caveats are:
> 
> *)  It hasn't been thoroughly tested (I lack sufficient caffeine at
> this point in time top get my head to cover the edge cases),
> especially with regards to line lengths extending over 1000
> characters, newline inclusion, and the intermingling of the two (see
> "Caveat 2" below).
> 
> *)  Tested/compiled only on FreeBSD 4.10 (my only 32 bit platform
> handy). You may have to link in an RE library or some other dark
> magic.
> 
> *)  It uses POSIX regex, which I seem to recall can have a degenerate
> runtime performance with certain patterns and backtracking (although
> I can't lay my hands on any references ATM).
> 
> Caveat 2:
> 
> In fact, a quick test appears to show that the newline is included
> even on line lengths exceeding 1000 characters.
> 
> $ perl -e 'print "a" x 2000, "\n";' | command/multilog '-*' '~a$'
> /tmp/log
> 
> does NOT match (I would expect "a" to match null at end of buffer),
> whereas
> 
> $ perl -e 'print "a" x 2000, "\n";' | command/multilog '-*' '~a.$'
> /tmp/log
> 
> does match (. matching newline).  Hmmm, needs more thought and
> understanding of the buffering routine, but the time I've allocated to
> this exercise is running out.  Any feedback on this would be welcome.
> 
> > P.S.: this is a typical example of "A computer does what you ask,
> > not what you want" :-)
> 
> Well, the following patch implements what your implying you want, but
> might not be what you need.  (And remember, "you can't always get what
> you want" according to the philosopher Jagger :-).
> 
> 


-- 
___________________________________________________________________
It is better to remain silent and be thought a fool, than to speak
aloud and remove all doubt.

+------------------------------------------------------------------+
| Richard Lucassen, Utrecht                                        |
| Public key and email address:                                    |
| http://www.lucassen.org/mail-pubkey.html                         |
+------------------------------------------------------------------+