Re: multilog filtering issues

richard lucassen wrote:

> Yep, that's it. And I miss completly the essence of this filtering
> method. I'm convinced that Dan has good reasons for programming multilog
> like this, but for me there is no _human_ logic in it.

I think your issue here is putting "Dan" and "human" in the same sentence. :-)
(I kid because I love :-P ).

For instance, something that continually "surprises" me is that the
timestamp gets included in the pattern, so adding 't' to the multilog
script means you have to adjust the patterns to take this into account.
I'm not sure under what circumstances I would want to include the
timestamp in the pattern matching, but without the design rationale
behind the program we can only guess the intent from the source code,
which is fraught with peril and can lead to wailing and gnashing of
teeth.

> I understand that the '*a' means that the *whole line* 'f abcde'
> matches, because there is _no_ 'a' somewhere else in the line. Bit weird
> IMHO ;-)

(* Honking big alert - the following is pure conjecture on my part *)

Well, yes, but bear in mind that multilog was used to log the output
of programs that Dan had written, so he may have adjusted the producing
program's log format accordingly.  And I suspect that he tokenises the
patterns somewhat by using white space to jump to the appropriate
position to match the strings he's after, ergo you'll see patterns of
the form '+* foo'.

> But as I have already said: what is the use of it? This looks more like
> odd behaviour of a bug. The way this filter works might be logical to a
> computer, but not for me as a human being.

I thought I had read somewhere about match() being written to be linear
in its run time, but all I can currently find is the current caveat on
the multilog page about its memory constraint.

> So, if I use it as a simple filter (as explained in the docs) and if I
> take all your explanations in consideration (like using the * with a
> space or a puntuation), I fear there is still a big chance that the
> filter will not always behave as expected. This is of course due to my
> unability to oversee all consequences of writing a correct filter
> statement.

I know, what you really want is regular expression matching.  See the
attached patch, which adds actions '~' and '^' to select and deselect
lines based on regular expressions, in addition to '-' and '+'.

Simple usage:

$ echo 'foo1234: hello' | command/multilog '-*' '~foo[0-9]*:' /tmp/log

will select, where as:

$ echo 'foo1234: hello' | command/multilog '+*' '^foo[0-9]*:' /tmp/log

will deselect.  Intermingling +, -, ~, ^ should behave as expected
(well, as I expected anyway).

My caveats are:

*)  It hasn't been thoroughly tested (I lack sufficient caffeine at this
point in time top get my head to cover the edge cases), especially with
regards to line lengths extending over 1000 characters, newline inclusion,
and the intermingling of the two (see "Caveat 2" below).

*)  Tested/compiled only on FreeBSD 4.10 (my only 32 bit platform handy).
You may have to link in an RE library or some other dark magic.

*)  It uses POSIX regex, which I seem to recall can have a degenerate
runtime performance with certain patterns and backtracking (although
I can't lay my hands on any references ATM).

Caveat 2:

In fact, a quick test appears to show that the newline is included even
on line lengths exceeding 1000 characters.

$ perl -e 'print "a" x 2000, "\n";' | command/multilog '-*' '~a$' /tmp/log

does NOT match (I would expect "a" to match null at end of buffer), whereas

$ perl -e 'print "a" x 2000, "\n";' | command/multilog '-*' '~a.$' /tmp/log

does match (. matching newline).  Hmmm, needs more thought and understanding
of the buffering routine, but the time I've allocated to this exercise is
running out.  Any feedback on this would be welcome.

> P.S.: this is a typical example of "A computer does what you ask, not
> what you want" :-)

Well, the following patch implements what your implying you want, but might
not be what you need.  (And remember, "you can't always get what you want"
according to the philosopher Jagger :-).


-- 
Regards,
  Daryl Tester

"Oh Christmas tree, oh Christmas tree!  From hell's heart I stab at thee."
  -- A very Kaaahn! Christmas

Multilog patch to allow selecting patterns using POSIX regular expressions.

In addition to actions '+' and '-', action:
'~RE' selects line if RE matches the line,
'^RE' deselects the line if RE matches the line.

Due to original design, RE can only match on the first 1000 characters.
Note that if log line length is < 1000 characters, the terminating newline
is included and will need to be accounted for in the RE if matching for end
of line.  Insufficient coffee prevented me from coming up with an elegant
solution for this, if any (feedback welcome).  I've also not tested this
as thoroughly as I would have liked (feedback welcome).

Cheers,
  Daryl Tester
  [email protected]


diff -ru admin-orig/daemontools-0.76/src/multilog.c admin/daemontools-0.76/src/multilog.c
--- admin-orig/daemontools-0.76/src/multilog.c	Fri Jul 13 02:19:49 2001
+++ admin/daemontools-0.76/src/multilog.c	Sun Dec  7 09:26:46 2008
@@ -20,6 +20,7 @@
 #include "sig.h"
 #include "match.h"
 #include "deepsleep.h"
+#include <regex.h>
 
 #define FATAL "multilog: fatal: "
 #define WARNING "multilog: warning: "
@@ -69,6 +70,33 @@
   }
 }
 
+regex_t *r;
+
+void r_init(char **script)
+{
+  int i, rnum, rc;
+  char errbuf[512];
+
+  for (rnum = i = 0; script[i]; i++)
+    if ((script[i][0] == '~') || (script[i][0] == '^'))
+      rnum++;
+
+  r = (regex_t *) alloc(rnum * sizeof(regex_t));
+  if (!r) strerr_die2x(111, FATAL, "out of memory");
+
+  for (rnum = i = 0; script[i]; i++) {
+    if ((script[i][0] == '~') || (script[i][0] == '^')) {
+      rc = regcomp(r + rnum, script[i] + 1, REG_EXTENDED | REG_NOSUB);
+      if (rc) {
+	regerror(rc, r + rnum, errbuf, sizeof(errbuf));
+	strerr_die5x(111, FATAL, "regcomp error: '", script[i] + 1, "': ",
+            errbuf);
+      }
+      rnum++;
+    }
+  }
+}
+
 struct cyclog {
   char buf[512];
   buffer ss;
@@ -480,7 +508,7 @@
 char inbuf[1024];
 buffer ssin = BUFFER_INIT(flushread,0,inbuf,sizeof inbuf);
 
-char line[1001];
+char line[1002];
 int linelen; /* 0 <= linelen <= 1000 */
 
 void doit(char **script)
@@ -492,6 +520,8 @@
   char *action;
   int flagselected;
   int flagtimestamp;
+  int rnum, rc;
+  char errbuf[512];
 
   flagtimestamp = 0;
   if (script[0])
@@ -525,6 +555,7 @@
 
     flagselected = 1;
     j = 0;
+    rnum = 0;
     for (i = 0;action = script[i];++i)
       switch(*action) {
         case '+':
@@ -532,11 +563,37 @@
             if (match(action + 1,line,linelen))
               flagselected = 1;
           break;
+	case '~':
+	  if (! flagselected) {
+	    line[linelen + 1] = '\0';
+	    rc = regexec(r + rnum, line, 0, NULL, 0);
+	    if (rc == 0) flagselected = 1;
+	    else if (rc != REG_NOMATCH) {
+	      regerror(rc, r + rnum, errbuf, sizeof(errbuf));
+	      strerr_die5x(111, FATAL, "regexec error: '", script[i] + 1,
+                  "': ", errbuf);
+	    }
+	  }
+	  rnum++;
+	  break;
         case '-':
           if (flagselected)
             if (match(action + 1,line,linelen))
               flagselected = 0;
           break;
+	case '^':
+	  if (flagselected) {
+	    line[linelen + 1] = '\0';
+	    rc = regexec(r + rnum, line, 0, NULL, 0);
+	    if (rc == 0) flagselected = 0;
+	    else if (rc != REG_NOMATCH) {
+	      regerror(rc, r + rnum, errbuf, sizeof(errbuf));
+	      strerr_die5x(111, FATAL, "regexec error: '", script[i] + 1,
+                  "': ", errbuf);
+	    }
+	  }
+	  rnum++;
+	  break;
         case 'e':
           if (flagselected) {
             if (linelen > 200) {
@@ -611,6 +668,7 @@
   ++argv;
   f_init(argv);
   c_init(argv);
+  r_init(argv);
   doit(argv);
   c_quit();
   _exit(0);