Local sources seem not to be working

Hi

Any news regarding this issue?

Making a recap of the findings:


  *   Using a Debian 10 buster with first release with 3.36.1;

  *   After some time "system()" source logs are not getting written to the destinations;
  *   The log messages from other sources, internal() and syslog(...) continue to work fine, being written to the destinations;
  *   One the things I noticed is that the socket to the journal seems to vanish during the error situation:

It seems that somehow syslog-ng in unable to read from linux journal.
Have you ever experienced this problem?
Do know what can be wrong with the system?


root@machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system*
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
systemd-j 1723 root  mem    REG   0,19  8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
systemd-j 1723 root  mem    REG   0,19  8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root   16u   REG   0,19  8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root   24u   REG   0,19  8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root  mem    REG   0,19  8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
syslog-ng 3201 root  mem    REG   0,19  8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root   14r   REG   0,19  8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
syslog-ng 3201 root   15r   REG   0,19  8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
journalct 6861 root  mem    REG   0,19  8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
journalct 6861 root  mem    REG   0,19  8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root    5r   REG   0,19  8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root    6r   REG   0,19  8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
root@ machine:~# lsof /run/log/journal/98101a328524447d88917bea845a8966/system*
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
systemd-j 1723 root  mem    REG   0,19  8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
systemd-j 1723 root  mem    REG   0,19  8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root   16u   REG   0,19  8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
systemd-j 1723 root   24u   REG   0,19  8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root  mem    REG   0,19  8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
journalct 6861 root  mem    REG   0,19  8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root    5r   REG   0,19  8388608 31745 /run/log/journal/98101a328524447d88917bea845a8966/system.journal
journalct 6861 root    6r   REG   0,19  8388608 26165 /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal

Thanks in advance,
Alex

From: syslog-ng <[email protected]> On Behalf Of Alexandre Santos
Sent: 19 de maio de 2022 09:25
To: Syslog-ng users' and developers' mailing list <[email protected]>
Subject: Re: [syslog-ng] Local sources seem not to be working

Hi Szilard,

There is no filter:

source syslog_ng_src {
    internal();
};

destination d_localfile_syslog_ng {
    program("/opt/machine/local/bin/write_with_rotation.sh /var/log/syslog-ng-internal.log 10 10"
        flags(syslog-protocol)
        suppress(5)
        disk-buffer(
            mem-buf-size(2097152)
            disk-buf-size(4194304)
            reliable(yes)
            dir("/tmp")
        )
    );
};
log {
    source(syslog_ng_src);
    destination(d_localfile_syslog_ng);
    flags(flow-control);
};

Thanks and Regards,
Alex

From: syslog-ng <[email protected]> On Behalf Of Szilard Parrag (sparrag)
Sent: 19 de maio de 2022 08:59
To: [email protected]
Subject: Re: [syslog-ng] Local sources seem not to be working

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi Alex,

We've checked it too and syslog-ng does not release the file descriptor of journald even with flow-control enabled.

Also, your internal logs seem rather terse, maybe there is a filter which filters out the important parts. Could you please check it?

Szilard

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq