Re: Local sources seem not to be working

Hi,

I haven't seen anything like this. We are reading the journal files using
libsystemd.

Try to remove the syslog-ng persist file and check if reading the journal
restarts.

Also there's can be two ways of local messages getting to syslog-ng,

1) /dev/log forwarding
2) reading the journal files

The first one is actively done by journald. Which one syslog-ng uses is
automatically detected by our system() source.

To see which one syslog-ng is trying to use, try to run it with
--preprocess-into=some-file and check how system() source is expanded.

I am unable to check the source code at the moment, so this is all from the
top-of-my-head, but I hope this already helps to troubleshoot the issue.

On Fri, Jun 24, 2022, 18:21 Alexandre Santos <[email protected]> wrote:

> Hi
>
>
>
> Any news regarding this issue?
>
>
>
> Making a recap of the findings:
>
>
>
>    - Using a Debian 10 buster with first release with 3.36.1;
>
>
>    - After some time “system()” source logs are not getting written to
>    the destinations;
>    - The log messages from other sources, internal() and syslog(…)
>    continue to work fine, being written to the destinations;
>    - One the things I noticed is that the socket to the journal seems to
>    vanish during the error situation:
>
>
>
> It seems that somehow syslog-ng in unable to read from linux journal.
>
> *Have you ever experienced this problem?*
>
> *Do know what can be wrong with the system?*
>
>
>
>
>
> root@machine:~# lsof
> /run/log/journal/98101a328524447d88917bea845a8966/system*
>
> COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
>
> systemd-j 1723 root  mem    REG   0,19  8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> systemd-j 1723 root  mem    REG   0,19  8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> systemd-j 1723 root   16u   REG   0,19  8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> systemd-j 1723 root   24u   REG   0,19  8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> syslog-ng 3201 root  mem    REG   0,19  8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> syslog-ng 3201 root  mem    REG   0,19  8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> syslog-ng 3201 root   14r   REG   0,19  8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> syslog-ng 3201 root   15r   REG   0,19  8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> journalct 6861 root  mem    REG   0,19  8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> journalct 6861 root  mem    REG   0,19  8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> journalct 6861 root    5r   REG   0,19  8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> journalct 6861 root    6r   REG   0,19  8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> root@ machine:~# lsof
> /run/log/journal/98101a328524447d88917bea845a8966/system*
>
> COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
>
> systemd-j 1723 root  mem    REG   0,19  8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> systemd-j 1723 root  mem    REG   0,19  8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> systemd-j 1723 root   16u   REG   0,19  8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> systemd-j 1723 root   24u   REG   0,19  8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> journalct 6861 root  mem    REG   0,19  8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
> journalct 6861 root  mem    REG   0,19  8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> journalct 6861 root    5r   REG   0,19  8388608 31745
> /run/log/journal/98101a328524447d88917bea845a8966/system.journal
>
> journalct 6861 root    6r   REG   0,19  8388608 26165
> /run/log/journal/98101a328524447d88917bea845a8966/system@3721b31246e54dc0baab1ac0f68c3f43-0000000000000001-000581d7e3fe20ba.journal
>
>
>
> Thanks in advance,
>
> Alex
>
>
>
> *From:* syslog-ng <[email protected]> * On Behalf Of *Alexandre
> Santos
> *Sent:* 19 de maio de 2022 09:25
> *To:* Syslog-ng users' and developers' mailing list <
> [email protected]>
> *Subject:* Re: [syslog-ng] Local sources seem not to be working
>
>
>
> Hi Szilard,
>
>
>
> There is no filter:
>
>
>
> source syslog_ng_src {
>
>     internal();
>
> };
>
>
>
> destination d_localfile_syslog_ng {
>
>     program("/opt/machine/local/bin/write_with_rotation.sh
> /var/log/syslog-ng-internal.log 10 10"
>
>         flags(syslog-protocol)
>
>         suppress(5)
>
>         disk-buffer(
>
>             mem-buf-size(2097152)
>
>             disk-buf-size(4194304)
>
>             reliable(yes)
>
>             dir("/tmp")
>
>         )
>
>     );
>
> };
>
> log {
>
>     source(syslog_ng_src);
>
>     destination(d_localfile_syslog_ng);
>
>     flags(flow-control);
>
> };
>
>
>
> Thanks and Regards,
>
> Alex
>
>
>
> *From:* syslog-ng <[email protected]> * On Behalf Of *Szilard
> Parrag (sparrag)
> *Sent:* 19 de maio de 2022 08:59
> *To:* [email protected]
> *Subject:* Re: [syslog-ng] Local sources seem not to be working
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> Hi Alex,
>
>
>
> We've checked it too and syslog-ng does not release the file descriptor of
> journald even with flow-control enabled.
>
>
>
> Also, your internal logs seem rather terse, maybe there is a filter which
> filters out the important parts. Could you please check it?
>
>
>
> Szilard
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq