Re: Local sources seem not to be working

Alexandre Santos <[email protected]> Wed, 30 Mar 2022 15:07:22 +0000

Newsgroups	gmane.comp.syslog-ng
Message-ID	<CO6PR10MB5602BFB8F446B54C91863F0FD01F9@CO6PR10MB5602.namprd10.prod.outlook.com>

Hi Gabor,

Thank you for you feedback.

Can you share the config, when the issue cannot be seen?
I am sending the configuration in attachment.

I would still like to see 2 "syslog-ng-ctl stats" outputs when the issue happens.
The issue is hard to reproduce, the next time the error is seen, I try to run it.
But how can I run syslog-ng-ctl stats for the 2nd syslog-ng instance?
root@machine:/~# ps -ewfH | grep syslog-ng
root      2582     1  0 09:06 ?        00:00:34   /usr/sbin/syslog-ng -F --caps cap_net_bind_service,cap_net_broadcast,cap_net_raw,cap_dac_read_search,cap_chown,cap_fowner=p cap_dac_override,cap_syslog=ep
root      4018     1  0 09:07 ?        00:00:00   /usr/sbin/syslog-ng -F --cfgfile=/etc/syslog-ng/mgmt-syslog-ng.conf --pidfile=/var/lib/syslog-ng/mgmt-syslog-ng.pid --persist-file=/var/lib/syslog-ng/mgmt-syslog-ng.persist --control=/var/lib/syslog-ng/mgmt-syslog-ng.ctl
syslog-ng-ctl, seems to only show stats for job 2582.

Regards, Alex


From: Gabor Nagy (gnagy) <[email protected]>
Sent: 29 de março de 2022 12:48
To: Alexandre Santos <[email protected]>; Syslog-ng users' and developers' mailing list <[email protected]>
Subject: Re: Local sources seem not to be working

Hi Alex,

Using regular disk-buffer vs. using reliable disk-buffer shouldn't cause symptoms like that. It sounds like reliable(yes) would turn on a flow-control-like behaviour, which it doesn't.
(And as you said it only affects local sources).
The main difference between the two kinds of disk-buffers is, that while reliable disk-buffer write every message to the disk-buffer, a normal disk-buffer has memory-only buffers for performance reasons (and flow-control reasons too).
You can still lose logs with a reliable disk-buffer if no flow-control is used: when the disk-buffer has reached it's maximum size and new messages keep arriving, then syslog-ng drops those messages.

We have more detailed documentation about disk-buffers in the admin guide, where you can see the structure of disk-buffers:
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.36/administration-guide/61#TOPIC-1768724<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.syslog-ng.com%2Ftechnical-documents%2Fdoc%2Fsyslog-ng-open-source-edition%2F3.36%2Fadministration-guide%2F61%23TOPIC-1768724&data=04%7C01%7CASantos%40infinera.com%7C888d04fa66b245d8a46008da117a0a88%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C1%7C637841513109614956%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=7M%2BxYTvsi1hu01apm%2BE1gNndB%2BZVzPnjbfvkWiOOBdA%3D&reserved=0>

Can you share the config, when the issue cannot be seen?
I would still like to see 2 "syslog-ng-ctl stats" outputs when the issue happens.

Regards,
Gabor
________________________________
From: Alexandre Santos <[email protected]<mailto:[email protected]>>
Sent: Monday, March 28, 2022 13:45
To: Gabor Nagy (gnagy) <[email protected]<mailto:[email protected]>>; Syslog-ng users' and developers' mailing list <[email protected]<mailto:[email protected]>>
Subject: RE: Local sources seem not to be working

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.


Hi Gabor,



"This is strange: the d_localfile destinations (as well as the vrf-socket destination "d_mgmt_vrf_socket") receive messages from the syslog() source, but not from the internal() or system() sources?"

Yes.



And the issue vanishes when "d_mgmt_vrf_socket" destination is removed?

Yes.



I could not test the 2 last suggestions that you made.



We did however another test, which was to remove the reliable option from d_mgmt_vrf_socket, and it seems the problem is not seen again.



Besides from what it is written in the manual, in other which cases/conditions can syslog-ng loose logs?



reliable()

Type:

yes|no

Default:

no

Description: If set to yes, syslog-ng OSE cannot lose logs in case of reload/restart, unreachable destination or syslog-ng OSE crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. If set to no, the normal disk-buffer will be used. This provides a faster, but less reliable disk-buffer option.



Thanks in advance,

Alex



From: Gabor Nagy (gnagy) <[email protected]<mailto:[email protected]>>
Sent: 25 de março de 2022 14:44
To: Alexandre Santos <[email protected]<mailto:[email protected]>>; Syslog-ng users' and developers' mailing list <[email protected]<mailto:[email protected]>>
Subject: Re: Local sources seem not to be working



CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.



Hi Alex,

Sorry I haven't answered yet. I'll have a few ideas I would like to try out next week.

This is strange: the d_localfile destinations (as well as the vrf-socket destination "d_mgmt_vrf_socket") receive messages from the syslog() source, but not from the internal() or system() sources?

And the issue vanishes when "d_mgmt_vrf_socket" destination is removed?

If it would be soft flow-control, then the syslog() source would be suspended too.

Just a tip: would you switch out the unix-dgram() destination to syslog() destination, please? Maybe that's not possible with the VRF in-place...

In the stats output, do you see an increased number of dropped messages?



I would still suggest increasing the 4MB disk-buffer. You should make an estimation of how long could the mgmt syslog-ng be down (i.e not receiving from the unix-dgram), what is the average incoming EPS and an average message size, that could give a hint about the required disk-buffer size.

Regards,

Gabor

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

42116.no.reliable.syslog-ng.conf (application/octet-stream, 14.3 KB)

@version: 3.32

# First, set some global options.
options {
    chain_hostnames(off);
    flush_lines(0);
    use_dns(no);
    use_fqdn(no);
    owner("root");
    group("root");
    perm(0644);
    stats_freq(0);
    bad_hostname("^gconfd$");
    ts-format("iso");
    time_zone("UTC");
    mark-freq(0);
    frac-digits(3);
    keep-hostname(yes);
};

# common block end


source s_src {
    system();
    internal();
    syslog(ip(10.20.30.40) transport("udp") port(514) keep-alive(no));
};

rewrite r_host { set("MACHINE-${HOST}", value("HOST")); };

# localFile: apps
destination d_localfile_apps {
    program("/opt/machine/local/bin/write_with_rotation.sh /var/log/apps.log 10 10"
        flags(syslog-protocol)
        suppress(5)
        disk-buffer(
            mem-buf-size(2097152)
            disk-buf-size(4194304)
            reliable(yes)
            dir("/tmp")
        )
    );
};
filter f_localfile_apps_local7 {
    facility(local7) and level(info .. emerg);
};
filter f_localfile_apps {
    filter(f_localfile_apps_local7); };
log {
    source(s_src);
    filter(f_localfile_apps);
    destination(d_localfile_apps);
    flags(flow-control);
};
# end of localFile: apps

# localFile: kernel
destination d_localfile_kernel {
    program("/opt/machine/local/bin/write_with_rotation.sh /var/log/kernel.log 10 10"
        flags(syslog-protocol)
        suppress(5)
        disk-buffer(
            mem-buf-size(2097152)
            disk-buf-size(4194304)
            reliable(yes)
            dir("/tmp")
        )
    );
};
filter f_localfile_kernel_kern {
    facility(kern) and level(info .. emerg);
};
filter f_localfile_kernel {
    filter(f_localfile_kernel_kern); };
log {
    source(s_src);
    filter(f_localfile_kernel);
    destination(d_localfile_kernel);
    flags(flow-control);
};
# end of localFile: kernel

# localFile: messages
destination d_localfile_messages {
    program("/opt/machine/local/bin/write_with_rotation.sh /var/log/messages.log 10 10"
        flags(syslog-protocol)
        suppress(5)
        disk-buffer(
            mem-buf-size(2097152)
            disk-buf-size(4194304)
            reliable(yes)
            dir("/tmp")
        )
    );
};
filter f_localfile_messages_mail {
    facility(mail) and level(info .. emerg);
};
filter f_localfile_messages_daemon {
    facility(daemon) and level(info .. emerg);
};
filter f_localfile_messages_syslog {
    facility(syslog) and level(info .. emerg);
};
filter f_localfile_messages_lpr {
    facility(lpr) and level(info .. emerg);
};
filter f_localfile_messages_uucp {
    facility(uucp) and level(info .. emerg);
};
filter f_localfile_messages_cron {
    facility(cron) and level(info .. emerg);
};
filter f_localfile_messages_ftp {
    facility(ftp) and level(info .. emerg);
};
filter f_localfile_messages_ntp {
    facility(ntp) and level(info .. emerg);
};
filter f_localfile_messages_security {
    facility(security) and level(info .. emerg);
};
filter f_localfile_messages_console {
    facility(console) and level(info .. emerg);
};
filter f_localfile_messages_solaris-cron {
    facility(solaris-cron) and level(info .. emerg);
};
filter f_localfile_messages {
    filter(f_localfile_messages_mail) or filter(f_localfile_messages_daemon) or filter(f_localfile_messages_syslog) or filter(f_localfile_messages_lpr) or filter(f_localfile_messages_uucp) or filter(f_localfile_messages_cron) or filter(f_localfile_messages_ftp) or filter(f_localfile_messages_ntp) or filter(f_localfile_messages_security) or filter(f_localfile_messages_console) or filter(f_localfile_messages_solaris-cron); };
log {
    source(s_src);
    filter(f_localfile_messages);
    destination(d_localfile_messages);
    flags(flow-control);
};
# end of localFile: messages

# localFile: security
destination d_localfile_security {
    program("/opt/machine/local/bin/write_with_rotation.sh /var/log/security.log 10 10"
        flags(syslog-protocol)
        suppress(5)
        disk-buffer(
            mem-buf-size(2097152)
            disk-buf-size(4194304)
            reliable(yes)
            dir("/tmp")
        )
    );
};
filter f_localfile_security_auth {
    facility(auth) and level(info .. emerg);
};
filter f_localfile_security_authpriv {
    facility(authpriv) and level(info .. emerg);
};
filter f_localfile_security {
    filter(f_localfile_security_auth) or filter(f_localfile_security_authpriv); };
log {
    source(s_src);
    filter(f_localfile_security);
    destination(d_localfile_security);
    flags(flow-control);
};
# end of localFile: security

# localFile: user
destination d_localfile_user {
    program("/opt/machine/local/bin/write_with_rotation.sh /var/log/user.log 10 10"
        flags(syslog-protocol)
        suppress(5)
        disk-buffer(
            mem-buf-size(2097152)
            disk-buf-size(4194304)
            reliable(yes)
            dir("/tmp")
        )
    );
};
filter f_localfile_user_user {
    facility(user) and level(info .. emerg);
};
filter f_localfile_user {
    filter(f_localfile_user_user); };
log {
    source(s_src);
    filter(f_localfile_user);
    destination(d_localfile_user);
    flags(flow-control);
};
# end of localFile: user

# localFile: alarm
destination d_localfile_alarm {
    program("/opt/machine/local/bin/write_with_rotation.sh /var/log/alarm.log 10 10"
        flags(syslog-protocol)
        suppress(5)
        disk-buffer(
            mem-buf-size(2097152)
            disk-buf-size(4194304)
            reliable(yes)
            dir("/tmp")
        )
    );
};
filter f_localfile_alarm_local1 {
    facility(local1) and level(info .. emerg);
};
filter f_localfile_alarm {
    filter(f_localfile_alarm_local1); };
log {
    source(s_src);
    filter(f_localfile_alarm);
    destination(d_localfile_alarm);
    flags(flow-control);
};
# end of localFile: alarm

# localFile: cli-command
destination d_localfile_cli-command {
    program("/opt/machine/local/bin/write_with_rotation.sh /var/log/cli-command.log 10 10"
        flags(syslog-protocol)
        suppress(5)
        disk-buffer(
            mem-buf-size(2097152)
            disk-buf-size(4194304)
            reliable(yes)
            dir("/tmp")
        )
    );
};
filter f_localfile_cli-command_match {
    match("<CLI>" value("MESSAGE"));
};
filter f_localfile_cli-command_local3 {
    facility(local3) and level(info .. emerg) and filter(f_localfile_cli-command_match);
};
filter f_localfile_cli-command {
    filter(f_localfile_cli-command_local3); };
log {
    source(s_src);
    filter(f_localfile_cli-command);
    destination(d_localfile_cli-command);
    flags(flow-control);
};
# end of localFile: cli-command

# localFile: configuration
destination d_localfile_configuration {
    program("/opt/machine/local/bin/write_with_rotation.sh /var/log/configuration.log 10 10"
        flags(syslog-protocol)
        suppress(5)
        disk-buffer(
            mem-buf-size(2097152)
            disk-buf-size(4194304)
            reliable(yes)
            dir("/tmp")
        )
    );
};
filter f_localfile_configuration_local2 {
    facility(local2) and level(info .. emerg);
};
filter f_localfile_configuration {
    filter(f_localfile_configuration_local2); };
log {
    source(s_src);
    filter(f_localfile_configuration);
    destination(d_localfile_configuration);
    flags(flow-control);
};
# end of localFile: configuration

# localFile: event
destination d_localfile_event {
    program("/opt/machine/local/bin/write_with_rotation.sh /var/log/event.log 10 10"
        flags(syslog-protocol)
        suppress(5)
        disk-buffer(
            mem-buf-size(2097152)
            disk-buf-size(4194304)
            reliable(yes)
            dir("/tmp")
        )
    );
};
filter f_localfile_event_local6 {
    facility(local6) and level(info .. emerg);
};
filter f_localfile_event {
    filter(f_localfile_event_local6); };
log {
    source(s_src);
    filter(f_localfile_event);
    destination(d_localfile_event);
    flags(flow-control);
};
# end of localFile: event

# localFile: grpc-command
destination d_localfile_grpc-command {
    program("/opt/machine/local/bin/write_with_rotation.sh /var/log/grpc-command.log 10 10"
        flags(syslog-protocol)
        suppress(5)
        disk-buffer(
            mem-buf-size(2097152)
            disk-buf-size(4194304)
            reliable(yes)
            dir("/tmp")
        )
    );
};
filter f_localfile_grpc-command_match {
    match("<GRPC>" value("MESSAGE"));
};
filter f_localfile_grpc-command_local3 {
    facility(local3) and level(info .. emerg) and filter(f_localfile_grpc-command_match);
};
filter f_localfile_grpc-command {
    filter(f_localfile_grpc-command_local3); };
log {
    source(s_src);
    filter(f_localfile_grpc-command);
    destination(d_localfile_grpc-command);
    flags(flow-control);
};
# end of localFile: grpc-command

# localFile: linecard
destination d_localfile_linecard {
    program("/opt/machine/local/bin/write_with_rotation.sh /var/log/linecard.log 10 10"
        flags(syslog-protocol)
        suppress(5)
        disk-buffer(
            mem-buf-size(2097152)
            disk-buf-size(4194304)
            reliable(yes)
            dir("/tmp")
        )
    );
};
filter f_localfile_linecard_local4 {
    facility(local4) and level(info .. emerg);
};
filter f_localfile_linecard {
    filter(f_localfile_linecard_local4); };
log {
    source(s_src);
    filter(f_localfile_linecard);
    destination(d_localfile_linecard);
    flags(flow-control);
};
# end of localFile: linecard

# localFile: netconf-command
destination d_localfile_netconf-command {
    program("/opt/machine/local/bin/write_with_rotation.sh /var/log/netconf-command.log 10 10"
        flags(syslog-protocol)
        suppress(5)
        disk-buffer(
            mem-buf-size(2097152)
            disk-buf-size(4194304)
            reliable(yes)
            dir("/tmp")
        )
    );
};
filter f_localfile_netconf-command_match {
    match("<NETCONF>" value("MESSAGE"));
};
filter f_localfile_netconf-command_local3 {
    facility(local3) and level(info .. emerg) and filter(f_localfile_netconf-command_match);
};
filter f_localfile_netconf-command {
    filter(f_localfile_netconf-command_local3); };
log {
    source(s_src);
    filter(f_localfile_netconf-command);
    destination(d_localfile_netconf-command);
    flags(flow-control);
};
# end of localFile: netconf-command

# localFile: restconf-command
destination d_localfile_restconf-command {
    program("/opt/machine/local/bin/write_with_rotation.sh /var/log/restconf-command.log 10 10"
        flags(syslog-protocol)
        suppress(5)
        disk-buffer(
            mem-buf-size(2097152)
            disk-buf-size(4194304)
            reliable(yes)
            dir("/tmp")
        )
    );
};
filter f_localfile_restconf-command_match {
    match("<RESTCONF>" value("MESSAGE"));
};
filter f_localfile_restconf-command_local3 {
    facility(local3) and level(info .. emerg) and filter(f_localfile_restconf-command_match);
};
filter f_localfile_restconf-command {
    filter(f_localfile_restconf-command_local3); };
log {
    source(s_src);
    filter(f_localfile_restconf-command);
    destination(d_localfile_restconf-command);
    flags(flow-control);
};
# end of localFile: restconf-command

# localFile: shell-command
destination d_localfile_shell-command {
    program("/opt/machine/local/bin/write_with_rotation.sh /var/log/shell-command.log 10 10"
        flags(syslog-protocol)
        suppress(5)
        disk-buffer(
            mem-buf-size(2097152)
            disk-buf-size(4194304)
            reliable(yes)
            dir("/tmp")
        )
    );
};
filter f_localfile_shell-command_match {
    match("<SHELL>" value("MESSAGE"));
};
filter f_localfile_shell-command_local3 {
    facility(local3) and level(info .. emerg) and filter(f_localfile_shell-command_match);
};
filter f_localfile_shell-command {
    filter(f_localfile_shell-command_local3); };
log {
    source(s_src);
    filter(f_localfile_shell-command);
    destination(d_localfile_shell-command);
    flags(flow-control);
};
# end of localFile: shell-command

# localFile: sysinfo
destination d_localfile_sysinfo {
    program("/opt/machine/local/bin/write_with_rotation.sh /var/log/sysinfo.log 10 10"
        flags(syslog-protocol)
        suppress(5)
        disk-buffer(
            mem-buf-size(2097152)
            disk-buf-size(4194304)
            reliable(yes)
            dir("/tmp")
        )
    );
};
filter f_localfile_sysinfo_local5 {
    facility(local5) and level(info .. emerg);
};
filter f_localfile_sysinfo {
    filter(f_localfile_sysinfo_local5); };
log {
    source(s_src);
    filter(f_localfile_sysinfo);
    destination(d_localfile_sysinfo);
    flags(flow-control);
};
# end of localFile: sysinfo

# localFile: tl1-command
destination d_localfile_tl1-command {
    program("/opt/machine/local/bin/write_with_rotation.sh /var/log/tl1-command.log 10 10"
        flags(syslog-protocol)
        suppress(5)
        disk-buffer(
            mem-buf-size(2097152)
            disk-buf-size(4194304)
            reliable(yes)
            dir("/tmp")
        )
    );
};
filter f_localfile_tl1-command_match {
    match("<TL1>" value("MESSAGE"));
};
filter f_localfile_tl1-command_local3 {
    facility(local3) and level(info .. emerg) and filter(f_localfile_tl1-command_match);
};
filter f_localfile_tl1-command {
    filter(f_localfile_tl1-command_local3); };
log {
    source(s_src);
    filter(f_localfile_tl1-command);
    destination(d_localfile_tl1-command);
    flags(flow-control);
};
# end of localFile: tl1-command

# localFile: ztp
destination d_localfile_ztp {
    program("/opt/machine/local/bin/write_with_rotation.sh /var/log/ztp.log 10 10"
        flags(syslog-protocol)
        suppress(5)
        disk-buffer(
            mem-buf-size(2097152)
            disk-buf-size(4194304)
            reliable(yes)
            dir("/tmp")
        )
    );
};
filter f_localfile_ztp_match {
    match("<ZTP>" value("MESSAGE"));
};
filter f_localfile_ztp_local7 {
    facility(local7) and level(info .. emerg) and filter(f_localfile_ztp_match);
};
filter f_localfile_ztp {
    filter(f_localfile_ztp_local7); };
log {
    source(s_src);
    filter(f_localfile_ztp);
    destination(d_localfile_ztp);
    flags(flow-control);
};
# end of localFile: ztp


destination d_mgmt_vrf_socket {
    unix-stream("/dev/uds_log"
        flags(syslog-protocol, no-multi-line)
        disk-buffer(
            mem-buf-size(2097152)
            disk-buf-size(4194304)
            dir("/tmp")
        )
    );
};
log {
    source(s_src);
    rewrite(r_host);
    destination(d_mgmt_vrf_socket);
};