Re: cfengine watchdog is not biting

 Hi Xander,

From that output it doesn't appear that the edited watchdog is looking at the same modified location for the pidfile as you're using, since the first test should have produced an "abort" message.

Could you please run those same tests again but convert the inline comments to echo commands as shown below, so we can see which branch is being taken?

Expected results: Your first test should give "Pidfile is definitely correct" and an abort message; second test should say "Pidfile is stale, ignore it"; third test should say "No current process matching pid in file".

Best,
--Mike Weilgart

######
if [ -s $PIDFILE ]; then
    echo We have a pidfile
    if ps -p $(cat $PIDFILE) > /dev/null 2>&1 ; then
        echo 'There is a process with the PID in the file, but is it stale?'
        if [ -d /proc ]; then
            echo "We can know for sure if it's stale"
            actual_process="/proc/$(cat "$PIDFILE")"
            newer="$(ls -1dt "$PIDFILE" "$actual_process" | head -n 1)"
            if [ "$actual_process" = "$newer" ]; then
                echo Pidfile is stale, ignore it
                echo $$ > $PIDFILE
            else
                echo Pidfile is definitely correct
                echo "$(date) Aborting execution of watchdog $$, existing watchdog process $(cat $PIDFILE) running" >> ${LOGFILE}
                exit 1
            fi
        else
            echo "No /proc, pidfile shows a running process, we'll assume it's valid"
            echo "$(date) Aborting execution of watchdog $$, existing watchdog process $(cat $PIDFILE) running" >> ${LOGFILE}
            exit 1
        fi
    else
        echo No current process matching pid in file
        echo $$ > $PIDFILE
    fi
else
    echo No pidfile at all
    echo $$ > $PIDFILE
fi
######

On Feb 26, 2024, at 1:44 AM, Xander Cage <[email protected]> wrote:

did the requested tests...


root@aixtest01: /root # bash -c 'echo $$ > /var/cfengine/watchdog_changed.pid && exec /opt/freeware/bin/sleep infinity' &
[2] 22020394
root@aixtest01: /root # /var/cfengine/bin/watchdog_changed

log output:

Mon Feb 26 10:36:45 CET 2024 Initiating watchdog 4456948
Mon Feb 26 10:36:47 CET 2024 Found 0 symptoms, threshold (0) not breached, no remediation or collection performed
Mon Feb 26 10:36:47 CET 2024 DONE watchdog 4456948


root@aixtest01: /root # bash -c 'echo $$ > /var/cfengine/watchdog_changed.pid'
root@aixtest01: /root # /var/cfengine/bin/watchdog_changed

log output:

Mon Feb 26 10:37:52 CET 2024 Initiating watchdog 17432960
Mon Feb 26 10:37:53 CET 2024 Found 0 symptoms, threshold (0) not breached, no remediation or collection performed
Mon Feb 26 10:37:53 CET 2024 DONE watchdog 17432960


root@aixtest01: /root # bash -c 'echo $$ > /var/cfengine/watchdog_changed.pid && touch -t 02210927 /var/cfengine/watchdog_changed.pid && exec /opt/freeware/bin/sleep infinity' &
[3] 32964974
root@aixtest01: /root # /var/cfengine/bin/watchdog_changed

[3]+  Stopped                 bash -c 'echo $$ > /var/cfengine/watchdog_changed.pid && touch -t 02210927 /var/cfengine/watchdog_changed.pid && exec /opt/freeware/bin/sleep infinity'

log output:

Mon Feb 26 10:39:28 CET 2024 Initiating watchdog 12321050
Mon Feb 26 10:39:31 CET 2024 Found 0 symptoms, threshold (0) not breached, no remediation or collection performed
Mon Feb 26 10:39:31 CET 2024 DONE watchdog 12321050

expected something more exiting...but it is what is i guess....


On Friday, February 23, 2024 at 9:20:49 AM UTC+1 Xander Cage wrote:
no problem...will do the tests next week...

On Thursday, February 22, 2024 at 6:32:06 PM UTC+1 [email protected] <applewebdata://C128395F-ACE0-4755-BE5D-395EF71C2696> wrote:
Thanks Xander,

You could test it by running something like:

bash -c 'echo $$ > /var/cfengine/watchdog.pid && exec sleep infinity' &

Running bash separately ensures the pid will be different, and the exec makes the sleep run with the same PID as the shell was running with (i.e. that PID is taken oven by the sleep) and the & makes it all happen in the background.

Then you've got a pidfile showing a valid, running process that is definitely using that pid.

To simulate stale pidfile with no process using that pid is easy, just put a number in there and then exit:

bash -c 'echo $$ > /var/cfengine/watchdog.pid'

To simulate stale pidfile where there is a process running on that pid but it's newer than the pidfile is also possible; it requires use of touch to set an old mtime.  Checking AIX docs (https://www.ibm.com/docs/en/aix/7.3?topic=t-touch-command <https://www.ibm.com/docs/en/aix/7.3?topic=t-touch-command>) it looks like it should be something like:

bash -c 'echo $$ > /var/cfengine/watchdog.pid && touch -t 02210927 /var/cfengine/watchdog.pid && exec sleep infinity' &

(That would set the mtime to 24 hours ago at this writing - Feb 21 at 9:27 in the morning.)

Then in each case run the watchdog and see what happens, maybe convert my inline comments to echo commands so you can see which branch is taken.  (E.g. echo "Pidfile is definitely correct")

I much prefer submitting fully tested code but unfortunately I don't have access to any AIX systems, so if you could try the above it would be very much appreciated.  :)  And this way any issues can be found before the updated code is included in any release.

Best,
--Mike Weilgart

On Feb 22, 2024, at 5:26 AM, Xander Cage <[email protected] <>> wrote:

forgot to post the error:

root@aixtest01: /root # /var/cfengine/bin/watchdog_changed
/var/cfengine/bin/watchdog_changed[52]: syntax error at line 64 : `else' unexpected

On Thursday, February 22, 2024 at 2:25:03 PM UTC+1 Xander Cage wrote:
i dont think the problematic situation can be simulated, but running the changed script shows a missing "then" ;-)

https://github.com/cfengine/masterfiles/blob/70eb84ad845af716fdc53b1299544e7b521a563d/cfe_internal/core/watchdog/templates/watchdog.mustache#L60 <https://github.com/cfengine/masterfiles/blob/70eb84ad845af716fdc53b1299544e7b521a563d/cfe_internal/core/watchdog/templates/watchdog.mustache#L60>

after adding it, the scripts runs without errors....

On Thursday, February 22, 2024 at 2:47:24 AM UTC+1 [email protected] <> wrote:
I opened a pull request: https://github.com/cfengine/masterfiles/pull/2841 <https://github.com/cfengine/masterfiles/pull/2841>

Would like some help with testing.  Would be especially great to test on some of these systems where the existing pidfile logic isn't working, besides testing the usual cases.

Best,
--Mike Weilgart
On Wednesday, February 21, 2024 at 6:59:41 AM UTC-8 [email protected] <http://northern.tech/> wrote:
Thanks for filing the ticket. :D

On Tuesday, February 20, 2024 at 6:34:15 AM UTC-6 Xander Cage wrote:
ticket created -> https://northerntech.atlassian.net/browse/CFE-4335 <https://northerntech.atlassian.net/browse/CFE-4335>


On Tuesday, February 20, 2024 at 11:11:20 AM UTC+1 Xander Cage wrote:
Hi,

At times the cfengine watchdog script is unable to resolve hanging cf-agents. the reason is that if the system is under high load the watchdog "hangs" because it reads his own stale pid file and is doing simply nothing. 

example:

i had 332 hanging cf-agent processe but the watchdog log just shows this.

Tue Feb 20 10:57:00 CET 2024 Initiating watchdog 6816488
Tue Feb 20 10:57:00 CET 2024 Aborting execution of watchdog 6816488, existing watchdog process 16450014 running
Tue Feb 20 10:58:00 CET 2024 Initiating watchdog 44499432
Tue Feb 20 10:58:00 CET 2024 Aborting execution of watchdog 44499432, existing watchdog process 16450014 running

after deleting the pid file...

Tue Feb 20 10:59:00 CET 2024 Initiating watchdog 25756150
Tue Feb 20 10:59:01 CET 2024 Found cf-execd not running
Tue Feb 20 10:59:03 CET 2024 Found 332 occurrences of cf-execd terminating unresponsive cf-agent
Tue Feb 20 10:59:04 CET 2024 Found 2 symptoms, threshold (0) breached.
Tue Feb 20 10:59:06 CET 2024 Initiating apoptosis
Tue Feb 20 10:59:10 CET 2024 Initiating anastasis

i guess there is room for impovement, as this can bring a system down quite easyily.

wbr

chris


-- 
You received this message because you are subscribed to a topic in the Google Groups "help-cfengine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/help-cfengine/ddLSAt7qsos/unsubscribe <https://groups.google.com/d/topic/help-cfengine/ddLSAt7qsos/unsubscribe>.
To unsubscribe from this group and all its topics, send an email to [email protected] <>.
To view this discussion on the web visit https://groups.google.com/d/msgid/help-cfengine/24484aa8-cf0f-4882-a8b5-47894c81af5cn%40googlegroups.com <https://groups.google.com/d/msgid/help-cfengine/24484aa8-cf0f-4882-a8b5-47894c81af5cn%40googlegroups.com?utm_medium=email&utm_source=footer>.


-- 
You received this message because you are subscribed to a topic in the Google Groups "help-cfengine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/help-cfengine/ddLSAt7qsos/unsubscribe <https://groups.google.com/d/topic/help-cfengine/ddLSAt7qsos/unsubscribe>.
To unsubscribe from this group and all its topics, send an email to [email protected] <mailto:[email protected]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/help-cfengine/0292d488-b8f5-44ac-b442-2f721dd82163n%40googlegroups.com <https://groups.google.com/d/msgid/help-cfengine/0292d488-b8f5-44ac-b442-2f721dd82163n%40googlegroups.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups "help-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web visit https://groups.google.com/d/msgid/help-cfengine/D3B8410E-1B32-4DEF-8E8A-57AE3AAE6EEC%40gmail.com.