Re: cfengine watchdog is not biting

Thanks Xander,

You could test it by running something like:

bash -c 'echo $$ > /var/cfengine/watchdog.pid && exec sleep infinity' &

Running bash separately ensures the pid will be different, and the exec makes the sleep run with the same PID as the shell was running with (i.e. that PID is taken oven by the sleep) and the & makes it all happen in the background.

Then you've got a pidfile showing a valid, running process that is definitely using that pid.

To simulate stale pidfile with no process using that pid is easy, just put a number in there and then exit:

bash -c 'echo $$ > /var/cfengine/watchdog.pid'

To simulate stale pidfile where there is a process running on that pid but it's newer than the pidfile is also possible; it requires use of touch to set an old mtime.  Checking AIX docs (https://www.ibm.com/docs/en/aix/7.3?topic=t-touch-command) it looks like it should be something like:

bash -c 'echo $$ > /var/cfengine/watchdog.pid && touch -t 02210927 /var/cfengine/watchdog.pid && exec sleep infinity' &

(That would set the mtime to 24 hours ago at this writing - Feb 21 at 9:27 in the morning.)

Then in each case run the watchdog and see what happens, maybe convert my inline comments to echo commands so you can see which branch is taken.  (E.g. echo "Pidfile is definitely correct")

I much prefer submitting fully tested code but unfortunately I don't have access to any AIX systems, so if you could try the above it would be very much appreciated.  :)  And this way any issues can be found before the updated code is included in any release.

Best,
--Mike Weilgart

On Feb 22, 2024, at 5:26 AM, Xander Cage <[email protected]> wrote:

forgot to post the error:

root@aixtest01: /root # /var/cfengine/bin/watchdog_changed
/var/cfengine/bin/watchdog_changed[52]: syntax error at line 64 : `else' unexpected

On Thursday, February 22, 2024 at 2:25:03 PM UTC+1 Xander Cage wrote:
i dont think the problematic situation can be simulated, but running the changed script shows a missing "then" ;-)

https://github.com/cfengine/masterfiles/blob/70eb84ad845af716fdc53b1299544e7b521a563d/cfe_internal/core/watchdog/templates/watchdog.mustache#L60 <https://github.com/cfengine/masterfiles/blob/70eb84ad845af716fdc53b1299544e7b521a563d/cfe_internal/core/watchdog/templates/watchdog.mustache#L60>

after adding it, the scripts runs without errors....

On Thursday, February 22, 2024 at 2:47:24 AM UTC+1 [email protected] <applewebdata://CA3B58C6-2B21-4F31-8E4F-5D7E5A644ED3> wrote:
I opened a pull request: https://github.com/cfengine/masterfiles/pull/2841 <https://github.com/cfengine/masterfiles/pull/2841>

Would like some help with testing.  Would be especially great to test on some of these systems where the existing pidfile logic isn't working, besides testing the usual cases.

Best,
--Mike Weilgart
On Wednesday, February 21, 2024 at 6:59:41 AM UTC-8 [email protected] wrote:
Thanks for filing the ticket. :D

On Tuesday, February 20, 2024 at 6:34:15 AM UTC-6 Xander Cage wrote:
ticket created -> https://northerntech.atlassian.net/browse/CFE-4335 <https://northerntech.atlassian.net/browse/CFE-4335>


On Tuesday, February 20, 2024 at 11:11:20 AM UTC+1 Xander Cage wrote:
Hi,

At times the cfengine watchdog script is unable to resolve hanging cf-agents. the reason is that if the system is under high load the watchdog "hangs" because it reads his own stale pid file and is doing simply nothing. 

example:

i had 332 hanging cf-agent processe but the watchdog log just shows this.

Tue Feb 20 10:57:00 CET 2024 Initiating watchdog 6816488
Tue Feb 20 10:57:00 CET 2024 Aborting execution of watchdog 6816488, existing watchdog process 16450014 running
Tue Feb 20 10:58:00 CET 2024 Initiating watchdog 44499432
Tue Feb 20 10:58:00 CET 2024 Aborting execution of watchdog 44499432, existing watchdog process 16450014 running

after deleting the pid file...

Tue Feb 20 10:59:00 CET 2024 Initiating watchdog 25756150
Tue Feb 20 10:59:01 CET 2024 Found cf-execd not running
Tue Feb 20 10:59:03 CET 2024 Found 332 occurrences of cf-execd terminating unresponsive cf-agent
Tue Feb 20 10:59:04 CET 2024 Found 2 symptoms, threshold (0) breached.
Tue Feb 20 10:59:06 CET 2024 Initiating apoptosis
Tue Feb 20 10:59:10 CET 2024 Initiating anastasis

i guess there is room for impovement, as this can bring a system down quite easyily.

wbr

chris


-- 
You received this message because you are subscribed to a topic in the Google Groups "help-cfengine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/help-cfengine/ddLSAt7qsos/unsubscribe <https://groups.google.com/d/topic/help-cfengine/ddLSAt7qsos/unsubscribe>.
To unsubscribe from this group and all its topics, send an email to [email protected] <mailto:[email protected]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/help-cfengine/24484aa8-cf0f-4882-a8b5-47894c81af5cn%40googlegroups.com <https://groups.google.com/d/msgid/help-cfengine/24484aa8-cf0f-4882-a8b5-47894c81af5cn%40googlegroups.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups "help-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web visit https://groups.google.com/d/msgid/help-cfengine/D9036E04-4834-430E-B95C-37A8E17006B1%40gmail.com.