Re: cfengine watchdog is not biting

no problem...will do the tests next week...

On Thursday, February 22, 2024 at 6:32:06 PM UTC+1 [email protected] 
wrote:

> Thanks Xander,
>
> You could test it by running something like:
>
> bash -c 'echo $$ > /var/cfengine/watchdog.pid && exec sleep infinity' &
>
> Running bash separately ensures the pid will be different, and the exec 
> makes the sleep run with the same PID as the shell was running with (i.e. 
> that PID is taken oven by the sleep) and the & makes it all happen in the 
> background.
>
> Then you've got a pidfile showing a valid, running process that is 
> definitely using that pid.
>
> To simulate stale pidfile with no process using that pid is easy, just put 
> a number in there and then exit:
>
> bash -c 'echo $$ > /var/cfengine/watchdog.pid'
>
> To simulate stale pidfile where there is a process running on that pid but 
> it's newer than the pidfile is also possible; it requires use of touch to 
> set an old mtime.  Checking AIX docs (
> https://www.ibm.com/docs/en/aix/7.3?topic=t-touch-command) it looks like 
> it should be something like:
>
> bash -c 'echo $$ > /var/cfengine/watchdog.pid && touch -t 
> 02210927 /var/cfengine/watchdog.pid && exec sleep infinity' &
>
> (That would set the mtime to 24 hours ago at this writing - Feb 21 at 9:27 
> in the morning.)
>
> Then in each case run the watchdog and see what happens, maybe convert my 
> inline comments to echo commands so you can see which branch is taken. 
>  (E.g. echo "Pidfile is definitely correct")
>
> I much prefer submitting fully tested code but unfortunately I don't have 
> access to any AIX systems, so if you could try the above it would be very 
> much appreciated.  :)  And this way any issues can be found before the 
> updated code is included in any release.
>
> Best,
> --Mike Weilgart
>
> On Feb 22, 2024, at 5:26 AM, Xander Cage <[email protected]> wrote:
>
> forgot to post the error:
>
> root@aixtest01: /root # /var/cfengine/bin/watchdog_changed
> /var/cfengine/bin/watchdog_changed[52]: syntax error at line 64 : `else' 
> unexpected
>
> On Thursday, February 22, 2024 at 2:25:03 PM UTC+1 Xander Cage wrote:
>
>> i dont think the problematic situation can be simulated, but running the 
>> changed script shows a missing "then" ;-)
>>
>>
>> https://github.com/cfengine/masterfiles/blob/70eb84ad845af716fdc53b1299544e7b521a563d/cfe_internal/core/watchdog/templates/watchdog.mustache#L60
>>
>> after adding it, the scripts runs without errors....
>>
>> On Thursday, February 22, 2024 at 2:47:24 AM UTC+1 [email protected] 
>> wrote:
>>
>>> I opened a pull request: 
>>> https://github.com/cfengine/masterfiles/pull/2841
>>>
>>> Would like some help with testing.  Would be especially great to test on 
>>> some of these systems where the existing pidfile logic isn't working, 
>>> besides testing the usual cases.
>>>
>>> Best,
>>> --Mike Weilgart
>>> On Wednesday, February 21, 2024 at 6:59:41 AM UTC-8 nick.a...@
>>> northern.tech wrote:
>>>
>>>> Thanks for filing the ticket. :D
>>>>
>>>> On Tuesday, February 20, 2024 at 6:34:15 AM UTC-6 Xander Cage wrote:
>>>>
>>>>> ticket created -> https://northerntech.atlassian.net/browse/CFE-4335
>>>>>
>>>>>
>>>>> On Tuesday, February 20, 2024 at 11:11:20 AM UTC+1 Xander Cage wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> At times the cfengine watchdog script is unable to resolve hanging 
>>>>>> cf-agents. the reason is that if the system is under high load the watchdog 
>>>>>> "hangs" because it reads his own stale pid file and is doing simply nothing.
>>>>>>  
>>>>>>
>>>>>> example:
>>>>>>
>>>>>> i had 332 hanging cf-agent processe but the watchdog log just shows 
>>>>>> this.
>>>>>>
>>>>>> Tue Feb 20 10:57:00 CET 2024 Initiating watchdog 6816488
>>>>>> Tue Feb 20 10:57:00 CET 2024 Aborting execution of watchdog 6816488, 
>>>>>> existing watchdog process 16450014 running
>>>>>> Tue Feb 20 10:58:00 CET 2024 Initiating watchdog 44499432
>>>>>> Tue Feb 20 10:58:00 CET 2024 Aborting execution of watchdog 44499432, 
>>>>>> existing watchdog process 16450014 running
>>>>>>
>>>>>> after deleting the pid file...
>>>>>>
>>>>>> Tue Feb 20 10:59:00 CET 2024 Initiating watchdog 25756150
>>>>>> Tue Feb 20 10:59:01 CET 2024 Found cf-execd not running
>>>>>> Tue Feb 20 10:59:03 CET 2024 Found 332 occurrences of cf-execd 
>>>>>> terminating unresponsive cf-agent
>>>>>> Tue Feb 20 10:59:04 CET 2024 Found 2 symptoms, threshold (0) breached.
>>>>>> Tue Feb 20 10:59:06 CET 2024 Initiating apoptosis
>>>>>> Tue Feb 20 10:59:10 CET 2024 Initiating anastasis
>>>>>>
>>>>>> i guess there is room for impovement, as this can bring a system down 
>>>>>> quite easyily.
>>>>>>
>>>>>> wbr
>>>>>>
>>>>>> chris
>>>>>>
>>>>>>
> -- 
> You received this message because you are subscribed to a topic in the 
> Google Groups "help-cfengine" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/help-cfengine/ddLSAt7qsos/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to 
> [email protected].
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/help-cfengine/24484aa8-cf0f-4882-a8b5-47894c81af5cn%40googlegroups.com 
> <https://groups.google.com/d/msgid/help-cfengine/24484aa8-cf0f-4882-a8b5-47894c81af5cn%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
>

-- 
You received this message because you are subscribed to the Google Groups "help-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web visit https://groups.google.com/d/msgid/help-cfengine/04ecb255-c3d1-44d8-a815-5e634123ef1dn%40googlegroups.com.