Monit triggering restart storm

Hi,

I have a bunch of Monit rules to perform check on a service

   1. One check process rule (existence and port checks)
      1. does not exist for 5 cycles then start
      2.  failed port XXXX for 6 times within 8 cycles then restart
      3.  failed port YYYY for 6 times within 8 cycles then restart
      4.  failed port ZZZZ for 6 times within 8 cycles then restart
      2. Three check program rules with custom checks
   1. if status != 0 for 5 times within 10 cycles then restart
      2. if status != 0 for 5 times within 10 cycles then restart
      3. if status != 0 for 5 times within 10 cycles then restart
      3. One to check log content
      1. check file  + if content = "BIG ERROR" then restart

start/stop rules are

start program = "/bin/systemctl start myservice"
stop program = "/bin/systemctl stop myservice"

There are no dependency at Monit level but checks are part of the same
bunch of groups.

Problem, is that due to multiple issues, I got a "restart" storm as

   1. some  port check failed -> restart issued
   2. lead to error at custom script -> restart issued
   3. content log reading has some lags -> restart issued

Myservice or system.d configuration/feature are not well designed so got
"already bind exception" as system.d tried to start several instance at the
same time🤔

So port check failed again, system.d killed the wrong one, MyService was
blocked, restart again. etc.....

I had to shutdown Monit to prevent further action (I could have monit -g
group unmonitor also), kill every instance of my service, start it
correctly, then reactivate Monit


Question:

   - Is there a native way to prevent Monit to issue the same start/stop
   commands in a defined time-frame ?
   - Does Monit dependency feature between checks could help as I don't see
   how it could help ?
   - Any other hint/proposal (aside increasing the values of "for N times
   within T cycles" to delay the restart)

Remark: maybe exploring system.D features StartLimitIntervalSe &
StartLimitBurst could help.


Best Regards.

-- 
To unsubscribe:
https://lists.nongnu.org/mailman/listinfo/monit-general