RE: Why are the services not started automatically on reboot?

Hello Lutz
Thanks for your comments.

> You are using RHEL I think?
Yes

> Are you using Systemd to start your applications or handle Monit the application startup only?
Monit

> Any dependency defined for the services?
Yes

> There is no reason to do a "monit start all" after your system is up and running. As long as you use the default setting ("onreboot start") Monit start all defined applications.
Good news, that is also my understanding.

If changed 2 service checks from PROGRAM to PROCESS and now the debug output in monit.log looks much more understandable.
I do not know why service check PROGRAM generated strange behavior like calling several time the START method on the same service  but it could be related to having the check program taking time to check if the applications run as they are heavy and take its time to start up. The check status is only collected after the next poll cycle.
I will investigate in more detail when I find time for it.

What still puzzles me that monit is not calling the START method for services which have no dependency and for the service which is at the base of the dependency tree (all other services depend from it) but calls the RESTART method.
Here the definition of the base service others depend on it
CHECK PROCESS mqm_TBCOM MATCHING "^/opt/mqm/bin/amqzxma0 -m TBCOM -u mqm$"
    START   = "/bin/bash -c '. /u01/app/bcom/settings/bash_profile; /bin/strmqm TBCOM'" UID mqm GID mqm
    STOP    = "/bin/bash -c '. /u01/app/bcom/settings/bash_profile; /bin/endmqm -i TBCOM'" UID mqm GID mqm
    RESTART = "/bin/bash -c '. /u01/app/bcom/settings/bash_profile; /opt/mqm/bin/endmqm -p TBCOM; /opt/mqm/bin/strmqm TBCOM'" UID mqm GID mqm
    IF NOT EXIST THEN RESTART
    IF NOT EXIST THEN EXEC "/u01/app/bcom/monit/logger.sh CRITICAL"
    IF 2 RESTARTS WITHIN 5 CYCLES THEN UNMONITOR
    IF 2 RESTARTS WITHIN 5 CYCLES THEN EXEC "/u01/app/bcom/monit/logger.sh FATAL"

The log shows that the RESTART method gets called.
[CEST Aug 14 22:07:14] error    : 'mqm_TBCOM' process is not running
[CEST Aug 14 22:07:14] info     : 'mqm_TBCOM' exec: '/u01/app/bcom/monit/logger.sh CRITICAL'
[CEST Aug 14 22:07:14] error    : 'mqm_TBCOM' process is not running
[CEST Aug 14 22:07:14] info     : 'mqm_TBCOM' trying to restart
[CEST Aug 14 22:07:14] debug    : 'appia_APPIA' process test failed [pid=22195] -- No such process
[CEST Aug 14 22:07:14] debug    : Cannot open proc file '/proc/2377/stat' -- No such file or directory
[CEST Aug 14 22:07:14] debug    : system statistic error -- cannot read /proc/2377/stat
[CEST Aug 14 22:07:14] info     : 'mqm_TBCOM' restart: '/bin/bash -c . /u01/app/bcom/settings/bash_profile; /opt/mqm/bin/endmqm -p TBCOM; /opt/mqm/bin/strmqm TBCOM'
[CEST Aug 14 22:07:22] debug    : AMQ8146E: IBM MQ queue manager not available.
...

For all other services which are dependent of the base service monit correctly calls the START method.
Any ideas why this happens?
Could it be that the services do have an EXEC method defined. I use it to log RESTART of services into syslog calling the logger command.
It would be very nice if monit ALERT would be configurable in a sense that not only mailing is possible but another transport could be called e.g. logger command using syslog.

I also noticed that
CHECK DIRECTORY mqsi_mqsibackout_CS WITH PATH /u01/app/mqsi/filenodes/CcpcsfileToReconSwift940/CCPCSFILE.TO.RECON.SWIFT.940/mqsibackout
    IF CHANGED TIMESTAMP THEN EXEC "/u01/app/bcom/monit/logger.sh CRITICAL"
generates the following log entry I do not understand as the directory has not changed
[CEST Aug 14 22:07:27] debug    : 'mqsi_mqsibackout_CS' directory exists
[CEST Aug 14 22:07:27] debug    : 'mqsi_mqsibackout_CS' is directory
[CEST Aug 14 22:07:27] error    : 'mqsi_mqsibackout_CS' modify/change time for /u01/app/mqsi/filenodes/CcpcsfileToReconSwift940/CCPCSFILE.TO.RECON.SWIFT.940/mqsibackout changed from N/A to Fri, 21 Jun 2019 16:54:48
[CEST Aug 14 22:07:27] info     : 'mqsi_mqsibackout_CS' exec: '/u01/app/bcom/monit/logger.sh CRITICAL'

Many thanks for any help.
Regards
Markus

-----Original Message-----
From: monit-general <monit-general-bounces+markus.sonderegger=juliusbaer.com@nongnu.org> On Behalf Of Lutz Mader
Sent: Sonntag, 11. August 2019 18:19
To: This is the general mailing list for monit <[email protected]>
Subject: Re: Why are the services not started automatically on reboot?

Hello Markus,
let us know the used Monit version (the output from monit -V) and the Linux version you used.

With regards, Lutz

p.s.
You are using RHEL I think?

Are you using Systemd to start your applications or handle Monit the application startup only?

Any dependency defined for the services?

p.s.
There is no reason to do a "monit start all" after your system is up and running. As long as you use the default setting ("onreboot start") Monit start all defined applications.

To prevent the default behaviour you can use "onreboot laststate", but Monit does not start stopped applications after a system restart then.
But you can stop applications and do some maintenance after a system restart and start the aplications manually.
You can use "onreboot" with any service definition also.

I use something like this, for example (with Monit 5.25.2).
check process Postgres with pidfile /var/run/pgsql/data.pid
  onreboot laststate
  start program "/bin/launchctl load -F
/Library/LaunchDaemons/org.postgresql.postmaster.plist"
  stop program "/bin/launchctl unload
/Library/LaunchDaemons/org.postgresql.postmaster.plist"
  if failed unixsocket /var/run/pgsql/.s.PGSQL.5432 then restart #  if failed host macmini.local port 5432 then restart
  if 3 restarts within 30 cycles then unmonitor


*****JuliusBaer Disclaimer***** This e-mail is for the intended recipient only and may contain confidential or privileged information. If you have received this e-mail by mistake, please contact us immediately and completely delete it (and any attachments) and do not forward it or inform any other person of its contents. If you send us messages by e-mail, we take this as your authorisation to correspond with you by e-mail, however, we reserve the right not to execute orders and instructions transmitted by e-mail at any time and without further explanation. If you do not wish to receive any further e-mail correspondence please let us know. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, amended, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Neither the Julius Baer Group nor the sender accept liability for any errors or omissions in the content of this message which arise as a result of its e-mail transmission. Please note that all e-mail communications to and from the Julius Baer Group may be monitored. This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction.