Re: issue with launching screen process with monit
Kacper Kłys DMCS <[email protected]> Wed, 22 Oct 2025 14:10:34 +0200
| Newsgroups | gmane.comp.monitoring.monit.general |
|---|---|
| Message-ID | <[email protected]> |
This is a multi-part message in MIME format. --------------BKYr92lIPzNX00H6Wqam2nTy Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit So, after I upgraded Monit to 5.35.2, I rebooted the system five times—four reboots were successful. I could see that the process was up and monitored by Monit. On the fifth reboot, Monit stopped monitoring the process (due to five restarts in five cycles) even though the process was up and running. It’s a Buildroot-based system running on a Microchip SAM9X60 (ARM926EJ-S, ARMv5TE). Kernel 6.1.22 from linux4microchip-2023.04. I will send all logs to the email you mentioned. Cheers Kacper W dniu 22.10.2025 o 13:59, [email protected] pisze: > Please can you provide more details? > > You upgraded monit to 5.35.2, then rebooted the server 6 times and the > problem persists? (~5 boots success, after ~6th error after first > cycle despite the "if 5 restarts within 5 cycles then unmonitor" > settings?) > > What platform it exactly is? (linux or BSD version?) > > Please can you add the process PID to the "screen process started > successfully" log message? > > Please can you send full monit log to [email protected]? > > Best regards, > Martin > > > >> On 22. 10. 2025, at 13:35, Kacper Kłys DMCS <[email protected]> wrote: >> >> Hi Martin, >> >> no changes for 5.35.0 and 5.35.2 >> >> Kacper >> >> >> W dniu 22.10.2025 o 13:05, [email protected] pisze: >>> Hi Kacper, >>> >>> it looks like a problem which was fixed in monit 5.35.0: The pidfile >>> may contain a PID from the previous system boot, which belongs to a >>> thread of different process after reboot. >>> >>> Please can you upgrade monit? (5.35.2 recommended) >>> >>> Best regards, >>> Martin >>> >>> >>> >>>> On 22. 10. 2025, at 10:54, [email protected] wrote: >>>> >>>> Hi guys, >>>> I have a problem with the monit daemon on my ARM system. Once every >>>> 5-6 times when I'm restarting the whole system, it happens that the >>>> screen process controlled by monit fails to start. I do not >>>> understand why. In monit logs or in system logs, there is no >>>> helpful information. Additionally, when monit claims that the >>>> process fails (and unmonitors it), I can see the proper screen >>>> process with a correct pid file available on the system. When I >>>> manually press "start service" in monit web gui, the correct >>>> process is found by monit. My monit version is 5.34.3. >>>> >>>> I also do not understand how fast monit claims that the process >>>> fails to start and unmonitors it (in monit logs look like it takes >>>> 1-2 sec while timeout in monitrc for this process is 10 seconds). >>>> >>>> *monitrc file:* >>>> set daemon 20 >>>> set onreboot start >>>> set log /var/log/monit.log >>>> >>>> check process screenprocess with pidfile /var/run/screenprocess.pid >>>> if not exist then start >>>> start program = "/etc/init.d/screenprocess start" with timeout >>>> 10 seconds >>>> stop program = "/etc/init.d/screenprocess stop" with timeout 10 >>>> seconds >>>> if 5 restarts within 5 cycles then unmonitor >>>> >>>> *screenprocess init.d script:* >>>> >>>> #!/bin/sh >>>> >>>> PIDFILE=/var/run/screenprocess.pid >>>> SCREEN_PATH=/tmp/screen >>>> >>>> case "$1" in >>>> start) >>>> logger -t screenprocess "Starting ioc" >>>> echo "Starting ioc" >>>> if [ -f "$PIDFILE" ]; then >>>> logger -t screenprocess "Screen process is already running." >>>> echo "Screen process is already running." >>>> exit 1 >>>> fi >>>> # create dir for screen socket files >>>> mkdir -p $SCREEN_PATH >>>> chmod 700 $SCREEN_PATH >>>> export SCREENDIR=$SCREEN_PATH >>>> # set stack limit >>>> ulimit -s 1024 >>>> # run screen >>>> screen -c /etc/screenprocesslogs.conf -dmSL screenprocess >>>> ./start.cmd && sleep 1 >>>> # get pid of screen >>>> screen -ls | grep -oE "[0-9]+\.screenprocess" | sed -e >>>> "s/\..*$//g" | tail -n 1 > $PIDFILE >>>> if [ -f "$PIDFILE" ]; then >>>> NUM_PIDS=$(grep -E '^[0-9]+$' "$PIDFILE" | wc -l) >>>> # check if only one screen session is active >>>> if [ $NUM_PIDS -eq 1 ]; then >>>> logger -t screenprocess "Screen process started successfully." >>>> echo "Screen process started successfully." >>>> else >>>> logger -t screenprocess "Failed to start Screen process. >>>> Multiple PIDs or none detected" >>>> echo "Failed to start Screen process. Multiple PIDs or none >>>> detected" >>>> exit 1 >>>> fi >>>> else >>>> logger -t screenprocess "Failed to start Screen process. PID >>>> file $PIDFILE not created" >>>> echo "Failed to start Screen process. PID file $PIDFILE not >>>> created" >>>> exit 1 >>>> fi >>>> ;; >>>> >>>> *Logs from monit:* >>>> [2025-10-22T08:27:22+0000] debug : pidfile >>>> '/var/run/screenprocess.pid' does not exist >>>> [2025-10-22T08:27:22+0000] error : 'screenprocess' process is >>>> not running >>>> [2025-10-22T08:27:22+0000] debug : >>>> ------------------------------------------------------------------------------- >>>> [2025-10-22T08:27:22+0000] debug : >>>> ------------------------------------------------------------------------------- >>>> [2025-10-22T08:27:22+0000] debug : pidfile >>>> '/var/run/screenprocess.pid' does not exist >>>> [2025-10-22T08:27:22+0000] info : 'screenprocess' start: >>>> '/etc/init.d/screenprocess start' >>>> [2025-10-22T08:27:22+0000] info : 'mgmt' start on user request >>>> [2025-10-22T08:27:22+0000] info : 'screenprocess' start on user >>>> request >>>> [2025-10-22T08:27:22+0000] info : Monit daemon with PID 251 >>>> awakened >>>> [2025-10-22T08:27:23+0000] debug : Starting screen process >>>> screen process started successfully. >>>> [2025-10-22T08:27:24+0000] error : 'screenprocess' service >>>> restarted 5 times within 5 cycles(s) - unmonitor >>>> >>>> I'd be grateful for any tips on what can be wrong. >>>> >>>> Cheers >>>> >>>> Kacper >>>> >>>> >>>> >>> > --------------BKYr92lIPzNX00H6Wqam2nTy Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit <!DOCTYPE html> <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> </head> <body> <p>So, after I upgraded Monit to 5.35.2, I rebooted the system five times—four reboots were successful. I could see that the process was up and monitored by Monit. </p> <p>On the fifth reboot, Monit stopped monitoring the process (due to five restarts in five cycles) even though the process was up and running.</p> <p><br> </p> <p>It’s a Buildroot-based system running on a Microchip SAM9X60 (ARM926EJ-S, ARMv5TE). Kernel 6.1.22 from linux4microchip-2023.04.<br> <br> I will send all logs to the email you mentioned.</p> <p>Cheers</p> <p>Kacper</p> <p><br> </p> <div class="moz-cite-prefix">W dniu 22.10.2025 o 13:59, <a class="moz-txt-link-abbreviated" href="mailto:[email protected]">[email protected]</a> pisze:<br> </div> <blockquote type="cite" cite="mid:[email protected]"> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> Please can you provide more details? <div><br> </div> <div>You upgraded monit to 5.35.2, then rebooted the server 6 times and the problem persists? (~5 boots success, after ~6th error after first cycle despite the "if 5 restarts within 5 cycles then unmonitor" settings?)</div> <div><br> </div> <div>What platform it exactly is? (linux or BSD version?)</div> <div><br> </div> <div>Please can you add the process PID to the "screen process started successfully" log message?</div> <div><br> </div> <div>Please can you send full monit log to <a class="moz-txt-link-abbreviated" href="mailto:[email protected]">[email protected]</a>?</div> <div><br> </div> <div>Best regards,</div> <div>Martin</div> <div><br> </div> <div><br id="lineBreakAtBeginningOfMessage"> <div><br> <blockquote type="cite"> <div>On 22. 10. 2025, at 13:35, Kacper Kłys DMCS <a class="moz-txt-link-rfc2396E" href="mailto:[email protected]"><[email protected]></a> wrote:</div> <br class="Apple-interchange-newline"> <div> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> <div> <p>Hi Martin, </p> <p>no changes for 5.35.0 and 5.35.2</p> <p>Kacper</p> <p><br> </p> <div class="moz-cite-prefix">W dniu 22.10.2025 o 13:05, <a class="moz-txt-link-abbreviated moz-txt-link-freetext" href="mailto:[email protected]" moz-do-not-send="true">[email protected]</a> pisze:<br> </div> <blockquote type="cite" cite="mid:[email protected]"> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <div>Hi Kacper,</div> <div><br> </div> <div>it looks like a problem which was fixed in monit 5.35.0: The pidfile may contain a PID from the previous system boot, which belongs to a thread of different process after reboot.</div> <div><br> </div> <div>Please can you upgrade monit? (5.35.2 recommended)</div> <div><br> </div> <div>Best regards,</div> <div>Martin</div> <div><br> </div> <div><br> </div> <br> <div> <blockquote type="cite"> <div>On 22. 10. 2025, at 10:54, <a class="moz-txt-link-abbreviated moz-txt-link-freetext" href="mailto:[email protected]" moz-do-not-send="true">[email protected]</a> wrote:</div> <div> <div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><br> </div> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <div> <p>Hi guys, <br> I have a problem with the monit daemon on my ARM system. Once every 5-6 times when I'm restarting the whole system, it happens that the screen process controlled by monit fails to start. I do not understand why. In monit logs or in system logs, there is no helpful information. Additionally, when monit claims that the process fails (and unmonitors it), I can see the proper screen process with a correct pid file available on the system. When I manually press "start service" in monit web gui, the correct process is found by monit. My monit version is 5.34.3.</p> <p>I also do not understand how fast monit claims that the process fails to start and unmonitors it (in monit logs look like it takes 1-2 sec while timeout in monitrc for this process is 10 seconds).</p> <p><b>monitrc file:</b><br> set daemon 20<br> set onreboot start<br> set log /var/log/monit.log<br> <br> check process screenprocess with pidfile /var/run/screenprocess.pid<br> if not exist then start<br> start program = "/etc/init.d/screenprocess start" with timeout 10 seconds<br> stop program = "/etc/init.d/screenprocess stop" with timeout 10 seconds<br> if 5 restarts within 5 cycles then unmonitor<br> </p> <p><b>screenprocess init.d script:</b></p> <p>#!/bin/sh<br> <br> PIDFILE=/var/run/screenprocess.pid<br> SCREEN_PATH=/tmp/screen<br> <br> case "$1" in<br> start)<br> logger -t screenprocess "Starting ioc"<br> echo "Starting ioc"<br> if [ -f "$PIDFILE" ]; then<br> logger -t screenprocess "Screen process is already running."<br> echo "Screen process is already running."<br> exit 1<br> fi<br> # create dir for screen socket files<br> mkdir -p $SCREEN_PATH<br> chmod 700 $SCREEN_PATH<br> export SCREENDIR=$SCREEN_PATH<br> # set stack limit<br> ulimit -s 1024<br> # run screen<br> screen -c /etc/screenprocesslogs.conf -dmSL screenprocess ./start.cmd && sleep 1<br> # get pid of screen<br> screen -ls | grep -oE "[0-9]+\.screenprocess" | sed -e "s/\..*$//g" | tail -n 1 > $PIDFILE<br> if [ -f "$PIDFILE" ]; then<br> NUM_PIDS=$(grep -E '^[0-9]+$' "$PIDFILE" | wc -l)<br> # check if only one screen session is active<br> if [ $NUM_PIDS -eq 1 ]; then<br> logger -t screenprocess "Screen process started successfully."<br> echo "Screen process started successfully."<br> else<br> logger -t screenprocess "Failed to start Screen process. Multiple PIDs or none detected"<br> echo "Failed to start Screen process. Multiple PIDs or none detected"<br> exit 1<br> fi<br> else<br> logger -t screenprocess "Failed to start Screen process. PID file $PIDFILE not created"<br> echo "Failed to start Screen process. PID file $PIDFILE not created"<br> exit 1<br> fi<br> ;;<br> <br> </p> <p><b>Logs from monit:</b><br> [2025-10-22T08:27:22+0000] debug : pidfile '/var/run/screenprocess.pid' does not exist<br> [2025-10-22T08:27:22+0000] error : 'screenprocess' process is not running<br> [2025-10-22T08:27:22+0000] debug : -------------------------------------------------------------------------------<br> [2025-10-22T08:27:22+0000] debug : -------------------------------------------------------------------------------<br> [2025-10-22T08:27:22+0000] debug : pidfile '/var/run/screenprocess.pid' does not exist<br> [2025-10-22T08:27:22+0000] info : 'screenprocess' start: '/etc/init.d/screenprocess start'<br> [2025-10-22T08:27:22+0000] info : 'mgmt' start on user request<br> [2025-10-22T08:27:22+0000] info : 'screenprocess' start on user request<br> [2025-10-22T08:27:22+0000] info : Monit daemon with PID 251 awakened<br> [2025-10-22T08:27:23+0000] debug : Starting screen process<br> screen process started successfully.<br> [2025-10-22T08:27:24+0000] error : 'screenprocess' service restarted 5 times within 5 cycles(s) - unmonitor<br> </p> <p>I'd be grateful for any tips on what can be wrong.</p> <p>Cheers</p> <p>Kacper</p> </div> <br> <br> </div> </blockquote> </div> <br> </blockquote> </div> </div> </blockquote> </div> <br> </div> </blockquote> </body> </html> --------------BKYr92lIPzNX00H6Wqam2nTy--