[education/kstars] kstars: fix(scheduler): Add wall-clock timeout to guiding stage to prevent infinite...

Git commit 99625d6ca585b8de801d64c26d3702abc36bd9b5 by Jasem Mutlaq, on behalf of Andreas R..
Committed on 20/07/2026 at 07:23.
Pushed by mutlaqja into branch 'master'.

fix(scheduler): Add wall-clock timeout to guiding stage to prevent infinite...

## Summary

Add a wall-clock timeout to the scheduler's guiding stage, reusing the existing `CaptureOperationsTimeout` setting (default 300s), to prevent infinite retry loops that waste entire nights on a single target.

## Problem

When PHD2 fails to find a guide star, the scheduler retries guiding indefinitely because:

1. PHD2 emits a transient `GUIDE_GUIDING` (settle begins) before `GUIDE_ABORTED` (settle fails) — resets the failure counter to 0 every cycle
2. PHD2's 600s calibration timeout absorbs "Lock Position Lost" events internally without reporting failure

Result: scheduler stuck for **2 hours** on one target while 93 other jobs waited.

## Evidence (user log, 94-job session)
* \[00:30:41\] PHD2: Settling failed (failed to find a suitable guide star). \[00:30:41\] Warning: job 'V0421 And' guiding failed.
* \[00:30:41\] Job 'V0421 And' is guiding, procedure will be restarted in 5s
* \[00:30:50\] Calibration is cleared.
* \[00:31:05\] PHD2: Calibrating, timing out in 600s.
* \[00:31:05\] PHD2: Lock Position Lost, continuing calibration. ... (repeats every 3s for 600s, then cycle restarts) ...
* \[02:25:03\] PHD2: Calibration Failed (Calibration manually stopped).

Previous target (V0418 Cas) captured successfully — the issue is target-specific guide star availability.


## Fix

Track total elapsed time in the guiding stage (across all retries). If it exceeds `CaptureOperationsTimeout` without reaching `GUIDING_COMPLETE`, abort the job and move on.

- Reuses existing `CaptureOperationsTimeout` option — no new settings
- Does not change existing failure counter or retry logic
- Works for both PHD2 and internal guider
- Timer starts once per target, resets on success or job transition

## Verified Behavior

Tested locally on StellarMate (Mele 4C/N150, x64) with:
- CCD Simulator as imaging camera + Telescope Simulator
- ToupTek GPM174M as real guide camera with lens cap on (no stars visible)
- PHD2 as external guider (connected to real guide cam via INDI)
- Magnus's 94-job `miror.esl` scheduler file

Results:
- **Timeout fires correctly**: After `CaptureOperationsTimeout` seconds without guiding success, the scheduler logs "guiding stage exceeded operations timeout" and marks the job aborted.
- **With "Aborted Job Management: None"**: Scheduler skips the failed target and moves to the next job. ✓
- **With "Aborted Job Management: Queue"**: Scheduler re-queues the same job immediately and loops on it (the greedy scheduler picks it again as highest priority with 0 captures).

## Open Question

When the guiding timeout fires, should the job state be `SCHEDJOB_ABORTED` (current, respects "Queue" re-scheduling) or `SCHEDJOB_ERROR` (never re-queued unless "Re-schedule errors" is enabled)?

Our view: `SCHEDJOB_ERROR` is semantically correct — no guide star at that field position is a persistent failure for this session, not a transient interruption. With `SCHEDJOB_ABORTED` + "Queue" mode, the scheduler loops on the same target indefinitely, defeating the purpose of the timeout.

Alternatively, the "Queue" mode's greedy re-scheduling could be improved to re-attempt aborted jobs only after all other eligible jobs have been tried (not immediately as highest priority). This would be a broader change to the greedy scheduler's job selection logic.

M  +1    -1    kstars/ekos/capture/opsmiscsettings.ui
M  +22   -0    kstars/ekos/scheduler/schedulermodulestate.cpp
M  +22   -0    kstars/ekos/scheduler/schedulermodulestate.h
M  +20   -0    kstars/ekos/scheduler/schedulerprocess.cpp
M  +1    -1    kstars/kstars.kcfg

https://invent.kde.org/education/kstars/-/commit/99625d6ca585b8de801d64c26d3702abc36bd9b5

diff --git a/kstars/ekos/capture/opsmiscsettings.ui b/kstars/ekos/capture/opsmiscsettings.ui
index 95d01a2c47..bb2aae88d0 100644
--- a/kstars/ekos/capture/opsmiscsettings.ui
+++ b/kstars/ekos/capture/opsmiscsettings.ui
@@ -170,7 +170,7 @@
      <item row="3" column="0">
       <widget class="QLabel" name="label_2">
        <property name="toolTip">
-        <string>Maximum number of seconds to wait before aborting the capture if operations like filter wheel changes or meridian flips take too long</string>
+        <string>Maximum number of seconds to wait before aborting the capture if operations like filter wheel changes or meridian flips take too long. Also bounds the time the scheduler spends establishing guiding (calibration and lock) before starting capture.</string>
        </property>
        <property name="text">
         <string>Capture Operations Timeout</string>
diff --git a/kstars/ekos/scheduler/schedulermodulestate.cpp b/kstars/ekos/scheduler/schedulermodulestate.cpp
index 075c427087..b08a7cc326 100644
--- a/kstars/ekos/scheduler/schedulermodulestate.cpp
+++ b/kstars/ekos/scheduler/schedulermodulestate.cpp
@@ -444,6 +444,28 @@ void SchedulerModuleState::startGuidingTimer(int milliseconds)
     m_restartGuidingTime = KStarsData::Instance()->ut();
 }
 
+void SchedulerModuleState::startGuidingStageTimer()
+{
+    if (!m_guidingStageStarted)
+    {
+        m_guidingStageStartTime = KStarsData::Instance()->ut();
+        m_guidingStageStarted = true;
+    }
+}
+
+void SchedulerModuleState::resetGuidingStageTimer()
+{
+    m_guidingStageStarted = false;
+    m_guidingStageStartTime = KStarsDateTime();
+}
+
+qint64 SchedulerModuleState::guidingStageTotalMsec() const
+{
+    if (!m_guidingStageStarted)
+        return 0;
+    return m_guidingStageStartTime.msecsTo(KStarsData::Instance()->ut());
+}
+
 // Allows for unit testing of static Scheduler methods,
 // as can't call KStarsData::Instance() during unit testing.
 KStarsDateTime *SchedulerModuleState::storedLocalTime = nullptr;
diff --git a/kstars/ekos/scheduler/schedulermodulestate.h b/kstars/ekos/scheduler/schedulermodulestate.h
index 9e70bb1486..53d018b052 100644
--- a/kstars/ekos/scheduler/schedulermodulestate.h
+++ b/kstars/ekos/scheduler/schedulermodulestate.h
@@ -513,6 +513,23 @@ class SchedulerModuleState : public QObject
         bool isGuidingTimerActive();
         void startGuidingTimer(int milliseconds);
 
+        /**
+         * @brief startGuidingStageTimer Records the wall-clock time when the guiding stage
+         * was first entered for the current target. Unlike the restart timer, this is NOT
+         * reset on retries — it tracks total elapsed time in the guiding phase.
+         */
+        void startGuidingStageTimer();
+        /**
+         * @brief resetGuidingStageTimer Clears the guiding stage timer (called when moving
+         * to next job or when guiding succeeds and capture starts).
+         */
+        void resetGuidingStageTimer();
+        /**
+         * @brief guidingStageTotalMsec Returns milliseconds since the guiding stage was
+         * first entered for this target. Returns 0 if timer not started.
+         */
+        qint64 guidingStageTotalMsec() const;
+
         /** @brief Setter used in testing to fix the local time. Otherwise getter gets from KStars instance. */
         /** @{ */
         static KStarsDateTime getLocalTime();
@@ -853,6 +870,11 @@ class SchedulerModuleState : public QObject
         // Delay for restarting the guider
         int m_restartGuidingInterval { -1 };
         KStarsDateTime m_restartGuidingTime;
+        // Wall-clock timestamp for when guiding was first attempted on the current target.
+        // Unlike currentOperationTime (which resets on each retry), this tracks the absolute
+        // start to detect infinite guide-retry loops (e.g. PHD2 calibration timeout cycles).
+        KStarsDateTime m_guidingStageStartTime;
+        bool m_guidingStageStarted { false };
         // Used in testing, instead of KStars::Instance() resources
         static KStarsDateTime *storedLocalTime;
         // The various preemptiveShutdown states are controlled by this one variable.
diff --git a/kstars/ekos/scheduler/schedulerprocess.cpp b/kstars/ekos/scheduler/schedulerprocess.cpp
index 98dca75b46..9c627fc68c 100644
--- a/kstars/ekos/scheduler/schedulerprocess.cpp
+++ b/kstars/ekos/scheduler/schedulerprocess.cpp
@@ -171,6 +171,7 @@ void SchedulerProcess::findNextJob()
     // Reset failed count
     moduleState()->resetAlignFailureCount();
     moduleState()->resetGuideFailureCount();
+    moduleState()->resetGuidingStageTimer();
     moduleState()->resetFocusFailureCount();
     moduleState()->resetCaptureFailureCount();
 
@@ -1372,6 +1373,9 @@ void SchedulerProcess::startGuiding(bool resetCalibration)
 
     appendLogText(i18n("Starting guiding procedure for %1 ...", activeJob()->getName()));
 
+    // Start the wall-clock timer for the guiding stage (only records on first call,
+    // not reset on retries — used to detect infinite guide-retry loops).
+    moduleState()->startGuidingStageTimer();
     moduleState()->startCurrentOperationTimer();
 }
 
@@ -2827,6 +2831,20 @@ void SchedulerProcess::checkJobStageEpilogue()
             break;
 
         case SCHEDSTAGE_GUIDING:
+            // Safety net: if the total time spent in the guiding stage exceeds the
+            // capture operations timeout, abort the job and move on. This catches infinite
+            // retry loops where PHD2 calibration timeouts + transient GUIDE_GUIDING states
+            // keep resetting the failure counter, preventing the normal MAX_FAILURE_ATTEMPTS
+            // mechanism from working.
+            if (moduleState()->guidingStageTotalMsec() > static_cast<qint64>(Options::captureOperationsTimeout()) * 1000)
+            {
+                appendLogText(i18n("Warning: job '%1' guiding stage exceeded operations timeout (%2 seconds), marking aborted.",
+                                   activeJob()->getName(), Options::captureOperationsTimeout()));
+                stopGuiding();
+                activeJob()->setState(SCHEDJOB_ERROR);
+                findNextJob();
+                break;
+            }
             // Let's make sure guide module does not become unresponsive
             if (moduleState()->getCurrentOperationMsec() > GUIDE_INACTIVITY_TIMEOUT)
             {
@@ -3714,6 +3732,8 @@ void SchedulerProcess::setGuideStatus(GuideState status)
             moduleState()->resetGuideFailureCount();
             // if guiding recovered while we are waiting, abort the restart
             moduleState()->cancelGuidingTimer();
+            // Guiding succeeded — reset the stage timer for this target
+            moduleState()->resetGuidingStageTimer();
 
             moduleState()->updateJobStage(SCHEDSTAGE_GUIDING_COMPLETE);
             getNextAction();
diff --git a/kstars/kstars.kcfg b/kstars/kstars.kcfg
index d0b46c6153..453390fb11 100644
--- a/kstars/kstars.kcfg
+++ b/kstars/kstars.kcfg
@@ -2355,7 +2355,7 @@
          <default>60</default>
       </entry>
       <entry name="CaptureOperationsTimeout" type="UInt">
-         <label>Maximum number of seconds to wait before aborting the capture if operations like filter wheel changes or meridian flips take too long.</label>
+         <label>Maximum number of seconds to wait before aborting the capture if operations like filter wheel changes or meridian flips take too long. Also bounds the time the scheduler spends establishing guiding (calibration and lock) before starting capture.</label>
          <default>300</default>
       </entry>
       <entry name="MinFlipDuration" type="UInt">