[PATCH kgl v2] Fix thread tracking not delivering new messages after a few days

Thread tracking was using a msgid:<msgid> query and --threads which
interacts badly with how lei update works.

After every run, lei update remembers the time for the most recent
successfully retrieved email (in the lastresult entry of the
lei.saved-search file). In subsequent runs, the user's query is limited
by lei to only 2 days prior to that time (see the --remote-fudge-time
argument of lei update). Therefore, for thread tracking, all lei updates
after the very first did:
- query msgid:<msgid> AND dt:<2 days before last msg>..
- for every message returned by the query, fetch the whole thread
If the thread kept getting new messages past that 2 day boundary the
date filter caused the query to return no message and no more updates
would be delivered. --threads does not matter as it only acts on
messages returned by the query and there was no message returned.

Note that this behavior applies to all lei searches. In practice it
works out because searches for a mailing list or a sender or recipient
will keep matching new messages so the query will not end up with no
results, searching repeatedly for a Message-ID is the one that makes
this noticeable.

Signed-off-by: Catalin Iacob <[email protected]>
---
The dt:19700101000000.. looks a bit odd for saying "all" but I didn't
find a way to do it better.

Considered alternatives for fixing this which I believe would all work
from inspecting the code and the reasons why I did not pick them

Alternative 1
lei q offers a --thread-id which triggers the server side to do the
search we want (any message in the thread of this Message-ID) rather
than the search lei currently does (full thread of message with this
Message-ID that is newer than ...). However that argument is not
persisted in the lei.saved-search file so lei update can't use it. Going
this route would have required changing lei and waiting for that to
trickle to all users with no real advantage as that --thread-id seems to
trigger on the server side the same code as the one triggered by the
/all/<msgid>/ URL.

Alternative 2
Call lei up with a higher --remote-fudge-time to basically cancel the
date filter that lei adds. Also works but seems more indirect/hacky and
it suffers from needing to pick a value, is 10 years enough etc. I did
check public-inbox code and as far as I see there is no command line
argument or config to disable this date filter for lei up except for a
high --remote-fudge-time.

Alternative 3
Remove the lastresult line from the lei.saved-search file every time
before invoking lei up. Needs knowledge of the path to the file,
violates layering by reaching into lei implementation details.

BTW, thanks for korgalore (and b4). I'm getting started with kernel
development and they are an absolute boon for watching the lists and
sending patches easily.
---
Changes in v2:
- Switched the datetime for "search all" to 19700101000000. I ran with
  the v1 patch locally since I sent it and noticed that at some point
  right after midnight lei update failed because it passed 1970-01-01
  through git approxidate which led to weird results, probably because it
  was so close to the Epoch, did not trace the code much as my Perl
  knowledge is very limited. Spelling out 19700101000000 fixes this by
  making lei use the date as-is as seen from the code [1] and from my
  testing
  [1](https://repo.or.cz/public-inbox.git/blob/8a3c04bb01b243c28515df77e8ca647ec54dec01:/lib/PublicInbox/Search.pm#l381)
- Link to v1: https://patch.msgid.link/20260531-thread-track-msgid-in-only-url-v1-1-f68f8e5d2497@gmail.com
---
 docs/usage.rst            |  2 +-
 src/korgalore/tracking.py | 10 +++++++---
 2 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/docs/usage.rst b/docs/usage.rst
index 66c6ecf4a098..779b190aaa8d 100644
--- a/docs/usage.rst
+++ b/docs/usage.rst
@@ -539,7 +539,7 @@ How Thread Tracking Works
 
 1. When you run ``track add``, korgalore:
 
-   * Creates a lei search for the thread using ``lei q "mid:<msgid>" --threads``
+   * Creates a lei search for the thread
    * Populates the search with current thread messages
    * Delivers all existing messages to your target
    * Saves tracking metadata in ``~/.local/share/korgalore/tracking.json``
diff --git a/src/korgalore/tracking.py b/src/korgalore/tracking.py
index 2d12e484aa8a..cb71a7f36b70 100644
--- a/src/korgalore/tracking.py
+++ b/src/korgalore/tracking.py
@@ -6,6 +6,7 @@ storing metadata in a separate manifest file from the main configuration.
 import json
 import logging
 import shutil
+import urllib.parse
 from dataclasses import dataclass
 from datetime import datetime, timezone, timedelta
 from enum import Enum
@@ -345,7 +346,7 @@ class TrackingManifest:
 def create_lei_thread_search(msgid: str, output_path: Path) -> Tuple[int, bytes]:
     """Create a new lei search for a thread by message ID.
 
-    Uses: lei q "mid:<msgid>" --threads --only https://lore.kernel.org/all -o v2:<output_path>
+    Uses: lei q "dt:19700101000000.." --only https://lore.kernel.org/all/<msgid> -o v2:<output_path>
 
     Args:
         msgid: The message ID to search for (without angle brackets).
@@ -360,8 +361,11 @@ def create_lei_thread_search(msgid: str, output_path: Path) -> Tuple[int, bytes]
     # Ensure output directory's parent exists
     output_path.parent.mkdir(parents=True, exist_ok=True)
 
-    args = ['q', f'mid:{msgid}', '--threads',
-            '--only', 'https://lore.kernel.org/all',
+    # We use the dt:19700101000000.. query to mean "search for all", it's the Message-ID in the url that
+    # returns the full thread of that message.
+    qmsgid = urllib.parse.quote_plus(msgid)
+    args = ['q', 'dt:19700101000000..',
+            '--only', f'https://lore.kernel.org/all/{qmsgid}',
             '-o', f'v2:{output_path}']
     logger.debug('Creating lei thread search: lei %s', ' '.join(args))
 

---
base-commit: 9060fd4d0d24895e29ca6d982bdd22b64ea45220
change-id: 20260531-thread-track-msgid-in-only-url-07ff53b0fca9

Best regards,
--  
Catalin Iacob <[email protected]>