Re: RFC - fixing comparison feature #2 - words

On Wed, Jul 29, 2026 at 11:27:40AM -0400, Richard Kimberly Heck wrote:
> If it weren't too difficult, a preference setting for word-only diffs might
> be welcome. That would speed things up a ton.

Right, at the same time, while chasing the littering in the result can be
written as small surgical touch - the heavy lifting is really to get the
heuristics right conceptually, not the code - the shift to word-based diff
will be somewhat massive :/

Anyway, we are not there yet, as Vincent rightly pointed out in the bug, the
same problem re-occurs on the word-level, I spent some time playing with
examples to nail down the rules for word-based spillover. Below is the attempt
for word-level heuristics.

Pavel
-----


  IVb. Detector for "this phrase's word diff is the scattered-noise"

   The scatter which IVa removes inside a word is not special to characters. Myers' algorithm
   fragments the diff wherever old and new coincide by chance rather than by meaning, and that
   recurs one level up: coincidental common words scatter phrase just as coincidental common 
   characters scattered word. Compare:
       characters (IVa):  unique -> single           {u}[si]n{iqu}[gl]e          ("n" by chance)
       words      (IVb):  buy a car -> sell a truck  {buy}[sell] a {car}[truck]  ("a" by chance)

   Both split one conceptual replacement around a meaningless survivor. IVa collapsed the
   character survivor; IVb collapses the word survivor - same idea, the next level up.

   Now IVa and IVb aren't symmetric because word-level survivors are harder to classify than
   char-level crumbs - a single absolute threshold 3 suffices for characters, but words need
   "small relative to the surrounding edits" and "not too large in absolute terms".

   The good news is that we can stop at this level. The probability of coincidentally shared word
   is still decent, but probability of such paragraph is nil - that's our 50 character cap.

    * Terminology (the same objects as IVa, one level up):
       "Continuous run" - one edit as a whole: at most one deleted block {...} then one inserted
                          block [...] (ie "{...}", "[...]", or "{...}[...]"), exactly as in IVa.

                          Its length is max(deleted chars, inserted chars) - how much it replaced.
                          E.g. {understanding}[memorizing] is one run, length max(13,10)=13.
                          We use max, not sum: the two sides are two versions of the same replaced
                          chunk.

       "Survivor" - an unchanged stretch (whole words + spaces) between two continuous runs;
                    the word-level version of IVa's interior survivor.

    * Now the heuristic:
       Collapse a survivor into one combined {old}[new] run when it is both

         1) short RELATIVELY:  (survivor <= left_run) && (survivor <= right_run)
              eg1: {eat}[cook] a {pie}[cake] -> {eat a pie}[cook a cake]
                                     (survivor " a "=3 <= runs 4,4 -> short -> collapse)
              eg2: {buy}[rent] a house near {work}[town]
                                     (survivor " a house near "=14 > runs 4,4 -> not short -> keep)

         2) short ABSOLUTELY: survivor <= 50 chars (the only knob of IVb - max collapsed survivor size)
              eg: an identical clause "the meeting is confirmed for next Tuesday afternoon"
                  (51 chars between two edits exceeds 50 -> keep)

       Otherwise keep the survivor shown as unchanged.
       
       Applied repeatedly until nothing more collapses (a merge grows the run and may then swallow
       the next survivor).

       Insets and paragraph breaks are hard anchors: never collapsed or crossed (so a footnote or
       graphic is never duplicated, and every rewrite stays within one paragraph).

    * Consequences
       1. A short accidental survivor between two larger rewrites is collapsed into one clean
          run - the relative test.
       2. Meaningful common text between small edits is kept: the relative test self-scales,
          so " a house near " survives when only the words around it changed.
       3. A long identical span is always kept, even when flanked by still-larger rewrites:
          the absolute cap wins, because a long span cannot be an accidental coincidence.
       4. The trigger is the survivor's length against its neighbours and the cap, never the
          amount of surrounding change.

-- 
lyx-devel mailing list
[email protected]
https://lists.lyx.org/mailman/listinfo/lyx-devel