Use window buffer history to infer buffer ownership: how reliable?

Hello,

I'm exploring a way to define buffer "ownership" by windows: a buffer
is considered owned by a window if it has appeared in that window at least once.
A buffer may be owned by multiple windows simultaneously. This is how
most modern editors like GNOME Builder and VS Codium behave.

One simple approach I'm currently following is to inspect the window's
current history together with
the lists returned by `window-prev-buffers' and `window-next-buffers'. I
have some concerns, though:

1. How stable are these lists as a source of truth? They can be
   freely manipulated via `set-window-prev-buffers' and
   `set-window-next-buffers', which makes them feel somewhat fragile.
   I also noticed that `window-prev-buffers' retains killed buffers
   is there a point at which Emacs cleans those up automatically, or
   does that accumulate indefinitely?

2. Scalability: if I want to find all owners of a given buffer, I
   would need to walk every live window's history lists. Is this
   likely to be a performance concern in practice, or is the number
   of windows small enough that it doesn't matter?
   
Any suggestion is greatly appreciated. Thank you in advance.