Re: Temporary vector file during merging

--000000000000a0c9560638908694
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Mike, I believe that the answer to your question is in this PR review
comment: https://github.com/apache/lucene/pull/601#discussion_r783711025.

Merging is currently implemented by looping over fields once, and merging
them. Writing the vec file first would require merging flat vectors for all
fields first, and then doing a second pass over all fields to create their
HNSW graph. This sounds doable, but we never got to it.



On Fri, Jun 27, 2025 at 2:19=E2=80=AFPM Michael Sokolov <[email protected]=
> wrote:

> Without this temp file we would need to load the entire set of vectors
> for the new merged segment into RAM in order to support building an
> HNSW graph from it. This way we can read the vectors off the disk in
> the same way we would do during normal searches.  I'm not sure, but I
> think the temp file simply gets renamed into the new segment and
> doesn't have to be physically copied a second time.  It would be good
> to confirm that.
>
> On Thu, Jun 26, 2025 at 4:52=E2=80=AFPM Viliam =C4=8Eurina <viliam.durina=
@gmail.com>
> wrote:
> >
> > Hi all,
> >
> > I noticed that during merging in an index that contains vector fields,
> the
> > new segment contains a temporary file with ".vec_temp_N.tmp" extension,
> > which contains all the vectors being merged. This file is used to searc=
h
> > for neighbors for the new HNSW graph. It is later deleted, and the
> segment
> > will contain a ".vec" file with the same vectors. So vectors are copied
> two
> > times and more space is temporarily needed on disk.
> >
> > In my index, the ".vec" file is 98% of the index size and the index is
> many
> > GB. Is it really necessary to have the temp file? Couldn't Lucene query
> the
> > "vec" file directly? I checked the code around it, one temp file is
> created
> > per field and the temp file is probably deleted before starting the nex=
t
> > field, but still, there is another copy of the vector, so the temp file
> > seems unnecessary.
> >
> > Is there some specific need for the temp file? I might try to do a PR
> > removing the need for it.
> >
> > Viliam
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
>

--=20
Adrien

--000000000000a0c9560638908694--