Re: Temporary vector file during merging

Without this temp file we would need to load the entire set of vectors
for the new merged segment into RAM in order to support building an
HNSW graph from it. This way we can read the vectors off the disk in
the same way we would do during normal searches.  I'm not sure, but I
think the temp file simply gets renamed into the new segment and
doesn't have to be physically copied a second time.  It would be good
to confirm that.

On Thu, Jun 26, 2025 at 4:52=E2=80=AFPM Viliam =C4=8Eurina <viliam.durina@g=
mail.com> wrote:
>
> Hi all,
>
> I noticed that during merging in an index that contains vector fields, th=
e
> new segment contains a temporary file with ".vec_temp_N.tmp" extension,
> which contains all the vectors being merged. This file is used to search
> for neighbors for the new HNSW graph. It is later deleted, and the segmen=
t
> will contain a ".vec" file with the same vectors. So vectors are copied t=
wo
> times and more space is temporarily needed on disk.
>
> In my index, the ".vec" file is 98% of the index size and the index is ma=
ny
> GB. Is it really necessary to have the temp file? Couldn't Lucene query t=
he
> "vec" file directly? I checked the code around it, one temp file is creat=
ed
> per field and the temp file is probably deleted before starting the next
> field, but still, there is another copy of the vector, so the temp file
> seems unnecessary.
>
> Is there some specific need for the temp file? I might try to do a PR
> removing the need for it.
>
> Viliam