Re: Increasing precision for sparse vector search

Forgot to close the loop on this one and was traveling last week.

I've gone back to the science folks with the suggestion that we experiment
with 16 bits. I suspect that you're right and 16 bits will be good enough.

It is good to know that we can play with it a bit, provided that we keep
under the field length overflow. For example, if we know that there are
fewer than 127 non-zero values for a given doc, I think we could expand to
24 bits. (I'm probably off-by-one in the count of values or bits.)
Hopefully it won't be necessary, though.

Thanks Adrien!

On Fri, May 16, 2025 at 1:51 PM Adrien Grand <[email protected]> wrote:

> Your understanding is correct. For reference, using only 16 bits was
> convenient to avoid overflowing the field length, but storage efficiency
> was a motivation as well. Using 32 bits per frequency instead of 16 would
> significantly increase the size of the inverted index.
>
> We could look into what it would take to support greater precision, on the
> other hand it would be interesting to check if more precision is actually
> needed. Historically, some people have raised concerns about e.g. how
> Lucene encodes length normalization factors on a single byte, but no
> benchmark that I'm aware of ever concluded that it hurts in practice. My
> understanding is also that weights are commonly stored as bfloat16, which
> have less precision (one less bit of mantissa) than FeatureField.
>
>
> On Fri, May 16, 2025 at 9:30 PM Michael Froh <[email protected]> wrote:
>
>> Hi folks,
>>
>> I was recently (yesterday) pulled into a conversation with a data science
>> colleague about sparse vector search, which led me to learn how
>> FeatureFields work.
>>
>> While it sounds like they should work to get things started, the science
>> guy was concerned about precision on their weights. Apparently their
>> experiments have been using 32-bit floats (with a 23 bit mantissa), whereas
>> FeatureField is stuck with an 8-bit mantissa. In theory, we could modify
>> FeatureField to use IEEE-754 half-precision floats (versus the current
>> truncated single-precision floats) giving us 10 bits of mantissa.
>>
>> I found a thread (
>> https://lists.apache.org/thread/dcwm71rnjz7mz69ntwy04co3615zdl6w) from
>> Ankur Goel from 2020 on a similar subject. Not really sure where he ended
>> up going with it, though. (Though it sounds like his use-case might have
>> been different.)
>>
>> I'm guessing the 16-bit limitation on FeatureField values is because the
>> custom term frequencies get added to the invertState.length at
>> https://github.com/apache/lucene/blob/b0ebaaed56505048d2c0bdfb7d71a3a082d23822/lucene/core/src/java/org/apache/lucene/index/IndexingChain.java#L1284,
>> and we're trying to avoid overflow?
>>
>> Are there any other good options for sparse vector storage in Lucene? I'd
>> love something kind of like FeatureField, but where the values are legit
>> "payload" values stored in postings without a couple of extra hops of
>> indirection, but also without pretending that they're term frequencies.
>> Does anyone else have a use for something like that?
>>
>> Thanks!
>> Froh
>>
>
>
> --
> Adrien
>