Performance Difference between files getting opened with IoContext.RANDOM vs IoContext.READ
Navneet Verma <[email protected]> Sun, 29 Sep 2024 01:04:28 -0700
| Newsgroups | gmane.comp.jakarta.lucene.user |
|---|---|
| Message-ID | <CAN1Xc8-C3Pd6AbC_Ly2=2y_4pqEW8ijFQe4CxzUpt+YPKdBDdQ@mail.gmail.com> |
Hi Lucene Experts, I wanted to understand the performance difference between opening and reading the whole file using an IndexInput with IoContext as RANDOM vs READ. I can see .vec files(storing the flat vectors) are opened with RANDOM and whereas dvd files are opened as READ. As per my testing with files close to size 5GB storing (~1.6M docs with each doc 3072 bytes), I can see that when full file checksum validation is happening for a file opened via READ context it is faster than RANDOM. The amount of time difference I am seeing is close to 50%. Hence the performance question is coming up, I wanted to understand is this understanding correct? Thanks Navneet