Re: Add option for forcing a 64bit hash on 32bit systems

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.

--8323329-1543701896-1609103477=:29310
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8BIT

On Sat, 26 Dec 2020, Érico Nogueira via The New Aspell Develpment Mailing List wrote:

> On Sat Dec 26, 2020 at 5:26 PM -03, Kevin Atkinson wrote:
>> On Tue, 22 Dec 2020, Érico Nogueira wrote:
>>
>> Aspell compiled dictionary formats are not really meant to be portable.
>> Compiling a dictionary is now very fast, it is even faster if checks are
>> disabled. One thing I will be open to is the creation of a portable text
>> based format which can be compiled very quickly on startup. This will
>> take some refactoring though to make it work.
>
> Allowing compilation at run time would allow for simpler packaging,
> especially when cross compiling. So this would be very nice :)

The basic idea behind the format is just to store the sorted word list 
(optionally after passing though "aspell clean", and maybe with soundslikes) 
in a file with a special header.  The file will then be compressed using 
prezip (a special compression for sorted wordlists) and then gzip.

When Aspell is given a portable format it will first check if a compiled 
dictionary already exists in either $XDG_CACHE_HOME/aspell/ or 
/var/cache/aspell/ (or maybe /usr/lib/aspell) and if so use that, otherwise it 
will attempt to compile it to $XDG_CACHE_HOME/aspell/ and if that fails try 
compiling it to /tmp/aspell/.

I am not sure when I will get around to doing this.  Patches are welcome. :)

>>> The only compatibility option offered is forcing 32bit hashes for all
>>> systems, which makes 64bit systems incapable of reading 64bit dictionaries.
>>
>> To be clear the only thing this does is change the type the hash function
>> used from size_t to u32int in modules/speller/default/readonly_ws.cpp:
<< ...
>> All integers used in the dictionary are 32 bit as 64 bit integers will be an
>> overkill. The fact that a 64 bit hash function is used on a 32 bit integer is
>> an oversight. I would rather that a 32 bit hash function is used on all
>> systems. The only reason that option exists is to avoid breaking dictionary
>> compatibility on 64bit systems. I am open to enabling 32 bit hashes by
>> default on the next major version bump.
>>
>> Note that on Debian Aspell is compiled with --enable-32-bit-hash-fun.

I should also note that Debian compiles the dictionary as part of the 
post-install process.  This saves a lot of space (often an order of 
magnetite) and avoids having to package architecture specific dictionary 
packages.

> I was looking into simplifying how we build the dictionaries in Void
> Linux.  Since our (32-bit) ARM packages are cross compiled from 64-bit
> hosts, including aspell dictionaries, I thought standardizing on a
> 64-bit format would be best. It might make sense, then, to clear up the
> explanation in [1], since it isn't clear that 32-bit hashes are actually
> preferred. Given your explanation, I will simply force the 32-bit hashes
> for all platforms, which is definitely simpler.

How about if a add something like this:

   A 32-bit hash function is preferred as the hash table uses 32-bit integers on
   all platforms.   Future versions of Aspell may default to forcing a 32-bit
   hash function.

Kevin
--8323329-1543701896-1609103477=:29310--