RE: Re: QRP question (new Gnutella developer)

> De : [email protected] [mailto:[email protected]] De la part
> de [email protected]
> Envoyé : mardi 4 septembre 2007 12:26
> À : [email protected]
> Objet : [the_gdf] Re: QRP question (new Gnutella developer)
> 
> Quoting mikeyv_b <[email protected]> from ml.gnutella.dev-forum:
> :So, here are my questions - I hope someone can help!
> :1. Do the message structures described in the second document match
> :the structures used in actual current QRP?
> 
> Yes, the QRP messages have not changed.
> 
> :2. I used these structures, and am having a problem. My client is not
> :reliably receiving searches from Phex, which I'm using as a host to
> :connect it to for testing.
> 
> Have you tried with gtk-gnutella as well?  You may have issues with
> internationalization and UTF-8 representations.  Gtk-gnutella has the
> correct algorithm implemented to compute the QRP hash value, especially
> in the presence of surrogates.  (this is not documented in the original
> LimeWire specs -- look at the qrp_hashcode() function in src/core/qrp.c
> for a correct C implementation).

For the case of surrogates, I had discussed this issue years ago, and had
described the problem in this list after discussing with a past Japanese
developer working at LimeWire (can't remember his name sorry) who worked
with me on making it international.

I had initially suggested a better way to handle non ISO-8859-1 characters
(because the default QRP hashing function discards all but the lowest 8 bits
of the UTF-16 code units.

But what should be known is that LimeWire internally handles all strings
using UTF-16 (not UTF-8) even if UTF-8 is used in encoded strings
transmitted on the network. This is important to understand because this
makes a big difference even with ISO-8859-1 only in the effective result of
the QRP hash function.

For characters out of the BMP (most often in the ideographic supplementary
plane), they are computed as if they were two surrogates, because of the
UTF-16 internal representation. This means that UTF-8 strings need to be
converted to UTF-16 for correct computing of the QRP hash function after
applying the NFKD substitution and removal of most diacritics and
simplification of spaces and punctuation.

I had wanted to change the way UTF-16 strings are used, so that it would not
ignore the 8 most significant bits of UTF-16 units, but would still preserve
the value of the computed QRP function for ISO-8859-1 only characters.
However this was not changed, and this causes bad indexing behaviour for
Japanese and Korean. The only change needed was to apply an OR to the low
and high byte of each UTF-16 code unit instead of just ignoring the high
byte, or some similar transform that preserves the computed value if the
high byte is null, but this would have needed to change the QRP table
content anyway (such change was applied in some third-party illegal clone of
LimeWire for Japanese, Korean and Chinese users, but not sourced back to
LimeWire because this clone was using LimeWire code without respecting the
GPL licence).

A revamped QRP hash function should be rediscussed now. It is possible to
preserve most of the compatibility with exisint QPR hash function so that it
will be mostly interoperable with legacy implementations for files shared
with Latin-written names.

The existing function works well for Latin, and other small alphabets that
can fit in a single 256-positions block (Arabic, Cyrillic, Greek), but it
fails miserably with large East-Asian scripts that span several 256-position
blocks (especially in Korean and Japanese), with too many collisions (and so
with poor QRP filtering). Preserving the interoperability for Arabic,
Hebrew, Cyrillic and Greek users would require handling some values for the
high bytes as if it was null. I suppose this can be done safely for all
values of high bytes (in UTF-16 code units) lower than 0x20.

My change was then:
* Read an UTF-16 code units into W;
* split W into high byte H and low byte L;
* if H < 0x20, set H := 0;
* use byte (L*(H*2+1)) instead of just L in the existing QRP function.