RE: Searching - duplicates

Joerg, Yuri,

I think I've got this coming from the right address - if not I
apologize.

I'm Tom, BTW.  I'm a C++ Wireless Networking on Windows developer from
Toronto.  And if no one else has already claimed this task, I'd like to
write the C++ (Windows) client code for these new features.

The issue, which you both raised, of duplicates is very important in my
opinion.  Once the search functionality is built into the server it will
be used much more (it appears to me to be the most requested feature),
and duplicates can often make search results quite ugly.  For an extreme
example try 'Beuna Vista Social Club'.  For a more reasonable example
try 'Bob Seger/Greatest Hits'.  There are 6 editions of this album
listed in the results (ignoring the `various` entry), where there should
be only two (one is a 'bonus tracks' edition).  This problem affects the
query command as well.

I like your solution Joerg: linking.  I imagine it working something
like this:

For each set of matching albums, one could be randomly chosen as the
master (or choose the one with the most extended data?).  A field could
be added to each album that pointed to its master.  For the master it
would point to itself or, even better, contain a list of the albums
pointing to it.  If the entry is new and this field isn't set yet then
just assume it is its own master.

For new entries this field could be set by the same process that updates
the index.  For any two albums, if the # of tracks are equal, and the
track names and lengths are close, then they are the same album.  This
would be a time consuming computation because it could not use the
existing fuzzy query (track lengths can be +-10 sec in different
editions of the same album).

The text search index could be significantly shrunk because it would
only include 'master' albums, not 'edition' albums.

If, in the future, we provided the ability to add and view additional
fields thru the web, we would only show the 'master' albums.
Eventually, 'edition' albums could be stripped of all data except tracks
offsets.

The new field (the link) could be implemented as a list of
<genre>/<discID> pairs.  For non-masters, the list would only contain
one item.

The web search currently returns:
	 <genre>, <discID>, <artist>, <album title>
for each album.  I would suggest that it also return this new link field
so that you don't have to read each discID to figure out which albums to
hide from your user.  It would also be nice if it returned the # of
tracks to help in identifying 'bonus track' editions (but I guess that
can be derived from the discID).

Tom.

> -----Original Message-----
> From: [email protected] 
> [mailto:[email protected]] On Behalf Of Joerg Hevers
> Sent: August 25, 2002 11:36 AM
> To: [email protected]
> Subject: Re: [fdb-dev] Searching
> 
> 
> Hello,
> 
> Monday, August 26, 2002, 1:34:13 AM, Yuri wrote:
> 
> > The last week or two have been rather hectic, so I haven't gotten 
> > around to slogging through the indexing code for the current system 
> > (besides which, I find perl scrypt amazingly annoying to 
> decrypt :0 - 
> > the lack of any and all comments in the source doesn't help...) but 
> > from the html doc, the whole thing looks a little clunky 
> (no offense 
> > to anyone/anything, that's just how it looks)... if someone 
> can just 
> > tell me the index's table structure the I'd be quite grateful ;)
> 
> I guess the only person who can tell you is Gerhard Gonter, 
> the author. He is also on this list - let's hope he answers 
> to your mail.
> 
> > On the upside, I've put together a near-optimal solution to this 
> > particular problem (I think ;) )
> 
> The solution you described on the board looks great for me - 
> if you can put it to code and it works as intended ;)
> 
> > About the issues raised about the current search:
> >    Firstly, the ascii-issue (also the alternate spellings issue):
> >       This is trivial for things like accents and missing/extra 
> > apostrophies:
> > strip the character down to it's base letter for the 
> former, and discard the 
> > shorter string for the latter ( Michael's -> michael, 
> L'Industrie -> 
> > indtustrie ). If this is done on the requested keyword as 
> well as the index, 
> > there won't be a problem.
> 
> Sounds good. But what will we do about entries in UTF-8, once 
> that's implemented?
> 
> >    About the duplicate elimination:
> >       I had kind of assumed that this would be one of the main 
> > administrative
> > uses of the full text search, as it is a trivial task, 
> especially with a 
> > hashed search as I've described on the dev board. You don't 
> even need any 
> > misspelling correction: the hashed search as described is 
> quick enough to 
> > check the similarity of the whole file, and > 75% word 
> matches are almost 
> > certainly the same thing (assuming the user didn't misspell 
> more than 25% of 
> > the words that is) so we do 
> > for each record in the database:
> >    if there are matches with more than 50% relevance:
> >       print a list of matches in order of relevance
> >       let the user decide what to do
> > and thats practically a python program - perl users take note :)
> 
> The problem with duplicates is, that even though something is 
> a duplicate regarding the titles etc. it is most likely not 
> regarding the discid and track offsets. This may be because 
> of different pressings of a CD being available or because 
> someone submitted info for a CD which he burned from MP3s 
> himself. We cannot delete such duplicates, if we want the 
> original CD to be recognized as an exact match. That's the 
> problem. A good idea would be a possibility to link several 
> entries (but not with a hardlink in the filesystem, like the 
> current server software allows - the track offsets of each 
> entry should be preserved, but titles should be the same for 
> all "linked" entries. I'd imagine, that this could be solved 
> quite well if we move to a relational database. As for real 
> duplicates: we have a script that checks for them and we run 
> it from time to time. Currently we would be able to remove 
> about 3500 dupliactes from the database by running this 
> script - but that's just a fraction of the duplicates with 
> different discids and track offsets...
> 
> btw: you didn't send the email from the address you 
> registered with on this list - therefore I had to approve it. 
> Approving every post to a mailinglist manually is quite 
> annoying, so I'd like to ask everyone to make sure that he 
> uses the right sender address.
> 
> Joerg
> 
> _______________________________________________
> fdb-dev mailing list
> [email protected]
> http://dtype.org/mailman/listinfo/fdb-dev
>