Sorting many small data sets

I have a database merge problem. It is the player databases from the
play by email Diplomacy hobby, but I think some pretty code might be
generically useful... Unless the project parameters make it a
non-starter.

Each record is only a about a dozen fields (~250 bytes each). There are
about 15,000 records distributed over a dozen or so flat file
databases, with much duplication between databases and a little
duplication within. If it's helpful, there are 15,000+ other docs (game
summaries) that contain an average of 7 records each. These might be
useful in that they can be accessed chronologically and are keyed to
the main databases. While the main databases only show the last update,
the summaries will show transitional states. This would be somewhat
analogous of accessing invoices to help determine if similar entries in
a database are separate customers or a duplicate account.

What I have so far is a parser that can read the flat file database and
sort the values into MySQL. It's a short hop from there to being able
to pull registration data off summaries. 

I could cross reference the key sets that are almost guarantees of
linkage - email address, database and local keys composite key, name and
birth date composite key - in order to clump the data sets and create a
corpus for each likely individual user. From there, I could create
classification files for each clump after comparing the data in each
corpus to the classifications already created, but I'm pretty sure that
involves doing more than strictly necessary.

I'm thinking that it might be better to do some combination of
automated and interactive training. If I maintain an even level of
training across the classifiers by automating the training of the
records that have already been classified then I could make the user
choices for  "Accept this decision?" be "'Yes', 'No', and 'Always accept
for this Pr or better.'" Intuitively, hyperspace seems to be the most
likely candidate because of the speed of learning and the small disk
foot print. Each corpus will be very small, the documents small and
fairly similar to begin with, and there will be 15,000+ classification
files floating around at the end of it, which would be nice to keep for
reducing future duplications. Is there a reasonable chance of this
approach working given the obvious weaknesses in the quality of the
data? 

Chris


------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you.  Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/