Re: Fwd: Musicbrainz database deduplication

Hey Wieland,

at the moment I can't share code or details about the framework but my
plans are to release the framework under an open source license in a few
months. This will happen after I finished my master thesis, which is
about this framework.

A huge problem for me is to understand the schema of the musicbrainz db.
It is hard for me to distinguish duplicates and no-duplicates because I
have no experience with the complex schema. At the moment I use the data
exported by the query mentioned in my first mail.

For now the framework is still under construction, so there is no result
at the moment worth sharing. But I would like to share the results, when
you are interested.

What I need is a sensible db query to create a csv file from, which will
be my input data for the deduplication process. Important is, that any
row should be a unique entity and only duplicates are similar. If you
can provide such a query I can process the data and publish a csv file
with duplicate pair ids or something like that.

Regards,
Niklas

On 02.02.2015 14:10, Wieland Hoffmann wrote:
> Hallo Niklas Wilcke:
>> I'm a computer scientist from Germany. At the moment I am developing a
>> big data deduplication framework for Apache Spark. For test purposes I
>> dumped [0] the musicbrainz database to a csv file.
> 
> This sounds interesting. Did you run the framework against the data as
> well? If so, can you share the results openly (or even more details
> about the framework itself)?
>