Re: metadatabase

On Sun, 2003-10-12 at 11:10, David Hough wrote:
> On Sun, 12 Oct 2003 11:04:55 +0200, Ralf Engels <[email protected]> wrote:
> 
> > Hi,
> > I am just trying to clean up playlist and playlistmanager.
> > Some things are still missing but I have some questions to the meta 
> > handling:
> >
> > 1. We have a playlist format (PLM) that stores meta informations. Why do 
> > we need a special database for this.
> >
> > 2. We already have a metadatabase (mdb) why do we need another (database 
> > musiccatalog) ?
> >
> > Can we please agree on the future stategy?
> >
> > I know that Kristian is in favour of his database. In principle I think 
> > that using a database is a good idea. Just not in this case.
> >
> > I would prefere using PLM and MDB for the future. The principle is sound 
> > and the implementation (expecially mdb) is quite good.
> >
> > BR,
> > Ralf
> 
> Hi,
> 
> I've been using the new database structure mdb for a while now and I agree 
> that its seems well thought out and should be what zinf uses for accessing 
> metadata. However, I have come across a number of issues with it that I 
> think should be resolved.
> 
> 
> First off, the new database seems to take a random view of when to ignore 
> case and when not too. For instance, My Music now lists both 'doves' and 
> 'Doves' as artists, yet they both have exactly the same tracks. In fact 
> they both have an album called 'lost souls' and an album called 'Lost 
> Souls' which both also have exactly the same tracks in.

This is a Bug.


> It appears that when importing file metadata it takes case into account 
> when deciding if to add a new artist/album, yet when searching for tracks 
> in these artists it dosn't take case into account. I would of thought that 
> it should ignore case at all times.
> 
> 
> Next, reading metadata about a particular track seems to have got rather 
> slow for a lot of my tracks. From the output from zinf it appears that as 
> a lot of my tracks don't have full tags (most of them are missing the year 
> tag for instance), metadata is being read from the database, then when not 
> all tags have been found its reading the metadata in the file again, only 
> to find once again that the tag is missing as the file hasn't actually 
> changed.
>
> I thought one of the reasons that we had the metadata stored in a database 
> was so that we didn't have to keep spending time reading the metadata from 
> the file whenever we needed it. The idea of looking for the extra metadata 
> in another place when you can't find it in one is a good idea, but the 
> current implementation seems extreamly inefficient.
> 
> Perhaps the database should store a file modified time as well as the 
> other metadata, and only bother reading the metadata from the file again 
> if this has changed. This, or some other idea, really needs to implemented 
> as otherwise the browser can seem quite unresponsive while waiting for 
> metadata.

This is exactly the model I had planned... Upper levels are supposed
to cache metadata from lower level and read them only when there
are missing items that are available in the lower levels. I haven't seen
a really good way know that all levels are consistent.

For example, currently you can add the tags that interest you
in readMetadata.. The system will try to find the value for any empty
tag at a level and stops when it has satisfied the complete request.   
However, I need some marker to say that I have already looked 
and no value is available.. this could be some fixed string or
a cache update time, etc.. I've got some uncommitted code on 
a branch that deals with this.  I'll clean it up and commit soon.


> Come to think of it, when the MusicBrainzDB is finally implemented, it is 
> also likely going to take time when requesting metadata from the servers. 
> So perhaps we need some common way of dealing with slow metadata sources.

I have been thinking we should only do MusicBrainz searches on request
i.e. during Music search, but this should really be a user option.


> Maybe the readMetadata call should return immediatly with the information 
>  from the database only, and start off threads for the other metadata 
> sources to read metadata from. Then when all the metadata sources return, 
> a event can be triggered that causes the browser/player to re-read the 
> metadata and get the now cached metadata from all sources.

The old metadata system would spawn a thread, just as you say.  I am 
not sure that the new browser actually needs this as we only request
the metadata when actually displayed (this can slow down scrolling
on slower machines).   However, the fixes mentioned above which
should stop slower/costly search at the end of the metadata hierarchy
should alleviate this.

> Exactly how such a system is implemented obviously needs quite a bit more 
> thought but it seems a resonable idea to me in principle.
> 
> 
> Finally, it would make sense if the readEntries function could not only 
> return the files and dirs in a directory, but could be made to read all 
> tracks recursively in all the sub-dirs as well. This is really just a 
> feature request that I will probably end up implementing my self 
> eventually, but I just thought I'd mention in case anyone else wants to 
> have a go. This would really be useful in enabling the browser to add 
> entire dir trees to the playlist.

This was already somewhat in the plan for the new playlist system.
I would rather the readEntries still returns only one level,
but that each time the player reads a non-song item, it would
try to re-expand (i.e. recursively expand).

That is  I would like to see playlist as set of URLs.. Each URL is a
either a playable file, a set of other URLs or a combination of the two.

The player would ask the MDB system to expand a playlist.  The
playlist could contain urls of the form
http://music.com/stream.mp3  (a stream)
or
file://home/music/yes       (a directory of yes albums and songs)
or even
zinf://tracks?genre=blues   (set of available blues tracks)

These would be dynamically expanded by the main player.   The
question I have is how to have a consistent view of the songs
being played and what is displayed in the browser and how
to keep the two synchronized.

Choices are:

1. The expanded list of tracks (current model)
2. A contracted view of high level URLs.
3. A tree view..

I started a rewrite of the playlist system a while back, 
but it became unwieldly  and I have yet to clean it up..

The cool thing would be that you could make really
dynamic playlists that change automatically as your
music collections grows and changes..



> Anyway, all in all I think the new mdb should be the standard way that 
> metadata is cached/read in zinf from now on.

Yes,  I hope new code will use the new system even though it 
is still in  a "work in progress".   The old code needs to
stay in as some people might want to use it and I still
don't want to give up on win32 which uses it.

However,  I hope one day someone will write a WxWindows version using
your gtkmusicbrowsermm as a model or we try try to see if the win32
port of gtkmm actually works.  If it does, the old code will go away
rather quickly.


-- 
Kristian G. Kvilekval
email:[email protected] office:(805)893-4276 http://www.cs.ucsb.edu/~kris



-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php