music catalog overhaul (long)

Over the last year, I have seen several requests for 
both more and less browser functionality.   In the
less cateogory, several users have requested the ability
to turn off the id3/ogg tags and navigate solely
by dir hierarchy.  Others have suggested that the directories
themselves represent the metadata (think artist/album/tracks)
and should be used to gather information about the tracks.
I use tag metadata and would like poorly tagged data to be fixed
from either the tag editor, automatic inference (based on
path/filename), and external sources (musicbrainz).

Furthermore, I believe that the musicbrowser is tied too tightly with
both the core player and the music catalog.  In particular, the use of
Playlist, PlaylistItem as a means of caching metadata have bothered
me.

Over the last few weeks, I have been revamping the whole musiccatalog
to permit a bunch of new functionality and I am hoping to simplify the
musicbrowser also.  Just to make things complicated, the musiccatalog
should also support on-demand loading of metadata instead of the
current practice of reading the database in on startup (not very
scalable).   The basic idea is that the catalog will be used to collect
metadata and offer metadata to the browser and player through a
simpler interface.  (Alot of this work began with the abandoned 
lazydb branch).

The system is keyed around two simple concepts:
a track and a playlist.   A track is simply some
URL and a playlist is list of URLs.   The music
catalog is responsible for converting URLs to 
metadata and converting URL to playlists.   

Given a track URL string, the music catalog will return a metadata
map, i.e. map<string,string>.  It can also be used to read playlists
i.e. returning a vector<string> where the contents are themselves
other URLs.  A simple extension would be to permit directory URLs i.e.
file://home/music/artist/ and the catalog would return a list of URL
for the directory.

Things get more interesting when we permit zinf psuedo-urls for
metadata.  For example zinf://tracks?artist=aphex_twin should return a
playlist consisting of all tracks with the tag "artist" set to
"aphex_twin".  Similarly, zinf://albums?artist=aphex_twin would return
a list of psuedo url's of the form
  zinf://album?artist=aphex_twin;title=...  
The expansion of a zinf url end with the a set of tracks/streams URLs.
Notice this allows arbitrary metadata to be queried including
zinf://tracks?genre=classical;performer=yoyoma
In the next section, I'll handling address non-standard tags
such as performer..


The music browser would need to communicate to the catalog using these
only URL and expected only a list of URLs back.  On-demand loading
would be quite simply as the browser would respond to expand/collapse
event by requesting simple URLs for each expanded node in the tree.
Each tree node would simple store some sort of url.  Supporting
directory browsing would be trivial.  


Simply stated the music catalog's main interaction functions
would be 
   readMetadata(const string& url, map<string,string>& meta);
   and 
   readEntries (const string&url, vector<string>& playlist);

The above interface is just to give the idea, the actual datastructures
are in classes with common accessor functions available to make things
easier.
Of course other catalog functionality would be maintained, including
writes, search, etc..

Event notification from the catalog should be disabled or curtailed,
the browser should always just reread only the presently displayed
items from the music catalog whenever appropiate
(i.e. INFO_CatalogChanged).


A good portion of the above has been completed and is running
on my system.   Currently I am working on the ....


MusicCatalog:

The music catalog needs to support diverse forms of metadata
from the none whatsoever to remote metadata services.
The redesign calls for a set of layered metadata sources.

Each metadata source is a MetadataDB.  The music catalog is
responsible for translating standard and zinf URLs to database calls
through a set of MetadataDBs.

These MetadataDBs can be local (gdbm, mk4), inferred (file
system structered), or remote (musicbrainz).  The can also be linked
.i.e.  a filesystem structure is the weakest metadata, with cached-tag
metadata overriding and finally linked to the musicbrainzDB.

It sort of looks like this:

musiccatalog->  
-> Filesystem   -+
-> Filesystem   -+
-> StreamServer -+
                 +-local metadata  ->  remotemeta database
-> CD database  -> local CD database -> remote CD database

   Priority is set in each database, request are found left->right.

Each database may (dis)allow writes to be recorded.  This would permit
zinf to contribute metadata to the musicbrainz library if the user
wishes or allow the user to disable metadata completely.

Each database exports the supported tags, in an open ended form.  For
example, the musicbrainz database needs to store song fingerprints in
order to perform lookups, while ogg files may store arbitrary tags.

Every database supports some standard queries, tracks, artists,
albums, genres, etc.. The most general form of query asks for a target
(i.e tracks) and passes in a list of constraints as map of (tag,value)
pairs. The MetadataDB can translate this to it's internal style and
perform the query. Unkown tags are simply ignored.

Track Metadata queries are similar, given a track url and a list
of tags the database will try to satisfy the tags.   

   getMetadata(const string&url, map<string,string>& meta) 
   setMetadata(...)


Each level fills in the values it knows about and pass the request
to its parent level.  It's a limited more complicated with 
read/write priority setting, but not much.

Currently I see the need for 

FileMetadataDB   -- use files for storage of metadata
GDBMMetadataDB   -- read/store metadata in old style db.
Mk4MetadataDB    -- use arbitray tags for new style DB
StructureDB      -- parse pathnames as metadata (artist/album/track).
MBMetadataDB     -- Musicbrainz metadata catalog.
MemoryDB         -- in memory db

The current functionaility achieved with (coded as db:priority)

  GDBM-DB:0 -> File-DB:1  
    check gdbm but override wtih file data.
    It might be seen as initializing MemoryDB with GDBM-DB ..
  
A non-tag database might be:
  Mk4M-DB:0  -> Structure-DB:1 
     saved inferred data in the a local database, notice this
     will never read/write tag data.

A top end configuration might be:

  Mk4-DB:1 -> FileDB:2 -> StructureDB:0 -> MB-DB:4

Use db metadata, then tag, then structure, and finally musicbrainz.
Notice that Structure data never overrides any field already found.


Conclusion:
--------------------------------

The music catalog needs a rewrite to increase functionality,
reduce component cross-talk, and simplify on-demand
browsing.   Some of this work has been completed and
I think I will be working on it a bit more often in
the future.

Comments?   


-- 
Kristian G. Kvilekval
email:[email protected] office:(805)893-4276 http://www.cs.ucsb.edu/~kris



-------------------------------------------------------
This SF.net email is sponsored by: VM Ware
With VMware you can run multiple operating systems on a single machine.
WITHOUT REBOOTING! Mix Linux / Windows / Novell virtual machines
at the same time. Free trial click here:http://www.vmware.com/wl/offer/358/0