Re: music catalog overhaul (long)

This sounds like a very good idea, the musiccatalog definetly needs to be 
made more flexible. I don't know how easy this would be, but would it be 
possible to document or at lay down specifics of how such an updated 
musiccatlog would interface with the rest of zinf (actual public class 
header files or something) so that I could write the new gtkmm musicbrowser 
to work with what the new musiccatlog will be. Obviously it would be better 
to write the new code to support the new interfaces, instead of limiting it 
to the old musiccatalog.

On Fri, 22 Aug 2003 14:34:40 -0700, Kristian Kvilekval <[email protected]> 
wrote:

>
>
>
> Over the last year, I have seen several requests for both more and less 
> browser functionality.   In the
> less cateogory, several users have requested the ability
> to turn off the id3/ogg tags and navigate solely
> by dir hierarchy.  Others have suggested that the directories
> themselves represent the metadata (think artist/album/tracks)
> and should be used to gather information about the tracks.
> I use tag metadata and would like poorly tagged data to be fixed
> from either the tag editor, automatic inference (based on
> path/filename), and external sources (musicbrainz).
>
> Furthermore, I believe that the musicbrowser is tied too tightly with
> both the core player and the music catalog.  In particular, the use of
> Playlist, PlaylistItem as a means of caching metadata have bothered
> me.
>
> Over the last few weeks, I have been revamping the whole musiccatalog
> to permit a bunch of new functionality and I am hoping to simplify the
> musicbrowser also.  Just to make things complicated, the musiccatalog
> should also support on-demand loading of metadata instead of the
> current practice of reading the database in on startup (not very
> scalable).   The basic idea is that the catalog will be used to collect
> metadata and offer metadata to the browser and player through a
> simpler interface.  (Alot of this work began with the abandoned lazydb 
> branch).
>
> The system is keyed around two simple concepts:
> a track and a playlist.   A track is simply some
> URL and a playlist is list of URLs.   The music
> catalog is responsible for converting URLs to metadata and converting URL 
> to playlists.
>
> Given a track URL string, the music catalog will return a metadata
> map, i.e. map<string,string>.  It can also be used to read playlists
> i.e. returning a vector<string> where the contents are themselves
> other URLs.  A simple extension would be to permit directory URLs i.e.
> file://home/music/artist/ and the catalog would return a list of URL
> for the directory.
>
> Things get more interesting when we permit zinf psuedo-urls for
> metadata.  For example zinf://tracks?artist=aphex_twin should return a
> playlist consisting of all tracks with the tag "artist" set to
> "aphex_twin".  Similarly, zinf://albums?artist=aphex_twin would return
> a list of psuedo url's of the form
> zinf://album?artist=aphex_twin;title=...  The expansion of a zinf url end 
> with the a set of tracks/streams URLs.
> Notice this allows arbitrary metadata to be queried including
> zinf://tracks?genre=classical;performer=yoyoma
> In the next section, I'll handling address non-standard tags
> such as performer..
>
>
> The music browser would need to communicate to the catalog using these
> only URL and expected only a list of URLs back.  On-demand loading
> would be quite simply as the browser would respond to expand/collapse
> event by requesting simple URLs for each expanded node in the tree.
> Each tree node would simple store some sort of url.  Supporting
> directory browsing would be trivial.
>
>
> Simply stated the music catalog's main interaction functions
> would be readMetadata(const string& url, map<string,string>& meta);
> and readEntries (const string&url, vector<string>& playlist);
>
> The above interface is just to give the idea, the actual datastructures
> are in classes with common accessor functions available to make things
> easier.
> Of course other catalog functionality would be maintained, including
> writes, search, etc..
>
> Event notification from the catalog should be disabled or curtailed,
> the browser should always just reread only the presently displayed
> items from the music catalog whenever appropiate
> (i.e. INFO_CatalogChanged).
>
>
> A good portion of the above has been completed and is running
> on my system.   Currently I am working on the ....
>
>
> MusicCatalog:
>
> The music catalog needs to support diverse forms of metadata
> from the none whatsoever to remote metadata services.
> The redesign calls for a set of layered metadata sources.
>
> Each metadata source is a MetadataDB.  The music catalog is
> responsible for translating standard and zinf URLs to database calls
> through a set of MetadataDBs.
>
> These MetadataDBs can be local (gdbm, mk4), inferred (file
> system structered), or remote (musicbrainz).  The can also be linked
> .i.e.  a filesystem structure is the weakest metadata, with cached-tag
> metadata overriding and finally linked to the musicbrainzDB.
>
> It sort of looks like this:
>
> musiccatalog->  -> Filesystem   -+
> -> Filesystem   -+
> -> StreamServer -+
> +-local metadata  ->  remotemeta database
> -> CD database  -> local CD database -> remote CD database
>
> Priority is set in each database, request are found left->right.
>
> Each database may (dis)allow writes to be recorded.  This would permit
> zinf to contribute metadata to the musicbrainz library if the user
> wishes or allow the user to disable metadata completely.
>
> Each database exports the supported tags, in an open ended form.  For
> example, the musicbrainz database needs to store song fingerprints in
> order to perform lookups, while ogg files may store arbitrary tags.
>
> Every database supports some standard queries, tracks, artists,
> albums, genres, etc.. The most general form of query asks for a target
> (i.e tracks) and passes in a list of constraints as map of (tag,value)
> pairs. The MetadataDB can translate this to it's internal style and
> perform the query. Unkown tags are simply ignored.
>
> Track Metadata queries are similar, given a track url and a list
> of tags the database will try to satisfy the tags.
>
> getMetadata(const string&url, map<string,string>& meta) setMetadata(...)
>
>
> Each level fills in the values it knows about and pass the request
> to its parent level.  It's a limited more complicated with read/write 
> priority setting, but not much.
>
> Currently I see the need for
>
> FileMetadataDB   -- use files for storage of metadata
> GDBMMetadataDB   -- read/store metadata in old style db.
> Mk4MetadataDB    -- use arbitray tags for new style DB
> StructureDB      -- parse pathnames as metadata (artist/album/track).
> MBMetadataDB     -- Musicbrainz metadata catalog.
> MemoryDB         -- in memory db
>
> The current functionaility achieved with (coded as db:priority)
>
> GDBM-DB:0 -> File-DB:1  check gdbm but override wtih file data.
> It might be seen as initializing MemoryDB with GDBM-DB ..
> A non-tag database might be:
> Mk4M-DB:0  -> Structure-DB:1 saved inferred data in the a local database, 
> notice this
> will never read/write tag data.
>
> A top end configuration might be:
>
> Mk4-DB:1 -> FileDB:2 -> StructureDB:0 -> MB-DB:4
>
> Use db metadata, then tag, then structure, and finally musicbrainz.
> Notice that Structure data never overrides any field already found.
>
>
> Conclusion:
> --------------------------------
>
> The music catalog needs a rewrite to increase functionality,
> reduce component cross-talk, and simplify on-demand
> browsing.   Some of this work has been completed and
> I think I will be working on it a bit more often in
> the future.
>
> Comments?
>
>



-- 
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/


-------------------------------------------------------
This SF.net email is sponsored by: VM Ware
With VMware you can run multiple operating systems on a single machine.
WITHOUT REBOOTING! Mix Linux / Windows / Novell virtual machines
at the same time. Free trial click here:http://www.vmware.com/wl/offer/358/0