Re: metadatabase

David Hough wrote:
> On Tue, 14 Oct 2003 17:48:41 -0400, Ed Sweetman <[email protected]> 
> wrote:
> 
>> Kristian G. Kvilekval wrote:
>>
>>> On Tue, 2003-10-14 at 11:34, David Hough wrote:
>>>
>>>> On Mon, 13 Oct 2003 01:24:32 -0400, Ed Sweetman 
>>>> <[email protected]> wrote:
> 
> ...
> 
>>>
>>>>> So yea, lets see some definitions and specifications on what a 
>>>>> playlist should be, why does it have to be different from a 
>>>>> metadata manager and  under what method should the UI be hooked 
>>>>> into the whole thing and anything else relating to it.
>>>>>
>>>>> to me :
>>>>>
>>>>> Playlist: human readable text list of streams playable to the the 
>>>>> player
>>>>> Playlist Manager: container that creates a list in the player 
>>>>> consisting of translated playlist entries such that a UI can 
>>>>> correctly display the entries.
>>>>> metadata manager: container that retrieves, accesses, removes, and 
>>>>> possibly writes metadata related to lists of streams given to it.
>>>>>
>>>>>
>>>>> Playlist would be an object or array of objects private to Playlist 
>>>>> Manager.  Metadata manager would be inherited by Playlist Manager. 
>>>>> Such that say to get the current track to be played you'd have a 
>>>>> call to the Playlist manager like 
>>>>> playlistManager->getStream("file:///home/mysong.ogg"); Retrieving 
>>>>> metadata if available is taken care of implicitly by getStream by 
>>>>> simply calling 
>>>>> this->metaManager->getStream("file:///home/mysong.ogg"); which only 
>>>>> accesses the database.  A separate update method can be used in 
>>>>> metadata Manager to update entries in the database as opposed to 
>>>>> researching directories for files and such and adding them to the 
>>>>> database.  These should be done explicitly by the user and not used 
>>>>> when simply reading the db for displaying a playlist.   etc etc.  
>>>>> Some feedback would be cool.
>>>>
>>>>
>>>> Personally, I don't reckon the playlist manager should be concerned 
>>>> with metadata at all. The playlist manager should be concerned with 
>>>> providing a way to move about in a playlist (next, prev, set index, 
>>>> etc...) and providing the player with a url for the current track.  
>>>> The playlist manager should also provide an interface for 
>>>> manipulating a playlist (add, remove, move, etc...) that will be 
>>>> accessed by the playist editing UI.
>>>>
>>>> There is no reason why the playlist manager should ever need to know 
>>>> metadata about any url as far as I can see, so it dosn't make sense 
>>>> (to me at least) that metadata should be read through the playlist 
>>>> manager. As far as I know the player itself never needs to know any 
>>>> metadata about a track, it just plays it. The only place metadata 
>>>> becomes relevant is in the ui plugins, so why mix the metadata in 
>>>> with basic player functionallity (i.e. playling a playlist)?
>>>
>>>
>>>
>>> I certainly agree.  I prefer the playlist to manipulate a list
>>> of URLs and nothing else.   Both the player and UI can retrieve 
>>> needed metadata on demand.  However, There are a few utility functions
>>> that don't seem fit elsewhere (i.e. total playlist time).
>>> I certainly don't want the playlist to cache metadata.
>>
>>
>> let me clarify:
>>
>> a playlist would be an object of streams, given in std::string URI 
>> format for instance.  stored in a plain text file.
>>
>> Metadata is an object containing fields pertaining to a given stream. 
>> stored in a database type file.
>>
>> a playlist manager is more than just a playlist, it basically acts as 
>> a backend to the playlist UI.  This means it controls both the 
>> metadata manager (separate object i'll explain later) and the 
>> playlists and gives all sorts of features involving the two.  This 
>> means the UI's dont actually impliment any code except that dealing 
>> with the Interface.  The playlist manager inherits the metadata 
>> manager because the data in the metadata manager directly relate to 
>> the streams in the playlist manager's playlist.  It makes sense to 
>> keep that all coherent.  Your playlist manager gives you the ability 
>> to handle an arbituary number of playlists, methods for retrieving: 
>> full playlist time, shuffling, repeating, next stream, prev stream, 
>> adding, removing, clearing, everything and anything to do with 
>> manipulation of the playlists and retrieval of data related to the 
>> streams zinf plays.
>>
>> Personally, I dont think zinf should be able to write to the files, 
>> it's a security risk and really has nothing to do with a "audio 
>> player" but since it's a requested feature by some, this I believe 
>> should be handled with the metadata manager. These functions normally 
>> wouldn't be accessed by the playlist manager so you'd call them 
>> directly with something like, 
>> playlistManager::metaManager->writeData("streamname",data); where data 
>> would be a MetaData object containing all the fields the user has 
>> specified etc etc.
>>
>> The MetaData manager is just an object that does the database data 
>> retrieval and gives the playlist manager a simple way of accessing 
>> that data for UI use. It really shouldn't be anywhere outside the 
>> playlist Manager.  There isn't any reason why the rest of the player 
>> would need to see it. Having everything behind the Playlist Manager's 
>> api makes everything much more simple and easy to follow.
>>
>> I rather prefer having the playlist manager take on all the duties of 
>> deciding what to tell the input system to play and what to allow the 
>> UI to display regarding the stream.  I think it concentrates all that 
>> code in one spot in zinf and allows programmers to more easily 
>> optimize the whole process and it gives all UI's equal access to 
>> features.
>>
>>
>> And just to be clear, i'm talking about a manager of playlists, not 
>> the playlist objects themselves.  I agree they should be std::string's 
>> of URI's, I just think the manager should be _THE_ api that the player 
>> uses to do anything related to the streams the user selects up to 
>> actually opening the files. And i think it can do this without being 
>> bloated or unweildy.
> 
> 
> Are you suggesting that the playlist manager should be an all in one 
> interface to both the library and editing playlists? What I mean by that 
> is are you suggesting that you would have a single PlaylistManager 
> object created which had an api that allowed any UI that wanted to query 
> the library about artists, albums, tracks, as well as read metadata 
> about any particular track. It would also maintain a copy of the current 
> playlist to control the player. Something along the lines of:
> 
> vector<std::string> playlist;
> context->playlistmanager->read_playlist("somelist.m3u", &playlist);
> context->playlistmanager->set_current_playlist(playlist);
> 
> could be done by the UI, and then the player could do something like
> 
> std::string url;
> context->playlistmanager->get_current_track(&url);
> context->playlistmanager->goto_next_track();
> 
> And this single object would also provide a way of editing playlists, like
> typedef Playlist;

I hate typedefs   eww.

Playlist would just be a vector of strings.
> Playlist new_list;
> playlistmanager->add_track(new_list, "file://somefile.ogg");

Commands like typedef just obfuscate code.  keep things obvious. If 
something is a std::vector<std::string> dont typedef it to something 
like Playlist just to make it quicker to type over and over again. 
Because debugging and maintainance work now increases and that's far 
more important than the time it took to originally write it due to some 
extra keystrokes.

> 
> 
> Or, are you suggesting that each playlist manager should have a single 
> playlist maintained internally (much like the current playlist manager) 
> and seperate instances are created for each playlist you want to edit. 
> Then reading metadata about a track is done through the playlist manager 
> as well, but it dosn't provide access to the underlying database so that 
> you can't query it for lists of artists or something.

more like a vector of playlists which are just vectors of strings. 
Probably more efficiently implimented as a map of strings and Metadata. 
   The Playlist Manager actually only has the first half of the map, It 
returns a full map or full map entry depending on the active options by 
sending the half filled map to the metaManager who adds the second entry 
and returns the full map which is returned immediately from the 
playlistManager back to the caller.  There isn't a double copy of any 
data in the PlaylistManager, but certain options would cause a double 
copy in the UI and PlaylistManager.  The undelying database is never 
even seen by the rest of the program.  Only the MetaManager access it 
and it can only be accessed by the Playlist Manager.

> Personally, even if the second option did provide a way to access the 
> library, it wouldn't make sense to be reading the library through what 
> is in affect a playlist. And if it didn't provide access to the library, 
> you would need to provide a seperate interface to read metadata from the 
> library. So in effect you would have two interfaces for reading 
> metadata, one for tracks in a playlist, and one for tracks only in the 
> library. Either way the second option doesn't seem like a particularly 
> good idea.


> The first option on the other hand, does seem like a good idea in 
> theory, as it would mean that all the musicbrowser or similar type ui 
> would be doing is implementing the playlist manager interface.

yea.   The playlistManager object directly handles all things to do with 
the playlist(s), including handling multiple lists at once.  This 
includes micro-lists that contain regex queries.  It indirectly handles 
via the MetaManager object all metadata related functions deemed 
necessary to zinf.  I personally dont like the idea of zinf writing 
metadata, so in my world, the MetaManager is responsible for reading 
from various sources and making that data available to the playlist 
manager is a single unified form.  The playlist manager shouldn't care 
what type of codec the file being queried about is. This focuses the 
"need to know" aspect down to the lowest possible layer. All upper 
layers do not really need any knowledge regarding the details of file or 
stream formats and features.


> ...
> 
>>>
>>>> I reckon the metadata manager should be completey seperate from the 
>>>> other base functionality of zinf as all it is doing is providing 
>>>> data to possible UIs, it is never actually used to play tracks. This 
>>>> is also part of the argument for not loading the entire metadata 
>>>> database at startup in the base section of zinf as there is no 
>>>> guarentee that it will be actually needed by the UIs, although it is 
>>>> highly likely.
>>
>>
>> Loading the entire database is cheaper than loading it little at a 
>> time.   You could not load it at all or just query the database per 
>> track in the current playlist, and either way on a newer computer this 
>> wont be noticed at all.  but on older computers where the proponents 
>> to the "delayed reading" of the database argue their method is better, 
>> I believe it actually eats more clock cycles by having to have the 
>> harddrive go back to the database to read more of it rather than 
>> taking advantage of the multi-block sequential aspects of the hdd's 
>> buffers and reading it all at once.  It's just faster that way, even 
>> though you take the hit all at once, it's all while the program loads 
>> anyway, not in the middle of a stream playback or something.  And 
>> besides, the error that the whole idea was cooked up to fix has long 
>> been fixed and was found to not be a problem with the size of the 
>> database at all, but rather a bug in zinf.
> 
> 
> It has been mentioned that the whole database does not really need to be 
> read, taking up resources, if the current ui isn't even going to use. If 
> reading the entire database is less costly then reading it incrementelly 
> (on average), then how about doing a bit of both.
> No part of the database is read into memory until the first time a query 
> is made upon it, and this is when the entire database is read in. This 
> way, if the database is not used its never read, and if it is there is a 
> slightly costly first call on the database, then after that no extra 
> cost of reading anything from disk is incurred.

Deciding when to read from the database is a tricky part though.

Basically your metadata object will have 5 std::string's.
path, artist, album, date, time length
path can be a max of 4KB, we'll give artist and album both 1KB, and last 
tiwe a total of 512bytes.  This means 5.5KB in just the data for each 
metadata object and we'll give it a few bytes container overhead.  Now 
say you have 6000 entries in your database (a fairly large collection), 
you now have 30MB in use.  This is a lot and of course it's a max value.

Perhaps instead we should base database queries on active playlists. Any 
and all active playlists cause the matadatamanager to query and hold 
Metadata objects for all the entries in the playlists.  When a playlist 
is closed all the matadata objects related to just that list are 
destroyed, any overlapping entries remain. As files are added if they're 
added together at one time the query command is globbed together as if 
it was loading a playlist.  Single additions to the list are done one at 
a time from the db.  In essence, you get no lost speed since you would 
have had to read the metada from either the file or the db anyway and 
it's likely the db is faster. Also, with this method it's easy to have 
options such as these.

Meta on demand :  always read metadata once and do not save any copies.

No metadata: dont bother querying the metamanager at all.

normal : like mentioned in above paragraph.


Most importantly about this whole playlist metadata speal is how the 
metadata is written to the database.   We basically have to re-impliment 
metadata readers for every decoder codec we have.  I believe it is 
possible to overload the Decoder's thread start function such that 
instead of normal decoder operation, it sets things up to just read the 
metadata and exit.  Of course this is not really an option if writing of 
metadata is still going to be used.  But by using the Decoders as our 
metadata readers, it's a one step deal to get both decoder support and 
full metadata support since who better to give support for metadata than 
the people writing the decoder.   The search function can send a 
MetaData reference to the decoder starts up the input subsystem and the 
decoder eventually exits with a filled in Metadata object which just 
gets pushed onto the Metadata Manager who files it away.

> 
> 
> David
> 




-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php