Re: thoughts on meta-data (was: importing entries)

On Apr 7, 2005 2:10 PM, will guaraldi <[email protected]> wrote:
> On Thu, 7 Apr 2005, Bill Mill wrote:
> >>
> >> Also, I'd stay away from metadata names that might conflict with
> >> existing entry variables.
> >
> > Well, I wrote my own metadata parser which puts them into a dictionary
> > of their own, so they can't conflict - they also can't be referenced
> > directly from the entry (it's *meta* data).
> 
> That's an interesting idea.  We could do that by storing all metadata
> under metadata::<key-name> kind of thing.
> 

Yup. If it's data, it should be in the entry - metadata should only be
accessible to pyblosxom, which may, in turn, give it back to the
entry. This is how my metatime plugin works.

I also think that

#key word1, a phrase1, another phrase1

should parse into  {key: ['word1', 'a phrase1', 'a phrase2']}. Commas
could be escaped. That's how my plugin works (although I've never
added comma escaping because I never needed it).

> 
> >> This would get fixed if we were looking at fully parsed entries (i.e.
> >> the file has been read and parsed out) before we sorted and sliced the
> >> list. Then we could allow metadata to override the file mtime.
> >
> > Or we could just parse the metadata (not the whole story, it'd be very
> > fast, esp since the metadata is right at the top of the file) look for a
> > specific key, and use it to sort with if it exists.
> 
> The issue is that when we're sorting all we have is the filename and the
> mtime--that's it.  We haven't built entry objects at that point.  So we
> definitely have to fix that.  I haven't worked through the particulars
> yet.
> 

Not true, check out the relevant lines of blosxom_file_handler. You
have a FileEntry object (which, admittedly, contains little other than
the filename and mtime):

    for ourfile in filelist:
        e = FileEntry(request, ourfile, data['root_datadir'])
        entrylist.append((e._mtime, e))

    # this sorts entries by mtime in reverse order.  entries that have
    # no mtime get sorted to the top.
    entrylist.sort()
    entrylist.reverse()

if you added metadata parsing to the __init__ call of the FileEntry
object, you could have the metadata available too, at a fairly small
cost. Pseudocode (from memory) of a function that would do it:

class FileEntry(BaseEntry):
    def __init__(...):
        ...
        self.meta = getMeta()

    def getMeta(self):
        meta = {}
        f = open(self._filename, 'r')
        f.next()
        for line in f:
            if not line.startswith('#'): break
            key, value = line[1:].split(' ', 1)
            meta[key] = value
        return meta

Then, the above lines could become, again in pseudocode:

    for ourfile in filelist:
        e = FileEntry(request, ourfile, data['root_datadir'])
        if 'datetime' in e.meta:
            entrylist.append((magic_time_converter(e.meta['datetime']), e))
        else:
            entrylist.append((e._mtime, e))

    # this sorts entries by mtime in reverse order.  entries that have
    # no mtime get sorted to the top.
    entrylist.sort()
    entrylist.reverse()

where magic_time_converter converts the chose date format to a number
like time.time().

The cost of this is basically time(file open) * n_entries , because
the file opening time is gonna cost you (IMHO) much more time than is
the parsing of the entries.

> It's an interesting point that we could open the file for reading, pull
> the title, metadata, and then the rest of it we could stick in a data
> object to be absorbed when needed.  That does skip the potentially
> cycle-intensive portion of parsing the entry.  For blosxom entries, it's
> trivial, but for other entryparsers, it's more involved.
> 

What entryparsers? I ask because I really don't know what else is out
there. If you were going to import from a DB, you would store the
metadata in seperate columns of the table where you kept the story -
in that case metadata parsing would be *blazing*.

> I'll have to think about this some more, but this sounds like it'll work
> pretty well.
> 
> In regards to caching and such, by default PyBlosxom will not be doing
> file-based caching.  I really want to push all file-based caching to
> plugins so then people can implement what caching works for them.  So it's
> nice that people can cache things, but PyBlosxom should work pretty well
> without caching (to varying degrees).
> 

fair enough.

> 
> > I think you see now why we were on different pages.
> 
> I've been pretty vague about what I'm thinking.  I haven't had time to
> sort it out into a spec yet.
> 

Cool, I hope you do. I'll write lots of comments. I'm unlikely to
write one myself, because I prefer to just hack. I think it can be a
big improvement, though.

Peace
Bill Mill
bill.mill at gmail.com


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click