Re: thoughts on meta-data (was: importing entries)

On Apr 7, 2005 10:56 PM, Wari Wahab <[email protected]> wrote:
> > Well, then.  I guess to fix these issues, we'd need to adjust the
> > FileEntry as you mentioned so that it opens the file, pulls in the
> > contents of the the file, extrapolates the title, extrapolates the
> > metadata and leaves parsing the body until it's needed.
> 
> Sorry Will, have to jump in here. I don't think this idea will scale as
> you have to open every file in the directory to grab metadata, slowing
> things down. And let's say you'd just leave the files open until
> another plugin reads the data, you'll lose file handles faster than you
> can blink.
> 
> And talk about actual entries itself, what if the entry is a pure PDF
> (in itself full of metadata, in a different form), or a JPG (EXIF
> data), or even one could have an MP3 parser that reads out the ID3 Tags
> to get text/images (Podcasting anyone? :) can be slow to read ID3v2 for
> example) to display entries, and have the actual file as downloadable
> enclosures.
> 
> I would agree to have a metadata scanning function, probably the first
> time it reads a file, and store it in a DB somewhere, and the next time
> pyblosxom runs, only one file gets opened (much like how the cache is
> done right now).
> 
> I remember one time a while back, there was a problem with one of our
> plugins which reads data in the files. That slowed down the main (old)
> pyblosxom blog by a factor of 4.
> 
> 

Assuming that you can skip a filestat if the metadata of a file
contains a time, then you can save a little time. With no db and 1000
entries, if all of them have a date/time tag, your time will look
something like:

>>> t = timeit.Timer('os.stat("test.py")', 'import os')
>>> t.timeit(1000)
0.089999914169311523
>>> def test():
...     d = {}
...     for line in file('test.py'):
...             if line[0] != '#': break
...             s = line.find(' ')
...             d[line[1:s]] = line[s+1:]
...     return d
...
>>> t = timeit.Timer('test()', 'from __main__ import test')
>>> t.timeit(1000)
0.17999982833862305

In this test, the first 3 lines of 'test.py' are metadata lines.

First off, I'm going to assume that if the file contains datetime
meta-data, we won't need to do an os.stat() on it. If so, over 1000
entries, if all of the entries have a time in metadata, it's going to
cost you .09 seconds. If only half of your entries have a time in
metadata, and thus you have to os.stat() them, it'll cost you about
.114 seconds, and if none of them have a time in metadata, it'll cost
you about .119 seconds.

I'm in favor of a cache DB too, but I think that those numbers are
small enough to warrant examination of file scanning as a possibility.
Of course they're nearly worthless, but I think there's hope for the
metadata scan not taking too long.

I think we should code it up, try it out, and measure how long it
takes in a more realistic scenario. Heck, I wrote a large majority of
the code required in my email yesterday.

Peace
Bill Mill
bill.mill at gmail.com


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click