Re: thoughts on meta-data

On Apr 8, 2005 10:58 AM, Wari Wahab <[email protected]> wrote:
> > In this test, the first 3 lines of 'test.py' are metadata lines.
> 
> :) Before I go to sleep: You should try to test 1000 different files
> instead of testing one test.py, and you'll see more realistic numbers.
> 

Well, I don't think these numbers are really good for too much, but
here it is with 1000 different files of various length (630 bytes ->
6.3k at random distribution), all with metadata:

==========begin test_meta.py========
import time, random

def makefiles():
    r = random.Random().randint
    for i in range(1000):
        f = file('test' + str(i), 'w')
        print >> f, """
#every file starts
#with some meta data
#in a couple of lines"""
        n = r(0, 100)
        print >> f, """
Some files contain more data
than that though, at random
sizes""" * n

def test():
    ran = range(1000)
    totaltime = 0.
    for i in ran:
        fname = 'test' + str(i)
        t1 = time.time()
        test_meta(fname)
        t2 = time.time()
        totaltime += t2-t1
    print "test took %f seconds" % totaltime


def test_meta(f):
    d = {}
    f = file(f)
    f.readline()
    for line in f:
        if line[0] != '#': break
        s = line.find(' ')
        d[line[1:s]] = line[s+1:]
    return d

makefiles()
test()
==========end test_meta.py===========

11:20 AM /d/code/Python/testfiles$ python test_meta.py
test took 0.205000 seconds

So there's not really a lot of difference that I see. If I change the
file so that ~50% of the generated files contain metadata, the results
get even better (but, of course IRL those files would need to be
os.stat()ed):

11:26 AM /d/code/Python/testfiles$ python test_meta.py
test took 0.160000 seconds
11:26 AM /d/code/Python/testfiles$ python test_meta.py
test took 0.150000 seconds

Peace
Bill Mill
bill.mill at gmail.com


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click