Re: MaJorToM version 1.2.0 released

On Sun, Apr 17, 2011 at 7:40 AM, Robert Barta <[email protected]> wrote:
> On Sun, Apr 17, 2011 at 12:10:56AM +0200, Lars Heuer wrote:
>> Hi Benjamin,
>>
>> > no, it's not.
>>
>> This statement would have been sufficient. Too much names and dancing
>> around. ;)

Yes and now. It would answer the initial question, but it gives you
some information how the "benchmarks" are fundamentally different:

Size of data:
* Lars tested separate imports of about 1000 tiny files, the biggest
of them still being smaller than 2Kb. The total raw test input data is
about 3MB uncompressed.

* We tested 6 to 30 imports per run, the biggest single import file of
them being around 500 MB, in total about one GB for one run.

Diversity of the data:
* Lars' benchmark checked probably every aspect: variants, reification
of roles, etc.

* We had much more uniform data: no variants, no reification on roles,
less than 300 scoping topics in total, etc.; still the e.g. freebase
data created a lot of different topic types, association types and
role types.

Differences in access:
* In the CXTM tests data gets imported and then exported one time.

* Our project had also one import, but we also tested consecutive
reads on parts of the data.

> I will not touch the benchmark either. There are too many variations
> in the process described to render the comparison ... uncomparable.

Our goal was not to create a benchmark (otherwise we would most likely
have published one, or at least the exact results), but to get the
project done and learn about the behavior of our tools with topic maps
bigger than a few kilobytes.

It's left to the reader if a eclectic test of 3MB testdata with one time export
or a more homogeneous test case with 1000MB test data and multiple
access patterns
is more representative for their needs.

> Still: The dancing was pretty. ;-)

:-)

Benjamin