Re: Problems importing huge amounts of data

On 3/3/06, Toby Dickenson <[email protected]> wrote:
> On Friday 03 Mar 2006 10:32, Santi Camps wrote:
>
> > Then we began to debug the DS itself, and found that the "problem" was
> > that DS was reading all the files during the commit process (more than
> > 30000, because there is a .c and .oid file per object),    Obviously,
> > it's not an error, but it has a tremendous cost.
>
> Interesting. What percentage difference did this make to performance?
>

Each commit was taken about 7m 30s, and after that change it downs to
only 20s.   Rest of the process takes about 30s per transaction, so,
the result was 8m -> 50s per transaction, about 90% of improvement.


> > So, we patch the DS in order not to do this readings when parameter
> > sync = 0.    I attach the patch, hopping it could be included in next
> > releases
>
> This reading is part of a mechanism to prevent 'dangling references' in the
> object database. You can suppress this whole mechanism by changing
>
> check_dangling_references: 1
>
> to
>
> check_dangling_references: 0
>
> in your configuration file. This should eliminate the same disk access as your
> patch, plus some other CPU time.
>

Ups, I will ask next time :-)

Thanks
--
Santi Camps
Earcon S.L. - http://www.earcon.com
                  - http://www.kmkey.com


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid0944&bid$1720&dat1642