Re: Searching without retaining all results
Graham Barr <[email protected]> Thu, 3 Aug 2017 11:55:10 -0500
| Newsgroups | gmane.comp.lang.perl.modules.ldap |
|---|---|
| Message-ID | <[email protected]> |
--Apple-Mail=_924DA4B7-E3B5-4473-BA8D-01B33B4ECA43 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On Aug 3, 2017, at 10:59 , Chris Ridd <[email protected]> wrote: >=20 > I had a requirement to extract 30+ million entries from an ldap = server, and naively thought that using callbacks would be useful. >=20 > While they are useful in getting me each result quickly, I didn't = realise that Search still captures each result in self. So after reading = several million entries Linux's oom killer kindly killed a) the ldap = server and then b) my script to try and get back some memory! >=20 > I ended up patching Search.pm to avoid capturing each result when = using callbacks. Is there a better way to do this? Did you call $mesg->pop_entry; in you callback. See = http://search.cpan.org/~marschap/perl-ldap-0.65/lib/Net/LDAP/FAQ.pod#USING= _THE_CALLBACK_SUBROUTINE_APPROACH = <http://search.cpan.org/~marschap/perl-ldap-0.65/lib/Net/LDAP/FAQ.pod#USIN= G_THE_CALLBACK_SUBROUTINE_APPROACH> for an example Graham. --Apple-Mail=_924DA4B7-E3B5-4473-BA8D-01B33B4ECA43 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On Aug 3, 2017, at 10:59 , Chris Ridd <<a = href=3D"mailto:[email protected]" class=3D"">[email protected]</a>> = wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div = class=3D"">I had a requirement to extract 30+ million entries from an = ldap server, and naively thought that using callbacks would be = useful.<br class=3D""><br class=3D"">While they are useful in getting me = each result quickly, I didn't realise that Search still captures each = result in self. So after reading several million entries Linux's oom = killer kindly killed a) the ldap server and then b) my script to try and = get back some memory!<br class=3D""><br class=3D"">I ended up patching = Search.pm to avoid capturing each result when using callbacks. Is there = a better way to do this?<br class=3D""></div></div></blockquote><div><br = class=3D""></div>Did you call $mesg->pop_entry; in you = callback. See <a = href=3D"http://search.cpan.org/~marschap/perl-ldap-0.65/lib/Net/LDAP/FAQ.p= od#USING_THE_CALLBACK_SUBROUTINE_APPROACH" = class=3D"">http://search.cpan.org/~marschap/perl-ldap-0.65/lib/Net/LDAP/FA= Q.pod#USING_THE_CALLBACK_SUBROUTINE_APPROACH</a> for an = example</div><div><br class=3D""></div><div>Graham.</div><div><br = class=3D""></div></body></html>= --Apple-Mail=_924DA4B7-E3B5-4473-BA8D-01B33B4ECA43--