Re: Searching without retaining all results
Chris Ridd <[email protected]> Thu, 03 Aug 2017 18:31:19 +0100
| Newsgroups | gmane.comp.lang.perl.modules.ldap |
|---|---|
| Message-ID | <[email protected]> |
--Apple-Mail-F2CA09E7-3C88-42F4-8E6A-FB9A7F8AE9E1 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable > On 3 Aug 2017, at 17:55, Graham Barr <[email protected]> wrote: >=20 >=20 >> On Aug 3, 2017, at 10:59 , Chris Ridd <[email protected]> wrote: >>=20 >> I had a requirement to extract 30+ million entries from an ldap server, a= nd naively thought that using callbacks would be useful. >>=20 >> While they are useful in getting me each result quickly, I didn't realise= that Search still captures each result in self. So after reading several mi= llion entries Linux's oom killer kindly killed a) the ldap server and then b= ) my script to try and get back some memory! >>=20 >> I ended up patching Search.pm to avoid capturing each result when using c= allbacks. Is there a better way to do this? >=20 > Did you call $mesg->pop_entry; in you callback. See http://search.cpan.or= g/~marschap/perl-ldap-0.65/lib/Net/LDAP/FAQ.pod#USING_THE_CALLBACK_SUBROUTIN= E_APPROACH for an example Aha, that's exactly what I was missing. Thanks Graham! Chris= --Apple-Mail-F2CA09E7-3C88-42F4-8E6A-FB9A7F8AE9E1 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><head><meta http-equiv=3D"content-type" content=3D"text/html; charset=3D= utf-8"></head><body dir=3D"auto"><div></div><div><br></div><div><br>On 3 Aug= 2017, at 17:55, Graham Barr <<a href=3D"mailto:[email protected]">gbarr@po= box.com</a>> wrote:<br><br></div><blockquote type=3D"cite"><div><meta htt= p-equiv=3D"Content-Type" content=3D"text/html charset=3Dus-ascii"><br class=3D= ""><div><blockquote type=3D"cite" class=3D""><div class=3D"">On Aug 3, 2017,= at 10:59 , Chris Ridd <<a href=3D"mailto:[email protected]" class=3D"">c= [email protected]</a>> wrote:</div><br class=3D"Apple-interchange-newline"= ><div class=3D""><div class=3D"">I had a requirement to extract 30+ million e= ntries from an ldap server, and naively thought that using callbacks would b= e useful.<br class=3D""><br class=3D"">While they are useful in getting me e= ach result quickly, I didn't realise that Search still captures each result i= n self. So after reading several million entries Linux's oom killer kindly k= illed a) the ldap server and then b) my script to try and get back some memo= ry!<br class=3D""><br class=3D"">I ended up patching Search.pm to avoid capt= uring each result when using callbacks. Is there a better way to do this?<br= class=3D""></div></div></blockquote><div><br class=3D""></div>Did you call&= nbsp; $mesg->pop_entry; in you callback. See <a href=3D"http://= search.cpan.org/~marschap/perl-ldap-0.65/lib/Net/LDAP/FAQ.pod#USING_THE_CALL= BACK_SUBROUTINE_APPROACH" class=3D"">http://search.cpan.org/~marschap/perl-l= dap-0.65/lib/Net/LDAP/FAQ.pod#USING_THE_CALLBACK_SUBROUTINE_APPROACH</a>&nbs= p; for an example</div></div></blockquote><div><br></div><div>Aha, that's ex= actly what I was missing. Thanks Graham!</div><div><br></div><div>Chris</div= ></body></html>= --Apple-Mail-F2CA09E7-3C88-42F4-8E6A-FB9A7F8AE9E1--