Re: thread resource leaks if thread terminates with uncaught exception
Shiro Kawai <[email protected]> Tue, 17 Dec 2024 21:24:00 -1000
| Newsgroups | gmane.lisp.scheme.gauche |
|---|---|
| Message-ID | <CALN0JNF-XHG1y76VaHWaErYRYgxtL+8VZsc0rrb++aAF2ADrmQ@mail.gmail.com> |
--===============4116528252810103398== Content-Type: multipart/alternative; boundary="000000000000d4769a0629864c51" --000000000000d4769a0629864c51 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ok, I reproduced it. The threads leave the socket to the localhost:8081 open (You can see it with `lsof -p <pid>`), which consumes the fd pool. `thread-terminate!` doesn't give the thread to cleanup, so relying on it should generally be avoided. However, those orphan sockets must be closed by gc. The fact it is not the case suggests there remain some references to the ScmVM(). On Tue, Dec 17, 2024 at 12:24=E2=80=AFAM Jens Thiele <[email protected]> wrote= : > Jens Thiele <[email protected]> writes: > > > Jens Thiele <[email protected]> writes: > > > >> Hi, > >> > >> I have some resource leaks problems with a long running process. > >> > >> I still don't have a really simple test but I am getting closer to the > >> problem. > >> > >> Let's say we have some webserver where some requests are really > >> slow. For example using makiki: server.scm: > >> > >> #!/bin/sh > >> #| -*- mode: scheme; coding: utf-8; -*- > >> exec gosh -I. -- $0 "$@" > >> |# > >> (use makiki) > >> (use text.html-lite) > >> > >> (define (hello) > >> (list (html-doctype) > >> (html:html > >> (html:head (html:title "hello")) > >> (html:body (html:p "hello"))))) > >> > >> (define-http-handler (GET) "/" (^[req app] (respond/ok req > (hello)))) > >> (define-http-handler (GET) "/slow" (^[req app] (sys-sleep 10) > (respond/ok req (hello)))) > >> > >> (define (main args) > >> (start-http-server :port 8081) > >> 0) > >> > >> Then we have some reverse proxy also using makiki (that will be the > >> process with the leaks). It tries to fetch many urls with a timeout in > >> parallel and for some reason oaccasionally there might be an error (th= e > >> "(when (=3D i 100) (error "i=3D100"))"). > >> > >> #!/bin/sh > >> #| -*- mode: scheme; coding: utf-8; -*- > >> #export GC_PRINT_STATS=3D1 > >> exec gosh -I. -- $0 "$@" > >> |# > >> (use makiki) > >> (use text.html-lite) > >> (use rfc.http) > >> (use gauche.threads) > >> (use file.util) > >> (use runtime-compile) > >> > >> (compile-and-load > >> `((inline-stub > >> (define-cproc get-gc-no () > >> (let* ((r::(struct GC_prof_stats_s))) > >> (GC_get_prof_stats (& r) (sizeof r)) > >> (return (SCM_MAKE_INT (ref r gc_no))))))) > >> '(get-gc-no)) > >> > >> ;; todo: linux specific > >> (define (num-open-files) (length (directory-list "/proc/self/fd" > :children? #t))) > >> > >> (define-http-handler (GET) "/" (^[req app] > >> (receive (status headers body) > >> (http-get "localhost:8081" "/") > >> (respond/ok req body)))) > >> > >> (define-http-handler (GET) "/timeout" (^[req app] > >> (let1 threads (map (lambda(i) > >> (make-thread > >> (lambda() > >> ;; if i > comment this one it works > >> (when (= =3D i > 100) (error "i=3D100")) > >> (receive > (status headers body) > >> > (http-get "localhost:8081" "/slow") > >> body)))= ) > >> (iota 500)) > > > > if I use "only" 50 here - it looks like there is no problem > > after quite some time I also got the error with 50 threads. Looks like > it just takes much longer. > > Interestingly the first error was again a > "getaddrinfo failed: System error: Device or resource busy" > in one of the threads > I am not sure how this error could happen with a lookup for "localhost" > at all. > > Afterwards the main thread exited with > "SYSTEM-ERROR: accept(2) failed: Too many open files" > > Best regards > Jens > > > _______________________________________________ > Gauche-devel mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/gauche-devel > --000000000000d4769a0629864c51 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Ok, I reproduced it.=C2=A0 The threads leave the socket to= the localhost:8081 open=C2=A0 (You can see it with `lsof -p <pid>`),= which consumes the fd pool.=C2=A0 `thread-terminate!` doesn't give the= thread to cleanup, so relying on it should=C2=A0generally be avoided.=C2= =A0 However, those orphan sockets must be closed by gc.=C2=A0 The fact it i= s not the case suggests there remain some references to the ScmVM().<div><b= r></div></div><br><div class=3D"gmail_quote gmail_quote_container"><div dir= =3D"ltr" class=3D"gmail_attr">On Tue, Dec 17, 2024 at 12:24=E2=80=AFAM Jens= Thiele <<a href=3D"mailto:[email protected]">[email protected]</a>> wrote:= <br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8= ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Jens Thiele <= ;<a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a>>= writes:<br> <br> > Jens Thiele <<a href=3D"mailto:[email protected]" target=3D"_blank">ka= [email protected]</a>> writes:<br> ><br> >> Hi,<br> >><br> >> I have some resource leaks problems with a long running process.<b= r> >><br> >> I still don't have a really simple test but I am getting close= r to the<br> >> problem.<br> >><br> >> Let's say we have some webserver where some requests are reall= y<br> >> slow. For example using makiki: server.scm:<br> >><br> >> #!/bin/sh<br> >> #| -*- mode: scheme; coding: utf-8; -*-<br> >> exec gosh -I. -- $0 "$@"<br> >> |#<br> >> (use makiki)<br> >> (use text.html-lite)<br> >><br> >> (define (hello)<br> >>=C2=A0 =C2=A0(list (html-doctype)<br> >>=C2=A0 =C2=A0 =C2=A0 (html:html<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0(html:head (html:title "hello"= ))<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0(html:body (html:p "hello"))))= )<br> >><br> >> (define-http-handler (GET)=C2=A0 "/"=C2=A0 =C2=A0 =C2=A0= (^[req app] (respond/ok req (hello))))<br> >> (define-http-handler (GET)=C2=A0 "/slow" (^[req app] (sy= s-sleep 10) (respond/ok req (hello))))<br> >><br> >> (define (main args)<br> >>=C2=A0 =C2=A0(start-http-server :port 8081)<br> >>=C2=A0 =C2=A00)<br> >><br> >> Then we have some reverse proxy also using makiki (that will be th= e<br> >> process with the leaks). It tries to fetch many urls with a timeou= t in<br> >> parallel and for some reason oaccasionally there might be an error= (the<br> >> "(when (=3D i 100) (error "i=3D100"))").<br> >><br> >> #!/bin/sh<br> >> #| -*- mode: scheme; coding: utf-8; -*-<br> >> #export GC_PRINT_STATS=3D1<br> >> exec gosh -I. -- $0 "$@"<br> >> |#<br> >> (use makiki)<br> >> (use text.html-lite)<br> >> (use rfc.http)<br> >> (use gauche.threads)<br> >> (use file.util)<br> >> (use runtime-compile)<br> >><br> >> (compile-and-load<br> >>=C2=A0 `((inline-stub<br> >>=C2=A0 =C2=A0 =C2=A0(define-cproc get-gc-no ()<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0(let* ((r::(struct GC_prof_stats_s)))<br= > >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(GC_get_prof_stats (& r) (siz= eof r))<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(return (SCM_MAKE_INT (ref r gc_n= o)))))))<br> >>=C2=A0 '(get-gc-no))<br> >><br> >> ;; todo: linux specific<br> >> (define (num-open-files) (length (directory-list "/proc/self/= fd" :children? #t)))<br> >><br> >> (define-http-handler (GET) "/" (^[req app]<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(receive (status headers body)= <br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(http-get "= localhost:8081" "/")<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(respond/ok req body)))= )<br> >><br> >> (define-http-handler (GET) "/timeout" (^[req app]<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (let1 th= reads (map (lambda(i)<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(make-= thread<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (lamb= da()<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 ;; if i comment this one it works<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 (when (=3D i 100) (error "i=3D100"))<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 (receive (status headers body)<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 (http-get "localhost:8081" "/slow")<b= r> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 body))))<br> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(iota 500))<b= r> ><br> > if I use "only" 50 here - it looks like there is no problem<= br> <br> after quite some time I also got the error with 50 threads. Looks like<br> it just takes much longer.<br> <br> Interestingly the first error was again a<br> "getaddrinfo failed: System error: Device or resource busy"<br> in one of the threads<br> I am not sure how this error could happen with a lookup for "localhost= "<br> at all.<br> <br> Afterwards the main thread exited with<br> "SYSTEM-ERROR: accept(2) failed: Too many open files"<br> <br> Best regards<br> Jens<br> <br> <br> _______________________________________________<br> Gauche-devel mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">Gau= [email protected]</a><br> <a href=3D"https://lists.sourceforge.net/lists/listinfo/gauche-devel" rel= =3D"noreferrer" target=3D"_blank">https://lists.sourceforge.net/lists/listi= nfo/gauche-devel</a><br> </blockquote></div> --000000000000d4769a0629864c51-- --===============4116528252810103398== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============4116528252810103398== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Gauche-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/gauche-devel --===============4116528252810103398==--