Re: Apache server returns early before process is complete
Joseph He <[email protected]> Tue, 13 May 2025 09:46:25 -0500
| Newsgroups | gmane.comp.apache.mod-perl |
|---|---|
| Message-ID | <CAD16sHvKjkwcV-zHejm1ju0qZTYeYvZjs+kv-3yzWvVHOVyd5A@mail.gmail.com> |
--000000000000e4dedd0635057f42 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Many thanks to you all. I am still trying to figure out the issue. Let me re-explain the problem I experienced with some details. The environment is Ubuntu 22.04, Apache2, ModPerl. I run a Http::request with LWP::UserAgent, the server receives the request and starts to process it. But it takes much longer due to a stalled SFTP call to the remote server, the Apache server timeout and sends back failure, meanwhile,* the server actually is still trying to process this request*. On the calling side, after receiving the failure status, it initiates another http::request and the load balancer redirects this call to another server for processing. It turns out this same http::request is processed twice. On my production server the timeout happens at 300 seconds mark. On my QA and Dev server, the timeout happens at 600 seconds. I have not changed anything on my production server yet. But on my QA and DEV servers, I have tried to change Timeout in apache2.conf, have tried to add Timeout to the virtualhost config, also have tried to add SetPerlEnv MOD_PERL_TIMEOUT to the virtualhost config, none of them change the timeout behavior of my QA and DEV servers. So what exactly controls the Timeout? I am totally lost. Cheers, Joe On Wed, Apr 23, 2025 at 5:17=E2=80=AFPM Mithun Bhattacharya <[email protected]= om> wrote: > Okay agreed that is a valid time out basically it is saying that a client > has established tcp/ip connection but has not put its request either a ge= t > put or a post > > On Wed, Apr 23, 2025, 3:38=E2=80=AFPM Joseph He <[email protected]= > wrote: > >> On Apache2 doc, I found this. How does this timeout work? It looks like >> it can only wait for 300 seconds before failing a request. >> >> https://httpd.apache.org/docs/2.0/mod/core.html#timeout >> Description: >> <https://httpd.apache.org/docs/2.0/mod/directive-dict.html#Description> = Amount >> of time the server will wait for certain events before failing a request >> Syntax: >> <https://httpd.apache.org/docs/2.0/mod/directive-dict.html#Syntax> >> TimeOut seconds >> Default: >> <https://httpd.apache.org/docs/2.0/mod/directive-dict.html#Default> Time= Out >> 300 >> Context: >> <https://httpd.apache.org/docs/2.0/mod/directive-dict.html#Context> serv= er >> config, virtual host >> Status: >> <https://httpd.apache.org/docs/2.0/mod/directive-dict.html#Status> Core >> Module: >> <https://httpd.apache.org/docs/2.0/mod/directive-dict.html#Module> core >> >> The TimeOut directive currently defines the amount of time Apache will >> wait for three things: >> >> 1. The total amount of time it takes to receive a GET request. >> 2. The amount of time between receipt of TCP packets on a POST or PUT >> request. >> 3. The amount of time between ACKs on transmissions of TCP packets in >> responses. >> >> We plan on making these separately configurable at some point down the >> road. The timer used to default to 1200 before 1.2, but has been lowered= to >> 300 which is still far more than necessary in most situations. It is not >> set any lower by default because there may still be odd places in the co= de >> where the timer is not reset when a packet is sent. >> >> On Wed, Apr 23, 2025 at 3:07=E2=80=AFPM Mithun Bhattacharya <mithnb@gmai= l.com> >> wrote: >> >>> You configure timeout at the client side. Apache is at the server side. >>> Server doesn't have a concept of time it could take days to run and not >>> care. >>> >>> mod_perl code is where you are sending the http return status to make >>> sure the client doesn't timeout waiting for the server to respond. >>> >>> On Wed, Apr 23, 2025, 2:19=E2=80=AFPM Joseph He <[email protected]= om> >>> wrote: >>> >>>> Thanks, all. >>>> Is that Apache timeout controlled by its configuration "Timeout"? >>>> I don't think it has anything to do with modPerl. Am I missing >>>> something? >>>> Thanks. >>>> >>>> On Wed, Apr 23, 2025 at 1:41=E2=80=AFPM Mithun Bhattacharya <mithnb@gm= ail.com> >>>> wrote: >>>> >>>>> Timeout happens because of how we handle the request. Timeout is >>>>> basically no response came back. Why that happens is because we think= we >>>>> want to have a correct response. Unfortunately for long running reque= sts >>>>> the correct response shouldn't be via http response code or we face >>>>> situations like this. Instead reply with a 200 OK immediately and the= n >>>>> provide correct status in the message body. Once a response code/head= er has >>>>> been sent timeout won't trigger and you could potentially hold the >>>>> connection for hours without a problem. >>>>> >>>>> On Wed, Apr 23, 2025, 9:32=E2=80=AFAM Andreas Mock <andreas.mock@web.= de> >>>>> wrote: >>>>> >>>>>> Hi Joseph, >>>>>> >>>>>> your description is very vague, so can only answer on some >>>>>> assumptions: >>>>>> >>>>>> It sounds like a timeout is fired somewhere. >>>>>> >>>>>> Best advice in these situations: Log as many steps as you can. Keep >>>>>> your >>>>>> eyes open on TCP/IP and higher level timeouts. >>>>>> >>>>>> Declare only ONE instance responsible for a retry: Either the app >>>>>> server >>>>>> calling the dispatcher with several tries or the dispatcher trying >>>>>> for >>>>>> himself. Not both. >>>>>> >>>>>> Best regards >>>>>> Andreas >>>>>> >>>>>> >>>>>> Am 23.04.2025 um 16:21 schrieb Joseph He: >>>>>> > All, good day. >>>>>> > >>>>>> > Here is the issue I have. >>>>>> > My entire application is running on ModPerl/Apache environment. >>>>>> > I send Http::Request with data load from my App server to a >>>>>> dispatch >>>>>> > server thru LWP::UserAgent, I set the timeout 600 seconds. >>>>>> > >>>>>> > The dispatch server is supposed to manipulate the data and send th= e >>>>>> > data to an external SFTP server. Because the SFTP can fail, it wil= l >>>>>> > keep trying up to 4 times with 30 seconds sleep in case that SFTP >>>>>> > connection fails. >>>>>> > >>>>>> > Recently, I found that I uploaded the file twice sometimes. I >>>>>> figured >>>>>> > out the root cause is that my Dispatch server returns 'failure' at >>>>>> 6 >>>>>> > minutes while it keeps trying to do the SFTP. The App server >>>>>> > received HTTP::Response with error status so it issued another cal= l >>>>>> to >>>>>> > send data. It turns out I uploaded the identified file twice. >>>>>> > >>>>>> > Anybody has this sort of experience? Why does the dispatch server >>>>>> > return 'error' while it still processes the data? >>>>>> > >>>>>> > Thanks a lot, >>>>>> > Joseph >>>>>> > >>>>>> >>>>> --000000000000e4dedd0635057f42 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Many thanks to you all.<div><br><div>I am still trying to = figure out the issue. Let me re-explain the problem I experienced with some= details.</div><div><br>The environment is=C2=A0Ubuntu 22.04, Apache2,=C2= =A0ModPerl.<br>I run a Http::request with LWP::UserAgent, the server receiv= es the request and starts to process it.</div><div>But it takes much longer= due=C2=A0to a stalled SFTP call to the remote server, the Apache server ti= meout and sends back failure, meanwhile,<b> the server actually=C2=A0is sti= ll trying to process this request</b>.<br>On the calling side, after receiv= ing the failure status, it initiates another http::request=C2=A0and the loa= d balancer redirects this call to another server for processing.=C2=A0</div= ><div>It turns out this same http::request is processed twice.</div><div><b= r></div><div>On my production=C2=A0server the timeout happens at 300 second= s mark. On my QA and Dev server, the timeout happens at 600 seconds. I have= not changed anything on my production server yet.=C2=A0</div><div>But on m= y QA and DEV servers, I have tried to change Timeout in apache2.conf, have = tried to add Timeout to the virtualhost config, also have tried to add SetP= erlEnv MOD_PERL_TIMEOUT to the virtualhost config, none of them change the = timeout behavior of my QA and DEV servers.<br><br>So what exactly controls = the Timeout? I am totally lost.</div><div><br></div><div>Cheers,</div><div>= Joe=C2=A0</div><div><br></div></div></div><br><div class=3D"gmail_quote gma= il_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, Apr 23, 2= 025 at 5:17=E2=80=AFPM Mithun Bhattacharya <<a href=3D"mailto:mithnb@gma= il.com">[email protected]</a>> wrote:<br></div><blockquote class=3D"gmail= _quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204= ,204);padding-left:1ex"><div dir=3D"auto">Okay agreed that is a valid time = out basically it is saying that a client has established tcp/ip connection = but has not put its request either a get put or a post</div><br><div class= =3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, Apr 23, 2025= , 3:38=E2=80=AFPM Joseph He <<a href=3D"mailto:[email protected]"= target=3D"_blank">[email protected]</a>> wrote:<br></div><blockq= uote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1p= x solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr">On Apache2 doc,= I found this. How does this timeout work? It looks like it can only wait f= or 300 seconds before failing a request.=C2=A0<div><br></div><div><a href= =3D"https://httpd.apache.org/docs/2.0/mod/core.html#timeout" rel=3D"norefer= rer" target=3D"_blank">https://httpd.apache.org/docs/2.0/mod/core.html#time= out</a><br><div><table style=3D"font-size:14px;border:1px solid rgb(170,170= ,170);border-collapse:collapse;padding:2px;margin-top:0.5em;margin-bottom:1= em;color:rgb(0,51,102)"><tbody><tr><th style=3D"empty-cells:show;padding:0.= 1em 0.2em;vertical-align:top;text-align:left;line-height:1.3em"><a href=3D"= https://httpd.apache.org/docs/2.0/mod/directive-dict.html#Description" styl= e=3D"color:rgb(0,115,199);background-color:inherit" rel=3D"noreferrer" targ= et=3D"_blank">Description:</a></th><td style=3D"empty-cells:show;padding:0.= 1em 0.2em;vertical-align:top;line-height:1.3em">Amount of time the server w= ill wait for certain events before failing a request</td></tr><tr><th style= =3D"empty-cells:show;padding:0.1em 0.2em;vertical-align:top;text-align:left= ;line-height:1.3em"><a href=3D"https://httpd.apache.org/docs/2.0/mod/direct= ive-dict.html#Syntax" style=3D"color:rgb(0,115,199);background-color:inheri= t" rel=3D"noreferrer" target=3D"_blank">Syntax:</a></th><td style=3D"empty-= cells:show;padding:0.1em 0.2em;vertical-align:top;line-height:1.3em"><code = style=3D"font-family:"Courier New",Courier,monospace;font-size:1e= m">TimeOut=C2=A0<var>seconds</var></code></td></tr><tr><th style=3D"empty-c= ells:show;padding:0.1em 0.2em;vertical-align:top;text-align:left;line-heigh= t:1.3em"><a href=3D"https://httpd.apache.org/docs/2.0/mod/directive-dict.ht= ml#Default" style=3D"color:rgb(0,115,199);background-color:inherit" rel=3D"= noreferrer" target=3D"_blank">Default:</a></th><td style=3D"empty-cells:sho= w;padding:0.1em 0.2em;vertical-align:top;line-height:1.3em"><code style=3D"= font-family:"Courier New",Courier,monospace;font-size:1em">TimeOu= t 300</code></td></tr><tr><th style=3D"empty-cells:show;padding:0.1em 0.2em= ;vertical-align:top;text-align:left;line-height:1.3em"><a href=3D"https://h= ttpd.apache.org/docs/2.0/mod/directive-dict.html#Context" style=3D"color:rg= b(0,115,199);background-color:inherit" rel=3D"noreferrer" target=3D"_blank"= >Context:</a></th><td style=3D"empty-cells:show;padding:0.1em 0.2em;vertica= l-align:top;line-height:1.3em">server config, virtual host</td></tr><tr><th= style=3D"empty-cells:show;padding:0.1em 0.2em;vertical-align:top;text-alig= n:left;line-height:1.3em"><a href=3D"https://httpd.apache.org/docs/2.0/mod/= directive-dict.html#Status" style=3D"color:rgb(0,115,199);background-color:= inherit" rel=3D"noreferrer" target=3D"_blank">Status:</a></th><td style=3D"= empty-cells:show;padding:0.1em 0.2em;vertical-align:top;line-height:1.3em">= Core</td></tr><tr><th style=3D"empty-cells:show;padding:0.1em 0.2em;vertica= l-align:top;text-align:left;line-height:1.3em"><a href=3D"https://httpd.apa= che.org/docs/2.0/mod/directive-dict.html#Module" style=3D"color:rgb(0,115,1= 99);background-color:inherit" rel=3D"noreferrer" target=3D"_blank">Module:<= /a></th><td style=3D"empty-cells:show;padding:0.1em 0.2em;vertical-align:to= p;line-height:1.3em">core</td></tr></tbody></table><p style=3D"line-height:= 1.3em;margin:0px 0px 1em;padding:0px;color:rgb(0,51,102);font-size:14px">Th= e=C2=A0<code style=3D"font-family:"Courier New",Courier,monospace= ;font-size:1em;color:rgb(40,127,0);background-color:inherit">TimeOut</code>= =C2=A0directive currently defines the amount of time Apache will wait for t= hree things:</p><ol style=3D"color:rgb(0,51,102);font-size:14px"><li style= =3D"line-height:1.3em;margin-top:0.5em">The total amount of time it takes t= o receive a GET request.</li><li style=3D"line-height:1.3em;margin-top:0.5e= m">The amount of time between receipt of TCP packets on a POST or PUT reque= st.</li><li style=3D"line-height:1.3em;margin-top:0.5em">The amount of time= between ACKs on transmissions of TCP packets in responses.</li></ol><p sty= le=3D"line-height:1.3em;margin:0px 0px 1em;padding:0px;color:rgb(0,51,102);= font-size:14px">We plan on making these separately configurable at some poi= nt down the road. The timer used to default to 1200 before 1.2, but has bee= n lowered to 300 which is still far more than necessary in most situations.= It is not set any lower by default because there may still be odd places i= n the code where the timer is not reset when a packet is sent.</p></div></d= iv></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_att= r">On Wed, Apr 23, 2025 at 3:07=E2=80=AFPM Mithun Bhattacharya <<a href= =3D"mailto:[email protected]" rel=3D"noreferrer" target=3D"_blank">mithnb@gm= ail.com</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"= margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-lef= t:1ex"><div dir=3D"auto"><p dir=3D"ltr">You configure timeout at the client= side. Apache is at the server side. Server doesn't have a concept of t= ime it could take days to run and not care.</p><p dir=3D"ltr">mod_perl code= is where you are sending the http return status to make sure the client do= esn't timeout waiting for the server to respond.</p></div><br><div clas= s=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, Apr 23, 202= 5, 2:19=E2=80=AFPM Joseph He <<a href=3D"mailto:[email protected]= " rel=3D"noreferrer" target=3D"_blank">[email protected]</a>> wro= te:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px = 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"= ltr">Thanks, all.<div>Is that Apache timeout controlled by its configuratio= n=C2=A0"Timeout"?=C2=A0</div><div>I don't think it has anythi= ng to do with modPerl. Am I=C2=A0missing something?</div><div>Thanks.</div>= </div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">= On Wed, Apr 23, 2025 at 1:41=E2=80=AFPM Mithun Bhattacharya <<a href=3D"= mailto:[email protected]" rel=3D"noreferrer noreferrer" target=3D"_blank">mi= [email protected]</a>> wrote:<br></div><blockquote class=3D"gmail_quote" st= yle=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padd= ing-left:1ex"><div dir=3D"auto">Timeout happens because of how we handle th= e request. Timeout is basically no response came back. Why that happens is = because we think we want to have a correct response. Unfortunately for long= running requests the correct response shouldn't be via http response c= ode or we face situations like this. Instead reply with a 200 OK immediatel= y and then provide correct status in the message body. Once a response code= /header has been sent timeout won't trigger and you could potentially h= old the connection for hours without a problem.</div><br><div class=3D"gmai= l_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, Apr 23, 2025, 9:32= =E2=80=AFAM Andreas Mock <<a href=3D"mailto:[email protected]" rel=3D"= noreferrer noreferrer" target=3D"_blank">[email protected]</a>> wrote:= <br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8= ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Joseph,<br> <br> your description is very vague, so can only answer on some assumptions:<br> <br> It sounds like a timeout is fired somewhere.<br> <br> Best advice in these situations: Log as many steps as you can. Keep your <b= r> eyes open on TCP/IP and higher level timeouts.<br> <br> Declare only ONE instance responsible for a retry: Either the app server <b= r> calling the dispatcher with several tries or the dispatcher trying for <br> himself. Not both.<br> <br> Best regards<br> Andreas<br> <br> <br> Am 23.04.2025 um 16:21 schrieb Joseph He:<br> > All, good day.<br> ><br> > Here is the issue I have.<br> > My entire application is running on ModPerl/Apache environment.<br> > I send Http::Request with data load from my App server to a dispatch <= br> > server thru LWP::UserAgent, I set the timeout 600 seconds.<br> ><br> > The dispatch server is supposed to manipulate the data and send the <b= r> > data to an external SFTP server. Because the SFTP can fail, it will <b= r> > keep trying up to 4 times with 30 seconds sleep in case that SFTP <br> > connection fails.<br> ><br> > Recently, I found that I uploaded the file twice sometimes. I figured = <br> > out the root cause is that my Dispatch server returns 'failure'= ; at 6 <br> > minutes while it keeps trying to do the SFTP. The App server <br> > received=C2=A0HTTP::Response with error status so it issued another ca= ll to <br> > send data. It turns out I uploaded the identified file twice.<br> ><br> > Anybody has this sort of experience? Why does the dispatch=C2=A0server= <br> > return 'error' while it still processes the data?<br> ><br> > Thanks a lot,<br> > Joseph<br> ><br> </blockquote></div> </blockquote></div> </blockquote></div> </blockquote></div> </blockquote></div> </blockquote></div> --000000000000e4dedd0635057f42--