Five Day Run

David Hakim <dhakim-Gkm/TONP9n1Wk0Htik3J/[email protected]> Sat, 25 Jan 2003 14:21:26 -0500
Newsgroups gmane.comp.lang.moto.devel
Message-ID <[email protected]>
	Well after 5 days of 20-30K pageviews per day webcodex's apache 
crashed :( . We had some indication this was coming. Cory had 
discovered about 42 active MySQL connections on wednesday night, but 
only 20 or so httpd processes and only 3 connections active at any one 
time :) . It seemed that in rare cases connections were being leaked by 
moto page views. There were some other indications of problems in the 
logs of apache processes refusing to be killed. When it finally went 
down the logs started filling up with allocation failures and thats all 
she wrote ...

	I'm writing about this sad tale here because there are a important 
lessons in the fix I implemented that all moto developers should know. 
It turns out great evil lurked in src/mmc/mgen.properties, the shell 
used by mmc to build compiled apache DSOs. On line 630, right around 
the end of the CATCH_ALL block there was a 'return 0'. This was real 
bad news. What it meant was that in the case of uncaught exceptions, 
the memory for the page view that caused the exception would not be 
cleaned up.

	More than just memory leaks, it meant destructors would not be 
executed, i.e. MySQL connections would not be closed. Finally it meant 
that the exception stack for the process would not properly be popped. 
Which brings me to the first point all moto core and extension 
developers should take to heart :

Never Never Never return out of a TRY ... END_TRY block .
	
	Note that this doesn't apply when your coding IN moto. A return / 
break / or continue within a try block in moto code works just fine. 
But when your dealing with the C macros ... just don't do it :)

	Now I don't know for sure yet which pages of which application caused 
the exceptions but I know how I'm going to find out. I have yet to 
implement this part of this fix in the repository but what will happen 
is that uncaught exceptions -- an where they were thrown from -- will 
get logged in the apache error log.

	Of course it may not have been an exception at all. A signal could 
have been raised during page execution. Calling atoi(null) in moto code 
right now causes a segfault for instance. This would also cause page 
execution to stop before cleanup. So I've made the 
SignalsThrowExceptions configuration option available to compiled moto 
modules as well. If this option is turned on in httpd.conf then 
whenever a SIGSEGV or a SIGBUS is raised during page execution the 
signal handler will throw a SignalException and everything will work 
itself out. All extension authors should probably turn this config 
option on while they are debugging their extensions (unless of course 
you are using gdb to go through httpd core dumps) .

	I have a couple questions here though

	1) Should the SignalsThrowExceptions config option be ON by default 
everywhere ?
	2) What other signals apart from SIGSEGV and SIGBUS should I be 
catching and wrapping with exceptions ?
	3) I believe under certain circumstances the Apache parent process 
sends signals to its child processes to kill them, I probably should 
not wrap these :) . Does anyone know what those signals are ?

-Dave