Re: Excess memory usage growth for a process

Its all later ... if its broken right now you can still use it right now!

On Thu, Jul 30, 2009 at 2:03 PM, Eddie Lascu<[email protected]> wrote:
> Thank you, Guys. I will look into that as well. Do I need a special start=
 of
> the process if I am to use ADPlus or I can attach to the process later?
>
>
> -----Original Message-----
> From: Discussion of advanced .NET topics.
> [mailto:[email protected]] On Behalf Of Greg Young
> Sent: Thursday, July 30, 2009 1:52 PM
> To: [email protected]
> Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a process
>
> Yep exactly what ADPlus is meant for! you can even do this right
> within ADPlus :)
>
> On Thu, Jul 30, 2009 at 1:49 PM, Wilson, Phil<[email protected]>
> wrote:
>> Not answering this particular question, but I believe you could hook all
> this together to automatically get that dump when the memory gets large i=
f
> it's rare enough that you don't want to sit and look at it all day. Perfm=
on
> has triggers that can fire off a script, and you can fire off the debuggi=
ng
> script ADPlus to dump your process with the -hang option.
>>
>> Phil Wilson
>>
>>
>> -----Original Message-----
>> From: Discussion of advanced .NET topics.
> [mailto:[email protected]] On Behalf Of Eddie Lascu
>> Sent: Wednesday, July 29, 2009 11:40 AM
>> To: [email protected]
>> Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a process
>>
>> Based on my preliminary reading of what this has to offer, it seems you
> can
>> integrate it with your VS IDE (not an option for me as my processes are
>> already deployed) or use it via WinDbg, but you need to look at the debu=
g
>> version or the runtime version that has the .pdb file (not an option
> either
>> as I only have the sole runtime file deployed, but much easily fixable).
> Is
>> this statement accurate?
>>
>> Thanks,
>> Eddie
>>
>> -----Original Message-----
>> From: Discussion of advanced .NET topics.
>> [mailto:[email protected]] On Behalf Of Adam Tuliper
>> Sent: Wednesday, July 29, 2009 1:24 PM
>> To: [email protected]
>> Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a process
>>
>>
> http://blogs.msdn.com/alejacma/archive/2009/06/30/sos-cheat-sheet-net-2-0=
-3-
>> 0-3-5.aspx
>>
>> !DumpHeap -min 85000
>> !EEHeap -GC; !DumpHeap <LOH begin> <LOH allocated>
>> =A0 =A0-> Shows all objects in Large Object Heap (LOH/Gen 3).
>>
>>
>>
>>
>> On Wed, Jul 29, 2009 at 1:06 PM, Greg Young <[email protected]>
> wrote:
>>
>>> Yes with ants you have to wait for the problem to happen.
>>>
>>> if you google SOS you can find some tutorials on it (it is a separate
>>> download from windbg ... it can also be used in visual studio!). John
>>> Robbins also has a great book covering the subject that I highly
>>> recommend.
>>>
>>> Cheers,
>>>
>>> greg
>>>
>>> On Wed, Jul 29, 2009 at 1:01 PM, Eddie Lascu<[email protected]> wrote=
:
>>> > Shoot, I didn't know SOS can do that for you. Can you share more abou=
t
>>> this
>>> > tool (a link to download it)? Is it part of the WinDbg tool that come=
s
>>> from
>>> > Microsoft? It's true that with ANTS I am at the mercy of the system,
>>> hoping
>>> > it will go nuts again and will have the memory size shut up crazy hig=
h.
>>> >
>>> > Cheers,
>>> > Eddie
>>> >
>>> >
>>> >
>>> > -----Original Message-----
>>> > From: Discussion of advanced .NET topics.
>>> > [mailto:[email protected]] On Behalf Of Greg Young
>>> > Sent: Wednesday, July 29, 2009 12:46 PM
>>> > To: [email protected]
>>> > Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a proce=
ss
>>> >
>>> > The wonderful thing about SOS is you dont need to run a profiler ...
>>> > when it happens take a memory dump, look whats actually happening in
>>> > the running app, is your LOH fragmented? Otherwise you are just
>>> > sticking your thumb up your butt and saying it will rain next
>>> > thursday.
>>> >
>>> > On Wed, Jul 29, 2009 at 12:43 PM, Eddie Lascu<[email protected]>
>>> wrote:
>>> >> I am using Reg Gate ANTS to profile the app, but it may take too muc=
h
>> to
>>> >> happen again. Before the spike I saw last weekend, the processes ran
>> for
>>> > 3-4
>>> >> months with no significant change in the size of memory used.
>>> >> Will look into Process.WorkingSet.
>>> >>
>>> >> Thanks,
>>> >> Eddie
>>> >>
>>> >>
>>> >>
>>> >> -----Original Message-----
>>> >> From: Discussion of advanced .NET topics.
>>> >> [mailto:[email protected]] On Behalf Of Greg Youn=
g
>>> >> Sent: Wednesday, July 29, 2009 12:38 PM
>>> >> To: [email protected]
>>> >> Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a
> process
>>> >>
>>> >> Task manager shows you working set not total memory used but yes the
>>> >> process class can help you with that (Process.WorkingSet as an
>>> >> example).
>>> >>
>>> >> Have you verified the LOH being the problem by using SOS?
>>> >>
>>> >> Cheers,
>>> >>
>>> >> Greg
>>> >>
>>> >> On Wed, Jul 29, 2009 at 12:33 PM, Eddie Lascu<[email protected]>
>>> wrote:
>>> >>> Hi guys,
>>> >>>
>>> >>> In light of this article:
>>> >>>
>>> >>>
>>> >>
>>> >
>>>
>>
> http://www.simple-talk.com/dotnet/.net-framework/the-dangers-of-the-large=
-ob
>>> >> ject-heap/
>>> >>>
>>> >>> I am no longer sure what I have is a memory leak. I could see how
> some
>>> of
>>> >> the things that happen in my application match the scenarios describ=
ed
>>> in
>>> >> the above article. Hence, I am ready to think about a workaround. My
>>> >> services are set up to run automatically and restart in case they
>> crash.
>>> >> While they never crash, I could leverage that feature and use it to
> fix
>>> > this
>>> >> excess memory usage that I experience every once in a blue moon. Wha=
t
> I
>>> >> thought was to monitor the memory used and terminate the application
> in
>>> > case
>>> >> it exceeds a set threshold. The Windows Service Manager will then
>>> restart
>>> >> the service and all the memory will be reclaimed by the OS.
>>> >>>
>>> >>> My question at this point is whether the .NET Framework has some
>> metric
>>> >> classes that can help monitor the total amount of memory allocated
>>> within
>>> > a
>>> >> process. Surely System.Diagnostics must have something like that. Is
>>> > anyone
>>> >> aware of a way to find out what memory is used by a process, just li=
ke
>>> the
>>> >> Task Manager does?
>>> >>>
>>> >>> Cheers,
>>> >>> Eddie
>>> >>>
>>> >>>
>>> >>> -----Original Message-----
>>> >>> From: Discussion of advanced .NET topics.
>>> >> [mailto:[email protected]] On Behalf Of Shawn
>>> > Wildermuth
>>> >>> Sent: Monday, July 27, 2009 12:49 PM
>>> >>> To: [email protected]
>>> >>> Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a
>> process
>>> >>>
>>> >>> Also, your 'using' statement stuff should help, but are you sure th=
at
>>> all
>>> >>> your object (XML classes and db connection classes) that support
>>> >> IDisposable
>>> >>> are being cleaned up?
>>> >>>
>>> >>> Thanks,
>>> >>>
>>> >>> Shawn Wildermuth
>>> >>> http://wildermuth.com
>>> >>> https://agilitrain.com
>>> >>> Microsoft MVP (C#), MCSD.NET, Author and Speaker
>>> >>>
>>> >>> The Silverlight Tour is coming to a city near you!
>>> >>>
>>> >>>
>>> >>> -----Original Message-----
>>> >>> From: Discussion of advanced .NET topics.
>>> >>> [mailto:[email protected]] On Behalf Of John
> Warner
>>> >>> Sent: Monday, July 27, 2009 12:28 PM
>>> >>> To: [email protected]
>>> >>> Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a
>> process
>>> >>>
>>> >>> I don't disagree with you just note that before 'some change' worki=
ng
>>> OK
>>> >>> no memory leaks, after change, memory leak. We both know your code
> did
>>> > not
>>> >>> recompile itself to include a memory leak ...
>>> >>> That reminds has the framework/runtimes been updated/patched?
>>> >>>
>>> >>> John Warner
>>> >>>
>>> >>>
>>> >>>> -----Original Message-----
>>> >>>> From: Discussion of advanced .NET topics. [mailto:ADVANCED-
>>> >>>> [email protected]] On Behalf Of Eddie Lascu
>>> >>>> Sent: Monday, July 27, 2009 12:00 PM
>>> >>>> To: [email protected]
>>> >>>> Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a
>>> >>>> process
>>> >>>>
>>> >>>> Hi John,
>>> >>>>
>>> >>>> I could check that with our system administrators. Do you think th=
at
>>> can
>>> >>> be
>>> >>>> a possible explanation?
>>> >>>> I just talked with one of the DB Admin and she said she made some
>>> >>> changes
>>> >>>> on
>>> >>>> some Stored Produces, but nothing as dramatic as a patch update. S=
he
>>> >>> also
>>> >>>> said that the drivers for the network cards were updated last week=
,
>>> but
>>> >>> I
>>> >>>> can't see how that can affect the memory used by my processes.
>>> >>>>
>>> >>>> Thanks,
>>> >>>> Eddie
>>> >>>>
>>> >>>>
>>> >>>> -----Original Message-----
>>> >>>> From: Discussion of advanced .NET topics.
>>> >>>> [mailto:[email protected]] On Behalf Of
>>> >>>> John Warner
>>> >>>> Sent: Monday, July 27, 2009 11:49 AM
>>> >>>> To: [email protected]
>>> >>>> Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a
>>> >>>> process
>>> >>>>
>>> >>>> Not what you are seeking, but has anything on the server changed? =
An
>>> >>>> Oracle patch or something of that nature?
>>> >>>>
>>> >>>> John Warner
>>> >>>>
>>> >>>>
>>> >>>> > -----Original Message-----
>>> >>>> > From: Discussion of advanced .NET topics. [mailto:ADVANCED-
>>> >>>> > [email protected]] On Behalf Of Eddie Lascu
>>> >>>> > Sent: Monday, July 27, 2009 11:33 AM
>>> >>>> > To: [email protected]
>>> >>>> > Subject: [ADVANCED-DOTNET] Excess memory usage growth for a
>>> >>>> process
>>> >>>> >
>>> >>>> > Hi guys,
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> > I need your help with an issue I am having right now. I have
>> several
>>> >>>> windows
>>> >>>> > services deployed. These services gather some XML files from the
>>> >>>> internet
>>> >>>> > and then parse the content and save the information in the
>> database.
>>> >>> The
>>> >>>> > database is Oracle 11g, so the access is done through ODP.NET. I
>>> had
>>> >>>> some
>>> >>>> > issues before (and some of you with good memory may recall my
>>> previous
>>> >>>> > questions) where I noticed the memory used by these services
>> growing
>>> >>>> very
>>> >>>> > high. At the time, I understood that some classes in the ODP.NET
>>> >>> library
>>> >>>> use
>>> >>>> > some JAVA code and they are known for not cleaning properly afte=
r
>>> >>> them.
>>> >>>> As a
>>> >>>> > result I rewrote my code in such a way that every dynamic
>> allocation
>>> >>> of
>>> >>>> an
>>> >>>> > object from the ODP.NET library was encompassed in an "using"
>>> >>> statement,
>>> >>>> > like this:
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> > using (OracleCommand objDbCommand =3D new
>>> >>>> > OracleCommand(strSqlSelectStatement,
>>> >>>> > objConnection))
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> > All was fine for a very long time. All my services kept a consta=
nt
>>> >>> size
>>> >>>> when
>>> >>>> > it came to the total memory used, a size that hovered around 55-=
60
>>> MB.
>>> >>>> This
>>> >>>> > was the case for months. At the beginning I checked the size
> almost
>>> on
>>> >>> a
>>> >>>> > daily basis. After a while, my confidence grew and I only checke=
d
>> it
>>> >>>> once a
>>> >>>> > week or so. Every time I looked at the size of all my processes =
(I
>>> had
>>> >>> 4
>>> >>>> > that followed pretty much the same approach on dealing with thos=
e
>>> >>> Oracle
>>> >>>> > objects) the size was around that 55-60 MB mark.
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> > Now, all of a sudden, this last weekend, the size for some of th=
e
>>> >>>> services
>>> >>>> > shut up to about 690 MB, despite the fact the service was runnin=
g
>>> with
>>> >>>> the
>>> >>>> > exact same code as it did a month ago. This represents a potenti=
al
>>> >>> fatal
>>> >>>> > problem for the server that can get to a point where it can no
>>> longer
>>> >>>> > allocate memory, so I need to find a way to understand what the
>> heck
>>> >>> is
>>> >>>> > going on. Unfortunately, I do not have any metrics implemented i=
n
>> my
>>> >>>> > application to log the total amount of memory used. That would
> have
>>> >>> been
>>> >>>> a
>>> >>>> > very good indication on when exactly did this surge started to
>>> happen.
>>> >>> I
>>> >>>> > could have checked the Event Viewer to see if the server got hit
> by
>>> a
>>> >>>> > nuclear bomb or something on that precise moment. I really don't
>> get
>>> >>>> this. I
>>> >>>> > mean that's why we moved to .NET in the first place, to be rid o=
f
>>> such
>>> >>>> > concerns, like making sure all dynamically allocated objects are
>>> then
>>> >>>> > destroyed.
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> > Can anyone give me an idea on how to investigate this issue? Are
>>> there
>>> >>>> tools
>>> >>>> > out there that could be used to pick at the memory allocated by =
a
>>> >>>> process
>>> >>>> > and see what are those objects that take up so much =A0memory? F=
or
>> the
>>> >>>> time
>>> >>>> > being, I have the processes running, but two of them are at or
>> close
>>> >>> to
>>> >>>> 690
>>> >>>> > MB and I don't know how long will I be able to keep them running=
.
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> > If you had an issue like this, what would you try to do?
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> > Any suggestion will be greatly appreciated.
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> > TIA,
>>> >>>> >
>>> >>>> > Eddie
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> >>>> > View archives and manage your subscription(s) at
>>> >>>> > http://peach.ease.lsoft.com/archives
>>> >>>>
>>> >>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> >>>> View archives and manage your subscription(s) at
>>> >>>> http://peach.ease.lsoft.com/archives
>>> >>>>
>>> >>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> >>>> View archives and manage your subscription(s) at
>>> >>>> http://peach.ease.lsoft.com/archives
>>> >>>
>>> >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> >>> View archives and manage your subscription(s) at
>>> >>> http://peach.ease.lsoft.com/archives
>>> >>>
>>> >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> >>> View archives and manage your subscription(s) at
>>> >> http://peach.ease.lsoft.com/archives
>>> >>>
>>> >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> >>> View archives and manage your subscription(s) at
>>> >> http://peach.ease.lsoft.com/archives
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Les erreurs de grammaire et de syntaxe ont =E9t=E9 incluses pour m'a=
ssurer
>>> >> de votre attention
>>> >>
>>> >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> >> View archives and manage your subscription(s) at
>>> >> http://peach.ease.lsoft.com/archives
>>> >>
>>> >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> >> View archives and manage your subscription(s) at
>>> > http://peach.ease.lsoft.com/archives
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > Les erreurs de grammaire et de syntaxe ont =E9t=E9 incluses pour m'as=
surer
>>> > de votre attention
>>> >
>>> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> > View archives and manage your subscription(s) at
>>> > http://peach.ease.lsoft.com/archives
>>> >
>>> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> > View archives and manage your subscription(s) at
>>> http://peach.ease.lsoft.com/archives
>>> >
>>>
>>>
>>>
>>> --
>>> Les erreurs de grammaire et de syntaxe ont =E9t=E9 incluses pour m'assu=
rer
>>> de votre attention
>>>
>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> View archives and manage your subscription(s) at
>>> http://peach.ease.lsoft.com/archives
>>>
>>
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> View archives and manage your subscription(s) at
>> http://peach.ease.lsoft.com/archives
>>
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> View archives and manage your subscription(s) at
> http://peach.ease.lsoft.com/archives
>>
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> View archives and manage your subscription(s) at
> http://peach.ease.lsoft.com/archives
>>
>
>
>
> --
> Les erreurs de grammaire et de syntaxe ont =E9t=E9 incluses pour m'assure=
r
> de votre attention
>
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> View archives and manage your subscription(s) at
> http://peach.ease.lsoft.com/archives
>
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> View archives and manage your subscription(s) at http://peach.ease.lsoft.=
com/archives
>



--=20
Les erreurs de grammaire et de syntaxe ont =E9t=E9 incluses pour m'assurer
de votre attention

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
View archives and manage your subscription(s) at http://peach.ease.lsoft.com/archives