RE: seeing memory leak in cimprovagt process

We need to re-look at the conclusion drawn Rohini.

I think it is false positive, Though I have not debugged it fully. To be
100% sure, we need to debug it.
Here is the flow, The message to start a cimprovagt takes a reference to
the config properties which are allocated by the cimserver.
So when cimprovagt exits, valgrind thought addresses used by the cimprovagt
were lost and hence possible leak.
Whereas the real allocated address is still used by cimserver and will be
cleanup when cimserver shuts down itself
A simple fix will be to pass the config properties by value, not by
reference. But I don't know how much performance impact. it will have

Pialy, your earlier valgrind log points to your provider code. You can
debug your code, in the meantime.
pegasus team will check if what valgrind complains is  really a issue or a
false positive.

Warm Regards
D>L> Meetei
9538270270
(Embedded image moved to file: pic28591.gif)IBM
IBM India Software Lab, STG
6th Floor, A0202,
EGL -D,Bangalore, India



From:	"Deshpande, Rohini (IESL)" <[email protected]>
To:	Devchandra Leishangthem/India/IBM@IBMIN,
            "[email protected]" <[email protected]>
Cc:	"[email protected]"
            <[email protected]>, "[email protected]"
            <[email protected]>, "[email protected]"
            <[email protected]>, "[email protected]"
            <[email protected]>
Date:	09/14/2012 03:33 PM
Subject:	RE: seeing memory leak in cimprovagt process



Hi Pialy,

The valgrind tests are complete and a bug is logged for the leaks detected
at http://bugzilla.openpegasus.org/show_bug.cgi?id=9398 .

Regards,
Rohini

From: Devchandra Leishangthem [mailto:[email protected]]
Sent: Wednesday, September 12, 2012 2:38 PM
To: [email protected]
Cc: [email protected]; [email protected];
[email protected]; Deshpande, Rohini (IESL);
[email protected]
Subject: RE: seeing memory leak in cimprovagt process

Most probably the library was unloaded. so valgrind misses the symbol
table.
writing a client application testing the functionality of the provider and
running the valgrind on the application might help.

Warm Regards
D>L> Meetei
9538270270
(Embedded image moved to file: pic27128.gif)IBM
IBM India Software Lab, STG
6th Floor, A0202,
EGL -D,Bangalore, India



From:        <[email protected]>
To:        <[email protected]>, <[email protected]>,
<[email protected]>,
Cc:        <[email protected]>, <[email protected]>,
<[email protected]>
Date:        09/12/2012 01:30 PM
Subject:        RE: seeing memory leak in cimprovagt process




Hi,

We have not checked with any other version of OP. We are using OP 2.11.1
version. We have run our provider in a single application and no leak was
detected that time.We will check by running test provider.

We have make debug build of our library and other dependent library in our
code base and Pegasus also.  But then also ?? symbols are coming, not
getting details of it.

Thanks & Regards,
Pialy


From: Deshpande, Rohini (IESL) [mailto:[email protected]]
Sent: Wednesday, September 12, 2012 11:41 AM
To: Pialy Ghosh (WT01 - Manufacturing & Hi Tech); [email protected];
[email protected]
Cc: Anirban Bhattacharyya (WT01 - Manufacturing & Hi Tech); Sandip Mandal
(WT01 - Manufacturing & Hi Tech); [email protected]
Subject: RE: seeing memory leak in cimprovagt process

The log has many “???” where the details are not available. We should get
symbols for these to get more details. Check the details in the report that
I sent to you.

By the log, after Pegasus::BasicProviderManagerRouter::processMessage, the
flow is through defaultProviderManger and then to provider.

Since we did not get similar leaks I’m not sure if this is a cimserver
problem.

Have you encountered a similar problem in previous version of OP? Can you
try using OP and invoking one of the test providers to see if same issue
occurs? If not, it might be problem with your provider.
Have you checked your provider to see if all the memory being allocated is
freed? Maybe adding some log and trace messages in provider might help.

The test is not yet complete on our system. It has never taken so long, but
I’m not disturbing it right now in the hope that it’ll complete soon.

Regards,
Rohini

From: [email protected] [mailto:[email protected]]
Sent: Tuesday, September 11, 2012 4:27 PM
To: Deshpande, Rohini (IESL); [email protected]; [email protected]
Cc: [email protected]; [email protected];
[email protected]
Subject: RE: seeing memory leak in cimprovagt process

Hi Rohini,

Thanks. Attaching the same mail thread where I have attached the Valgrind
report previously.

Regards,
Pialy

From: Deshpande, Rohini (IESL) [mailto:[email protected]]
Sent: Tuesday, September 11, 2012 4:14 PM
To: Pialy Ghosh (WT01 - Manufacturing & Hi Tech); [email protected]
Cc: Anirban Bhattacharyya (WT01 - Manufacturing & Hi Tech); Sandip Mandal
(WT01 - Manufacturing & Hi Tech); [email protected]
Subject: RE: seeing memory leak in cimprovagt process

Hi Pialy,

The tests have not yet completed even after 4 days. I tried getting the
report for the memory leaks till now. These are the two leaks detected.
Hope this helps.

TestParser is reported twice and the cimprovagt multiple times.

I think these will be the final leaks too since the tests are running for
so long. I’ll wait for the tests to complete and report again.

Can you also send us the memory leaks that you are getting to pegasus-l
mailing list?

Regards,
Rohini

From: [email protected] [mailto:[email protected]]
Sent: Tuesday, September 11, 2012 12:49 PM
To: [email protected]; Deshpande, Rohini (IESL)
Cc: [email protected]; [email protected];
[email protected]
Subject: RE: seeing memory leak in cimprovagt process
Importance: High

Hi,

Any update on this. As depending upon your report we have to take a
decision in this issue asap and it’s a critical one for us now.
Looking forward for your support on this. Thanks in advance.

Regards,
Pialy

From: Devchandra L Meetei [mailto:[email protected]]
Sent: Friday, September 07, 2012 3:57 PM
To: Deshpande, Rohini (IESL)
Cc: Pialy Ghosh (WT01 - Manufacturing & Hi Tech); Anirban Bhattacharyya
(WT01 - Manufacturing & Hi Tech)
Subject: Re: seeing memory leak in cimprovagt process

I am really sorry to say that I will not be unable to test Valgrind test
for 2.11 due to ongoing CTC test for 2.12.

We are going to do CTC test for 12 different combinations for System P ,
System X and Zseries, So I wont be having enough time this week.

But for me valgrind test went though within 2 hours and 45 mint.

--Regards
Devchandra
On Fri, Sep 7, 2012 at 12:16 PM, Deshpande, Rohini (IESL) <
[email protected]> wrote:
Checked the system, the valgrind tests were running without SQL. But even
till today the tests were still at TestServer profile tests. I’ve stopped
it and will retstart to see if it finishes this time.

Dev,

Can you try running valgrind tests on 2.11?

Regards,
Rohini

From: Deshpande, Rohini (IESL)
Sent: Thursday, September 06, 2012 10:58 AM
To: Devchandra L Meetei; [email protected]

Cc: [email protected]
Subject: RE: seeing memory leak in cimprovagt process

Oh, ok. I’ll leave it till today to see if it completes. If not, I’ll
disable SQL and start again.

Regards,
Rohini

From: Devchandra L Meetei [mailto:[email protected]]
Sent: Thursday, September 06, 2012 10:55 AM
To: [email protected]
Cc: Deshpande, Rohini (IESL); [email protected]

Subject: Re: seeing memory leak in cimprovagt process

Rohini
running valgrind with SQLLite repository will take horribly long time.
Disabling SQLLite might help in reducing valgrind test.
On Thu, Sep 6, 2012 at 10:49 AM, <[email protected]> wrote:
Hi Rohini,

This issue is high priority one for us. So, now your valgrind report will
be very helpful input for us. Please can you provide this as earliest as
possible.
Thanks in advance.

Regards,
Pialy

From: Deshpande, Rohini (IESL) [mailto:[email protected]]
Sent: Tuesday, September 04, 2012 9:14 AM
To: Pialy Ghosh (WT01 - Manufacturing & Hi Tech); [email protected]

Cc: Anirban Bhattacharyya (WT01 - Manufacturing & Hi Tech)
Subject: RE: seeing memory leak in cimprovagt process

Hi Pialy,

We tried running valgrind tests on 2.11 but the 2.12 tests started before
2.11 could complete. Since we are close to 2.12 release we concentrated on
that.

I’ll start the 2.11 tests again. Apologies for the delay.

Regards,
Rohini

From: [email protected] [mailto:[email protected]]
Sent: Monday, September 03, 2012 4:04 PM
To: Deshpande, Rohini (IESL); [email protected]
Cc: [email protected]
Subject: RE: seeing memory leak in cimprovagt process

Hi Rohini,

Any update on this?  Please can you share the result of it.

Thanks & Regards,
Pialy

From: Deshpande, Rohini (IESL) [mailto:[email protected]]
Sent: Thursday, August 09, 2012 3:41 PM
To: Pialy Ghosh (WT01 - Manufacturing & Hi Tech); [email protected]
Cc: [email protected]; Anirban Bhattacharyya (WT01 - Manufacturing
& Hi Tech)
Subject: RE: seeing memory leak in cimprovagt process

Hi Pialy,

This is done on the current branch (2.12).

Valgrind has run on 2.11 before 2.12 branch was created. We haven’t done it
on 2.11.1.

We can run valgrind on 2.11.1 and check if any leaks are detected with the
build time options that you have mentioned. This will take some time and
we’ll let you know the results after this is complete.

Regards,
Rohini

From: [email protected] [mailto:[email protected]]
Sent: Thursday, August 09, 2012 3:12 PM
To: Deshpande, Rohini (IESL); [email protected]
Cc: [email protected]; [email protected]
Subject: RE: seeing memory leak in cimprovagt process

Hi Rohini,

We are using OP 2.11.1, I am checking it on Rhel5 32 bit Linux Platform.
My build variables are:

PEGASUS_ENABLE_CMPI_PROVIDER_MANAGER=true
PEGASUS_ENABLE_IPV6=true
PEGASUS_ENABLE_SLP=true
PEGASUS_SNIA_EXTENSIONS=true
PEGASUS_PLATFORM=LINUX_IX86_GNU
PEGASUS_HAS_SSL=true
PEGASUS_PAM_AUTHENTICATION=1

In which OP version this report is generated, please can you inform me.




Hi Devchandra,

No significant increase in our application is measured from the heap  and
stack memory usages by the application. Heap memory is increased by 80 KB
maximum and no increment in stack memory usages, so, it’s almost constant.

When we are running cimserver, then we can see cimprovagt process heap
usages is increased from 8876 KB to 22000KB in 18hours.

Thanks & Regards,
Pialy



From: Deshpande, Rohini (IESL) [mailto:[email protected]]
Sent: Thursday, August 09, 2012 2:19 PM
To: Devchandra L Meetei; Pialy Ghosh (WT01 - Manufacturing & Hi Tech)
Cc: [email protected]; Anirban Bhattacharyya (WT01 - Manufacturing
& Hi Tech)
Subject: RE: seeing memory leak in cimprovagt process

On which OP version are you seeing this?

We run nightly builds with valgrind. Here are the results
http://nbat.openpegasus.org/platform.php?id=372&archive=N&sort=0 .

I guess your build time options are different. Can you send us your
environment build variables?

Regards,
Rohini

From: Devchandra L Meetei [mailto:[email protected]]
Sent: Thursday, August 09, 2012 12:54 PM
To: [email protected]
Cc: [email protected]; [email protected]
Subject: Re: seeing memory leak in cimprovagt process

Thanks for all of your responses.
If we simply start and stop the server, then also we can see memory leak
from Valgrind report. To separate out the memory leak area, we created a
single application with our provider code. And by executing the application
with Valgrind, it’s not reporting any memory leak. We put our standalone
application  overnight and there is no significant increase in memory
usage.

How are you measuring exactly? No Significant increase? would appreciate a
quantification.

But when we run cimserver overnight, not doing any operation on it, we can
see increase in memory.
So, by summarizing this, can we suspect some memory leak in the serverside?

How you find that the memory being leaked is from cimserver?
If there is some serverside leak, it would have been detected by the
valgrind test, which have been runring on daily nightly build testing.


Please give your input on this asap.
On Thu, Aug 9, 2012 at 12:39 PM, <[email protected]> wrote:
Hi All,

Thanks for all of your responses.
If we simply start and stop the server, then also we can see memory leak
from Valgrind report. To separate out the memory leak area, we created a
single application with our provider code. And by executing the application
with Valgrind, it’s not reporting any memory leak. We put our standalone
application  overnight and there is no significant increase in memory
usage.

But when we run cimserver overnight, not doing any operation on it, we can
see increase in memory.
So, by summarizing this, can we suspect some memory leak in the serverside?


Please give your input on this asap.

Thanks & Regards,
Pialy



From: Karl Schopmeyer [mailto:[email protected]]
Sent: Wednesday, August 08, 2012 1:54 AM
To: Pialy Ghosh (WT01 - Manufacturing & Hi Tech); [email protected]
Cc: Anirban Bhattacharyya (WT01 - Manufacturing & Hi Tech)
Subject: RE: seeing memory leak in cimprovagt process

Note that there are still some memory issues in Pegasus shutdown so that
this may be some of what you are seeing.  To separate these out, simply
start and stop the server. with valgrind running. That will give you a clue
what things valgrind reports with without your provider being called.

Second, you might be better to test the provider in out-of-process mode
where you are running valgrind only on the particular provider and OOP
agent to determine if the error is from the provider or agent vs. the
server.
If you have issues setting this up, there is an example of a script to run
valgrind on just cimprovagt by replacing cimprovagt executable with a
script like the following


#!/bin/sh
## Original Author: Tim Potter
# move cimprovagt to cimprovagt.real
# move this file to cimprovagt
#
# mv /usr/sbin/cimprovagt /usr/sbin/cimprovagt.real
# cp cimprovagt.wrapper /usr/sbin/cimprovagt
## By default the script doesn't call valgrind - enable it by
## creating a semaphore file of the form /tmp/$MODULE.valgrind where module
## is the module name in the output of "cimprovider -l".
## Or create a file /tmp/LogAll.valgrind which valgrinds all providers.
module=$5
VALGRIND_ARGS="--leak-check=yes --trace-children=yes
--log-file=/tmp/$module.valgrind"
if [ -e /tmp/$module.valgrind -o -e /tmp/LogAll.valgrind ]; then
       exec /usr/bin/valgrind $VALGRIND_ARGS \
     /usr/sbin/cimprovagt.real "$@"
else
       exec /usr/sbin/cimprovagt.real "$@"
fi

Karl

Thanks for your message at 06:14 AM 8/7/2012. Your message was:
Hi All,

Thanks for all of your response.
We have configured cimserver ?forceProviderProcesses=false?, so that all
the provider processes will be in the same context.
Now, we run ?Valgrind? for cimserver process. And then, we have generated
one event for our provider and stop the cimserver process after that.

Then from ?Valgrind? memory leak report, we can see many definitely lost
memory issue are coming from Pegasus. Please check the attached file for
details of it. Is this a known issue?


Please give your valuable input. Thanks in advance.

Regards,
Pialy



From: Devchandra Leishangthem [ mailto:[email protected]]
Sent: Friday, August 03, 2012 9:52 PM
To: Pialy Ghosh (WT01 - Manufacturing & Hi Tech)
Cc: Anirban Bhattacharyya (WT01 - Product Engineering Services);
[email protected]
Subject: Re: seeing memory leak in cimprovagt process

Is the leak seen in a particular cimprovagt or for all the provider
processes?
If It is seen in some particular agent processes, Then most probably it is
leak in the corresponding provider.
See if running under a memory debugger helps for that particular agt
processes.

Warm Regards
D>L> Meetei
9538270270
(Embedded image moved to file: pic18623.gif)IBM
IBM India Software Lab, STG
6th Floor, A0202,
EGL -D,Bangalore, India



From:        <[email protected]>
To:        <[email protected]>,
Cc:        <[email protected]>
Date:        08/03/2012 05:37 PM
Subject:        seeing memory leak in cimprovagt process





Hi,

We are seeing some memory leak in cimprovagt process. We are running open
Pegasus, cimserver in a windows 64 bit OS. Our provider is running on
cimserver.
After few days long run we can see, memory usages is increased. I have
searched in internet and find out same kind of problem is mentioned by
others also. So, just wanted to know, is this a known issue in Pegasus? If
not, then we will come to know that, issue is coming from our provider.

Your valuable input will be very helpful for us to analyze the issue.

Thanks & Regards,
Pialy



Please do not print this email unless it is absolutely necessary.

The information contained in this electronic message and any attachments to
this message are intended for the exclusive use of the addressee(s) and may
contain proprietary, confidential or privileged information. If you are not
the intended recipient, you should not disseminate, distribute or copy this
e-mail. Please notify the sender immediately and destroy all copies of this
message and any attachments.

WARNING: Computer viruses can be transmitted via email. The recipient
should check this email and any attachments for the presence of viruses.
The company accepts no liability for any damage caused by any virus
transmitted by this email.

www.wipro.com

Please do not print this email unless it is absolutely necessary.

The information contained in this electronic message and any attachments to
this message are intended for the exclusive use of the addressee(s) and may
contain proprietary, confidential or privileged information. If you are not
the intended recipient, you should not disseminate, distribute or copy this
e-mail. Please notify the sender immediately and destroy all copies of this
message and any attachments.

WARNING: Computer viruses can be transmitted via email. The recipient
should check this email and any attachments for the presence of viruses.
The company accepts no liability for any damage caused by any virus
transmitted by this email.

www.wipro.com


Karl Schopmeyer                   Inova Development Inc.
305 Spring Creek Village, Suite 475 -  Dallas TX, 75248 USA
EMAIL: [email protected]       FAX: 1-972-239-0326
Phone 1-972-814-5581
Skype: kschopmeyer         Skype Phone: (214) 556-5971


Please do not print this email unless it is absolutely necessary.


The information contained in this electronic message and any attachments to
this message are intended for the exclusive use of the addressee(s) and may
contain proprietary, confidential or privileged information. If you are not
the intended recipient, you should not disseminate, distribute or copy this
e-mail. Please notify the sender immediately and destroy all copies of this
message and any attachments.


WARNING: Computer viruses can be transmitted via email. The recipient
should check this email and any attachments for the presence of viruses.
The company accepts no liability for any damage caused by any virus
transmitted by this email.


www.wipro.com



Please do not print this email unless it is absolutely necessary.


The information contained in this electronic message and any attachments to
this message are intended for the exclusive use of the addressee(s) and may
contain proprietary, confidential or privileged information. If you are not
the intended recipient, you should not disseminate, distribute or copy this
e-mail. Please notify the sender immediately and destroy all copies of this
message and any attachments.


WARNING: Computer viruses can be transmitted via email. The recipient
should check this email and any attachments for the presence of viruses.
The company accepts no liability for any damage caused by any virus
transmitted by this email.


www.wipro.com


Please do not print this email unless it is absolutely necessary.


The information contained in this electronic message and any attachments to
this message are intended for the exclusive use of the addressee(s) and may
contain proprietary, confidential or privileged information. If you are not
the intended recipient, you should not disseminate, distribute or copy this
e-mail. Please notify the sender immediately and destroy all copies of this
message and any attachments.


WARNING: Computer viruses can be transmitted via email. The recipient
should check this email and any attachments for the presence of viruses.
The company accepts no liability for any damage caused by any virus
transmitted by this email.


www.wipro.com


Please do not print this email unless it is absolutely necessary.


The information contained in this electronic message and any attachments to
this message are intended for the exclusive use of the addressee(s) and may
contain proprietary, confidential or privileged information. If you are not
the intended recipient, you should not disseminate, distribute or copy this
e-mail. Please notify the sender immediately and destroy all copies of this
message and any attachments.


WARNING: Computer viruses can be transmitted via email. The recipient
should check this email and any attachments for the presence of viruses.
The company accepts no liability for any damage caused by any virus
transmitted by this email.


www.wipro.com




The information contained in this electronic message and any attachments to
this message are intended for the exclusive use of the addressee(s) and may
contain proprietary, confidential or privileged information. If you are not
the intended recipient, you should not disseminate, distribute or copy this
e-mail. Please notify the sender immediately and destroy all copies of this
message and any attachments.


WARNING: Computer viruses can be transmitted via email. The recipient
should check this email and any attachments for the presence of viruses.
The company accepts no liability for any damage caused by any virus
transmitted by this email.


www.wipro.com


Please do not print this email unless it is absolutely necessary.


The information contained in this electronic message and any attachments to
this message are intended for the exclusive use of the addressee(s) and may
contain proprietary, confidential or privileged information. If you are not
the intended recipient, you should not disseminate, distribute or copy this
e-mail. Please notify the sender immediately and destroy all copies of this
message and any attachments.


WARNING: Computer viruses can be transmitted via email. The recipient
should check this email and any attachments for the presence of viruses.
The company accepts no liability for any damage caused by any virus
transmitted by this email.


www.wipro.com


Please do not print this email unless it is absolutely necessary.


The information contained in this electronic message and any attachments to
this message are intended for the exclusive use of the addressee(s) and may
contain proprietary, confidential or privileged information. If you are not
the intended recipient, you should not disseminate, distribute or copy this
e-mail. Please notify the sender immediately and destroy all copies of this
message and any attachments.


WARNING: Computer viruses can be transmitted via email. The recipient
should check this email and any attachments for the presence of viruses.
The company accepts no liability for any damage caused by any virus
transmitted by this email.


www.wipro.com


The information contained in this electronic message and any attachments to
this message are intended for the exclusive use of the addressee(s) and may
contain proprietary, confidential or privileged information. If you are not
the intended recipient, you should not disseminate, distribute or copy this
e-mail. Please notify the sender immediately and destroy all copies of this
message and any attachments.


WARNING: Computer viruses can be transmitted via email. The recipient
should check this email and any attachments for the presence of viruses.
The company accepts no liability for any damage caused by any virus
transmitted by this email.


www.wipro.com