[Mondo-devel] mindi/mondo virtual memory

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Mondo-devel] mindi/mondo virtual memory

Michael White-2
Hello all,

I'm seeing an issue on my RHEL 6.8 serves where something is acquiring virtual memory but doesn't appear to release it.
My virtual memory graph stair steps up each day (see attachment) and it appears its happing during when mondoarchive runs. Though I don't know how to prove or dis-prove.
 
Could mindi/mondo not be releasing virtual memory when done?

Anyone have any thoughts on how this can be figured out?

Mindi v3.0.2-r3578
Mondo v 3.2.2-r3578

Thanks.
Michael White

****************************************************************************************************
CONFIDENTIALITY NOTICE
This communication and any files or attachments transmitted with it may
contain information that is confidential, privileged and exempt from disclosure under
applicable law. It is intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient, you are hereby notified that any use,
dissemination, copying, or action taken in reliance on the contents of this
communication is strictly prohibited. If you have received this communication in error,
please promptly delete it and notify the sender of the delivery error by e-mail so that
the appropriate action may be taken to avoid troubling you further.
Thank you for your cooperation. V2
****************************************************************************************************



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Mondo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mondo-devel

virtmem.PNG (37K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Mondo-devel] mindi/mondo virtual memory

Bruno Cornec-4
Michael White said on Fri, Jun 02, 2017 at 04:21:25PM +0000:
>I'm seeing an issue on my RHEL 6.8 serves where something is acquiring virtual memory but doesn't appear to release it.
>My virtual memory graph stair steps up each day (see attachment) and it appears its happing during when mondoarchive runs. Though I don't know how to prove or dis-prove.
>
>Could mindi/mondo not be releasing virtual memory when done?

I don't think this is due to mondoarchive, except if the command doesn't end correctly. Check with a ps auxww if you have remaining processes called that way. If not, it could be a command launched by mondoarchive/mindi which doesn't end correctly itself. Again the ps would help, as well as the log from the archiving run.

Bruno.
--
HPE EMEA EG FLOSS Technology Strategist http://www.hpe.com/engage/opensource
Open Source Profession, WW Linux Community Lead    http://github.com/bcornec   
FLOSS projects:    http://mondorescue.org         http://project-builder.org
Musique ancienne?   http://www.musique-ancienne.org  http://www.medieval.org

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Mondo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mondo-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Mondo-devel] mindi/mondo virtual memory

Michael White-2
Thanks for the reply Bruno.
I'm not following you on what to look for with the command "ps auxww".

Over the weekend I stopped Mondoarchive from running and the memory graphs flatted out until MondoArchive ran Monday night and the graph stepped up like the example I sent.
I'll be back in the office Wednesday and I'll gather the Midi/Mondo logs and attach.

Thanks again for the groups help.

Michael White

-----Original Message-----
From: Bruno Cornec [mailto:[hidden email]]
Sent: Tuesday, June 06, 2017 10:13 AM
To: Mondo mailing list <[hidden email]>
Subject: Re: [Mondo-devel] mindi/mondo virtual memory

Michael White said on Fri, Jun 02, 2017 at 04:21:25PM +0000:
>I'm seeing an issue on my RHEL 6.8 serves where something is acquiring virtual memory but doesn't appear to release it.
>My virtual memory graph stair steps up each day (see attachment) and it appears its happing during when mondoarchive runs. Though I don't know how to prove or dis-prove.
>
>Could mindi/mondo not be releasing virtual memory when done?

I don't think this is due to mondoarchive, except if the command doesn't end correctly. Check with a ps auxww if you have remaining processes called that way. If not, it could be a command launched by mondoarchive/mindi which doesn't end correctly itself. Again the ps would help, as well as the log from the archiving run.

Bruno.
--
HPE EMEA EG FLOSS Technology Strategist http://www.hpe.com/engage/opensource
Open Source Profession, WW Linux Community Lead    http://github.com/bcornec   
FLOSS projects:    http://mondorescue.org         http://project-builder.org
Musique ancienne?   http://www.musique-ancienne.org  http://www.medieval.org

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________
Mondo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mondo-devel

****************************************************************************************************
CONFIDENTIALITY NOTICE
This communication and any files or attachments transmitted with it may
contain information that is confidential, privileged and exempt from disclosure under
applicable law. It is intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient, you are hereby notified that any use,
dissemination, copying, or action taken in reliance on the contents of this
communication is strictly prohibited. If you have received this communication in error,
please promptly delete it and notify the sender of the delivery error by e-mail so that
the appropriate action may be taken to avoid troubling you further.
Thank you for your cooperation. V2
****************************************************************************************************



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Mondo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mondo-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Mondo-devel] mindi/mondo virtual memory

Bruno Cornec-4
Michael White said on Tue, Jun 06, 2017 at 04:50:51PM +0000:
>I'm not following you on what to look for with the command "ps auxww".

Well, I try to understand which process can be eating memory. That command could help.

>Over the weekend I stopped Mondoarchive from running and the memory graphs flatted out until MondoArchive ran Monday night and the graph stepped up like the example I sent.

How long is mondoarchive running for you ?
Of course, when it's running, it's consuming more memory, even if I try to do my best to avoid unnecessary consumption.
But when the command ends, memory should come back again. Is that the case ?

Bruno.
--
HPE EMEA EG FLOSS Technology Strategist http://www.hpe.com/engage/opensource
Open Source Profession, WW Linux Community Lead    http://github.com/bcornec   
FLOSS projects:    http://mondorescue.org         http://project-builder.org
Musique ancienne?   http://www.musique-ancienne.org  http://www.medieval.org

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Mondo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mondo-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Mondo-devel] mindi/mondo virtual memory

Michael White-2
Bruno here are the logs you ask for as you can see in the graph when I don't run mondoarchive the virtual memory flattens out, as soon as I run it vmem starts stepping back up again.
I scanned through the logs but I don't see anything jumping out at me.

Thanks
Michael White

-----Original Message-----
From: Bruno Cornec [mailto:[hidden email]]
Sent: Tuesday, June 06, 2017 11:56 AM
To: Mondo mailing list <[hidden email]>
Subject: Re: [Mondo-devel] mindi/mondo virtual memory

Michael White said on Tue, Jun 06, 2017 at 04:50:51PM +0000:
>I'm not following you on what to look for with the command "ps auxww".

Well, I try to understand which process can be eating memory. That command could help.

>Over the weekend I stopped Mondoarchive from running and the memory graphs flatted out until MondoArchive ran Monday night and the graph stepped up like the example I sent.

How long is mondoarchive running for you ?
Of course, when it's running, it's consuming more memory, even if I try to do my best to avoid unnecessary consumption.
But when the command ends, memory should come back again. Is that the case ?

Bruno.
--
HPE EMEA EG FLOSS Technology Strategist http://www.hpe.com/engage/opensource
Open Source Profession, WW Linux Community Lead    http://github.com/bcornec   
FLOSS projects:    http://mondorescue.org         http://project-builder.org
Musique ancienne?   http://www.musique-ancienne.org  http://www.medieval.org

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________
Mondo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mondo-devel

****************************************************************************************************
CONFIDENTIALITY NOTICE
This communication and any files or attachments transmitted with it may
contain information that is confidential, privileged and exempt from disclosure under
applicable law. It is intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient, you are hereby notified that any use,
dissemination, copying, or action taken in reliance on the contents of this
communication is strictly prohibited. If you have received this communication in error,
please promptly delete it and notify the sender of the delivery error by e-mail so that
the appropriate action may be taken to avoid troubling you further.
Thank you for your cooperation. V2
****************************************************************************************************



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Mondo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mondo-devel

mondo.PNG (40K) Download Attachment
hnaap1_mindi.log.gz (79K) Download Attachment
hnaap1_mondot2.log.gz (66K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Mondo-devel] mindi/mondo virtual memory

Michael White-2
In reply to this post by Bruno Cornec-4
Hi Bruno,

I'm still trying to figure out why something is holding on to virtual memory after the job finishes.  As I noted in a previous post - if I stop mono archive from running virtual memory does not increase that day but when mondo archive starts again virtual memory bumps up another notch again.

I'm wondering could it possibly be mindi? Doesn't mindi use a ram disk to build the boot image? I've looked at mindi's log (attached) but I really don't see anything about memory allocation/deallocation.

Another odd thing is I have 8 Linux servers basically identical to each other that I run mondo archive on but only two production servers are showing this virtual memory issue.

Thanks
Michael White

-----Original Message-----
From: Bruno Cornec [mailto:[hidden email]]
Sent: Tuesday, June 06, 2017 11:56 AM
To: Mondo mailing list <[hidden email]>
Subject: Re: [Mondo-devel] mindi/mondo virtual memory

Michael White said on Tue, Jun 06, 2017 at 04:50:51PM +0000:
>I'm not following you on what to look for with the command "ps auxww".

Well, I try to understand which process can be eating memory. That command could help.

>Over the weekend I stopped Mondoarchive from running and the memory graphs flatted out until MondoArchive ran Monday night and the graph stepped up like the example I sent.

How long is mondoarchive running for you ?
Of course, when it's running, it's consuming more memory, even if I try to do my best to avoid unnecessary consumption.
But when the command ends, memory should come back again. Is that the case ?

Bruno.
--
HPE EMEA EG FLOSS Technology Strategist http://www.hpe.com/engage/opensource
Open Source Profession, WW Linux Community Lead    http://github.com/bcornec   
FLOSS projects:    http://mondorescue.org         http://project-builder.org
Musique ancienne?   http://www.musique-ancienne.org  http://www.medieval.org

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________
Mondo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mondo-devel

****************************************************************************************************
CONFIDENTIALITY NOTICE
This communication and any files or attachments transmitted with it may
contain information that is confidential, privileged and exempt from disclosure under
applicable law. It is intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient, you are hereby notified that any use,
dissemination, copying, or action taken in reliance on the contents of this
communication is strictly prohibited. If you have received this communication in error,
please promptly delete it and notify the sender of the delivery error by e-mail so that
the appropriate action may be taken to avoid troubling you further.
Thank you for your cooperation. V2
****************************************************************************************************



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Mondo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mondo-devel

mindi.gz (76K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Mondo-devel] mindi/mondo virtual memory

Rick Stevens
On 06/14/2017 10:54 AM, Michael White wrote:
> Hi Bruno,
>
> I'm still trying to figure out why something is holding on to virtual memory after the job finishes.  As I noted in a previous post - if I stop mono archive from running virtual memory does not increase that day but when mondo archive starts again virtual memory bumps up another notch again.
>
> I'm wondering could it possibly be mindi? Doesn't mindi use a ram disk to build the boot image? I've looked at mindi's log (attached) but I really don't see anything about memory allocation/deallocation.
>
> Another odd thing is I have 8 Linux servers basically identical to each other that I run mondo archive on but only two production servers are showing this virtual memory issue.

Suggestion from left field: try running mondoarchive under memusage:

        memusage mondoarchive <all your wonderful options>

or perhaps using operf/opreport instead of memusage. It might even be
better to run operf in system-wide mode to catch everything going on.
See "man operf" for details.

memusage is part of the glibc-utils RPM on Fedora/CentOS/Red Hat.

operf/opreport is part of the oprofile RPM on Fedora/CentOS/Red Hat.
Can't speak about other distros, but they're probably similarly named.

> -----Original Message-----
> From: Bruno Cornec [mailto:[hidden email]]
> Sent: Tuesday, June 06, 2017 11:56 AM
> To: Mondo mailing list <[hidden email]>
> Subject: Re: [Mondo-devel] mindi/mondo virtual memory
>
> Michael White said on Tue, Jun 06, 2017 at 04:50:51PM +0000:
>> I'm not following you on what to look for with the command "ps auxww".
>
> Well, I try to understand which process can be eating memory. That command could help.
>
>> Over the weekend I stopped Mondoarchive from running and the memory graphs flatted out until MondoArchive ran Monday night and the graph stepped up like the example I sent.
>
> How long is mondoarchive running for you ?
> Of course, when it's running, it's consuming more memory, even if I try to do my best to avoid unnecessary consumption.
> But when the command ends, memory should come back again. Is that the case ?
>
> Bruno.
> --
> HPE EMEA EG FLOSS Technology Strategist http://www.hpe.com/engage/opensource
> Open Source Profession, WW Linux Community Lead    http://github.com/bcornec   
> FLOSS projects:    http://mondorescue.org         http://project-builder.org
> Musique ancienne?   http://www.musique-ancienne.org  http://www.medieval.org
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________
> Mondo-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/mondo-devel
>
> ****************************************************************************************************
> CONFIDENTIALITY NOTICE
> This communication and any files or attachments transmitted with it may
> contain information that is confidential, privileged and exempt from disclosure under
> applicable law. It is intended solely for the use of the individual or entity to which it is
> addressed. If you are not the intended recipient, you are hereby notified that any use,
> dissemination, copying, or action taken in reliance on the contents of this
> communication is strictly prohibited. If you have received this communication in error,
> please promptly delete it and notify the sender of the delivery error by e-mail so that
> the appropriate action may be taken to avoid troubling you further.
> Thank you for your cooperation. V2
> ****************************************************************************************************
>
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
>
> _______________________________________________
> Mondo-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/mondo-devel
>


--
----------------------------------------------------------------------
- Rick Stevens, Systems Engineer, AllDigital    [hidden email] -
- AIM/Skype: therps2        ICQ: 226437340           Yahoo: origrps2 -
-                                                                    -
-        Brain:  The organ with which we think that we think.        -
----------------------------------------------------------------------

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Mondo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mondo-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Mondo-devel] mindi/mondo virtual memory

Michael White-2
Thanks Rick, doesn't look like I have those available but I'll look into them.

Michael White

-----Original Message-----
From: Rick Stevens [mailto:[hidden email]]
Sent: Wednesday, June 14, 2017 4:14 PM
To: [hidden email]
Subject: Re: [Mondo-devel] mindi/mondo virtual memory

On 06/14/2017 10:54 AM, Michael White wrote:
> Hi Bruno,
>
> I'm still trying to figure out why something is holding on to virtual memory after the job finishes.  As I noted in a previous post - if I stop mono archive from running virtual memory does not increase that day but when mondo archive starts again virtual memory bumps up another notch again.
>
> I'm wondering could it possibly be mindi? Doesn't mindi use a ram disk to build the boot image? I've looked at mindi's log (attached) but I really don't see anything about memory allocation/deallocation.
>
> Another odd thing is I have 8 Linux servers basically identical to each other that I run mondo archive on but only two production servers are showing this virtual memory issue.

Suggestion from left field: try running mondoarchive under memusage:

        memusage mondoarchive <all your wonderful options>

or perhaps using operf/opreport instead of memusage. It might even be better to run operf in system-wide mode to catch everything going on.
See "man operf" for details.

memusage is part of the glibc-utils RPM on Fedora/CentOS/Red Hat.

operf/opreport is part of the oprofile RPM on Fedora/CentOS/Red Hat.
Can't speak about other distros, but they're probably similarly named.

> -----Original Message-----
> From: Bruno Cornec [mailto:[hidden email]]
> Sent: Tuesday, June 06, 2017 11:56 AM
> To: Mondo mailing list <[hidden email]>
> Subject: Re: [Mondo-devel] mindi/mondo virtual memory
>
> Michael White said on Tue, Jun 06, 2017 at 04:50:51PM +0000:
>> I'm not following you on what to look for with the command "ps auxww".
>
> Well, I try to understand which process can be eating memory. That command could help.
>
>> Over the weekend I stopped Mondoarchive from running and the memory graphs flatted out until MondoArchive ran Monday night and the graph stepped up like the example I sent.
>
> How long is mondoarchive running for you ?
> Of course, when it's running, it's consuming more memory, even if I try to do my best to avoid unnecessary consumption.
> But when the command ends, memory should come back again. Is that the case ?
>
> Bruno.
> --
> HPE EMEA EG FLOSS Technology Strategist http://www.hpe.com/engage/opensource
> Open Source Profession, WW Linux Community Lead    http://github.com/bcornec   
> FLOSS projects:    http://mondorescue.org         http://project-builder.org
> Musique ancienne?   http://www.musique-ancienne.org  http://www.medieval.org
>
> ----------------------------------------------------------------------
> -------- Check out the vibrant tech community on one of the world's
> most engaging tech sites, Slashdot.org! http://sdm.link/slashdot 
> _______________________________________________
> Mondo-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/mondo-devel
>
> **********************************************************************
> ******************************
> CONFIDENTIALITY NOTICE
> This communication and any files or attachments transmitted with it
> may contain information that is confidential, privileged and exempt
> from disclosure under applicable law. It is intended solely for the
> use of the individual or entity to which it is addressed. If you are
> not the intended recipient, you are hereby notified that any use,
> dissemination, copying, or action taken in reliance on the contents of
> this communication is strictly prohibited. If you have received this
> communication in error, please promptly delete it and notify the sender of the delivery error by e-mail so that the appropriate action may be taken to avoid troubling you further.
> Thank you for your cooperation. V2
> **********************************************************************
> ******************************
>
>
>
>
> ----------------------------------------------------------------------
> -------- Check out the vibrant tech community on one of the world's
> most engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
>
> _______________________________________________
> Mondo-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/mondo-devel
>


--
----------------------------------------------------------------------
- Rick Stevens, Systems Engineer, AllDigital    [hidden email] -
- AIM/Skype: therps2        ICQ: 226437340           Yahoo: origrps2 -
-                                                                    -
-        Brain:  The organ with which we think that we think.        -
----------------------------------------------------------------------

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________
Mondo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mondo-devel

****************************************************************************************************
CONFIDENTIALITY NOTICE
This communication and any files or attachments transmitted with it may
contain information that is confidential, privileged and exempt from disclosure under
applicable law. It is intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient, you are hereby notified that any use,
dissemination, copying, or action taken in reliance on the contents of this
communication is strictly prohibited. If you have received this communication in error,
please promptly delete it and notify the sender of the delivery error by e-mail so that
the appropriate action may be taken to avoid troubling you further.
Thank you for your cooperation. V2
****************************************************************************************************



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Mondo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mondo-devel
Loading...