Memory usage

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory usage

tringate
I have noticed now for about a month or so that when an avatar arrives on a region, the simulator memory usage skyrockets.  Usually the idle memory usage has been 200MB or so for a region with no prims at all on it.  Just the terrain with nothing added.  Now the same region will skyrocket to over 1.5MB when an avatar arrives, then gradually decrees a small amount once the region has no avatars on it.
 
SIM PING goes nuts on all my regions when an avatar arrives and lag usually becomes very noticeable.  I am running Verizon FiOS with a solid 50/50MB data rate.
 
I have been running my regions for several years now using “monit” to monitor them and have used the memory setting of 1.3GB as a point to indicate there might be some kind of memory issue and to restart the region.  About a month ago all my regions started doing these restarts because of memory usage exceeding that value.  I now have them set to trip at 2.5GB and yet they still will occasionally reach that memory limit.
 
What the heck changed in opensim that has caused memory usage to skyrocket when an avatar arrives and why does it now drop back to a reasonable level?
 
Did I miss some setting change that was incorporated in opensim?
 
The same is true for the CPU usage.  I never saw the kind of CPU usage that the current version of opensim is using.   Usually a region will idle with no avatars on it between 0.2-1.3% of CPU.  Now they will idle somewhat higher than that but when an avatar arrives the CPU will shoot up to in the 5-10% range with some times going as high as 50%.
 
I run AMD 4 and 8 core servers all with 8GB of ram using Fedora 19, 20 and 21.  The server configurations have been consistent.
 
Is this the new normal for opensim?
 
Tom
 
 

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage

Dahlia Trimble
I have seen memory issues with mono 3.1-3.2.x; they were so bad that I stayed with 2.10.8.1 until I tried 3.99.x which worked quite well. I haven't seen the CPU issues you mention at all. Note that measuring memory is difficult with managed applications as many memory monitoring systems cannot account for memory in the Mono VM which is unreferenced but not yet garbage collected; this might explain why you gradually see memory use decrease.

If you have high memory or CPU when a particular avatar arrives it could be poorly scripted attachments or perhaps a corrupt mesh or other asset. If you see it with the default "Ruth" avatar, then there are likely problems with your installation and/or configuration. Also keep in mind that if it's a megaregion then it will use much more memory for the large terrain.

On Tue, May 26, 2015 at 10:20 AM, Thomas Ringate <[hidden email]> wrote:
I have noticed now for about a month or so that when an avatar arrives on a region, the simulator memory usage skyrockets.  Usually the idle memory usage has been 200MB or so for a region with no prims at all on it.  Just the terrain with nothing added.  Now the same region will skyrocket to over 1.5MB when an avatar arrives, then gradually decrees a small amount once the region has no avatars on it.
 
SIM PING goes nuts on all my regions when an avatar arrives and lag usually becomes very noticeable.  I am running Verizon FiOS with a solid 50/50MB data rate.
 
I have been running my regions for several years now using “monit” to monitor them and have used the memory setting of 1.3GB as a point to indicate there might be some kind of memory issue and to restart the region.  About a month ago all my regions started doing these restarts because of memory usage exceeding that value.  I now have them set to trip at 2.5GB and yet they still will occasionally reach that memory limit.
 
What the heck changed in opensim that has caused memory usage to skyrocket when an avatar arrives and why does it now drop back to a reasonable level?
 
Did I miss some setting change that was incorporated in opensim?
 
The same is true for the CPU usage.  I never saw the kind of CPU usage that the current version of opensim is using.   Usually a region will idle with no avatars on it between 0.2-1.3% of CPU.  Now they will idle somewhat higher than that but when an avatar arrives the CPU will shoot up to in the 5-10% range with some times going as high as 50%.
 
I run AMD 4 and 8 core servers all with 8GB of ram using Fedora 19, 20 and 21.  The server configurations have been consistent.
 
Is this the new normal for opensim?
 
Tom
 
 

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users



_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage

tringate
To help clarify a few of your points let me add that I am running mono 2.10.8.  The CPU and memory increases when “any” avatar arrives. My regions all run in a single instance on four different servers.  The test region I am working with is running in a dedicated server with nothing running other than the one test region which is nothing but a flat region with no prims at all.  My avatar has no scripted attachments and is the same as it was back in 2012
 
This all started 4-6 weeks ago when I updated opensim software.  My .ini files have not changed other than the one change OSgrid said to change which was a change to one URI.
 
I keep all my regions at the latest package release OSgrid makes available.
 
The memory usage “monit” is reporting is not total system memory usage but rather memory usage of that specific pid.  So whatever memory mono is using is not the issue as I see it.
 
If the general consensus is that it has nothing to do with opensim changes, I will attempt to go back and find what the last release was where this problem was not noticeable.  Since no one has been complaining about memory usage, I was thinking it was some default or some new parameter that had been changed and I missed making the change.
 
Tom
 
 
Sent: Tuesday, May 26, 2015 4:01 PM
Subject: Re: [Opensim-users] Memory usage
 
I have seen memory issues with mono 3.1-3.2.x; they were so bad that I stayed with 2.10.8.1 until I tried 3.99.x which worked quite well. I haven't seen the CPU issues you mention at all. Note that measuring memory is difficult with managed applications as many memory monitoring systems cannot account for memory in the Mono VM which is unreferenced but not yet garbage collected; this might explain why you gradually see memory use decrease.

If you have high memory or CPU when a particular avatar arrives it could be poorly scripted attachments or perhaps a corrupt mesh or other asset. If you see it with the default "Ruth" avatar, then there are likely problems with your installation and/or configuration. Also keep in mind that if it's a megaregion then it will use much more memory for the large terrain.
 
On Tue, May 26, 2015 at 10:20 AM, Thomas Ringate <[hidden email]> wrote:
I have noticed now for about a month or so that when an avatar arrives on a region, the simulator memory usage skyrockets.  Usually the idle memory usage has been 200MB or so for a region with no prims at all on it.  Just the terrain with nothing added.  Now the same region will skyrocket to over 1.5MB when an avatar arrives, then gradually decrees a small amount once the region has no avatars on it.
 
SIM PING goes nuts on all my regions when an avatar arrives and lag usually becomes very noticeable.  I am running Verizon FiOS with a solid 50/50MB data rate.
 
I have been running my regions for several years now using “monit” to monitor them and have used the memory setting of 1.3GB as a point to indicate there might be some kind of memory issue and to restart the region.  About a month ago all my regions started doing these restarts because of memory usage exceeding that value.  I now have them set to trip at 2.5GB and yet they still will occasionally reach that memory limit.
 
What the heck changed in opensim that has caused memory usage to skyrocket when an avatar arrives and why does it now drop back to a reasonable level?
 
Did I miss some setting change that was incorporated in opensim?
 
The same is true for the CPU usage.  I never saw the kind of CPU usage that the current version of opensim is using.   Usually a region will idle with no avatars on it between 0.2-1.3% of CPU.  Now they will idle somewhat higher than that but when an avatar arrives the CPU will shoot up to in the 5-10% range with some times going as high as 50%.
 
I run AMD 4 and 8 core servers all with 8GB of ram using Fedora 19, 20 and 21.  The server configurations have been consistent.
 
Is this the new normal for opensim?
 
Tom
 
 

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users

 


_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage

Dahlia Trimble
Yes, unfortunately memory the operating system says a particular process is using is inaccurate with a managed environment like Mono provides and doesn't take Mono's internal memory management and garbage collection into account. Typing "show stats" in the OpenSim console can provide some memory statistics as reported by the Mono runtime, but recently the accuracy of these numbers has also been called into question. To get a feel of how garbage collection is working on your system you can type "show stats", then "force gc" and "show stats" again which, if done often enough, can provide much insight.

If you are seeing issues that others aren't seeing then it's probable that there may be something particular to your installation rather than to the code base. Anyway, others may have ideas or have noticed similar issues and just not said anything yet so this list or the opensim-dev list are a good place to discuss it.

On Tue, May 26, 2015 at 2:36 PM, Thomas Ringate <[hidden email]> wrote:
To help clarify a few of your points let me add that I am running mono 2.10.8.  The CPU and memory increases when “any” avatar arrives. My regions all run in a single instance on four different servers.  The test region I am working with is running in a dedicated server with nothing running other than the one test region which is nothing but a flat region with no prims at all.  My avatar has no scripted attachments and is the same as it was back in 2012
 
This all started 4-6 weeks ago when I updated opensim software.  My .ini files have not changed other than the one change OSgrid said to change which was a change to one URI.
 
I keep all my regions at the latest package release OSgrid makes available.
 
The memory usage “monit” is reporting is not total system memory usage but rather memory usage of that specific pid.  So whatever memory mono is using is not the issue as I see it.
 
If the general consensus is that it has nothing to do with opensim changes, I will attempt to go back and find what the last release was where this problem was not noticeable.  Since no one has been complaining about memory usage, I was thinking it was some default or some new parameter that had been changed and I missed making the change.
 
Tom
 
 
Sent: Tuesday, May 26, 2015 4:01 PM
Subject: Re: [Opensim-users] Memory usage
 
I have seen memory issues with mono 3.1-3.2.x; they were so bad that I stayed with 2.10.8.1 until I tried 3.99.x which worked quite well. I haven't seen the CPU issues you mention at all. Note that measuring memory is difficult with managed applications as many memory monitoring systems cannot account for memory in the Mono VM which is unreferenced but not yet garbage collected; this might explain why you gradually see memory use decrease.

If you have high memory or CPU when a particular avatar arrives it could be poorly scripted attachments or perhaps a corrupt mesh or other asset. If you see it with the default "Ruth" avatar, then there are likely problems with your installation and/or configuration. Also keep in mind that if it's a megaregion then it will use much more memory for the large terrain.
 
On Tue, May 26, 2015 at 10:20 AM, Thomas Ringate <[hidden email]> wrote:
I have noticed now for about a month or so that when an avatar arrives on a region, the simulator memory usage skyrockets.  Usually the idle memory usage has been 200MB or so for a region with no prims at all on it.  Just the terrain with nothing added.  Now the same region will skyrocket to over 1.5MB when an avatar arrives, then gradually decrees a small amount once the region has no avatars on it.
 
SIM PING goes nuts on all my regions when an avatar arrives and lag usually becomes very noticeable.  I am running Verizon FiOS with a solid 50/50MB data rate.
 
I have been running my regions for several years now using “monit” to monitor them and have used the memory setting of 1.3GB as a point to indicate there might be some kind of memory issue and to restart the region.  About a month ago all my regions started doing these restarts because of memory usage exceeding that value.  I now have them set to trip at 2.5GB and yet they still will occasionally reach that memory limit.
 
What the heck changed in opensim that has caused memory usage to skyrocket when an avatar arrives and why does it now drop back to a reasonable level?
 
Did I miss some setting change that was incorporated in opensim?
 
The same is true for the CPU usage.  I never saw the kind of CPU usage that the current version of opensim is using.   Usually a region will idle with no avatars on it between 0.2-1.3% of CPU.  Now they will idle somewhat higher than that but when an avatar arrives the CPU will shoot up to in the 5-10% range with some times going as high as 50%.
 
I run AMD 4 and 8 core servers all with 8GB of ram using Fedora 19, 20 and 21.  The server configurations have been consistent.
 
Is this the new normal for opensim?
 
Tom
 
 

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users

 


_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users


_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users



_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users