Memory usage

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory usage

aiaustin

>From: "Thomas Ringate" <[hidden email]>
>...I have noticed now for about a month or so that when an avatar
>arrives on a region, the simulator memory usage skyrockets.  Usually
>the idle memory usage has been 200MB or so for a region with no
>prims at all on it.  Just the terrain with nothing added.  Now the
>same region will skyrocket to over 1.5GB [<-- Ai correction] when an
>avatar arrives, then gradually decrees a small amount once the
>region has no avatars on it.
>...
>What the heck changed in opensim that has caused memory usage to
>skyrocket when an avatar arrives and why does it now [<-- Ai note:
>do you mean "not"?] drop back to a reasonable level?


Can you pin down more precisely when this started Thomas.  In the
last month or so there has been the BIG changes to how the initial
inventory loads (which fixed a serious initial inventory hang issue
some of us have been having for some time on OSGrid), and that is
having some less serious knock on issues that are still under
investigation.  But these would not have started until 6th to 8th
May.  So if your issues started 6 weeks ago then its something else.

It may be worth starting a Mantis to try to get some in pit from
others, but as a quick test I just used one addon test region on
OSGrid and observe these memory usage figures on a Windows 8.1 setup
using SQLite... I am  not sure how "normal" this all is though Thomas...

Idling single region       590MB
1st avatar joins            998MB
2nd avatar joins          1724MB
2nd avatar leaves        1448MB
     after 1 minute idle   1449MB
     after 3 minutes idle  1449MB
     after 5 minutes        1005MB
1st avatar leaves          1048MB
     after 1 minute          1049MB rising slowly
     after 3 minutes        1052MB rising slowly
     after 5 minutes         1053MB
1st avatar logs back in  1302MB
2nd avatar logs back in  1692MB
2nd avatar leaves          1770MB rising slowly
     after 1 minute           1774MB rising slowly
...





_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage

M.E. Verhagen
The memory consumption is troubling.

It seems to be an cross platform issue, it happens on linux, mac and also windows.

The script engine can be responsible for the skyrocketing of the memory usage when an avatar enters the region.
The inventory system is also called as a possible suspect.
Also the physics system can be un issuer. 

This throws up a question, how can the memory usage be broken down, to see what part is to blame ?

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage

Teravus Ovares
https://www.jetbrains.com/dotmemory/ dotmemory has been relatively successful at identifying memory issues in the past.    

That said, it's not free and there may be free alternatives.....  but be advised you want one that supports managed memory...    and one that works well with Mono.Addins.    Many free ones in the past have had issues dynamically loading modules with Mono.addins.   If the modules are not loaded, you will see the server come up with no regions and it will be pretty useless for testing.   Testing physics is a bit harder because it is both managed and unmanaged...   so you'll need to examine the pinned memory.    Also, note that while running the simulator in a profiler such as dotMemory, it will run noticeably slower depending on how you do the profiling.

Regards

Teravus

On Wed, May 27, 2015 at 3:01 AM, M.E. Verhagen <[hidden email]> wrote:
The memory consumption is troubling.

It seems to be an cross platform issue, it happens on linux, mac and also windows.

The script engine can be responsible for the skyrocketing of the memory usage when an avatar enters the region.
The inventory system is also called as a possible suspect.
Also the physics system can be un issuer. 

This throws up a question, how can the memory usage be broken down, to see what part is to blame ?

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users



_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage

Luisillo Contepomi
I have not problems or issues with memory.
My memory test results:

Grid and sim in the same machine.
Windows7 - 64 - RAM 8GB
Intel Core i7-3770 at 3.40GHz
Users in this test is out in other machines connecting by a 100Mb network.

All users in this test are using Radegast. All have Inventory with
1200 Items aprox.
and worn scripts as AO, radar and resizers in hair and shoes.

Continente is a Var Region 1024x1024
Prims used 55.048
Scripts 4.499

Region (Continente) # xengine status
Status of XEngine instance for Continente
Scripts loaded             : 4499
Scripts waiting for load   : 0
Max threads                : 100
Min threads                : 2
Allocated threads          : 100
In use threads             : 42
Work items waiting         : 0
Events queued              : 636
Events processed           : 1028273
Sensors                    : 157
Dataserver requests        : 0
Timers                     : 715
Listeners                  : 85

From tast manager in K
CPU 1 from 8 at 70 - 75% all time during test.
-----------------
Memory OpenSim.exe       /         MySql server
Start region at. 9.39h
------------------------------------------------------------
1.613.652 no users    9.40h                652.976
2.140.920 no users    9.45h            "
------------------------------------------------------------
finish scripts load       9.46h            "

I order in console "fcache clear file" 9.47h Im not using memory cache. "
------------------------------------------------------------
2.142.964 no users 9.47h            "
2.134.240 no users 9.48h                  653.140
1.468.540 no users 9.49h
1.148.276 no users 9.50h (fcache clear finished.)

Test with 2 users
-------------------------------------------------------------
1.148.384 Login in one user 9.54H (RADEGAST)
1.185.828 with 1 user  9.56h
1.337.280 with 2 users 9.58h            734.548

1.389.148 2 users one is searching in his inventory     735.900

1.411.896 logout one user 10.01h        736.452
1.515.700 1 user 10.02H
1.271.528 1 user 10.03H                    736.468
---------------------------------------------------
1.257.620 logout user No users in 10.04H    736.692
1.221.208 no users 10.05h            736.692
1.207.296 no users 10.07h            736.692
1.204.980 no users 10.07h            735.660
1.204.736 no users 10.09h            735.660

------------simultaneus 8 users login at 10.10h
---------------------------------------------------------------

1.228.172 10.10h.
1.427.396 10.11h  5 users login
1.699.788 10.12h   8 users login
1.937.552 10.13h 8 users in home
2.008.496 10.14h 8 users in home        745.468
2.010.358 10.15h 8 users in home        745.928
1.960.540 10.14h 8 users in home        745.944
1.887.942 10.15h 8 users in home        "
1.946.372 10.16h 8 users in home        "
1.902.660 10.17h 8 users in home location    "
1.927.866 10.18h    "            "
1.872.444 10.19h    "            "
1.885.264 10.20h    "            "
1.838.972 10.23h    "            "
1.908.056 10.25h    "

Now logout all users one by minute.

1 - 1.906.736 10.26h                746.528
2 - 1.907.524 10.27h                746.532
3 - 1.876.456 10.28h                746.612
4 - 1.884.836 10.29h                "
5 - 1.788.868 10.30h                746.852
6 - 1.732.212 10.31h                "
7 - 1.756.340 10.32h                "
8 - 1.648.408 10.33h                "
NO USERS
1.629.464 10.34h                "
1.618.988 10.36h
1.434.444 10.37h

Robust.exe always near 136.570K
---------------------------------------------------------
10.17h When have 8 users STATS:
---------------------------------------------------------
Region (Continente) # show stats


CONNECTION STATISTICS
Client logouts due to no data receive timeout: 0


SAMPLE FRAME STATISTICS
Dilatn  SimFPS  PhyFPS  AgntUp  RootAg  ChldAg  Prims   AtvPrm  AtvScr  ScrLPS
  1.00      55    55.6     0.0       8       0   56152      25    4501  10991070


PktsIn  PktOut  PendDl  PendUl  UnackB  TotlFt  NetFt   PhysFt  OthrFt  AgntFt
ImgsFt
    20     123       0       0    1142    18.1     0.0     0.4     0.0     0.6
   0.2


MEMORY STATISTICS
Heap allocated to OpenSim   : 1647 MB
Last heap allocation rate   : 8.061 MB/s
Average heap allocation rate: 9.906 MB/s
Process memory              : 2007 MB

10.46H NO USERS STATS
------------------------------------------------------
Region (Continente) # show stats


CONNECTION STATISTICS
Client logouts due to no data receive timeout: 0


SAMPLE FRAME STATISTICS
Dilatn  SimFPS  PhyFPS  AgntUp  RootAg  ChldAg  Prims   AtvPrm  AtvScr  ScrLPS
  1.00      55    55.3     0.0       0       0   54875      24    4451  13512840


PktsIn  PktOut  PendDl  PendUl  UnackB  TotlFt  NetFt   PhysFt  OthrFt  AgntFt
ImgsFt
    13      78       0       0       0    18.3     0.0     0.6     0.0     0.0
   0.0


MEMORY STATISTICS
Heap allocated to OpenSim   : 768 MB
Last heap allocation rate   : 1.732 MB/s
Average heap allocation rate: 2.493 MB/s
Process memory              : 1493 MB

For me all is ok.




2015-05-28 1:53 GMT+02:00 Teravus Ovares <[hidden email]>:

> https://www.jetbrains.com/dotmemory/ dotmemory has been relatively
> successful at identifying memory issues in the past.
>
> That said, it's not free and there may be free alternatives.....  but be
> advised you want one that supports managed memory...    and one that works
> well with Mono.Addins.    Many free ones in the past have had issues
> dynamically loading modules with Mono.addins.   If the modules are not
> loaded, you will see the server come up with no regions and it will be
> pretty useless for testing.   Testing physics is a bit harder because it is
> both managed and unmanaged...   so you'll need to examine the pinned memory.
> Also, note that while running the simulator in a profiler such as dotMemory,
> it will run noticeably slower depending on how you do the profiling.
>
> Regards
>
> Teravus
>
> On Wed, May 27, 2015 at 3:01 AM, M.E. Verhagen <[hidden email]> wrote:
>>
>> The memory consumption is troubling.
>>
>> It seems to be an cross platform issue, it happens on linux, mac and also
>> windows.
>>
>> The script engine can be responsible for the skyrocketing of the memory
>> usage when an avatar enters the region.
>> The inventory system is also called as a possible suspect.
>> Also the physics system can be un issuer.
>>
>> This throws up a question, how can the memory usage be broken down, to see
>> what part is to blame ?
>>
>> _______________________________________________
>> Opensim-users mailing list
>> [hidden email]
>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>>
>
>
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>
_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage

tringate
Very interesting.  My setup is quite different in that I do not run the grid
but rather use the OSgrid.  I only run the regions.  That could be a huge
difference.  However since OSgrid is officially designated as the "opensim"
test grid, I would hope the developers are serious about the results people
are having who use that grid.

I am in the process of trying to discover exactly when this problem started
for me as was earlier requested.

The different versions I can test with are these releases from OSgrid:
osgrid-opensim-01162015.v0.8.1.97ac80d
osgrid-opensim-01172015.v0.8.1.1f04e1b
osgrid-opensim-02212015.v0.8.1.7b9ad11
osgrid-opensim-03142015.v0.8.1.8b13e4e
osgrid-opensim-03172015.v0.8.2.83e58eb
osgrid-opensim-04052015.v0.8.2.8d66284
osgrid-opensim-04122015.v0.8.2.d96d31b
osgrid-opensim-05072015.v0.8.2.c74cef0
osgrid-opensim-05092015.v0.8.2.adf0f49
osgrid-opensim-05232015.v0.8.2.abb3bb6

Later today I will cycle all of these releases on one of my regions and run
a test using "monit" with my original parameters set to see which version it
was that started using so much simulator memory.  The change control on my
home network is not near as professional as what I lived with at IBM for 31
years.  Now I depend on my own memory.  That is not a very good choice in my
case.  Luisillo's testing was far more accurate with excellent metrics to
compare.  Bravo.

Tom


-----Original Message-----
From: Luisillo Contepomi
Sent: Thursday, May 28, 2015 5:07 AM
To: [hidden email]
Subject: Re: [Opensim-users] Memory usage

I have not problems or issues with memory.
My memory test results:

Grid and sim in the same machine.
Windows7 - 64 - RAM 8GB
Intel Core i7-3770 at 3.40GHz
Users in this test is out in other machines connecting by a 100Mb network.

All users in this test are using Radegast. All have Inventory with
1200 Items aprox.
and worn scripts as AO, radar and resizers in hair and shoes.

Continente is a Var Region 1024x1024
Prims used 55.048
Scripts 4.499

Region (Continente) # xengine status
Status of XEngine instance for Continente
Scripts loaded             : 4499
Scripts waiting for load   : 0
Max threads                : 100
Min threads                : 2
Allocated threads          : 100
In use threads             : 42
Work items waiting         : 0
Events queued              : 636
Events processed           : 1028273
Sensors                    : 157
Dataserver requests        : 0
Timers                     : 715
Listeners                  : 85

From tast manager in K
CPU 1 from 8 at 70 - 75% all time during test.
-----------------
Memory OpenSim.exe       /         MySql server
Start region at. 9.39h
------------------------------------------------------------
1.613.652 no users    9.40h                652.976
2.140.920 no users    9.45h            "
------------------------------------------------------------
finish scripts load       9.46h            "

I order in console "fcache clear file" 9.47h Im not using memory cache. "
------------------------------------------------------------
2.142.964 no users 9.47h            "
2.134.240 no users 9.48h                  653.140
1.468.540 no users 9.49h
1.148.276 no users 9.50h (fcache clear finished.)

Test with 2 users
-------------------------------------------------------------
1.148.384 Login in one user 9.54H (RADEGAST)
1.185.828 with 1 user  9.56h
1.337.280 with 2 users 9.58h            734.548

1.389.148 2 users one is searching in his inventory     735.900

1.411.896 logout one user 10.01h        736.452
1.515.700 1 user 10.02H
1.271.528 1 user 10.03H                    736.468
---------------------------------------------------
1.257.620 logout user No users in 10.04H    736.692
1.221.208 no users 10.05h            736.692
1.207.296 no users 10.07h            736.692
1.204.980 no users 10.07h            735.660
1.204.736 no users 10.09h            735.660

------------simultaneus 8 users login at 10.10h
---------------------------------------------------------------

1.228.172 10.10h.
1.427.396 10.11h  5 users login
1.699.788 10.12h   8 users login
1.937.552 10.13h 8 users in home
2.008.496 10.14h 8 users in home        745.468
2.010.358 10.15h 8 users in home        745.928
1.960.540 10.14h 8 users in home        745.944
1.887.942 10.15h 8 users in home        "
1.946.372 10.16h 8 users in home        "
1.902.660 10.17h 8 users in home location    "
1.927.866 10.18h    "            "
1.872.444 10.19h    "            "
1.885.264 10.20h    "            "
1.838.972 10.23h    "            "
1.908.056 10.25h    "

Now logout all users one by minute.

1 - 1.906.736 10.26h                746.528
2 - 1.907.524 10.27h                746.532
3 - 1.876.456 10.28h                746.612
4 - 1.884.836 10.29h                "
5 - 1.788.868 10.30h                746.852
6 - 1.732.212 10.31h                "
7 - 1.756.340 10.32h                "
8 - 1.648.408 10.33h                "
NO USERS
1.629.464 10.34h                "
1.618.988 10.36h
1.434.444 10.37h

Robust.exe always near 136.570K
---------------------------------------------------------
10.17h When have 8 users STATS:
---------------------------------------------------------
Region (Continente) # show stats


CONNECTION STATISTICS
Client logouts due to no data receive timeout: 0


SAMPLE FRAME STATISTICS
Dilatn  SimFPS  PhyFPS  AgntUp  RootAg  ChldAg  Prims   AtvPrm  AtvScr
ScrLPS
  1.00      55    55.6     0.0       8       0   56152      25    4501
10991070


PktsIn  PktOut  PendDl  PendUl  UnackB  TotlFt  NetFt   PhysFt  OthrFt
AgntFt
ImgsFt
    20     123       0       0    1142    18.1     0.0     0.4     0.0
0.6
   0.2


MEMORY STATISTICS
Heap allocated to OpenSim   : 1647 MB
Last heap allocation rate   : 8.061 MB/s
Average heap allocation rate: 9.906 MB/s
Process memory              : 2007 MB

10.46H NO USERS STATS
------------------------------------------------------
Region (Continente) # show stats


CONNECTION STATISTICS
Client logouts due to no data receive timeout: 0


SAMPLE FRAME STATISTICS
Dilatn  SimFPS  PhyFPS  AgntUp  RootAg  ChldAg  Prims   AtvPrm  AtvScr
ScrLPS
  1.00      55    55.3     0.0       0       0   54875      24    4451
13512840


PktsIn  PktOut  PendDl  PendUl  UnackB  TotlFt  NetFt   PhysFt  OthrFt
AgntFt
ImgsFt
    13      78       0       0       0    18.3     0.0     0.6     0.0
0.0
   0.0


MEMORY STATISTICS
Heap allocated to OpenSim   : 768 MB
Last heap allocation rate   : 1.732 MB/s
Average heap allocation rate: 2.493 MB/s
Process memory              : 1493 MB

For me all is ok.




2015-05-28 1:53 GMT+02:00 Teravus Ovares <[hidden email]>:

> https://www.jetbrains.com/dotmemory/ dotmemory has been relatively
> successful at identifying memory issues in the past.
>
> That said, it's not free and there may be free alternatives.....  but be
> advised you want one that supports managed memory...    and one that works
> well with Mono.Addins.    Many free ones in the past have had issues
> dynamically loading modules with Mono.addins.   If the modules are not
> loaded, you will see the server come up with no regions and it will be
> pretty useless for testing.   Testing physics is a bit harder because it
> is
> both managed and unmanaged...   so you'll need to examine the pinned
> memory.
> Also, note that while running the simulator in a profiler such as
> dotMemory,
> it will run noticeably slower depending on how you do the profiling.
>
> Regards
>
> Teravus
>
> On Wed, May 27, 2015 at 3:01 AM, M.E. Verhagen <[hidden email]>
> wrote:
>>
>> The memory consumption is troubling.
>>
>> It seems to be an cross platform issue, it happens on linux, mac and also
>> windows.
>>
>> The script engine can be responsible for the skyrocketing of the memory
>> usage when an avatar enters the region.
>> The inventory system is also called as a possible suspect.
>> Also the physics system can be un issuer.
>>
>> This throws up a question, how can the memory usage be broken down, to
>> see
>> what part is to blame ?
>>
>> _______________________________________________
>> Opensim-users mailing list
>> [hidden email]
>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>>
>
>
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>
_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users 

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage

Luisillo Contepomi
I have setup always false in "AppDomainLoading" in all my regions.
Is very important the difference.


Results with  "AppDomainLoading" in OpenSim.ini
( 4467 scripts in this region, no avatars login.)

AppDomainLoading = "false"
--------------------------------------------
MEMORY STATISTICS
Heap allocated to OpenSim   : 1456 MB
Last heap allocation rate   : 20.647 MB/s
Average heap allocation rate: 9.178 MB/s
Process memory              : 1868 MB

AppDomainLoading = "true"
----------------------------------------
MEMORY STATISTICS
Heap allocated to OpenSim   : 918 MB
Last heap allocation rate   : 20.23 MB/s
Average heap allocation rate: 12.022 MB/s
Process memory              : 3570 MB

---------------------------------------
Status of XEngine instance for Continente
Scripts loaded             : 4467
Scripts waiting for load   : 0
Max threads                : 100
Min threads                : 2
Allocated threads          : 100
In use threads             : 100
Work items waiting         : 104
Events queued              : 418
Events processed           : 165762
Sensors                    : 157
Dataserver requests        : 0
Timers                     : 702
Listeners                  : 78

Regards,
Luisillo

2015-05-28 17:06 GMT+02:00 Thomas Ringate <[hidden email]>:

> Very interesting.  My setup is quite different in that I do not run the grid
> but rather use the OSgrid.  I only run the regions.  That could be a huge
> difference.  However since OSgrid is officially designated as the "opensim"
> test grid, I would hope the developers are serious about the results people
> are having who use that grid.
>
> I am in the process of trying to discover exactly when this problem started
> for me as was earlier requested.
>
> The different versions I can test with are these releases from OSgrid:
> osgrid-opensim-01162015.v0.8.1.97ac80d
> osgrid-opensim-01172015.v0.8.1.1f04e1b
> osgrid-opensim-02212015.v0.8.1.7b9ad11
> osgrid-opensim-03142015.v0.8.1.8b13e4e
> osgrid-opensim-03172015.v0.8.2.83e58eb
> osgrid-opensim-04052015.v0.8.2.8d66284
> osgrid-opensim-04122015.v0.8.2.d96d31b
> osgrid-opensim-05072015.v0.8.2.c74cef0
> osgrid-opensim-05092015.v0.8.2.adf0f49
> osgrid-opensim-05232015.v0.8.2.abb3bb6
>
> Later today I will cycle all of these releases on one of my regions and run
> a test using "monit" with my original parameters set to see which version it
> was that started using so much simulator memory.  The change control on my
> home network is not near as professional as what I lived with at IBM for 31
> years.  Now I depend on my own memory.  That is not a very good choice in my
> case.  Luisillo's testing was far more accurate with excellent metrics to
> compare.  Bravo.
>
> Tom
>
>
> -----Original Message----- From: Luisillo Contepomi
> Sent: Thursday, May 28, 2015 5:07 AM
> To: [hidden email]
> Subject: Re: [Opensim-users] Memory usage
>
>
> I have not problems or issues with memory.
> My memory test results:
>
> Grid and sim in the same machine.
> Windows7 - 64 - RAM 8GB
> Intel Core i7-3770 at 3.40GHz
> Users in this test is out in other machines connecting by a 100Mb network.
>
> All users in this test are using Radegast. All have Inventory with
> 1200 Items aprox.
> and worn scripts as AO, radar and resizers in hair and shoes.
>
> Continente is a Var Region 1024x1024
> Prims used 55.048
> Scripts 4.499
>
> Region (Continente) # xengine status
> Status of XEngine instance for Continente
> Scripts loaded             : 4499
> Scripts waiting for load   : 0
> Max threads                : 100
> Min threads                : 2
> Allocated threads          : 100
> In use threads             : 42
> Work items waiting         : 0
> Events queued              : 636
> Events processed           : 1028273
> Sensors                    : 157
> Dataserver requests        : 0
> Timers                     : 715
> Listeners                  : 85
>
> From tast manager in K
> CPU 1 from 8 at 70 - 75% all time during test.
> -----------------
> Memory OpenSim.exe       /         MySql server
> Start region at. 9.39h
> ------------------------------------------------------------
> 1.613.652 no users    9.40h                652.976
> 2.140.920 no users    9.45h            "
> ------------------------------------------------------------
> finish scripts load       9.46h            "
>
> I order in console "fcache clear file" 9.47h Im not using memory cache. "
> ------------------------------------------------------------
> 2.142.964 no users 9.47h            "
> 2.134.240 no users 9.48h                  653.140
> 1.468.540 no users 9.49h
> 1.148.276 no users 9.50h (fcache clear finished.)
>
> Test with 2 users
> -------------------------------------------------------------
> 1.148.384 Login in one user 9.54H (RADEGAST)
> 1.185.828 with 1 user  9.56h
> 1.337.280 with 2 users 9.58h            734.548
>
> 1.389.148 2 users one is searching in his inventory     735.900
>
> 1.411.896 logout one user 10.01h        736.452
> 1.515.700 1 user 10.02H
> 1.271.528 1 user 10.03H                    736.468
> ---------------------------------------------------
> 1.257.620 logout user No users in 10.04H    736.692
> 1.221.208 no users 10.05h            736.692
> 1.207.296 no users 10.07h            736.692
> 1.204.980 no users 10.07h            735.660
> 1.204.736 no users 10.09h            735.660
>
> ------------simultaneus 8 users login at 10.10h
> ---------------------------------------------------------------
>
> 1.228.172 10.10h.
> 1.427.396 10.11h  5 users login
> 1.699.788 10.12h   8 users login
> 1.937.552 10.13h 8 users in home
> 2.008.496 10.14h 8 users in home        745.468
> 2.010.358 10.15h 8 users in home        745.928
> 1.960.540 10.14h 8 users in home        745.944
> 1.887.942 10.15h 8 users in home        "
> 1.946.372 10.16h 8 users in home        "
> 1.902.660 10.17h 8 users in home location    "
> 1.927.866 10.18h    "            "
> 1.872.444 10.19h    "            "
> 1.885.264 10.20h    "            "
> 1.838.972 10.23h    "            "
> 1.908.056 10.25h    "
>
> Now logout all users one by minute.
>
> 1 - 1.906.736 10.26h                746.528
> 2 - 1.907.524 10.27h                746.532
> 3 - 1.876.456 10.28h                746.612
> 4 - 1.884.836 10.29h                "
> 5 - 1.788.868 10.30h                746.852
> 6 - 1.732.212 10.31h                "
> 7 - 1.756.340 10.32h                "
> 8 - 1.648.408 10.33h                "
> NO USERS
> 1.629.464 10.34h                "
> 1.618.988 10.36h
> 1.434.444 10.37h
>
> Robust.exe always near 136.570K
> ---------------------------------------------------------
> 10.17h When have 8 users STATS:
> ---------------------------------------------------------
> Region (Continente) # show stats
>
>
> CONNECTION STATISTICS
> Client logouts due to no data receive timeout: 0
>
>
> SAMPLE FRAME STATISTICS
> Dilatn  SimFPS  PhyFPS  AgntUp  RootAg  ChldAg  Prims   AtvPrm  AtvScr
> ScrLPS
>  1.00      55    55.6     0.0       8       0   56152      25    4501
> 10991070
>
>
> PktsIn  PktOut  PendDl  PendUl  UnackB  TotlFt  NetFt   PhysFt  OthrFt
> AgntFt
> ImgsFt
>    20     123       0       0    1142    18.1     0.0     0.4     0.0 0.6
>   0.2
>
>
> MEMORY STATISTICS
> Heap allocated to OpenSim   : 1647 MB
> Last heap allocation rate   : 8.061 MB/s
> Average heap allocation rate: 9.906 MB/s
> Process memory              : 2007 MB
>
> 10.46H NO USERS STATS
> ------------------------------------------------------
> Region (Continente) # show stats
>
>
> CONNECTION STATISTICS
> Client logouts due to no data receive timeout: 0
>
>
> SAMPLE FRAME STATISTICS
> Dilatn  SimFPS  PhyFPS  AgntUp  RootAg  ChldAg  Prims   AtvPrm  AtvScr
> ScrLPS
>  1.00      55    55.3     0.0       0       0   54875      24    4451
> 13512840
>
>
> PktsIn  PktOut  PendDl  PendUl  UnackB  TotlFt  NetFt   PhysFt  OthrFt
> AgntFt
> ImgsFt
>    13      78       0       0       0    18.3     0.0     0.6     0.0 0.0
>   0.0
>
>
> MEMORY STATISTICS
> Heap allocated to OpenSim   : 768 MB
> Last heap allocation rate   : 1.732 MB/s
> Average heap allocation rate: 2.493 MB/s
> Process memory              : 1493 MB
>
> For me all is ok.
>
>
>
>
> 2015-05-28 1:53 GMT+02:00 Teravus Ovares <[hidden email]>:
>>
>> https://www.jetbrains.com/dotmemory/ dotmemory has been relatively
>> successful at identifying memory issues in the past.
>>
>> That said, it's not free and there may be free alternatives.....  but be
>> advised you want one that supports managed memory...    and one that works
>> well with Mono.Addins.    Many free ones in the past have had issues
>> dynamically loading modules with Mono.addins.   If the modules are not
>> loaded, you will see the server come up with no regions and it will be
>> pretty useless for testing.   Testing physics is a bit harder because it
>> is
>> both managed and unmanaged...   so you'll need to examine the pinned
>> memory.
>> Also, note that while running the simulator in a profiler such as
>> dotMemory,
>> it will run noticeably slower depending on how you do the profiling.
>>
>> Regards
>>
>> Teravus
>>
>> On Wed, May 27, 2015 at 3:01 AM, M.E. Verhagen <[hidden email]>
>> wrote:
>>>
>>>
>>> The memory consumption is troubling.
>>>
>>> It seems to be an cross platform issue, it happens on linux, mac and also
>>> windows.
>>>
>>> The script engine can be responsible for the skyrocketing of the memory
>>> usage when an avatar enters the region.
>>> The inventory system is also called as a possible suspect.
>>> Also the physics system can be un issuer.
>>>
>>> This throws up a question, how can the memory usage be broken down, to
>>> see
>>> what part is to blame ?
>>>
>>> _______________________________________________
>>> Opensim-users mailing list
>>> [hidden email]
>>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>>>
>>
>>
>> _______________________________________________
>> Opensim-users mailing list
>> [hidden email]
>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>>
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage

Dahlia Trimble
In reply to this post by tringate
OSGrid is not "officially" the opensim test grid. They are a seperete entity but happen to be very generous in helping test and reporting results. That said, I have very good memory usage in my OSGrid regions which are somewhat isolated, however I have seen regions which share borders with other people's regions in crowded areas use more memory. This is often the case in a grid as neighboring regions share a lot of information with each other.

On Thu, May 28, 2015 at 8:06 AM, Thomas Ringate <[hidden email]> wrote:
Very interesting.  My setup is quite different in that I do not run the grid but rather use the OSgrid.  I only run the regions.  That could be a huge difference.  However since OSgrid is officially designated as the "opensim" test grid, I would hope the developers are serious about the results people are having who use that grid.

I am in the process of trying to discover exactly when this problem started for me as was earlier requested.

The different versions I can test with are these releases from OSgrid:
osgrid-opensim-01162015.v0.8.1.97ac80d
osgrid-opensim-01172015.v0.8.1.1f04e1b
osgrid-opensim-02212015.v0.8.1.7b9ad11
osgrid-opensim-03142015.v0.8.1.8b13e4e
osgrid-opensim-03172015.v0.8.2.83e58eb
osgrid-opensim-04052015.v0.8.2.8d66284
osgrid-opensim-04122015.v0.8.2.d96d31b
osgrid-opensim-05072015.v0.8.2.c74cef0
osgrid-opensim-05092015.v0.8.2.adf0f49
osgrid-opensim-05232015.v0.8.2.abb3bb6

Later today I will cycle all of these releases on one of my regions and run a test using "monit" with my original parameters set to see which version it was that started using so much simulator memory.  The change control on my home network is not near as professional as what I lived with at IBM for 31 years.  Now I depend on my own memory.  That is not a very good choice in my case.  Luisillo's testing was far more accurate with excellent metrics to compare.  Bravo.

Tom


-----Original Message----- From: Luisillo Contepomi
Sent: Thursday, May 28, 2015 5:07 AM
To: [hidden email]
Subject: Re: [Opensim-users] Memory usage


I have not problems or issues with memory.
My memory test results:

Grid and sim in the same machine.
Windows7 - 64 - RAM 8GB
Intel Core i7-3770 at 3.40GHz
Users in this test is out in other machines connecting by a 100Mb network.

All users in this test are using Radegast. All have Inventory with
1200 Items aprox.
and worn scripts as AO, radar and resizers in hair and shoes.

Continente is a Var Region 1024x1024
Prims used 55.048
Scripts 4.499

Region (Continente) # xengine status
Status of XEngine instance for Continente
Scripts loaded             : 4499
Scripts waiting for load   : 0
Max threads                : 100
Min threads                : 2
Allocated threads          : 100
In use threads             : 42
Work items waiting         : 0
Events queued              : 636
Events processed           : 1028273
Sensors                    : 157
Dataserver requests        : 0
Timers                     : 715
Listeners                  : 85

From tast manager in K
CPU 1 from 8 at 70 - 75% all time during test.
-----------------
Memory OpenSim.exe       /         MySql server
Start region at. 9.39h
------------------------------------------------------------
1.613.652 no users    9.40h                652.976
2.140.920 no users    9.45h            "
------------------------------------------------------------
finish scripts load       9.46h            "

I order in console "fcache clear file" 9.47h Im not using memory cache. "
------------------------------------------------------------
2.142.964 no users 9.47h            "
2.134.240 no users 9.48h                  653.140
1.468.540 no users 9.49h
1.148.276 no users 9.50h (fcache clear finished.)

Test with 2 users
-------------------------------------------------------------
1.148.384 Login in one user 9.54H (RADEGAST)
1.185.828 with 1 user  9.56h
1.337.280 with 2 users 9.58h            734.548

1.389.148 2 users one is searching in his inventory     735.900

1.411.896 logout one user 10.01h        736.452
1.515.700 1 user 10.02H
1.271.528 1 user 10.03H                    736.468
---------------------------------------------------
1.257.620 logout user No users in 10.04H    736.692
1.221.208 no users 10.05h            736.692
1.207.296 no users 10.07h            736.692
1.204.980 no users 10.07h            735.660
1.204.736 no users 10.09h            735.660

------------simultaneus 8 users login at 10.10h
---------------------------------------------------------------

1.228.172 10.10h.
1.427.396 10.11h  5 users login
1.699.788 10.12h   8 users login
1.937.552 10.13h 8 users in home
2.008.496 10.14h 8 users in home        745.468
2.010.358 10.15h 8 users in home        745.928
1.960.540 10.14h 8 users in home        745.944
1.887.942 10.15h 8 users in home        "
1.946.372 10.16h 8 users in home        "
1.902.660 10.17h 8 users in home location    "
1.927.866 10.18h    "            "
1.872.444 10.19h    "            "
1.885.264 10.20h    "            "
1.838.972 10.23h    "            "
1.908.056 10.25h    "

Now logout all users one by minute.

1 - 1.906.736 10.26h                746.528
2 - 1.907.524 10.27h                746.532
3 - 1.876.456 10.28h                746.612
4 - 1.884.836 10.29h                "
5 - 1.788.868 10.30h                746.852
6 - 1.732.212 10.31h                "
7 - 1.756.340 10.32h                "
8 - 1.648.408 10.33h                "
NO USERS
1.629.464 10.34h                "
1.618.988 10.36h
1.434.444 10.37h

Robust.exe always near 136.570K
---------------------------------------------------------
10.17h When have 8 users STATS:
---------------------------------------------------------
Region (Continente) # show stats


CONNECTION STATISTICS
Client logouts due to no data receive timeout: 0


SAMPLE FRAME STATISTICS
Dilatn  SimFPS  PhyFPS  AgntUp  RootAg  ChldAg  Prims   AtvPrm  AtvScr ScrLPS
 1.00      55    55.6     0.0       8       0   56152      25    4501 10991070


PktsIn  PktOut  PendDl  PendUl  UnackB  TotlFt  NetFt   PhysFt  OthrFt AgntFt
ImgsFt
   20     123       0       0    1142    18.1     0.0     0.4     0.0 0.6
  0.2


MEMORY STATISTICS
Heap allocated to OpenSim   : 1647 MB
Last heap allocation rate   : 8.061 MB/s
Average heap allocation rate: 9.906 MB/s
Process memory              : 2007 MB

10.46H NO USERS STATS
------------------------------------------------------
Region (Continente) # show stats


CONNECTION STATISTICS
Client logouts due to no data receive timeout: 0


SAMPLE FRAME STATISTICS
Dilatn  SimFPS  PhyFPS  AgntUp  RootAg  ChldAg  Prims   AtvPrm  AtvScr ScrLPS
 1.00      55    55.3     0.0       0       0   54875      24    4451 13512840


PktsIn  PktOut  PendDl  PendUl  UnackB  TotlFt  NetFt   PhysFt  OthrFt AgntFt
ImgsFt
   13      78       0       0       0    18.3     0.0     0.6     0.0 0.0
  0.0


MEMORY STATISTICS
Heap allocated to OpenSim   : 768 MB
Last heap allocation rate   : 1.732 MB/s
Average heap allocation rate: 2.493 MB/s
Process memory              : 1493 MB

For me all is ok.




2015-05-28 1:53 GMT+02:00 Teravus Ovares <[hidden email]>:
https://www.jetbrains.com/dotmemory/ dotmemory has been relatively
successful at identifying memory issues in the past.

That said, it's not free and there may be free alternatives.....  but be
advised you want one that supports managed memory...    and one that works
well with Mono.Addins.    Many free ones in the past have had issues
dynamically loading modules with Mono.addins.   If the modules are not
loaded, you will see the server come up with no regions and it will be
pretty useless for testing.   Testing physics is a bit harder because it is
both managed and unmanaged...   so you'll need to examine the pinned memory.
Also, note that while running the simulator in a profiler such as dotMemory,
it will run noticeably slower depending on how you do the profiling.

Regards

Teravus

On Wed, May 27, 2015 at 3:01 AM, M.E. Verhagen <[hidden email]> wrote:

The memory consumption is troubling.

It seems to be an cross platform issue, it happens on linux, mac and also
windows.

The script engine can be responsible for the skyrocketing of the memory
usage when an avatar enters the region.
The inventory system is also called as a possible suspect.
Also the physics system can be un issuer.

This throws up a question, how can the memory usage be broken down, to see
what part is to blame ?

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users



_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users


_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage

M.E. Verhagen
I have got 15 normal regions on one simulator, that gives this stats:

MEMORY STATISTICS

Heap allocated to OpenSim   : 212 MB

Last heap allocation rate   : 0 MB/s

Average heap allocation rate: 0 MB/s

Process memory              : 282 MB


and one with 2 var regions with this stats:


MEMORY STATISTICS

Heap allocated to OpenSim   : 635 MB

Last heap allocation rate   : 0.016 MB/s

Average heap allocation rate: 19.894 MB/s

Process memory              : 1724 MB


The regions are running on a mac, which has got an activity view which shows another picture, it has got 1,22 gig memory reserved for the first, and up to 7,8 gig for second somehow in use. 

The second 7,8 starts at 1,61gig and rises very slowly over days. I am sure that has got something to do with physics engine or something else leaking memory. osx kills that process automatic when it goes over 8gig and then the second simulator is automatic restarted.

I have got the first one running on opendynamicsengin. It does not any significant increase in memory.

The second runs on bulletsim cause viraginous require bulletsim. 


_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage

Dahlia Trimble
Var regions will use more memory than standard 256x256 regions because of the memory needed for the terrain and associated overhead, caching, etc. However if the memory use far exceeds that which the larger terrain can justify, there could be some other issue(s) at play.

On Thu, May 28, 2015 at 2:53 PM, M.E. Verhagen <[hidden email]> wrote:
I have got 15 normal regions on one simulator, that gives this stats:

MEMORY STATISTICS

Heap allocated to OpenSim   : 212 MB

Last heap allocation rate   : 0 MB/s

Average heap allocation rate: 0 MB/s

Process memory              : 282 MB


and one with 2 var regions with this stats:


MEMORY STATISTICS

Heap allocated to OpenSim   : 635 MB

Last heap allocation rate   : 0.016 MB/s

Average heap allocation rate: 19.894 MB/s

Process memory              : 1724 MB


The regions are running on a mac, which has got an activity view which shows another picture, it has got 1,22 gig memory reserved for the first, and up to 7,8 gig for second somehow in use. 

The second 7,8 starts at 1,61gig and rises very slowly over days. I am sure that has got something to do with physics engine or something else leaking memory. osx kills that process automatic when it goes over 8gig and then the second simulator is automatic restarted.

I have got the first one running on opendynamicsengin. It does not any significant increase in memory.

The second runs on bulletsim cause viraginous require bulletsim. 


_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users



_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage

tringate
In reply to this post by Luisillo Contepomi
I tested the following 10 versions of opensim built for the OSgrid.

The times are from the monitor program "monit",and are the uptime.
The test process was;
Place a fresh copy of opensim as distributed, copy the needed .ini files
into the simulator,
Restart the region.  (This causes "monit" to detect a new PID and resets
it's "uptime" to be zero)
7 minute tick, login with "coolvl" with standard unscripted avatar, record
metrics.
10 minute tick, log off,  record metrics.
15 minute tick  record metrics.
The 7 minute tick metrics represents the IDLE usage of a freshly started and
un-visited region.
(the reason for waiting 7 minutes was to give ample time to compile the 626
scripts as an initial load)
The 10 minute tick metrics represents the usage after an avatar has logged
in and been on the region for approximately 3 minutes.
The 15 minute tick represents the usage 5 minutes after an avatar has left
the region and no other usage of the region has taken place.

Region used: Sanctuary  Prims= 7985  Scripts= 626
-------------------------------------------------------------------  idle
1 avatar                  avatar leave
-------------------------------------------------------- Time mem kb   CPU%
Time mem kb  CPU%  Time  mem kb   CPU%
osgrid-opensim-01162015.v0.8.1.97ac80d 7m   574952k   3.0
osgrid-opensim-01172015.v0.8.1.1f04e1b 7m   585076k   2.8  27m  513324k  2.9
30m   513364k   2.6
osgrid-opensim-02212015.v0.8.1.7b9ad11 7m   565480k   2.5  10m  560624k  2.9
15m   536468k   2.5
osgrid-opensim-03142015.v0.8.1.8b13e4e 7m   568560k   2.6  10m  576828k  2.9
15m   567436k   2.5
osgrid-opensim-03172015.v0.8.2.83e58eb 7m   580736k   2.4  10m  583992k  2.8
15m   572664k   2.5
osgrid-opensim-04052015.v0.8.2.8d66284 7m   564208k   2.5  10m  574464k  2.8
15m   574704k   2.5
osgrid-opensim-04122015.v0.8.2.d96d31b 7m   574392k   2.6  10m  585156k  3.0
15m   546868k   2.6
osgrid-opensim-05072015.v0.8.2.c74cef0 7m   583924k   2.6  10m 1139092k  2.7
15m  1141972k   2.5
osgrid-opensim-05092015.v0.8.2.adf0f49 7m   578632k   2.5  10m 1207972k  2.9
15m  1208696k   2.5
osgrid-opensim-05232015.v0.8.2.abb3bb6 7m   585384K   2.6  10m  584852k  5.5
15m  1165304k   2.5

Additional comments:

This problem appeared for me in the May 7, 2015 OSgrid release build.  It is
present in the latest build which was May 23, 2015.
The initial IDLE memory appears to be nearly the same for all versions
tested.
Prior to the May 7th release memory usage fluctuated very little when
avatars came and went.
After the May 7 release memory usage goes up dramatically, for EACH avatar
that arrives and does not go back down.
There was a change on how memory is used between the May 9 and May 23
builds.  The first builds used the memory much faster, and CPU usage would
rise quickly last for a short time and then drop back down.
In the May 23 release, CPU usage goes up much higher, holds at that high
usage level much longer but does drop back to the "idle" level.  Memory is
slower to be increased, but still rises to over double that of the idle rate
and does not drop back to the idle level ever.
The value of AppDomainLoading has been set to "false" in the OSgrid builds
for all the builds distributed this year that I have, that is what all of my
tests were run with.
However reading that parameter it reads to me that having it set to FALSE
would cause this exact problem.
That does not explain why it never caused this problem prior to the May 7th
build.

I have "monit" configured to allow both memory and CPU usage to go over a
set limit but if it remains above that limit for longer than what I believe
to be a reasonable time, in my case 3 minutes, it will restart the region.

From 2011 until May 7th 2015 it was rare any of my regions were restarted by
"monit".  After the May 9th release "monit" would restart my regions as soon
as any avatar visited it.  I had set the memory limit to be 1.3GB prior to
May 7th.  Now setting it at 5GB still results in some regions restarting a
few times every day.

Until the May 9th release none of my four servers had ever used the swap
file.  Now the usage of the swap file is in the mid 20%.  Lag has become a
problem on my regions, and system CPU usage is much higher than it ever was
prior to the May 7th release.

To suggest nothing is wrong with whatever was done to opensim between the
March release I tested and the May 7th release is very difficult to accept.

If the only problem was higher CPU usage I would accept it, but since
performance and reliability have also been diminished this is not something
that can be ignored.

Memory that gets used and not released is not a good thing.  This is
happening across three released of Fedora linux.  Fedora 19, 20, and 21, on
four different servers.

I do not know if I should report this in mantis .  Usually when I report
something it is already being worked on under some other report using
language I would not think to search on.

Tom



-----Original Message-----
From: Luisillo Contepomi
Sent: Thursday, May 28, 2015 12:00 PM
To: [hidden email]
Subject: Re: [Opensim-users] Memory usage

I have setup always false in "AppDomainLoading" in all my regions.
Is very important the difference.


Results with  "AppDomainLoading" in OpenSim.ini
( 4467 scripts in this region, no avatars login.)

AppDomainLoading = "false"
--------------------------------------------
MEMORY STATISTICS
Heap allocated to OpenSim   : 1456 MB
Last heap allocation rate   : 20.647 MB/s
Average heap allocation rate: 9.178 MB/s
Process memory              : 1868 MB

AppDomainLoading = "true"
----------------------------------------
MEMORY STATISTICS
Heap allocated to OpenSim   : 918 MB
Last heap allocation rate   : 20.23 MB/s
Average heap allocation rate: 12.022 MB/s
Process memory              : 3570 MB

---------------------------------------
Status of XEngine instance for Continente
Scripts loaded             : 4467
Scripts waiting for load   : 0
Max threads                : 100
Min threads                : 2
Allocated threads          : 100
In use threads             : 100
Work items waiting         : 104
Events queued              : 418
Events processed           : 165762
Sensors                    : 157
Dataserver requests        : 0
Timers                     : 702
Listeners                  : 78

Regards,
Luisillo

2015-05-28 17:06 GMT+02:00 Thomas Ringate <[hidden email]>:

> Very interesting.  My setup is quite different in that I do not run the
> grid
> but rather use the OSgrid.  I only run the regions.  That could be a huge
> difference.  However since OSgrid is officially designated as the
> "opensim"
> test grid, I would hope the developers are serious about the results
> people
> are having who use that grid.
>
> I am in the process of trying to discover exactly when this problem
> started
> for me as was earlier requested.
>
> The different versions I can test with are these releases from OSgrid:
> osgrid-opensim-01162015.v0.8.1.97ac80d
> osgrid-opensim-01172015.v0.8.1.1f04e1b
> osgrid-opensim-02212015.v0.8.1.7b9ad11
> osgrid-opensim-03142015.v0.8.1.8b13e4e
> osgrid-opensim-03172015.v0.8.2.83e58eb
> osgrid-opensim-04052015.v0.8.2.8d66284
> osgrid-opensim-04122015.v0.8.2.d96d31b
> osgrid-opensim-05072015.v0.8.2.c74cef0
> osgrid-opensim-05092015.v0.8.2.adf0f49
> osgrid-opensim-05232015.v0.8.2.abb3bb6
>
> Later today I will cycle all of these releases on one of my regions and
> run
> a test using "monit" with my original parameters set to see which version
> it
> was that started using so much simulator memory.  The change control on my
> home network is not near as professional as what I lived with at IBM for
> 31
> years.  Now I depend on my own memory.  That is not a very good choice in
> my
> case.  Luisillo's testing was far more accurate with excellent metrics to
> compare.  Bravo.
>
> Tom
>
>
> -----Original Message----- From: Luisillo Contepomi
> Sent: Thursday, May 28, 2015 5:07 AM
> To: [hidden email]
> Subject: Re: [Opensim-users] Memory usage
>
>
> I have not problems or issues with memory.
> My memory test results:
>
> Grid and sim in the same machine.
> Windows7 - 64 - RAM 8GB
> Intel Core i7-3770 at 3.40GHz
> Users in this test is out in other machines connecting by a 100Mb network.
>
> All users in this test are using Radegast. All have Inventory with
> 1200 Items aprox.
> and worn scripts as AO, radar and resizers in hair and shoes.
>
> Continente is a Var Region 1024x1024
> Prims used 55.048
> Scripts 4.499
>
> Region (Continente) # xengine status
> Status of XEngine instance for Continente
> Scripts loaded             : 4499
> Scripts waiting for load   : 0
> Max threads                : 100
> Min threads                : 2
> Allocated threads          : 100
> In use threads             : 42
> Work items waiting         : 0
> Events queued              : 636
> Events processed           : 1028273
> Sensors                    : 157
> Dataserver requests        : 0
> Timers                     : 715
> Listeners                  : 85
>
> From tast manager in K
> CPU 1 from 8 at 70 - 75% all time during test.
> -----------------
> Memory OpenSim.exe       /         MySql server
> Start region at. 9.39h
> ------------------------------------------------------------
> 1.613.652 no users    9.40h                652.976
> 2.140.920 no users    9.45h            "
> ------------------------------------------------------------
> finish scripts load       9.46h            "
>
> I order in console "fcache clear file" 9.47h Im not using memory cache. "
> ------------------------------------------------------------
> 2.142.964 no users 9.47h            "
> 2.134.240 no users 9.48h                  653.140
> 1.468.540 no users 9.49h
> 1.148.276 no users 9.50h (fcache clear finished.)
>
> Test with 2 users
> -------------------------------------------------------------
> 1.148.384 Login in one user 9.54H (RADEGAST)
> 1.185.828 with 1 user  9.56h
> 1.337.280 with 2 users 9.58h            734.548
>
> 1.389.148 2 users one is searching in his inventory     735.900
>
> 1.411.896 logout one user 10.01h        736.452
> 1.515.700 1 user 10.02H
> 1.271.528 1 user 10.03H                    736.468
> ---------------------------------------------------
> 1.257.620 logout user No users in 10.04H    736.692
> 1.221.208 no users 10.05h            736.692
> 1.207.296 no users 10.07h            736.692
> 1.204.980 no users 10.07h            735.660
> 1.204.736 no users 10.09h            735.660
>
> ------------simultaneus 8 users login at 10.10h
> ---------------------------------------------------------------
>
> 1.228.172 10.10h.
> 1.427.396 10.11h  5 users login
> 1.699.788 10.12h   8 users login
> 1.937.552 10.13h 8 users in home
> 2.008.496 10.14h 8 users in home        745.468
> 2.010.358 10.15h 8 users in home        745.928
> 1.960.540 10.14h 8 users in home        745.944
> 1.887.942 10.15h 8 users in home        "
> 1.946.372 10.16h 8 users in home        "
> 1.902.660 10.17h 8 users in home location    "
> 1.927.866 10.18h    "            "
> 1.872.444 10.19h    "            "
> 1.885.264 10.20h    "            "
> 1.838.972 10.23h    "            "
> 1.908.056 10.25h    "
>
> Now logout all users one by minute.
>
> 1 - 1.906.736 10.26h                746.528
> 2 - 1.907.524 10.27h                746.532
> 3 - 1.876.456 10.28h                746.612
> 4 - 1.884.836 10.29h                "
> 5 - 1.788.868 10.30h                746.852
> 6 - 1.732.212 10.31h                "
> 7 - 1.756.340 10.32h                "
> 8 - 1.648.408 10.33h                "
> NO USERS
> 1.629.464 10.34h                "
> 1.618.988 10.36h
> 1.434.444 10.37h
>
> Robust.exe always near 136.570K
> ---------------------------------------------------------
> 10.17h When have 8 users STATS:
> ---------------------------------------------------------
> Region (Continente) # show stats
>
>
> CONNECTION STATISTICS
> Client logouts due to no data receive timeout: 0
>
>
> SAMPLE FRAME STATISTICS
> Dilatn  SimFPS  PhyFPS  AgntUp  RootAg  ChldAg  Prims   AtvPrm  AtvScr
> ScrLPS
>  1.00      55    55.6     0.0       8       0   56152      25    4501
> 10991070
>
>
> PktsIn  PktOut  PendDl  PendUl  UnackB  TotlFt  NetFt   PhysFt  OthrFt
> AgntFt
> ImgsFt
>    20     123       0       0    1142    18.1     0.0     0.4     0.0 0.6
>   0.2
>
>
> MEMORY STATISTICS
> Heap allocated to OpenSim   : 1647 MB
> Last heap allocation rate   : 8.061 MB/s
> Average heap allocation rate: 9.906 MB/s
> Process memory              : 2007 MB
>
> 10.46H NO USERS STATS
> ------------------------------------------------------
> Region (Continente) # show stats
>
>
> CONNECTION STATISTICS
> Client logouts due to no data receive timeout: 0
>
>
> SAMPLE FRAME STATISTICS
> Dilatn  SimFPS  PhyFPS  AgntUp  RootAg  ChldAg  Prims   AtvPrm  AtvScr
> ScrLPS
>  1.00      55    55.3     0.0       0       0   54875      24    4451
> 13512840
>
>
> PktsIn  PktOut  PendDl  PendUl  UnackB  TotlFt  NetFt   PhysFt  OthrFt
> AgntFt
> ImgsFt
>    13      78       0       0       0    18.3     0.0     0.6     0.0 0.0
>   0.0
>
>
> MEMORY STATISTICS
> Heap allocated to OpenSim   : 768 MB
> Last heap allocation rate   : 1.732 MB/s
> Average heap allocation rate: 2.493 MB/s
> Process memory              : 1493 MB
>
> For me all is ok.
>
>
>
>
> 2015-05-28 1:53 GMT+02:00 Teravus Ovares <[hidden email]>:
>>
>> https://www.jetbrains.com/dotmemory/ dotmemory has been relatively
>> successful at identifying memory issues in the past.
>>
>> That said, it's not free and there may be free alternatives.....  but be
>> advised you want one that supports managed memory...    and one that
>> works
>> well with Mono.Addins.    Many free ones in the past have had issues
>> dynamically loading modules with Mono.addins.   If the modules are not
>> loaded, you will see the server come up with no regions and it will be
>> pretty useless for testing.   Testing physics is a bit harder because it
>> is
>> both managed and unmanaged...   so you'll need to examine the pinned
>> memory.
>> Also, note that while running the simulator in a profiler such as
>> dotMemory,
>> it will run noticeably slower depending on how you do the profiling.
>>
>> Regards
>>
>> Teravus
>>
>> On Wed, May 27, 2015 at 3:01 AM, M.E. Verhagen <[hidden email]>
>> wrote:
>>>
>>>
>>> The memory consumption is troubling.
>>>
>>> It seems to be an cross platform issue, it happens on linux, mac and
>>> also
>>> windows.
>>>
>>> The script engine can be responsible for the skyrocketing of the memory
>>> usage when an avatar enters the region.
>>> The inventory system is also called as a possible suspect.
>>> Also the physics system can be un issuer.
>>>
>>> This throws up a question, how can the memory usage be broken down, to
>>> see
>>> what part is to blame ?
>>>
>>> _______________________________________________
>>> Opensim-users mailing list
>>> [hidden email]
>>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>>>
>>
>>
>> _______________________________________________
>> Opensim-users mailing list
>> [hidden email]
>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>>
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users 

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage

Luisillo Contepomi
Im looking about this in mantis.
Was reported from 0.7.3.X but is closed.
Please report it again for the new versión.

http://opensimulator.org/mantis/view.php?id=6030
http://opensimulator.org/mantis/view.php?id=6031

Today I am observing cpu consuption... is hight : from 70% to 100% all
time for one cpu.



2015-05-29 2:24 GMT+02:00 Thomas Ringate <[hidden email]>:

> I tested the following 10 versions of opensim built for the OSgrid.
>
> The times are from the monitor program "monit",and are the uptime.
> The test process was;
> Place a fresh copy of opensim as distributed, copy the needed .ini files
> into the simulator,
> Restart the region.  (This causes "monit" to detect a new PID and resets
> it's "uptime" to be zero)
> 7 minute tick, login with "coolvl" with standard unscripted avatar, record
> metrics.
> 10 minute tick, log off,  record metrics.
> 15 minute tick  record metrics.
> The 7 minute tick metrics represents the IDLE usage of a freshly started and
> un-visited region.
> (the reason for waiting 7 minutes was to give ample time to compile the 626
> scripts as an initial load)
> The 10 minute tick metrics represents the usage after an avatar has logged
> in and been on the region for approximately 3 minutes.
> The 15 minute tick represents the usage 5 minutes after an avatar has left
> the region and no other usage of the region has taken place.
>
> Region used: Sanctuary  Prims= 7985  Scripts= 626
> -------------------------------------------------------------------  idle 1
> avatar                  avatar leave
> -------------------------------------------------------- Time mem kb   CPU%
> Time mem kb  CPU%  Time  mem kb   CPU%
> osgrid-opensim-01162015.v0.8.1.97ac80d 7m   574952k   3.0
> osgrid-opensim-01172015.v0.8.1.1f04e1b 7m   585076k   2.8  27m  513324k  2.9
> 30m   513364k   2.6
> osgrid-opensim-02212015.v0.8.1.7b9ad11 7m   565480k   2.5  10m  560624k  2.9
> 15m   536468k   2.5
> osgrid-opensim-03142015.v0.8.1.8b13e4e 7m   568560k   2.6  10m  576828k  2.9
> 15m   567436k   2.5
> osgrid-opensim-03172015.v0.8.2.83e58eb 7m   580736k   2.4  10m  583992k  2.8
> 15m   572664k   2.5
> osgrid-opensim-04052015.v0.8.2.8d66284 7m   564208k   2.5  10m  574464k  2.8
> 15m   574704k   2.5
> osgrid-opensim-04122015.v0.8.2.d96d31b 7m   574392k   2.6  10m  585156k  3.0
> 15m   546868k   2.6
> osgrid-opensim-05072015.v0.8.2.c74cef0 7m   583924k   2.6  10m 1139092k  2.7
> 15m  1141972k   2.5
> osgrid-opensim-05092015.v0.8.2.adf0f49 7m   578632k   2.5  10m 1207972k  2.9
> 15m  1208696k   2.5
> osgrid-opensim-05232015.v0.8.2.abb3bb6 7m   585384K   2.6  10m  584852k  5.5
> 15m  1165304k   2.5
>
> Additional comments:
>
> This problem appeared for me in the May 7, 2015 OSgrid release build.  It is
> present in the latest build which was May 23, 2015.
> The initial IDLE memory appears to be nearly the same for all versions
> tested.
> Prior to the May 7th release memory usage fluctuated very little when
> avatars came and went.
> After the May 7 release memory usage goes up dramatically, for EACH avatar
> that arrives and does not go back down.
> There was a change on how memory is used between the May 9 and May 23
> builds.  The first builds used the memory much faster, and CPU usage would
> rise quickly last for a short time and then drop back down.
> In the May 23 release, CPU usage goes up much higher, holds at that high
> usage level much longer but does drop back to the "idle" level.  Memory is
> slower to be increased, but still rises to over double that of the idle rate
> and does not drop back to the idle level ever.
> The value of AppDomainLoading has been set to "false" in the OSgrid builds
> for all the builds distributed this year that I have, that is what all of my
> tests were run with.
> However reading that parameter it reads to me that having it set to FALSE
> would cause this exact problem.
> That does not explain why it never caused this problem prior to the May 7th
> build.
>
> I have "monit" configured to allow both memory and CPU usage to go over a
> set limit but if it remains above that limit for longer than what I believe
> to be a reasonable time, in my case 3 minutes, it will restart the region.
>
> From 2011 until May 7th 2015 it was rare any of my regions were restarted by
> "monit".  After the May 9th release "monit" would restart my regions as soon
> as any avatar visited it.  I had set the memory limit to be 1.3GB prior to
> May 7th.  Now setting it at 5GB still results in some regions restarting a
> few times every day.
>
> Until the May 9th release none of my four servers had ever used the swap
> file.  Now the usage of the swap file is in the mid 20%.  Lag has become a
> problem on my regions, and system CPU usage is much higher than it ever was
> prior to the May 7th release.
>
> To suggest nothing is wrong with whatever was done to opensim between the
> March release I tested and the May 7th release is very difficult to accept.
>
> If the only problem was higher CPU usage I would accept it, but since
> performance and reliability have also been diminished this is not something
> that can be ignored.
>
> Memory that gets used and not released is not a good thing.  This is
> happening across three released of Fedora linux.  Fedora 19, 20, and 21, on
> four different servers.
>
> I do not know if I should report this in mantis .  Usually when I report
> something it is already being worked on under some other report using
> language I would not think to search on.
>
> Tom
>
>
>
> -----Original Message----- From: Luisillo Contepomi
> Sent: Thursday, May 28, 2015 12:00 PM
> To: [hidden email]
> Subject: Re: [Opensim-users] Memory usage
>
> I have setup always false in "AppDomainLoading" in all my regions.
> Is very important the difference.
>
>
> Results with  "AppDomainLoading" in OpenSim.ini
> ( 4467 scripts in this region, no avatars login.)
>
> AppDomainLoading = "false"
> --------------------------------------------
> MEMORY STATISTICS
> Heap allocated to OpenSim   : 1456 MB
> Last heap allocation rate   : 20.647 MB/s
> Average heap allocation rate: 9.178 MB/s
> Process memory              : 1868 MB
>
> AppDomainLoading = "true"
> ----------------------------------------
> MEMORY STATISTICS
> Heap allocated to OpenSim   : 918 MB
> Last heap allocation rate   : 20.23 MB/s
> Average heap allocation rate: 12.022 MB/s
> Process memory              : 3570 MB
>
> ---------------------------------------
> Status of XEngine instance for Continente
> Scripts loaded             : 4467
> Scripts waiting for load   : 0
> Max threads                : 100
> Min threads                : 2
> Allocated threads          : 100
> In use threads             : 100
> Work items waiting         : 104
> Events queued              : 418
> Events processed           : 165762
> Sensors                    : 157
> Dataserver requests        : 0
> Timers                     : 702
> Listeners                  : 78
>
> Regards,
> Luisillo
>
> 2015-05-28 17:06 GMT+02:00 Thomas Ringate <[hidden email]>:
>>
>> Very interesting.  My setup is quite different in that I do not run the
>> grid
>> but rather use the OSgrid.  I only run the regions.  That could be a huge
>> difference.  However since OSgrid is officially designated as the
>> "opensim"
>> test grid, I would hope the developers are serious about the results
>> people
>> are having who use that grid.
>>
>> I am in the process of trying to discover exactly when this problem
>> started
>> for me as was earlier requested.
>>
>> The different versions I can test with are these releases from OSgrid:
>> osgrid-opensim-01162015.v0.8.1.97ac80d
>> osgrid-opensim-01172015.v0.8.1.1f04e1b
>> osgrid-opensim-02212015.v0.8.1.7b9ad11
>> osgrid-opensim-03142015.v0.8.1.8b13e4e
>> osgrid-opensim-03172015.v0.8.2.83e58eb
>> osgrid-opensim-04052015.v0.8.2.8d66284
>> osgrid-opensim-04122015.v0.8.2.d96d31b
>> osgrid-opensim-05072015.v0.8.2.c74cef0
>> osgrid-opensim-05092015.v0.8.2.adf0f49
>> osgrid-opensim-05232015.v0.8.2.abb3bb6
>>
>> Later today I will cycle all of these releases on one of my regions and
>> run
>> a test using "monit" with my original parameters set to see which version
>> it
>> was that started using so much simulator memory.  The change control on my
>> home network is not near as professional as what I lived with at IBM for
>> 31
>> years.  Now I depend on my own memory.  That is not a very good choice in
>> my
>> case.  Luisillo's testing was far more accurate with excellent metrics to
>> compare.  Bravo.
>>
>> Tom
>>
>>
>> -----Original Message----- From: Luisillo Contepomi
>> Sent: Thursday, May 28, 2015 5:07 AM
>> To: [hidden email]
>> Subject: Re: [Opensim-users] Memory usage
>>
>>
>> I have not problems or issues with memory.
>> My memory test results:
>>
>> Grid and sim in the same machine.
>> Windows7 - 64 - RAM 8GB
>> Intel Core i7-3770 at 3.40GHz
>> Users in this test is out in other machines connecting by a 100Mb network.
>>
>> All users in this test are using Radegast. All have Inventory with
>> 1200 Items aprox.
>> and worn scripts as AO, radar and resizers in hair and shoes.
>>
>> Continente is a Var Region 1024x1024
>> Prims used 55.048
>> Scripts 4.499
>>
>> Region (Continente) # xengine status
>> Status of XEngine instance for Continente
>> Scripts loaded             : 4499
>> Scripts waiting for load   : 0
>> Max threads                : 100
>> Min threads                : 2
>> Allocated threads          : 100
>> In use threads             : 42
>> Work items waiting         : 0
>> Events queued              : 636
>> Events processed           : 1028273
>> Sensors                    : 157
>> Dataserver requests        : 0
>> Timers                     : 715
>> Listeners                  : 85
>>
>> From tast manager in K
>> CPU 1 from 8 at 70 - 75% all time during test.
>> -----------------
>> Memory OpenSim.exe       /         MySql server
>> Start region at. 9.39h
>> ------------------------------------------------------------
>> 1.613.652 no users    9.40h                652.976
>> 2.140.920 no users    9.45h            "
>> ------------------------------------------------------------
>> finish scripts load       9.46h            "
>>
>> I order in console "fcache clear file" 9.47h Im not using memory cache. "
>> ------------------------------------------------------------
>> 2.142.964 no users 9.47h            "
>> 2.134.240 no users 9.48h                  653.140
>> 1.468.540 no users 9.49h
>> 1.148.276 no users 9.50h (fcache clear finished.)
>>
>> Test with 2 users
>> -------------------------------------------------------------
>> 1.148.384 Login in one user 9.54H (RADEGAST)
>> 1.185.828 with 1 user  9.56h
>> 1.337.280 with 2 users 9.58h            734.548
>>
>> 1.389.148 2 users one is searching in his inventory     735.900
>>
>> 1.411.896 logout one user 10.01h        736.452
>> 1.515.700 1 user 10.02H
>> 1.271.528 1 user 10.03H                    736.468
>> ---------------------------------------------------
>> 1.257.620 logout user No users in 10.04H    736.692
>> 1.221.208 no users 10.05h            736.692
>> 1.207.296 no users 10.07h            736.692
>> 1.204.980 no users 10.07h            735.660
>> 1.204.736 no users 10.09h            735.660
>>
>> ------------simultaneus 8 users login at 10.10h
>> ---------------------------------------------------------------
>>
>> 1.228.172 10.10h.
>> 1.427.396 10.11h  5 users login
>> 1.699.788 10.12h   8 users login
>> 1.937.552 10.13h 8 users in home
>> 2.008.496 10.14h 8 users in home        745.468
>> 2.010.358 10.15h 8 users in home        745.928
>> 1.960.540 10.14h 8 users in home        745.944
>> 1.887.942 10.15h 8 users in home        "
>> 1.946.372 10.16h 8 users in home        "
>> 1.902.660 10.17h 8 users in home location    "
>> 1.927.866 10.18h    "            "
>> 1.872.444 10.19h    "            "
>> 1.885.264 10.20h    "            "
>> 1.838.972 10.23h    "            "
>> 1.908.056 10.25h    "
>>
>> Now logout all users one by minute.
>>
>> 1 - 1.906.736 10.26h                746.528
>> 2 - 1.907.524 10.27h                746.532
>> 3 - 1.876.456 10.28h                746.612
>> 4 - 1.884.836 10.29h                "
>> 5 - 1.788.868 10.30h                746.852
>> 6 - 1.732.212 10.31h                "
>> 7 - 1.756.340 10.32h                "
>> 8 - 1.648.408 10.33h                "
>> NO USERS
>> 1.629.464 10.34h                "
>> 1.618.988 10.36h
>> 1.434.444 10.37h
>>
>> Robust.exe always near 136.570K
>> ---------------------------------------------------------
>> 10.17h When have 8 users STATS:
>> ---------------------------------------------------------
>> Region (Continente) # show stats
>>
>>
>> CONNECTION STATISTICS
>> Client logouts due to no data receive timeout: 0
>>
>>
>> SAMPLE FRAME STATISTICS
>> Dilatn  SimFPS  PhyFPS  AgntUp  RootAg  ChldAg  Prims   AtvPrm  AtvScr
>> ScrLPS
>>  1.00      55    55.6     0.0       8       0   56152      25    4501
>> 10991070
>>
>>
>> PktsIn  PktOut  PendDl  PendUl  UnackB  TotlFt  NetFt   PhysFt  OthrFt
>> AgntFt
>> ImgsFt
>>    20     123       0       0    1142    18.1     0.0     0.4     0.0 0.6
>>   0.2
>>
>>
>> MEMORY STATISTICS
>> Heap allocated to OpenSim   : 1647 MB
>> Last heap allocation rate   : 8.061 MB/s
>> Average heap allocation rate: 9.906 MB/s
>> Process memory              : 2007 MB
>>
>> 10.46H NO USERS STATS
>> ------------------------------------------------------
>> Region (Continente) # show stats
>>
>>
>> CONNECTION STATISTICS
>> Client logouts due to no data receive timeout: 0
>>
>>
>> SAMPLE FRAME STATISTICS
>> Dilatn  SimFPS  PhyFPS  AgntUp  RootAg  ChldAg  Prims   AtvPrm  AtvScr
>> ScrLPS
>>  1.00      55    55.3     0.0       0       0   54875      24    4451
>> 13512840
>>
>>
>> PktsIn  PktOut  PendDl  PendUl  UnackB  TotlFt  NetFt   PhysFt  OthrFt
>> AgntFt
>> ImgsFt
>>    13      78       0       0       0    18.3     0.0     0.6     0.0 0.0
>>   0.0
>>
>>
>> MEMORY STATISTICS
>> Heap allocated to OpenSim   : 768 MB
>> Last heap allocation rate   : 1.732 MB/s
>> Average heap allocation rate: 2.493 MB/s
>> Process memory              : 1493 MB
>>
>> For me all is ok.
>>
>>
>>
>>
>> 2015-05-28 1:53 GMT+02:00 Teravus Ovares <[hidden email]>:
>>>
>>>
>>> https://www.jetbrains.com/dotmemory/ dotmemory has been relatively
>>> successful at identifying memory issues in the past.
>>>
>>> That said, it's not free and there may be free alternatives.....  but be
>>> advised you want one that supports managed memory...    and one that
>>> works
>>> well with Mono.Addins.    Many free ones in the past have had issues
>>> dynamically loading modules with Mono.addins.   If the modules are not
>>> loaded, you will see the server come up with no regions and it will be
>>> pretty useless for testing.   Testing physics is a bit harder because it
>>> is
>>> both managed and unmanaged...   so you'll need to examine the pinned
>>> memory.
>>> Also, note that while running the simulator in a profiler such as
>>> dotMemory,
>>> it will run noticeably slower depending on how you do the profiling.
>>>
>>> Regards
>>>
>>> Teravus
>>>
>>> On Wed, May 27, 2015 at 3:01 AM, M.E. Verhagen <[hidden email]>
>>> wrote:
>>>>
>>>>
>>>>
>>>> The memory consumption is troubling.
>>>>
>>>> It seems to be an cross platform issue, it happens on linux, mac and
>>>> also
>>>> windows.
>>>>
>>>> The script engine can be responsible for the skyrocketing of the memory
>>>> usage when an avatar enters the region.
>>>> The inventory system is also called as a possible suspect.
>>>> Also the physics system can be un issuer.
>>>>
>>>> This throws up a question, how can the memory usage be broken down, to
>>>> see
>>>> what part is to blame ?
>>>>
>>>> _______________________________________________
>>>> Opensim-users mailing list
>>>> [hidden email]
>>>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>>>>
>>>
>>>
>>> _______________________________________________
>>> Opensim-users mailing list
>>> [hidden email]
>>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>>>
>> _______________________________________________
>> Opensim-users mailing list
>> [hidden email]
>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>> _______________________________________________
>> Opensim-users mailing list
>> [hidden email]
>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage

tringate
Luisillo,

I have opened a new mantis on this issue.
[opensim 0007593]: High memory usage when avatar arrives and is never
released

My fingers are crossed that I have not duplicated some other mantis that is
reporting the same problem.

For my regions the problem is very easy to see.  I get the impression that
is not the case for many others who do not have this problem.

My CPU usage is different than before, but it does return to normal levels
soon after an avatar leaves.  What is different is that it goes much higher
now, and lasts longer.

I did watch it while three avatars arrived and both CPU and memory went
nuts.  CPU returned to normal but memory remained at a level four times the
idle level.  When they left, memory stayed at the high level.  It is now a
few hours later, and memory is still at the high level.  A region restart
put everything back to normal until the next time it is visited.

Tom


-----Original Message-----
From: Luisillo Contepomi
Sent: Friday, May 29, 2015 3:53 AM
To: [hidden email]
Subject: Re: [Opensim-users] Memory usage

Im looking about this in mantis.
Was reported from 0.7.3.X but is closed.
Please report it again for the new versión.

http://opensimulator.org/mantis/view.php?id=6030
http://opensimulator.org/mantis/view.php?id=6031

Today I am observing cpu consuption... is hight : from 70% to 100% all
time for one cpu.



2015-05-29 2:24 GMT+02:00 Thomas Ringate <[hidden email]>:

> I tested the following 10 versions of opensim built for the OSgrid.
>
> The times are from the monitor program "monit",and are the uptime.
> The test process was;
> Place a fresh copy of opensim as distributed, copy the needed .ini files
> into the simulator,
> Restart the region.  (This causes "monit" to detect a new PID and resets
> it's "uptime" to be zero)
> 7 minute tick, login with "coolvl" with standard unscripted avatar, record
> metrics.
> 10 minute tick, log off,  record metrics.
> 15 minute tick  record metrics.
> The 7 minute tick metrics represents the IDLE usage of a freshly started
> and
> un-visited region.
> (the reason for waiting 7 minutes was to give ample time to compile the
> 626
> scripts as an initial load)
> The 10 minute tick metrics represents the usage after an avatar has logged
> in and been on the region for approximately 3 minutes.
> The 15 minute tick represents the usage 5 minutes after an avatar has left
> the region and no other usage of the region has taken place.
>
> Region used: Sanctuary  Prims= 7985  Scripts= 626
> -------------------------------------------------------------------  idle
> 1
> avatar                  avatar leave
> -------------------------------------------------------- Time mem kb
> CPU%
> Time mem kb  CPU%  Time  mem kb   CPU%
> osgrid-opensim-01162015.v0.8.1.97ac80d 7m   574952k   3.0
> osgrid-opensim-01172015.v0.8.1.1f04e1b 7m   585076k   2.8  27m  513324k
> 2.9
> 30m   513364k   2.6
> osgrid-opensim-02212015.v0.8.1.7b9ad11 7m   565480k   2.5  10m  560624k
> 2.9
> 15m   536468k   2.5
> osgrid-opensim-03142015.v0.8.1.8b13e4e 7m   568560k   2.6  10m  576828k
> 2.9
> 15m   567436k   2.5
> osgrid-opensim-03172015.v0.8.2.83e58eb 7m   580736k   2.4  10m  583992k
> 2.8
> 15m   572664k   2.5
> osgrid-opensim-04052015.v0.8.2.8d66284 7m   564208k   2.5  10m  574464k
> 2.8
> 15m   574704k   2.5
> osgrid-opensim-04122015.v0.8.2.d96d31b 7m   574392k   2.6  10m  585156k
> 3.0
> 15m   546868k   2.6
> osgrid-opensim-05072015.v0.8.2.c74cef0 7m   583924k   2.6  10m 1139092k
> 2.7
> 15m  1141972k   2.5
> osgrid-opensim-05092015.v0.8.2.adf0f49 7m   578632k   2.5  10m 1207972k
> 2.9
> 15m  1208696k   2.5
> osgrid-opensim-05232015.v0.8.2.abb3bb6 7m   585384K   2.6  10m  584852k
> 5.5
> 15m  1165304k   2.5
>
> Additional comments:
>
> This problem appeared for me in the May 7, 2015 OSgrid release build.  It
> is
> present in the latest build which was May 23, 2015.
> The initial IDLE memory appears to be nearly the same for all versions
> tested.
> Prior to the May 7th release memory usage fluctuated very little when
> avatars came and went.
> After the May 7 release memory usage goes up dramatically, for EACH avatar
> that arrives and does not go back down.
> There was a change on how memory is used between the May 9 and May 23
> builds.  The first builds used the memory much faster, and CPU usage would
> rise quickly last for a short time and then drop back down.
> In the May 23 release, CPU usage goes up much higher, holds at that high
> usage level much longer but does drop back to the "idle" level.  Memory is
> slower to be increased, but still rises to over double that of the idle
> rate
> and does not drop back to the idle level ever.
> The value of AppDomainLoading has been set to "false" in the OSgrid builds
> for all the builds distributed this year that I have, that is what all of
> my
> tests were run with.
> However reading that parameter it reads to me that having it set to FALSE
> would cause this exact problem.
> That does not explain why it never caused this problem prior to the May
> 7th
> build.
>
> I have "monit" configured to allow both memory and CPU usage to go over a
> set limit but if it remains above that limit for longer than what I
> believe
> to be a reasonable time, in my case 3 minutes, it will restart the region.
>
> From 2011 until May 7th 2015 it was rare any of my regions were restarted
> by
> "monit".  After the May 9th release "monit" would restart my regions as
> soon
> as any avatar visited it.  I had set the memory limit to be 1.3GB prior to
> May 7th.  Now setting it at 5GB still results in some regions restarting a
> few times every day.
>
> Until the May 9th release none of my four servers had ever used the swap
> file.  Now the usage of the swap file is in the mid 20%.  Lag has become a
> problem on my regions, and system CPU usage is much higher than it ever
> was
> prior to the May 7th release.
>
> To suggest nothing is wrong with whatever was done to opensim between the
> March release I tested and the May 7th release is very difficult to
> accept.
>
> If the only problem was higher CPU usage I would accept it, but since
> performance and reliability have also been diminished this is not
> something
> that can be ignored.
>
> Memory that gets used and not released is not a good thing.  This is
> happening across three released of Fedora linux.  Fedora 19, 20, and 21,
> on
> four different servers.
>
> I do not know if I should report this in mantis .  Usually when I report
> something it is already being worked on under some other report using
> language I would not think to search on.
>
> Tom
>
>
>
> -----Original Message----- From: Luisillo Contepomi
> Sent: Thursday, May 28, 2015 12:00 PM
> To: [hidden email]
> Subject: Re: [Opensim-users] Memory usage
>
> I have setup always false in "AppDomainLoading" in all my regions.
> Is very important the difference.
>
>
> Results with  "AppDomainLoading" in OpenSim.ini
> ( 4467 scripts in this region, no avatars login.)
>
> AppDomainLoading = "false"
> --------------------------------------------
> MEMORY STATISTICS
> Heap allocated to OpenSim   : 1456 MB
> Last heap allocation rate   : 20.647 MB/s
> Average heap allocation rate: 9.178 MB/s
> Process memory              : 1868 MB
>
> AppDomainLoading = "true"
> ----------------------------------------
> MEMORY STATISTICS
> Heap allocated to OpenSim   : 918 MB
> Last heap allocation rate   : 20.23 MB/s
> Average heap allocation rate: 12.022 MB/s
> Process memory              : 3570 MB
>
> ---------------------------------------
> Status of XEngine instance for Continente
> Scripts loaded             : 4467
> Scripts waiting for load   : 0
> Max threads                : 100
> Min threads                : 2
> Allocated threads          : 100
> In use threads             : 100
> Work items waiting         : 104
> Events queued              : 418
> Events processed           : 165762
> Sensors                    : 157
> Dataserver requests        : 0
> Timers                     : 702
> Listeners                  : 78
>
> Regards,
> Luisillo
>
> 2015-05-28 17:06 GMT+02:00 Thomas Ringate <[hidden email]>:
>>
>> Very interesting.  My setup is quite different in that I do not run the
>> grid
>> but rather use the OSgrid.  I only run the regions.  That could be a huge
>> difference.  However since OSgrid is officially designated as the
>> "opensim"
>> test grid, I would hope the developers are serious about the results
>> people
>> are having who use that grid.
>>
>> I am in the process of trying to discover exactly when this problem
>> started
>> for me as was earlier requested.
>>
>> The different versions I can test with are these releases from OSgrid:
>> osgrid-opensim-01162015.v0.8.1.97ac80d
>> osgrid-opensim-01172015.v0.8.1.1f04e1b
>> osgrid-opensim-02212015.v0.8.1.7b9ad11
>> osgrid-opensim-03142015.v0.8.1.8b13e4e
>> osgrid-opensim-03172015.v0.8.2.83e58eb
>> osgrid-opensim-04052015.v0.8.2.8d66284
>> osgrid-opensim-04122015.v0.8.2.d96d31b
>> osgrid-opensim-05072015.v0.8.2.c74cef0
>> osgrid-opensim-05092015.v0.8.2.adf0f49
>> osgrid-opensim-05232015.v0.8.2.abb3bb6
>>
>> Later today I will cycle all of these releases on one of my regions and
>> run
>> a test using "monit" with my original parameters set to see which version
>> it
>> was that started using so much simulator memory.  The change control on
>> my
>> home network is not near as professional as what I lived with at IBM for
>> 31
>> years.  Now I depend on my own memory.  That is not a very good choice in
>> my
>> case.  Luisillo's testing was far more accurate with excellent metrics to
>> compare.  Bravo.
>>
>> Tom
>>
>>
>> -----Original Message----- From: Luisillo Contepomi
>> Sent: Thursday, May 28, 2015 5:07 AM
>> To: [hidden email]
>> Subject: Re: [Opensim-users] Memory usage
>>
>>
>> I have not problems or issues with memory.
>> My memory test results:
>>
>> Grid and sim in the same machine.
>> Windows7 - 64 - RAM 8GB
>> Intel Core i7-3770 at 3.40GHz
>> Users in this test is out in other machines connecting by a 100Mb
>> network.
>>
>> All users in this test are using Radegast. All have Inventory with
>> 1200 Items aprox.
>> and worn scripts as AO, radar and resizers in hair and shoes.
>>
>> Continente is a Var Region 1024x1024
>> Prims used 55.048
>> Scripts 4.499
>>
>> Region (Continente) # xengine status
>> Status of XEngine instance for Continente
>> Scripts loaded             : 4499
>> Scripts waiting for load   : 0
>> Max threads                : 100
>> Min threads                : 2
>> Allocated threads          : 100
>> In use threads             : 42
>> Work items waiting         : 0
>> Events queued              : 636
>> Events processed           : 1028273
>> Sensors                    : 157
>> Dataserver requests        : 0
>> Timers                     : 715
>> Listeners                  : 85
>>
>> From tast manager in K
>> CPU 1 from 8 at 70 - 75% all time during test.
>> -----------------
>> Memory OpenSim.exe       /         MySql server
>> Start region at. 9.39h
>> ------------------------------------------------------------
>> 1.613.652 no users    9.40h                652.976
>> 2.140.920 no users    9.45h            "
>> ------------------------------------------------------------
>> finish scripts load       9.46h            "
>>
>> I order in console "fcache clear file" 9.47h Im not using memory cache. "
>> ------------------------------------------------------------
>> 2.142.964 no users 9.47h            "
>> 2.134.240 no users 9.48h                  653.140
>> 1.468.540 no users 9.49h
>> 1.148.276 no users 9.50h (fcache clear finished.)
>>
>> Test with 2 users
>> -------------------------------------------------------------
>> 1.148.384 Login in one user 9.54H (RADEGAST)
>> 1.185.828 with 1 user  9.56h
>> 1.337.280 with 2 users 9.58h            734.548
>>
>> 1.389.148 2 users one is searching in his inventory     735.900
>>
>> 1.411.896 logout one user 10.01h        736.452
>> 1.515.700 1 user 10.02H
>> 1.271.528 1 user 10.03H                    736.468
>> ---------------------------------------------------
>> 1.257.620 logout user No users in 10.04H    736.692
>> 1.221.208 no users 10.05h            736.692
>> 1.207.296 no users 10.07h            736.692
>> 1.204.980 no users 10.07h            735.660
>> 1.204.736 no users 10.09h            735.660
>>
>> ------------simultaneus 8 users login at 10.10h
>> ---------------------------------------------------------------
>>
>> 1.228.172 10.10h.
>> 1.427.396 10.11h  5 users login
>> 1.699.788 10.12h   8 users login
>> 1.937.552 10.13h 8 users in home
>> 2.008.496 10.14h 8 users in home        745.468
>> 2.010.358 10.15h 8 users in home        745.928
>> 1.960.540 10.14h 8 users in home        745.944
>> 1.887.942 10.15h 8 users in home        "
>> 1.946.372 10.16h 8 users in home        "
>> 1.902.660 10.17h 8 users in home location    "
>> 1.927.866 10.18h    "            "
>> 1.872.444 10.19h    "            "
>> 1.885.264 10.20h    "            "
>> 1.838.972 10.23h    "            "
>> 1.908.056 10.25h    "
>>
>> Now logout all users one by minute.
>>
>> 1 - 1.906.736 10.26h                746.528
>> 2 - 1.907.524 10.27h                746.532
>> 3 - 1.876.456 10.28h                746.612
>> 4 - 1.884.836 10.29h                "
>> 5 - 1.788.868 10.30h                746.852
>> 6 - 1.732.212 10.31h                "
>> 7 - 1.756.340 10.32h                "
>> 8 - 1.648.408 10.33h                "
>> NO USERS
>> 1.629.464 10.34h                "
>> 1.618.988 10.36h
>> 1.434.444 10.37h
>>
>> Robust.exe always near 136.570K
>> ---------------------------------------------------------
>> 10.17h When have 8 users STATS:
>> ---------------------------------------------------------
>> Region (Continente) # show stats
>>
>>
>> CONNECTION STATISTICS
>> Client logouts due to no data receive timeout: 0
>>
>>
>> SAMPLE FRAME STATISTICS
>> Dilatn  SimFPS  PhyFPS  AgntUp  RootAg  ChldAg  Prims   AtvPrm  AtvScr
>> ScrLPS
>>  1.00      55    55.6     0.0       8       0   56152      25    4501
>> 10991070
>>
>>
>> PktsIn  PktOut  PendDl  PendUl  UnackB  TotlFt  NetFt   PhysFt  OthrFt
>> AgntFt
>> ImgsFt
>>    20     123       0       0    1142    18.1     0.0     0.4     0.0 0.6
>>   0.2
>>
>>
>> MEMORY STATISTICS
>> Heap allocated to OpenSim   : 1647 MB
>> Last heap allocation rate   : 8.061 MB/s
>> Average heap allocation rate: 9.906 MB/s
>> Process memory              : 2007 MB
>>
>> 10.46H NO USERS STATS
>> ------------------------------------------------------
>> Region (Continente) # show stats
>>
>>
>> CONNECTION STATISTICS
>> Client logouts due to no data receive timeout: 0
>>
>>
>> SAMPLE FRAME STATISTICS
>> Dilatn  SimFPS  PhyFPS  AgntUp  RootAg  ChldAg  Prims   AtvPrm  AtvScr
>> ScrLPS
>>  1.00      55    55.3     0.0       0       0   54875      24    4451
>> 13512840
>>
>>
>> PktsIn  PktOut  PendDl  PendUl  UnackB  TotlFt  NetFt   PhysFt  OthrFt
>> AgntFt
>> ImgsFt
>>    13      78       0       0       0    18.3     0.0     0.6     0.0 0.0
>>   0.0
>>
>>
>> MEMORY STATISTICS
>> Heap allocated to OpenSim   : 768 MB
>> Last heap allocation rate   : 1.732 MB/s
>> Average heap allocation rate: 2.493 MB/s
>> Process memory              : 1493 MB
>>
>> For me all is ok.
>>
>>
>>
>>
>> 2015-05-28 1:53 GMT+02:00 Teravus Ovares <[hidden email]>:
>>>
>>>
>>> https://www.jetbrains.com/dotmemory/ dotmemory has been relatively
>>> successful at identifying memory issues in the past.
>>>
>>> That said, it's not free and there may be free alternatives.....  but be
>>> advised you want one that supports managed memory...    and one that
>>> works
>>> well with Mono.Addins.    Many free ones in the past have had issues
>>> dynamically loading modules with Mono.addins.   If the modules are not
>>> loaded, you will see the server come up with no regions and it will be
>>> pretty useless for testing.   Testing physics is a bit harder because it
>>> is
>>> both managed and unmanaged...   so you'll need to examine the pinned
>>> memory.
>>> Also, note that while running the simulator in a profiler such as
>>> dotMemory,
>>> it will run noticeably slower depending on how you do the profiling.
>>>
>>> Regards
>>>
>>> Teravus
>>>
>>> On Wed, May 27, 2015 at 3:01 AM, M.E. Verhagen <[hidden email]>
>>> wrote:
>>>>
>>>>
>>>>
>>>> The memory consumption is troubling.
>>>>
>>>> It seems to be an cross platform issue, it happens on linux, mac and
>>>> also
>>>> windows.
>>>>
>>>> The script engine can be responsible for the skyrocketing of the memory
>>>> usage when an avatar enters the region.
>>>> The inventory system is also called as a possible suspect.
>>>> Also the physics system can be un issuer.
>>>>
>>>> This throws up a question, how can the memory usage be broken down, to
>>>> see
>>>> what part is to blame ?
>>>>
>>>> _______________________________________________
>>>> Opensim-users mailing list
>>>> [hidden email]
>>>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>>>>
>>>
>>>
>>> _______________________________________________
>>> Opensim-users mailing list
>>> [hidden email]
>>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>>>
>> _______________________________________________
>> Opensim-users mailing list
>> [hidden email]
>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>> _______________________________________________
>> Opensim-users mailing list
>> [hidden email]
>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users 

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage

Dahlia Trimble
I'm not sure if this is your problem but some recent changes that improve inventory loading may be contributing to this. As I'm told (I haven't verified this personally), prior versions would make many redundant requests to the inventory server. Now, fewer requests are made and they may be for larger chunks of data. Typically when many requests are made via http, some throttling (intentional and not intentional) occurs which would free the CPU for other tasks, or show it as idle. Now, less throttling may be necessary as fewer irrelevant requests are made and the ones that complete can be used effectively which could make the CPU usage appear higher and require more memory, especially for large inventories. The nature of the .NET managed runtime environment is such that the memory is only freed when the garbage collection process is invoked which could be significantly delayed, which is probably why you see the memory use decrease over time.

I'm not saying this is definitely your problem, only that it's something I'm aware of that might be a cause of your symptoms you describe. If so, it's likely not a bug but rather a difference in how things are working with the goal of improved performance.

On Fri, May 29, 2015 at 1:35 PM, Thomas Ringate <[hidden email]> wrote:
Luisillo,

I have opened a new mantis on this issue.
[opensim 0007593]: High memory usage when avatar arrives and is never released

My fingers are crossed that I have not duplicated some other mantis that is reporting the same problem.

For my regions the problem is very easy to see.  I get the impression that is not the case for many others who do not have this problem.

My CPU usage is different than before, but it does return to normal levels soon after an avatar leaves.  What is different is that it goes much higher now, and lasts longer.

I did watch it while three avatars arrived and both CPU and memory went nuts.  CPU returned to normal but memory remained at a level four times the idle level.  When they left, memory stayed at the high level.  It is now a few hours later, and memory is still at the high level.  A region restart put everything back to normal until the next time it is visited.

Tom


-----Original Message----- From: Luisillo Contepomi
Sent: Friday, May 29, 2015 3:53 AM

To: [hidden email]
Subject: Re: [Opensim-users] Memory usage

Im looking about this in mantis.
Was reported from 0.7.3.X but is closed.
Please report it again for the new versión.

http://opensimulator.org/mantis/view.php?id=6030
http://opensimulator.org/mantis/view.php?id=6031

Today I am observing cpu consuption... is hight : from 70% to 100% all
time for one cpu.



2015-05-29 2:24 GMT+02:00 Thomas Ringate <[hidden email]>:
I tested the following 10 versions of opensim built for the OSgrid.

The times are from the monitor program "monit",and are the uptime.
The test process was;
Place a fresh copy of opensim as distributed, copy the needed .ini files
into the simulator,
Restart the region.  (This causes "monit" to detect a new PID and resets
it's "uptime" to be zero)
7 minute tick, login with "coolvl" with standard unscripted avatar, record
metrics.
10 minute tick, log off,  record metrics.
15 minute tick  record metrics.
The 7 minute tick metrics represents the IDLE usage of a freshly started and
un-visited region.
(the reason for waiting 7 minutes was to give ample time to compile the 626
scripts as an initial load)
The 10 minute tick metrics represents the usage after an avatar has logged
in and been on the region for approximately 3 minutes.
The 15 minute tick represents the usage 5 minutes after an avatar has left
the region and no other usage of the region has taken place.

Region used: Sanctuary  Prims= 7985  Scripts= 626
-------------------------------------------------------------------  idle 1
avatar                  avatar leave
-------------------------------------------------------- Time mem kb CPU%
Time mem kb  CPU%  Time  mem kb   CPU%
osgrid-opensim-01162015.v0.8.1.97ac80d 7m   574952k   3.0
osgrid-opensim-01172015.v0.8.1.1f04e1b 7m   585076k   2.8  27m  513324k 2.9
30m   513364k   2.6
osgrid-opensim-02212015.v0.8.1.7b9ad11 7m   565480k   2.5  10m  560624k 2.9
15m   536468k   2.5
osgrid-opensim-03142015.v0.8.1.8b13e4e 7m   568560k   2.6  10m  576828k 2.9
15m   567436k   2.5
osgrid-opensim-03172015.v0.8.2.83e58eb 7m   580736k   2.4  10m  583992k 2.8
15m   572664k   2.5
osgrid-opensim-04052015.v0.8.2.8d66284 7m   564208k   2.5  10m  574464k 2.8
15m   574704k   2.5
osgrid-opensim-04122015.v0.8.2.d96d31b 7m   574392k   2.6  10m  585156k 3.0
15m   546868k   2.6
osgrid-opensim-05072015.v0.8.2.c74cef0 7m   583924k   2.6  10m 1139092k 2.7
15m  1141972k   2.5
osgrid-opensim-05092015.v0.8.2.adf0f49 7m   578632k   2.5  10m 1207972k 2.9
15m  1208696k   2.5
osgrid-opensim-05232015.v0.8.2.abb3bb6 7m   585384K   2.6  10m  584852k 5.5
15m  1165304k   2.5

Additional comments:

This problem appeared for me in the May 7, 2015 OSgrid release build.  It is
present in the latest build which was May 23, 2015.
The initial IDLE memory appears to be nearly the same for all versions
tested.
Prior to the May 7th release memory usage fluctuated very little when
avatars came and went.
After the May 7 release memory usage goes up dramatically, for EACH avatar
that arrives and does not go back down.
There was a change on how memory is used between the May 9 and May 23
builds.  The first builds used the memory much faster, and CPU usage would
rise quickly last for a short time and then drop back down.
In the May 23 release, CPU usage goes up much higher, holds at that high
usage level much longer but does drop back to the "idle" level.  Memory is
slower to be increased, but still rises to over double that of the idle rate
and does not drop back to the idle level ever.
The value of AppDomainLoading has been set to "false" in the OSgrid builds
for all the builds distributed this year that I have, that is what all of my
tests were run with.
However reading that parameter it reads to me that having it set to FALSE
would cause this exact problem.
That does not explain why it never caused this problem prior to the May 7th
build.

I have "monit" configured to allow both memory and CPU usage to go over a
set limit but if it remains above that limit for longer than what I believe
to be a reasonable time, in my case 3 minutes, it will restart the region.

From 2011 until May 7th 2015 it was rare any of my regions were restarted by
"monit".  After the May 9th release "monit" would restart my regions as soon
as any avatar visited it.  I had set the memory limit to be 1.3GB prior to
May 7th.  Now setting it at 5GB still results in some regions restarting a
few times every day.

Until the May 9th release none of my four servers had ever used the swap
file.  Now the usage of the swap file is in the mid 20%.  Lag has become a
problem on my regions, and system CPU usage is much higher than it ever was
prior to the May 7th release.

To suggest nothing is wrong with whatever was done to opensim between the
March release I tested and the May 7th release is very difficult to accept.

If the only problem was higher CPU usage I would accept it, but since
performance and reliability have also been diminished this is not something
that can be ignored.

Memory that gets used and not released is not a good thing.  This is
happening across three released of Fedora linux.  Fedora 19, 20, and 21, on
four different servers.

I do not know if I should report this in mantis .  Usually when I report
something it is already being worked on under some other report using
language I would not think to search on.

Tom



-----Original Message----- From: Luisillo Contepomi
Sent: Thursday, May 28, 2015 12:00 PM
To: [hidden email]
Subject: Re: [Opensim-users] Memory usage

I have setup always false in "AppDomainLoading" in all my regions.
Is very important the difference.


Results with  "AppDomainLoading" in OpenSim.ini
( 4467 scripts in this region, no avatars login.)

AppDomainLoading = "false"
--------------------------------------------
MEMORY STATISTICS
Heap allocated to OpenSim   : 1456 MB
Last heap allocation rate   : 20.647 MB/s
Average heap allocation rate: 9.178 MB/s
Process memory              : 1868 MB

AppDomainLoading = "true"
----------------------------------------
MEMORY STATISTICS
Heap allocated to OpenSim   : 918 MB
Last heap allocation rate   : 20.23 MB/s
Average heap allocation rate: 12.022 MB/s
Process memory              : 3570 MB

---------------------------------------
Status of XEngine instance for Continente
Scripts loaded             : 4467
Scripts waiting for load   : 0
Max threads                : 100
Min threads                : 2
Allocated threads          : 100
In use threads             : 100
Work items waiting         : 104
Events queued              : 418
Events processed           : 165762
Sensors                    : 157
Dataserver requests        : 0
Timers                     : 702
Listeners                  : 78

Regards,
Luisillo

2015-05-28 17:06 GMT+02:00 Thomas Ringate <[hidden email]>:

Very interesting.  My setup is quite different in that I do not run the
grid
but rather use the OSgrid.  I only run the regions.  That could be a huge
difference.  However since OSgrid is officially designated as the
"opensim"
test grid, I would hope the developers are serious about the results
people
are having who use that grid.

I am in the process of trying to discover exactly when this problem
started
for me as was earlier requested.

The different versions I can test with are these releases from OSgrid:
osgrid-opensim-01162015.v0.8.1.97ac80d
osgrid-opensim-01172015.v0.8.1.1f04e1b
osgrid-opensim-02212015.v0.8.1.7b9ad11
osgrid-opensim-03142015.v0.8.1.8b13e4e
osgrid-opensim-03172015.v0.8.2.83e58eb
osgrid-opensim-04052015.v0.8.2.8d66284
osgrid-opensim-04122015.v0.8.2.d96d31b
osgrid-opensim-05072015.v0.8.2.c74cef0
osgrid-opensim-05092015.v0.8.2.adf0f49
osgrid-opensim-05232015.v0.8.2.abb3bb6

Later today I will cycle all of these releases on one of my regions and
run
a test using "monit" with my original parameters set to see which version
it
was that started using so much simulator memory.  The change control on my
home network is not near as professional as what I lived with at IBM for
31
years.  Now I depend on my own memory.  That is not a very good choice in
my
case.  Luisillo's testing was far more accurate with excellent metrics to
compare.  Bravo.

Tom


-----Original Message----- From: Luisillo Contepomi
Sent: Thursday, May 28, 2015 5:07 AM
To: [hidden email]
Subject: Re: [Opensim-users] Memory usage


I have not problems or issues with memory.
My memory test results:

Grid and sim in the same machine.
Windows7 - 64 - RAM 8GB
Intel Core i7-3770 at 3.40GHz
Users in this test is out in other machines connecting by a 100Mb network.

All users in this test are using Radegast. All have Inventory with
1200 Items aprox.
and worn scripts as AO, radar and resizers in hair and shoes.

Continente is a Var Region 1024x1024
Prims used 55.048
Scripts 4.499

Region (Continente) # xengine status
Status of XEngine instance for Continente
Scripts loaded             : 4499
Scripts waiting for load   : 0
Max threads                : 100
Min threads                : 2
Allocated threads          : 100
In use threads             : 42
Work items waiting         : 0
Events queued              : 636
Events processed           : 1028273
Sensors                    : 157
Dataserver requests        : 0
Timers                     : 715
Listeners                  : 85

From tast manager in K
CPU 1 from 8 at 70 - 75% all time during test.
-----------------
Memory OpenSim.exe       /         MySql server
Start region at. 9.39h
------------------------------------------------------------
1.613.652 no users    9.40h                652.976
2.140.920 no users    9.45h            "
------------------------------------------------------------
finish scripts load       9.46h            "

I order in console "fcache clear file" 9.47h Im not using memory cache. "
------------------------------------------------------------
2.142.964 no users 9.47h            "
2.134.240 no users 9.48h                  653.140
1.468.540 no users 9.49h
1.148.276 no users 9.50h (fcache clear finished.)

Test with 2 users
-------------------------------------------------------------
1.148.384 Login in one user 9.54H (RADEGAST)
1.185.828 with 1 user  9.56h
1.337.280 with 2 users 9.58h            734.548

1.389.148 2 users one is searching in his inventory     735.900

1.411.896 logout one user 10.01h        736.452
1.515.700 1 user 10.02H
1.271.528 1 user 10.03H                    736.468
---------------------------------------------------
1.257.620 logout user No users in 10.04H    736.692
1.221.208 no users 10.05h            736.692
1.207.296 no users 10.07h            736.692
1.204.980 no users 10.07h            735.660
1.204.736 no users 10.09h            735.660

------------simultaneus 8 users login at 10.10h
---------------------------------------------------------------

1.228.172 10.10h.
1.427.396 10.11h  5 users login
1.699.788 10.12h   8 users login
1.937.552 10.13h 8 users in home
2.008.496 10.14h 8 users in home        745.468
2.010.358 10.15h 8 users in home        745.928
1.960.540 10.14h 8 users in home        745.944
1.887.942 10.15h 8 users in home        "
1.946.372 10.16h 8 users in home        "
1.902.660 10.17h 8 users in home location    "
1.927.866 10.18h    "            "
1.872.444 10.19h    "            "
1.885.264 10.20h    "            "
1.838.972 10.23h    "            "
1.908.056 10.25h    "

Now logout all users one by minute.

1 - 1.906.736 10.26h                746.528
2 - 1.907.524 10.27h                746.532
3 - 1.876.456 10.28h                746.612
4 - 1.884.836 10.29h                "
5 - 1.788.868 10.30h                746.852
6 - 1.732.212 10.31h                "
7 - 1.756.340 10.32h                "
8 - 1.648.408 10.33h                "
NO USERS
1.629.464 10.34h                "
1.618.988 10.36h
1.434.444 10.37h

Robust.exe always near 136.570K
---------------------------------------------------------
10.17h When have 8 users STATS:
---------------------------------------------------------
Region (Continente) # show stats


CONNECTION STATISTICS
Client logouts due to no data receive timeout: 0


SAMPLE FRAME STATISTICS
Dilatn  SimFPS  PhyFPS  AgntUp  RootAg  ChldAg  Prims   AtvPrm  AtvScr
ScrLPS
 1.00      55    55.6     0.0       8       0   56152      25    4501
10991070


PktsIn  PktOut  PendDl  PendUl  UnackB  TotlFt  NetFt   PhysFt  OthrFt
AgntFt
ImgsFt
   20     123       0       0    1142    18.1     0.0     0.4     0.0 0.6
  0.2


MEMORY STATISTICS
Heap allocated to OpenSim   : 1647 MB
Last heap allocation rate   : 8.061 MB/s
Average heap allocation rate: 9.906 MB/s
Process memory              : 2007 MB

10.46H NO USERS STATS
------------------------------------------------------
Region (Continente) # show stats


CONNECTION STATISTICS
Client logouts due to no data receive timeout: 0


SAMPLE FRAME STATISTICS
Dilatn  SimFPS  PhyFPS  AgntUp  RootAg  ChldAg  Prims   AtvPrm  AtvScr
ScrLPS
 1.00      55    55.3     0.0       0       0   54875      24    4451
13512840


PktsIn  PktOut  PendDl  PendUl  UnackB  TotlFt  NetFt   PhysFt  OthrFt
AgntFt
ImgsFt
   13      78       0       0       0    18.3     0.0     0.6     0.0 0.0
  0.0


MEMORY STATISTICS
Heap allocated to OpenSim   : 768 MB
Last heap allocation rate   : 1.732 MB/s
Average heap allocation rate: 2.493 MB/s
Process memory              : 1493 MB

For me all is ok.




2015-05-28 1:53 GMT+02:00 Teravus Ovares <[hidden email]>:


https://www.jetbrains.com/dotmemory/ dotmemory has been relatively
successful at identifying memory issues in the past.

That said, it's not free and there may be free alternatives.....  but be
advised you want one that supports managed memory...    and one that
works
well with Mono.Addins.    Many free ones in the past have had issues
dynamically loading modules with Mono.addins.   If the modules are not
loaded, you will see the server come up with no regions and it will be
pretty useless for testing.   Testing physics is a bit harder because it
is
both managed and unmanaged...   so you'll need to examine the pinned
memory.
Also, note that while running the simulator in a profiler such as
dotMemory,
it will run noticeably slower depending on how you do the profiling.

Regards

Teravus

On Wed, May 27, 2015 at 3:01 AM, M.E. Verhagen <[hidden email]>
wrote:



The memory consumption is troubling.

It seems to be an cross platform issue, it happens on linux, mac and
also
windows.

The script engine can be responsible for the skyrocketing of the memory
usage when an avatar enters the region.
The inventory system is also called as a possible suspect.
Also the physics system can be un issuer.

This throws up a question, how can the memory usage be broken down, to
see
what part is to blame ?

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users



_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users


_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users