What is going on with the latest release of opensim for OSgrid

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

What is going on with the latest release of opensim for OSgrid

tringate
I just did a update to the latest release of opensim released by OSgrid and it took 1 hour and 12 minutes from the time the command was give to load before it enabled logins.
 
This is on a 4 by 4 var region that previously took 17 minutes to load.  While that is nothing to write home about, an hour and 12 minutes is totally insane.
 
This region runs on a dedicated server with nothing else running on it at all.  It is a 6 core AMD with 16GB of ram.
 
Using “top” to view cpu usage the most I ever noticed it going to as 16% total with only one core hitting 100% for a few MS every now and then.
 
It looked like it too most og that time generating the 16 tile textures for the grid display, but scripts did take about 11 minutes to get started.  That part did not appear to be significantly different than before.
 
Here are some key log entries so you can see how long each thing is taking.
 
2017-06-25 10:34:15,481 INFO  [OPENSIM MAIN]: configured log4net using default OpenSim.exe.config  <—first log entry
2017-06-25 10:34:25,743 DEBUG [REGION DB]: Loaded 5694 objects using 47381 prims
2017-06-25 10:34:31,207 DEBUG [REGION DB]: Loaded inventory from 3563 objects
2017-06-25 10:34:31,209 INFO  [SCENE]: Loaded 5694 objects from the datastore
2017-06-25 10:34:40,244 DEBUG [WORLD MAP]: Generating map image for TSim 1
                                                                                                                                                        <—big delay
2017-06-25 11:17:53,580 DEBUG [WORLD MAP]: Storing map image c393c67c-eb69-43b1-b6bd-650d06c59e99 for TSim 1
2017-06-25 11:17:55,255 INFO  [SCENE]: Initializing script instances in TSim 1
2017-06-25 11:19:14,586 INFO  [SCENE]: Initialized 6604 script instances in TSim 1
2017-06-25 11:19:14,979 INFO  [ubOde] start processing pending actor operations
                                                                                                                                                        <—big delay
2017-06-25 11:29:23,776 INFO  [XEngine]: Started 50 scripts in TSim 1
                                                                                                                                                        <—big delay but not much different than previous version
2017-06-25 11:45:33,268 INFO  [XEngine]: Completed starting 6583 scripts on TSim 1
2017-06-25 11:45:57,478 DEBUG [RegionReady]: Region "TSim 1" is ready: "server_startup,1,21," on channel -800
 
This is a very limited log entry list as there are hundreds of entries many of them listing bad textures or other things indicating corrupt assets coming from the asset server.
 
This was a cold start where it does have to generate and compile stuff, but it has never taken any where near this long for a region to come online before.
 
No one else has noticed this huge increase in start-up time?
 
I’m running 4 servers, so I tried this same region on my test server, it only has 8gb or ram and a bit slower 6 core AMD cpu.  It took 2 hours on that server.
 
My other 4 by 4 region TSim 2 takes 12 minutes to come online.  This is not the cold start time, which does take longer.
 
2017-06-25 04:02:57,726 INFO  [OPENSIM MAIN]: configured log4net using default OpenSim.exe.config
2017-06-25 04:04:22,724 DEBUG [REGION DB]: Loaded 3068 objects using 33622 prims
2017-06-25 04:03:25,024 INFO  [XEngine]: Initializing scripts in region TSim 2
2017-06-25 04:09:34,146 INFO  [XEngine]: Started 50 scripts in TSim 2
2017-06-25 04:15:45,075 INFO  [XEngine]: Completed starting 4534 scripts on TSim 2
2017-06-25 04:15:45,138 DEBUG [RegionReady]: Region "TSim 2" is ready: "server_startup,1,27," on channel -800
 
 
 
 
 

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What is going on with the latest release of opensim for OSgrid

AJLDuarte

Hi

                What was the state of folder bin/AssetsCache on those 2 tests?

                According to options, map generation my require full assets  fetch from grid and that can take ages if not cached.

                Of course, make sure FloatSam cache is proper configure. It should no longer be optional, since it is needed and others don’t do the job now.

 

Also seems your t2 is not doing map and not using ubOde ?

                Even with all assets present in cache map is heavy, can take a bit

                ubOde does operations before scene Ready so that ready does mean it a bit more

Other engines do it later so you don’t see it and region Ready really doesn’t mean much.

Ubit

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of [hidden email]
Sent: Sunday, June 25, 2017 17:32
To: OS-Opensim Users
Subject: [Opensim-users] What is going on with the latest release of opensim for OSgrid

 

I just did a update to the latest release of opensim released by OSgrid and it took 1 hour and 12 minutes from the time the command was give to load before it enabled logins.

 

This is on a 4 by 4 var region that previously took 17 minutes to load.  While that is nothing to write home about, an hour and 12 minutes is totally insane.

 

This region runs on a dedicated server with nothing else running on it at all.  It is a 6 core AMD with 16GB of ram.

 

Using “top” to view cpu usage the most I ever noticed it going to as 16% total with only one core hitting 100% for a few MS every now and then.

 

It looked like it too most og that time generating the 16 tile textures for the grid display, but scripts did take about 11 minutes to get started.  That part did not appear to be significantly different than before.

 

Here are some key log entries so you can see how long each thing is taking.

 

2017-06-25 10:34:15,481 INFO  [OPENSIM MAIN]: configured log4net using default OpenSim.exe.config  <—first log entry

2017-06-25 10:34:25,743 DEBUG [REGION DB]: Loaded 5694 objects using 47381 prims

2017-06-25 10:34:31,207 DEBUG [REGION DB]: Loaded inventory from 3563 objects

2017-06-25 10:34:31,209 INFO  [SCENE]: Loaded 5694 objects from the datastore

2017-06-25 10:34:40,244 DEBUG [WORLD MAP]: Generating map image for TSim 1

                                                                                                                                                        <—big delay

2017-06-25 11:17:53,580 DEBUG [WORLD MAP]: Storing map image c393c67c-eb69-43b1-b6bd-650d06c59e99 for TSim 1

2017-06-25 11:17:55,255 INFO  [SCENE]: Initializing script instances in TSim 1

2017-06-25 11:19:14,586 INFO  [SCENE]: Initialized 6604 script instances in TSim 1

2017-06-25 11:19:14,979 INFO  [ubOde] start processing pending actor operations

                                                                                                                                                        <—big delay

2017-06-25 11:29:23,776 INFO  [XEngine]: Started 50 scripts in TSim 1

                                                                                                                                                        <—big delay but not much different than previous version

2017-06-25 11:45:33,268 INFO  [XEngine]: Completed starting 6583 scripts on TSim 1

2017-06-25 11:45:57,478 DEBUG [RegionReady]: Region "TSim 1" is ready: "server_startup,1,21," on channel -800

 

This is a very limited log entry list as there are hundreds of entries many of them listing bad textures or other things indicating corrupt assets coming from the asset server.

 

This was a cold start where it does have to generate and compile stuff, but it has never taken any where near this long for a region to come online before.

 

No one else has noticed this huge increase in start-up time?

 

I’m running 4 servers, so I tried this same region on my test server, it only has 8gb or ram and a bit slower 6 core AMD cpu.  It took 2 hours on that server.

 

My other 4 by 4 region TSim 2 takes 12 minutes to come online.  This is not the cold start time, which does take longer.

 

2017-06-25 04:02:57,726 INFO  [OPENSIM MAIN]: configured log4net using default OpenSim.exe.config

2017-06-25 04:04:22,724 DEBUG [REGION DB]: Loaded 3068 objects using 33622 prims

2017-06-25 04:03:25,024 INFO  [XEngine]: Initializing scripts in region TSim 2

2017-06-25 04:09:34,146 INFO  [XEngine]: Started 50 scripts in TSim 2

2017-06-25 04:15:45,075 INFO  [XEngine]: Completed starting 4534 scripts on TSim 2

2017-06-25 04:15:45,138 DEBUG [RegionReady]: Region "TSim 2" is ready: "server_startup,1,27," on channel -800

 

 

 

 

 


_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What is going on with the latest release ofopensim for OSgrid

tringate
Ubit,
 
Thanks for replying so fast.  I missed that TSim 2 had not been set to use ubOde so I change that setting and it is restarting now.
The cache is properly set up and is retained for “FileCacheTimeout = 48”.
 
Here is the new start time for TSim 2 with only the physic engine changed to ubOde;
2017-06-25 13:17:06,208 INFO  [OPENSIM MAIN]: configured log4net using default OpenSim.exe.config
2017-06-25 13:17:38,581 DEBUG [REGION DB]: Loaded 3068 objects using 33622 prims
2017-06-25 13:31:40,085 INFO  [ubOde] 19624 prim actors loaded
2017-06-25 13:31:55,090 INFO  [XEngine]: Performing initial script startup on TSim 2
2017-06-25 13:31:55,778 INFO  [XEngine]: Started 50 scripts in TSim 2
2017-06-25 13:35:17,226 INFO  [XEngine]: Completed starting 4534 scripts on TSim 2
2017-06-25 13:35:17,293 DEBUG [RegionReady]: Region "TSim 2" is ready: "server_startup,1,27," on channel –800
 
So with ubOde it took about 18 minutes vs. 12 minutes.
 
The “ready” message occurs at the same time the “logins enabled” message appears on the instance console and is when the region becomes actually alive on OSgrid for people to be able to not get the “reagion not available” and sent to a close by region.
 
Can I ask another question totally of this topic?
 
How hard would it be to fix the date and time stamps that are logged in things like “last login” and such in the group windows?  None of the date things have anything in them other than “unknown” or sometimes some old date back in the 1990’s.
 
From: AJLDuarte
Sent: Sunday, June 25, 2017 12:55 PM
Subject: Re: [Opensim-users] What is going on with the latest release ofopensim for OSgrid
 

Hi

                What was the state of folder bin/AssetsCache on those 2 tests?

                According to options, map generation my require full assets  fetch from grid and that can take ages if not cached.

                Of course, make sure FloatSam cache is proper configure. It should no longer be optional, since it is needed and others don’t do the job now.

 

Also seems your t2 is not doing map and not using ubOde ?

                Even with all assets present in cache map is heavy, can take a bit

                ubOde does operations before scene Ready so that ready does mean it a bit more

Other engines do it later so you don’t see it and region Ready really doesn’t mean much.

Ubit

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of [hidden email]
Sent: Sunday, June 25, 2017 17:32
To: OS-Opensim Users
Subject: [Opensim-users] What is going on with the latest release of opensim for OSgrid

 

I just did a update to the latest release of opensim released by OSgrid and it took 1 hour and 12 minutes from the time the command was give to load before it enabled logins.

 

This is on a 4 by 4 var region that previously took 17 minutes to load.  While that is nothing to write home about, an hour and 12 minutes is totally insane.

 

This region runs on a dedicated server with nothing else running on it at all.  It is a 6 core AMD with 16GB of ram.

 

Using “top” to view cpu usage the most I ever noticed it going to as 16% total with only one core hitting 100% for a few MS every now and then.

 

It looked like it too most og that time generating the 16 tile textures for the grid display, but scripts did take about 11 minutes to get started.  That part did not appear to be significantly different than before.

 

Here are some key log entries so you can see how long each thing is taking.

 

2017-06-25 10:34:15,481 INFO  [OPENSIM MAIN]: configured log4net using default OpenSim.exe.config  <—first log entry

2017-06-25 10:34:25,743 DEBUG [REGION DB]: Loaded 5694 objects using 47381 prims

2017-06-25 10:34:31,207 DEBUG [REGION DB]: Loaded inventory from 3563 objects

2017-06-25 10:34:31,209 INFO  [SCENE]: Loaded 5694 objects from the datastore

2017-06-25 10:34:40,244 DEBUG [WORLD MAP]: Generating map image for TSim 1

                                                                                                                                                        <—big delay

2017-06-25 11:17:53,580 DEBUG [WORLD MAP]: Storing map image c393c67c-eb69-43b1-b6bd-650d06c59e99 for TSim 1

2017-06-25 11:17:55,255 INFO  [SCENE]: Initializing script instances in TSim 1

2017-06-25 11:19:14,586 INFO  [SCENE]: Initialized 6604 script instances in TSim 1

2017-06-25 11:19:14,979 INFO  [ubOde] start processing pending actor operations

                                                                                                                                                        <—big delay

2017-06-25 11:29:23,776 INFO  [XEngine]: Started 50 scripts in TSim 1

                                                                                                                                                        <—big delay but not much different than previous version

2017-06-25 11:45:33,268 INFO  [XEngine]: Completed starting 6583 scripts on TSim 1

2017-06-25 11:45:57,478 DEBUG [RegionReady]: Region "TSim 1" is ready: "server_startup,1,21," on channel -800

 

This is a very limited log entry list as there are hundreds of entries many of them listing bad textures or other things indicating corrupt assets coming from the asset server.

 

This was a cold start where it does have to generate and compile stuff, but it has never taken any where near this long for a region to come online before.

 

No one else has noticed this huge increase in start-up time?

 

I’m running 4 servers, so I tried this same region on my test server, it only has 8gb or ram and a bit slower 6 core AMD cpu.  It took 2 hours on that server.

 

My other 4 by 4 region TSim 2 takes 12 minutes to come online.  This is not the cold start time, which does take longer.

 

2017-06-25 04:02:57,726 INFO  [OPENSIM MAIN]: configured log4net using default OpenSim.exe.config

2017-06-25 04:04:22,724 DEBUG [REGION DB]: Loaded 3068 objects using 33622 prims

2017-06-25 04:03:25,024 INFO  [XEngine]: Initializing scripts in region TSim 2

2017-06-25 04:09:34,146 INFO  [XEngine]: Started 50 scripts in TSim 2

2017-06-25 04:15:45,075 INFO  [XEngine]: Completed starting 4534 scripts on TSim 2

2017-06-25 04:15:45,138 DEBUG [RegionReady]: Region "TSim 2" is ready: "server_startup,1,27," on channel -800

 

 

 

 

 


_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What is going on with the latest release ofopensim for OSgrid

tringate
In reply to this post by AJLDuarte
Ubit,
 
After checking everything I decided to do a “warm” start, one where it is not a total upgrade with all the compiled stuff deleted.  This time the restart took 14 minutes which is what I normally observe.  I guess I never watched a cold restart before so I guess maybe it always took that long.  Usually I let the updates happen at the middle of the night restart I have scheduled.  I always found everything working just fine in the mornings.
 
Sorry for the alarm, but the 70 minute restart time took me by surprise.
 
One good thing I did learn from all of this was that it may not be necessary to do a reboot every night to prevent lag.  One of my servers ran 12 days with no reboot because I had forgot to turn the nightly reboot back on during a routine manual maintenance I did earlier this month.
 
I may go to a once a week reboot on one server and see how that goes.  If all goes well, I’ll change them all to once a week.
 
Tom
 
 
From: AJLDuarte
Sent: Sunday, June 25, 2017 12:55 PM
Subject: Re: [Opensim-users] What is going on with the latest release ofopensim for OSgrid
 

Hi

                What was the state of folder bin/AssetsCache on those 2 tests?

                According to options, map generation my require full assets  fetch from grid and that can take ages if not cached.

                Of course, make sure FloatSam cache is proper configure. It should no longer be optional, since it is needed and others don’t do the job now.

 

Also seems your t2 is not doing map and not using ubOde ?

                Even with all assets present in cache map is heavy, can take a bit

                ubOde does operations before scene Ready so that ready does mean it a bit more

Other engines do it later so you don’t see it and region Ready really doesn’t mean much.

Ubit

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of [hidden email]
Sent: Sunday, June 25, 2017 17:32
To: OS-Opensim Users
Subject: [Opensim-users] What is going on with the latest release of opensim for OSgrid

 

I just did a update to the latest release of opensim released by OSgrid and it took 1 hour and 12 minutes from the time the command was give to load before it enabled logins.

 

This is on a 4 by 4 var region that previously took 17 minutes to load.  While that is nothing to write home about, an hour and 12 minutes is totally insane.

 

This region runs on a dedicated server with nothing else running on it at all.  It is a 6 core AMD with 16GB of ram.

 

Using “top” to view cpu usage the most I ever noticed it going to as 16% total with only one core hitting 100% for a few MS every now and then.

 

It looked like it too most og that time generating the 16 tile textures for the grid display, but scripts did take about 11 minutes to get started.  That part did not appear to be significantly different than before.

 

Here are some key log entries so you can see how long each thing is taking.

 

2017-06-25 10:34:15,481 INFO  [OPENSIM MAIN]: configured log4net using default OpenSim.exe.config  <—first log entry

2017-06-25 10:34:25,743 DEBUG [REGION DB]: Loaded 5694 objects using 47381 prims

2017-06-25 10:34:31,207 DEBUG [REGION DB]: Loaded inventory from 3563 objects

2017-06-25 10:34:31,209 INFO  [SCENE]: Loaded 5694 objects from the datastore

2017-06-25 10:34:40,244 DEBUG [WORLD MAP]: Generating map image for TSim 1

                                                                                                                                                        <—big delay

2017-06-25 11:17:53,580 DEBUG [WORLD MAP]: Storing map image c393c67c-eb69-43b1-b6bd-650d06c59e99 for TSim 1

2017-06-25 11:17:55,255 INFO  [SCENE]: Initializing script instances in TSim 1

2017-06-25 11:19:14,586 INFO  [SCENE]: Initialized 6604 script instances in TSim 1

2017-06-25 11:19:14,979 INFO  [ubOde] start processing pending actor operations

                                                                                                                                                        <—big delay

2017-06-25 11:29:23,776 INFO  [XEngine]: Started 50 scripts in TSim 1

                                                                                                                                                        <—big delay but not much different than previous version

2017-06-25 11:45:33,268 INFO  [XEngine]: Completed starting 6583 scripts on TSim 1

2017-06-25 11:45:57,478 DEBUG [RegionReady]: Region "TSim 1" is ready: "server_startup,1,21," on channel -800

 

This is a very limited log entry list as there are hundreds of entries many of them listing bad textures or other things indicating corrupt assets coming from the asset server.

 

This was a cold start where it does have to generate and compile stuff, but it has never taken any where near this long for a region to come online before.

 

No one else has noticed this huge increase in start-up time?

 

I’m running 4 servers, so I tried this same region on my test server, it only has 8gb or ram and a bit slower 6 core AMD cpu.  It took 2 hours on that server.

 

My other 4 by 4 region TSim 2 takes 12 minutes to come online.  This is not the cold start time, which does take longer.

 

2017-06-25 04:02:57,726 INFO  [OPENSIM MAIN]: configured log4net using default OpenSim.exe.config

2017-06-25 04:04:22,724 DEBUG [REGION DB]: Loaded 3068 objects using 33622 prims

2017-06-25 04:03:25,024 INFO  [XEngine]: Initializing scripts in region TSim 2

2017-06-25 04:09:34,146 INFO  [XEngine]: Started 50 scripts in TSim 2

2017-06-25 04:15:45,075 INFO  [XEngine]: Completed starting 4534 scripts on TSim 2

2017-06-25 04:15:45,138 DEBUG [RegionReady]: Region "TSim 2" is ready: "server_startup,1,27," on channel -800

 

 

 

 

 


_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What is going on with the latest release ofopensim for OSgrid

AJLDuarte

Hi

                Good to read things are not so bad after all

                About reboots, do them as often as your use case allows (and backups of course).

                Regions memory use keeps piling up.. etc, etc

Ubit

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of [hidden email]
Sent: Monday, June 26, 2017 01:28
To: [hidden email]
Subject: Re: [Opensim-users] What is going on with the latest release ofopensim for OSgrid

 

Ubit,

 

After checking everything I decided to do a “warm” start, one where it is not a total upgrade with all the compiled stuff deleted.  This time the restart took 14 minutes which is what I normally observe.  I guess I never watched a cold restart before so I guess maybe it always took that long.  Usually I let the updates happen at the middle of the night restart I have scheduled.  I always found everything working just fine in the mornings.

 

Sorry for the alarm, but the 70 minute restart time took me by surprise.

 

One good thing I did learn from all of this was that it may not be necessary to do a reboot every night to prevent lag.  One of my servers ran 12 days with no reboot because I had forgot to turn the nightly reboot back on during a routine manual maintenance I did earlier this month.

 

I may go to a once a week reboot on one server and see how that goes.  If all goes well, I’ll change them all to once a week.

 

Tom

 

 

From: AJLDuarte

Sent: Sunday, June 25, 2017 12:55 PM

Subject: Re: [Opensim-users] What is going on with the latest release ofopensim for OSgrid

 

Hi

                What was the state of folder bin/AssetsCache on those 2 tests?

                According to options, map generation my require full assets  fetch from grid and that can take ages if not cached.

                Of course, make sure FloatSam cache is proper configure. It should no longer be optional, since it is needed and others don’t do the job now.

 

Also seems your t2 is not doing map and not using ubOde ?

                Even with all assets present in cache map is heavy, can take a bit

                ubOde does operations before scene Ready so that ready does mean it a bit more

Other engines do it later so you don’t see it and region Ready really doesn’t mean much.

Ubit

 

 

From: [hidden email] [[hidden email]] On Behalf Of [hidden email]
Sent: Sunday, June 25, 2017 17:32
To: OS-Opensim Users
Subject: [Opensim-users] What is going on with the latest release of opensim for OSgrid

 

I just did a update to the latest release of opensim released by OSgrid and it took 1 hour and 12 minutes from the time the command was give to load before it enabled logins.

 

This is on a 4 by 4 var region that previously took 17 minutes to load.  While that is nothing to write home about, an hour and 12 minutes is totally insane.

 

This region runs on a dedicated server with nothing else running on it at all.  It is a 6 core AMD with 16GB of ram.

 

Using “top” to view cpu usage the most I ever noticed it going to as 16% total with only one core hitting 100% for a few MS every now and then.

 

It looked like it too most og that time generating the 16 tile textures for the grid display, but scripts did take about 11 minutes to get started.  That part did not appear to be significantly different than before.

 

Here are some key log entries so you can see how long each thing is taking.

 

2017-06-25 10:34:15,481 INFO  [OPENSIM MAIN]: configured log4net using default OpenSim.exe.config  <—first log entry

2017-06-25 10:34:25,743 DEBUG [REGION DB]: Loaded 5694 objects using 47381 prims

2017-06-25 10:34:31,207 DEBUG [REGION DB]: Loaded inventory from 3563 objects

2017-06-25 10:34:31,209 INFO  [SCENE]: Loaded 5694 objects from the datastore

2017-06-25 10:34:40,244 DEBUG [WORLD MAP]: Generating map image for TSim 1

                                                                                                                                                        <—big delay

2017-06-25 11:17:53,580 DEBUG [WORLD MAP]: Storing map image c393c67c-eb69-43b1-b6bd-650d06c59e99 for TSim 1

2017-06-25 11:17:55,255 INFO  [SCENE]: Initializing script instances in TSim 1

2017-06-25 11:19:14,586 INFO  [SCENE]: Initialized 6604 script instances in TSim 1

2017-06-25 11:19:14,979 INFO  [ubOde] start processing pending actor operations

                                                                                                                                                        <—big delay

2017-06-25 11:29:23,776 INFO  [XEngine]: Started 50 scripts in TSim 1

                                                                                                                                                        <—big delay but not much different than previous version

2017-06-25 11:45:33,268 INFO  [XEngine]: Completed starting 6583 scripts on TSim 1

2017-06-25 11:45:57,478 DEBUG [RegionReady]: Region "TSim 1" is ready: "server_startup,1,21," on channel -800

 

This is a very limited log entry list as there are hundreds of entries many of them listing bad textures or other things indicating corrupt assets coming from the asset server.

 

This was a cold start where it does have to generate and compile stuff, but it has never taken any where near this long for a region to come online before.

 

No one else has noticed this huge increase in start-up time?

 

I’m running 4 servers, so I tried this same region on my test server, it only has 8gb or ram and a bit slower 6 core AMD cpu.  It took 2 hours on that server.

 

My other 4 by 4 region TSim 2 takes 12 minutes to come online.  This is not the cold start time, which does take longer.

 

2017-06-25 04:02:57,726 INFO  [OPENSIM MAIN]: configured log4net using default OpenSim.exe.config

2017-06-25 04:04:22,724 DEBUG [REGION DB]: Loaded 3068 objects using 33622 prims

2017-06-25 04:03:25,024 INFO  [XEngine]: Initializing scripts in region TSim 2

2017-06-25 04:09:34,146 INFO  [XEngine]: Started 50 scripts in TSim 2

2017-06-25 04:15:45,075 INFO  [XEngine]: Completed starting 4534 scripts on TSim 2

2017-06-25 04:15:45,138 DEBUG [RegionReady]: Region "TSim 2" is ready: "server_startup,1,27," on channel -800

 

 

 

 

 


_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users


_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What is going on with the latest release ofopensim for OSgrid

Chris M
I have been seeing similar issues myself between the past week and a half - two weeks on master

Though the previous responses in this thread mention map and asset cache population as being a possible issue I'm not so sure that this is the case in this example because there is very little rezzed out in my case; 2 of the regions are completely empty and the other 2 are sparsely populated with random cube prims here and there mostly for script testing.

This is being seen on a test install independent from my main grid (not running both at the same time), clean database, 4 regions, grid mode (Not HG, 1 OpenSim.exe instance), Windows 7 x64 under .NET, and on ubODE. I'm also using the default map tile module (That is, the old one, and not Warp3D)

Log File Examples:

17:05:47 - [REGION LOADER FILE SYSTEM]: Loaded config for region Test Sandbox
17:05:47 - [LOAD REGIONS PLUGIN]: Loading specific shared modules...
17:05:47 - [LOAD REGIONS PLUGIN]: Done.
17:05:47 - [LOAD REGIONS PLUGIN]: Creating Region: Monte Cristo (ThreadID: 1)
17:05:47 - [SCENE]: Default script engine XEngine
17:05:47 - [SCENE]: Region Monte Cristo, WORLD MAP refresh time set to 0 seconds

17:05:47 - [SCENE]: Using the BestAvatarResponsiveness prioritization scheme
17:05:47 - [HEIGHTMAP TERRAIN DATA]: VD2Gzip 296 bytes input
17:05:47 - [HEIGHTMAP TERRAIN DATA]: V2DGzip. Heightmap size=<256,256>. Region s
ize=<256,256>
17:05:47 - [HEIGHTMAP TERRAIN DATA]: HeightmapTerrainData create from Variable2D
Gzip serialization. Size=<256,256>
17:05:47 - [HEIGHTMAP TERRAIN DATA]: VD2Gzip 296 bytes input
17:05:47 - [HEIGHTMAP TERRAIN DATA]: V2DGzip. Heightmap size=<256,256>. Region s
ize=<256,256>
17:05:47 - [HEIGHTMAP TERRAIN DATA]: HeightmapTerrainData create from Variable2D
Gzip serialization. Size=<256,256>
17:05:47 - [MODULES]: Loading Region's modules (old style)
17:05:47 - [REGIONMODULES]: Loading Region's modules (new style)
17:05:47 - [CHAT]: Initialized for Monte Cristo w:10 s:20 S:100
17:05:47 - [MESSAGE TRANSFER]: Message transfer module active
17:05:47 - [REMOTE GRID USER CONNECTOR]: Enabled remote grid user for region Mon
te Cristo

(A lot of times it gets stuck for a long time here)



Another log from a different OS session:

17:07:55 - [LOAD REGIONS PLUGIN]: Creating Region: Monte Cristo (ThreadID: 1)
17:07:55 - [SCENE]: Default script engine XEngine
17:07:55 - [SCENE]: Region Monte Cristo, WORLD MAP refresh time set to 0 seconds

17:07:55 - [SCENE]: Using the BestAvatarResponsiveness prioritization scheme
17:07:55 - [HEIGHTMAP TERRAIN DATA]: VD2Gzip 296 bytes input
17:07:55 - [HEIGHTMAP TERRAIN DATA]: V2DGzip. Heightmap size=<256,256>. Region s
ize=<256,256>
17:07:55 - [HEIGHTMAP TERRAIN DATA]: HeightmapTerrainData create from Variable2D
Gzip serialization. Size=<256,256>
17:07:55 - [HEIGHTMAP TERRAIN DATA]: VD2Gzip 296 bytes input
17:07:55 - [HEIGHTMAP TERRAIN DATA]: V2DGzip. Heightmap size=<256,256>. Region s
ize=<256,256>
17:07:55 - [HEIGHTMAP TERRAIN DATA]: HeightmapTerrainData create from Variable2D
Gzip serialization. Size=<256,256>
17:07:55 - [MODULES]: Loading Region's modules (old style)
17:07:55 - [REGIONMODULES]: Loading Region's modules (new style)

(Sometimes it gets stuck for a long time here but not as often as the first example)



On 6/25/2017 7:40 PM, AJLDuarte wrote:

Hi

                Good to read things are not so bad after all

                About reboots, do them as often as your use case allows (and backups of course).

                Regions memory use keeps piling up.. etc, etc

Ubit

 

 

From: [hidden email] [[hidden email]] On Behalf Of [hidden email]
Sent: Monday, June 26, 2017 01:28
To: [hidden email]
Subject: Re: [Opensim-users] What is going on with the latest release ofopensim for OSgrid

 

Ubit,

 

After checking everything I decided to do a “warm” start, one where it is not a total upgrade with all the compiled stuff deleted.  This time the restart took 14 minutes which is what I normally observe.  I guess I never watched a cold restart before so I guess maybe it always took that long.  Usually I let the updates happen at the middle of the night restart I have scheduled.  I always found everything working just fine in the mornings.

 

Sorry for the alarm, but the 70 minute restart time took me by surprise.

 

One good thing I did learn from all of this was that it may not be necessary to do a reboot every night to prevent lag.  One of my servers ran 12 days with no reboot because I had forgot to turn the nightly reboot back on during a routine manual maintenance I did earlier this month.

 

I may go to a once a week reboot on one server and see how that goes.  If all goes well, I’ll change them all to once a week.

 

Tom

 

 

From: AJLDuarte

Sent: Sunday, June 25, 2017 12:55 PM

Subject: Re: [Opensim-users] What is going on with the latest release ofopensim for OSgrid

 

Hi

                What was the state of folder bin/AssetsCache on those 2 tests?

                According to options, map generation my require full assets  fetch from grid and that can take ages if not cached.

                Of course, make sure FloatSam cache is proper configure. It should no longer be optional, since it is needed and others don’t do the job now.

 

Also seems your t2 is not doing map and not using ubOde ?

                Even with all assets present in cache map is heavy, can take a bit

                ubOde does operations before scene Ready so that ready does mean it a bit more

Other engines do it later so you don’t see it and region Ready really doesn’t mean much.

Ubit

 

 

From: [hidden email] [[hidden email]] On Behalf Of [hidden email]
Sent: Sunday, June 25, 2017 17:32
To: OS-Opensim Users
Subject: [Opensim-users] What is going on with the latest release of opensim for OSgrid

 

I just did a update to the latest release of opensim released by OSgrid and it took 1 hour and 12 minutes from the time the command was give to load before it enabled logins.

 

This is on a 4 by 4 var region that previously took 17 minutes to load.  While that is nothing to write home about, an hour and 12 minutes is totally insane.

 

This region runs on a dedicated server with nothing else running on it at all.  It is a 6 core AMD with 16GB of ram.

 

Using “top” to view cpu usage the most I ever noticed it going to as 16% total with only one core hitting 100% for a few MS every now and then.

 

It looked like it too most og that time generating the 16 tile textures for the grid display, but scripts did take about 11 minutes to get started.  That part did not appear to be significantly different than before.

 

Here are some key log entries so you can see how long each thing is taking.

 

2017-06-25 10:34:15,481 INFO  [OPENSIM MAIN]: configured log4net using default OpenSim.exe.config  <—first log entry

2017-06-25 10:34:25,743 DEBUG [REGION DB]: Loaded 5694 objects using 47381 prims

2017-06-25 10:34:31,207 DEBUG [REGION DB]: Loaded inventory from 3563 objects

2017-06-25 10:34:31,209 INFO  [SCENE]: Loaded 5694 objects from the datastore

2017-06-25 10:34:40,244 DEBUG [WORLD MAP]: Generating map image for TSim 1

                                                                                                                                                        <—big delay

2017-06-25 11:17:53,580 DEBUG [WORLD MAP]: Storing map image c393c67c-eb69-43b1-b6bd-650d06c59e99 for TSim 1

2017-06-25 11:17:55,255 INFO  [SCENE]: Initializing script instances in TSim 1

2017-06-25 11:19:14,586 INFO  [SCENE]: Initialized 6604 script instances in TSim 1

2017-06-25 11:19:14,979 INFO  [ubOde] start processing pending actor operations

                                                                                                                                                        <—big delay

2017-06-25 11:29:23,776 INFO  [XEngine]: Started 50 scripts in TSim 1

                                                                                                                                                        <—big delay but not much different than previous version

2017-06-25 11:45:33,268 INFO  [XEngine]: Completed starting 6583 scripts on TSim 1

2017-06-25 11:45:57,478 DEBUG [RegionReady]: Region "TSim 1" is ready: "server_startup,1,21," on channel -800

 

This is a very limited log entry list as there are hundreds of entries many of them listing bad textures or other things indicating corrupt assets coming from the asset server.

 

This was a cold start where it does have to generate and compile stuff, but it has never taken any where near this long for a region to come online before.

 

No one else has noticed this huge increase in start-up time?

 

I’m running 4 servers, so I tried this same region on my test server, it only has 8gb or ram and a bit slower 6 core AMD cpu.  It took 2 hours on that server.

 

My other 4 by 4 region TSim 2 takes 12 minutes to come online.  This is not the cold start time, which does take longer.

 

2017-06-25 04:02:57,726 INFO  [OPENSIM MAIN]: configured log4net using default OpenSim.exe.config

2017-06-25 04:04:22,724 DEBUG [REGION DB]: Loaded 3068 objects using 33622 prims

2017-06-25 04:03:25,024 INFO  [XEngine]: Initializing scripts in region TSim 2

2017-06-25 04:09:34,146 INFO  [XEngine]: Started 50 scripts in TSim 2

2017-06-25 04:15:45,075 INFO  [XEngine]: Completed starting 4534 scripts on TSim 2

2017-06-25 04:15:45,138 DEBUG [RegionReady]: Region "TSim 2" is ready: "server_startup,1,27," on channel -800

 

 

 

 

 


_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users



_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users


-- 
OpenSim: 10 Region Standalone on 0.8.1 Dev
Physics: Open Dynamics Engine
OS: Windows 7 (x64)
CPU: AMD FX 8320 8-Core 3.5 GHz
Memory: 16 GB DDR3
Database: MySQL 5.1.63 (x64)

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What is going on with the latest release ofopensim for OSgrid

David Saunders-2
What kind of connection do you have between the asset server and the simulators. We have some very large regions that are pack to the brim, they rarely take 30 min from cold start, a warm start is in minutes before the login are enabled. Out asset servers are on the same lan as the simulator servers  It takes about 60 min to soft restart the server.  And this will delay start up of simulators keeps simulators from locking out the core code from each other.  We figured .8 to .9 its taks about double the memory and maybe 25% longer start times. Main due to physic engine  

Was looking at a few of our big regions, they take maybe 5 min to load assets at max, and maybe the same with scripts. This is with 0 cache hits. and that at teh largest regions we have right now with 30k objects/8k scripts 

We use an older version of .9 that 2 months old, and been waiting for the release version to upgrade, we have a  bit of work to incorporate our grid mods into the code.   were in a production environment and have to stay a step behind the bleeding edge or it will cut deep :)

even now we have to restart the region about 1 to 2 days because of memory leak and the "creep". The memory leak is when the region grow from 1.4gb used to 20gb use. The creep is something a little different,  it apears the region disconnects from the grid, and you can TP to it or out of it.  A soft restart and it working like new. All I have come with is it being disconnected from the Asset server and not reconnecting.  

Wish you luck, I might be making a new OSGrid region soon and maybe enable hypergrid who knows our little garden works for us. 

David.


On Tue, Jun 27, 2017 at 9:01 PM, Chris <[hidden email]> wrote:
I have been seeing similar issues myself between the past week and a half - two weeks on master

Though the previous responses in this thread mention map and asset cache population as being a possible issue I'm not so sure that this is the case in this example because there is very little rezzed out in my case; 2 of the regions are completely empty and the other 2 are sparsely populated with random cube prims here and there mostly for script testing.

This is being seen on a test install independent from my main grid (not running both at the same time), clean database, 4 regions, grid mode (Not HG, 1 OpenSim.exe instance), Windows 7 x64 under .NET, and on ubODE. I'm also using the default map tile module (That is, the old one, and not Warp3D)

Log File Examples:

17:05:47 - [REGION LOADER FILE SYSTEM]: Loaded config for region Test Sandbox
17:05:47 - [LOAD REGIONS PLUGIN]: Loading specific shared modules...
17:05:47 - [LOAD REGIONS PLUGIN]: Done.
17:05:47 - [LOAD REGIONS PLUGIN]: Creating Region: Monte Cristo (ThreadID: 1)
17:05:47 - [SCENE]: Default script engine XEngine
17:05:47 - [SCENE]: Region Monte Cristo, WORLD MAP refresh time set to 0 seconds

17:05:47 - [SCENE]: Using the BestAvatarResponsiveness prioritization scheme
17:05:47 - [HEIGHTMAP TERRAIN DATA]: VD2Gzip 296 bytes input
17:05:47 - [HEIGHTMAP TERRAIN DATA]: V2DGzip. Heightmap size=<256,256>. Region s
ize=<256,256>
17:05:47 - [HEIGHTMAP TERRAIN DATA]: HeightmapTerrainData create from Variable2D
Gzip serialization. Size=<256,256>
17:05:47 - [HEIGHTMAP TERRAIN DATA]: VD2Gzip 296 bytes input
17:05:47 - [HEIGHTMAP TERRAIN DATA]: V2DGzip. Heightmap size=<256,256>. Region s
ize=<256,256>
17:05:47 - [HEIGHTMAP TERRAIN DATA]: HeightmapTerrainData create from Variable2D
Gzip serialization. Size=<256,256>
17:05:47 - [MODULES]: Loading Region's modules (old style)
17:05:47 - [REGIONMODULES]: Loading Region's modules (new style)
17:05:47 - [CHAT]: Initialized for Monte Cristo w:10 s:20 S:100
17:05:47 - [MESSAGE TRANSFER]: Message transfer module active
17:05:47 - [REMOTE GRID USER CONNECTOR]: Enabled remote grid user for region Mon
te Cristo

(A lot of times it gets stuck for a long time here)



Another log from a different OS session:

17:07:55 - [LOAD REGIONS PLUGIN]: Creating Region: Monte Cristo (ThreadID: 1)
17:07:55 - [SCENE]: Default script engine XEngine
17:07:55 - [SCENE]: Region Monte Cristo, WORLD MAP refresh time set to 0 seconds

17:07:55 - [SCENE]: Using the BestAvatarResponsiveness prioritization scheme
17:07:55 - [HEIGHTMAP TERRAIN DATA]: VD2Gzip 296 bytes input
17:07:55 - [HEIGHTMAP TERRAIN DATA]: V2DGzip. Heightmap size=<256,256>. Region s
ize=<256,256>
17:07:55 - [HEIGHTMAP TERRAIN DATA]: HeightmapTerrainData create from Variable2D
Gzip serialization. Size=<256,256>
17:07:55 - [HEIGHTMAP TERRAIN DATA]: VD2Gzip 296 bytes input
17:07:55 - [HEIGHTMAP TERRAIN DATA]: V2DGzip. Heightmap size=<256,256>. Region s
ize=<256,256>
17:07:55 - [HEIGHTMAP TERRAIN DATA]: HeightmapTerrainData create from Variable2D
Gzip serialization. Size=<256,256>
17:07:55 - [MODULES]: Loading Region's modules (old style)
17:07:55 - [REGIONMODULES]: Loading Region's modules (new style)

(Sometimes it gets stuck for a long time here but not as often as the first example)




On 6/25/2017 7:40 PM, AJLDuarte wrote:

Hi

                Good to read things are not so bad after all

                About reboots, do them as often as your use case allows (and backups of course).

                Regions memory use keeps piling up.. etc, etc

Ubit

 

 

From: [hidden email] [[hidden email]] On Behalf Of [hidden email]
Sent: Monday, June 26, 2017 01:28
To: [hidden email]
Subject: Re: [Opensim-users] What is going on with the latest release ofopensim for OSgrid

 

Ubit,

 

After checking everything I decided to do a “warm” start, one where it is not a total upgrade with all the compiled stuff deleted.  This time the restart took 14 minutes which is what I normally observe.  I guess I never watched a cold restart before so I guess maybe it always took that long.  Usually I let the updates happen at the middle of the night restart I have scheduled.  I always found everything working just fine in the mornings.

 

Sorry for the alarm, but the 70 minute restart time took me by surprise.

 

One good thing I did learn from all of this was that it may not be necessary to do a reboot every night to prevent lag.  One of my servers ran 12 days with no reboot because I had forgot to turn the nightly reboot back on during a routine manual maintenance I did earlier this month.

 

I may go to a once a week reboot on one server and see how that goes.  If all goes well, I’ll change them all to once a week.

 

Tom

 

 

From: AJLDuarte

Sent: Sunday, June 25, 2017 12:55 PM

Subject: Re: [Opensim-users] What is going on with the latest release ofopensim for OSgrid

 

Hi

                What was the state of folder bin/AssetsCache on those 2 tests?

                According to options, map generation my require full assets  fetch from grid and that can take ages if not cached.

                Of course, make sure FloatSam cache is proper configure. It should no longer be optional, since it is needed and others don’t do the job now.

 

Also seems your t2 is not doing map and not using ubOde ?

                Even with all assets present in cache map is heavy, can take a bit

                ubOde does operations before scene Ready so that ready does mean it a bit more

Other engines do it later so you don’t see it and region Ready really doesn’t mean much.

Ubit

 

 

From: [hidden email] [[hidden email]] On Behalf Of [hidden email]
Sent: Sunday, June 25, 2017 17:32
To: OS-Opensim Users
Subject: [Opensim-users] What is going on with the latest release of opensim for OSgrid

 

I just did a update to the latest release of opensim released by OSgrid and it took 1 hour and 12 minutes from the time the command was give to load before it enabled logins.

 

This is on a 4 by 4 var region that previously took 17 minutes to load.  While that is nothing to write home about, an hour and 12 minutes is totally insane.

 

This region runs on a dedicated server with nothing else running on it at all.  It is a 6 core AMD with 16GB of ram.

 

Using “top” to view cpu usage the most I ever noticed it going to as 16% total with only one core hitting 100% for a few MS every now and then.

 

It looked like it too most og that time generating the 16 tile textures for the grid display, but scripts did take about 11 minutes to get started.  That part did not appear to be significantly different than before.

 

Here are some key log entries so you can see how long each thing is taking.

 

2017-06-25 10:34:15,481 INFO  [OPENSIM MAIN]: configured log4net using default OpenSim.exe.config  <—first log entry

2017-06-25 10:34:25,743 DEBUG [REGION DB]: Loaded 5694 objects using 47381 prims

2017-06-25 10:34:31,207 DEBUG [REGION DB]: Loaded inventory from 3563 objects

2017-06-25 10:34:31,209 INFO  [SCENE]: Loaded 5694 objects from the datastore

2017-06-25 10:34:40,244 DEBUG [WORLD MAP]: Generating map image for TSim 1

                                                                                                                                                        <—big delay

2017-06-25 11:17:53,580 DEBUG [WORLD MAP]: Storing map image c393c67c-eb69-43b1-b6bd-650d06c59e99 for TSim 1

2017-06-25 11:17:55,255 INFO  [SCENE]: Initializing script instances in TSim 1

2017-06-25 11:19:14,586 INFO  [SCENE]: Initialized 6604 script instances in TSim 1

2017-06-25 11:19:14,979 INFO  [ubOde] start processing pending actor operations

                                                                                                                                                        <—big delay

2017-06-25 11:29:23,776 INFO  [XEngine]: Started 50 scripts in TSim 1

                                                                                                                                                        <—big delay but not much different than previous version

2017-06-25 11:45:33,268 INFO  [XEngine]: Completed starting 6583 scripts on TSim 1

2017-06-25 11:45:57,478 DEBUG [RegionReady]: Region "TSim 1" is ready: "server_startup,1,21," on channel -800

 

This is a very limited log entry list as there are hundreds of entries many of them listing bad textures or other things indicating corrupt assets coming from the asset server.

 

This was a cold start where it does have to generate and compile stuff, but it has never taken any where near this long for a region to come online before.

 

No one else has noticed this huge increase in start-up time?

 

I’m running 4 servers, so I tried this same region on my test server, it only has 8gb or ram and a bit slower 6 core AMD cpu.  It took 2 hours on that server.

 

My other 4 by 4 region TSim 2 takes 12 minutes to come online.  This is not the cold start time, which does take longer.

 

2017-06-25 04:02:57,726 INFO  [OPENSIM MAIN]: configured log4net using default OpenSim.exe.config

2017-06-25 04:04:22,724 DEBUG [REGION DB]: Loaded 3068 objects using 33622 prims

2017-06-25 04:03:25,024 INFO  [XEngine]: Initializing scripts in region TSim 2

2017-06-25 04:09:34,146 INFO  [XEngine]: Started 50 scripts in TSim 2

2017-06-25 04:15:45,075 INFO  [XEngine]: Completed starting 4534 scripts on TSim 2

2017-06-25 04:15:45,138 DEBUG [RegionReady]: Region "TSim 2" is ready: "server_startup,1,27," on channel -800

 

 

 

 

 


_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users



_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users


-- 
OpenSim: 10 Region Standalone on 0.8.1 Dev
Physics: Open Dynamics Engine
OS: Windows 7 (x64)
CPU: AMD FX 8320 8-Core 3.5 GHz
Memory: 16 GB DDR3
Database: MySQL 5.1.63 (x64)

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users



_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What is going on with the latest release ofopensim for OSgrid

Chris M
Everything is local to the same system (MySQL, ROBUST, and OpenSim) and as it's a test install is configured to only establish connections within my LAN (i.e. no outside access and doesn't connect to anything outside of my LAN). But as stated previously there's almost nothing on the test install which is why I'm a bit perplexed as to why OpenSim start up would hang for long periods of times at seemingly random points. Slightly older revisions, I want to say perhaps 2 or 3 weeks older, don't do this on the same install.

On 6/28/2017 9:33 AM, David Saunders wrote:
What kind of connection do you have between the asset server and the simulators. We have some very large regions that are pack to the brim, they rarely take 30 min from cold start, a warm start is in minutes before the login are enabled. Out asset servers are on the same lan as the simulator servers  It takes about 60 min to soft restart the server.  And this will delay start up of simulators keeps simulators from locking out the core code from each other.  We figured .8 to .9 its taks about double the memory and maybe 25% longer start times. Main due to physic engine  

Was looking at a few of our big regions, they take maybe 5 min to load assets at max, and maybe the same with scripts. This is with 0 cache hits. and that at teh largest regions we have right now with 30k objects/8k scripts 

We use an older version of .9 that 2 months old, and been waiting for the release version to upgrade, we have a  bit of work to incorporate our grid mods into the code.   were in a production environment and have to stay a step behind the bleeding edge or it will cut deep :)

even now we have to restart the region about 1 to 2 days because of memory leak and the "creep". The memory leak is when the region grow from 1.4gb used to 20gb use. The creep is something a little different,  it apears the region disconnects from the grid, and you can TP to it or out of it.  A soft restart and it working like new. All I have come with is it being disconnected from the Asset server and not reconnecting.  

Wish you luck, I might be making a new OSGrid region soon and maybe enable hypergrid who knows our little garden works for us. 

David.


On Tue, Jun 27, 2017 at 9:01 PM, Chris <[hidden email]> wrote:
I have been seeing similar issues myself between the past week and a half - two weeks on master

Though the previous responses in this thread mention map and asset cache population as being a possible issue I'm not so sure that this is the case in this example because there is very little rezzed out in my case; 2 of the regions are completely empty and the other 2 are sparsely populated with random cube prims here and there mostly for script testing.

This is being seen on a test install independent from my main grid (not running both at the same time), clean database, 4 regions, grid mode (Not HG, 1 OpenSim.exe instance), Windows 7 x64 under .NET, and on ubODE. I'm also using the default map tile module (That is, the old one, and not Warp3D)

Log File Examples:

17:05:47 - [REGION LOADER FILE SYSTEM]: Loaded config for region Test Sandbox
17:05:47 - [LOAD REGIONS PLUGIN]: Loading specific shared modules...
17:05:47 - [LOAD REGIONS PLUGIN]: Done.
17:05:47 - [LOAD REGIONS PLUGIN]: Creating Region: Monte Cristo (ThreadID: 1)
17:05:47 - [SCENE]: Default script engine XEngine
17:05:47 - [SCENE]: Region Monte Cristo, WORLD MAP refresh time set to 0 seconds

17:05:47 - [SCENE]: Using the BestAvatarResponsiveness prioritization scheme
17:05:47 - [HEIGHTMAP TERRAIN DATA]: VD2Gzip 296 bytes input
17:05:47 - [HEIGHTMAP TERRAIN DATA]: V2DGzip. Heightmap size=<256,256>. Region s
ize=<256,256>
17:05:47 - [HEIGHTMAP TERRAIN DATA]: HeightmapTerrainData create from Variable2D
Gzip serialization. Size=<256,256>
17:05:47 - [HEIGHTMAP TERRAIN DATA]: VD2Gzip 296 bytes input
17:05:47 - [HEIGHTMAP TERRAIN DATA]: V2DGzip. Heightmap size=<256,256>. Region s
ize=<256,256>
17:05:47 - [HEIGHTMAP TERRAIN DATA]: HeightmapTerrainData create from Variable2D
Gzip serialization. Size=<256,256>
17:05:47 - [MODULES]: Loading Region's modules (old style)
17:05:47 - [REGIONMODULES]: Loading Region's modules (new style)
17:05:47 - [CHAT]: Initialized for Monte Cristo w:10 s:20 S:100
17:05:47 - [MESSAGE TRANSFER]: Message transfer module active
17:05:47 - [REMOTE GRID USER CONNECTOR]: Enabled remote grid user for region Mon
te Cristo

(A lot of times it gets stuck for a long time here)



Another log from a different OS session:

17:07:55 - [LOAD REGIONS PLUGIN]: Creating Region: Monte Cristo (ThreadID: 1)
17:07:55 - [SCENE]: Default script engine XEngine
17:07:55 - [SCENE]: Region Monte Cristo, WORLD MAP refresh time set to 0 seconds

17:07:55 - [SCENE]: Using the BestAvatarResponsiveness prioritization scheme
17:07:55 - [HEIGHTMAP TERRAIN DATA]: VD2Gzip 296 bytes input
17:07:55 - [HEIGHTMAP TERRAIN DATA]: V2DGzip. Heightmap size=<256,256>. Region s
ize=<256,256>
17:07:55 - [HEIGHTMAP TERRAIN DATA]: HeightmapTerrainData create from Variable2D
Gzip serialization. Size=<256,256>
17:07:55 - [HEIGHTMAP TERRAIN DATA]: VD2Gzip 296 bytes input
17:07:55 - [HEIGHTMAP TERRAIN DATA]: V2DGzip. Heightmap size=<256,256>. Region s
ize=<256,256>
17:07:55 - [HEIGHTMAP TERRAIN DATA]: HeightmapTerrainData create from Variable2D
Gzip serialization. Size=<256,256>
17:07:55 - [MODULES]: Loading Region's modules (old style)
17:07:55 - [REGIONMODULES]: Loading Region's modules (new style)

(Sometimes it gets stuck for a long time here but not as often as the first example)




On 6/25/2017 7:40 PM, AJLDuarte wrote:

Hi

                Good to read things are not so bad after all

                About reboots, do them as often as your use case allows (and backups of course).

                Regions memory use keeps piling up.. etc, etc

Ubit

 

 

From: [hidden email] [[hidden email]] On Behalf Of [hidden email]
Sent: Monday, June 26, 2017 01:28
To: [hidden email]
Subject: Re: [Opensim-users] What is going on with the latest release ofopensim for OSgrid

 

Ubit,

 

After checking everything I decided to do a “warm” start, one where it is not a total upgrade with all the compiled stuff deleted.  This time the restart took 14 minutes which is what I normally observe.  I guess I never watched a cold restart before so I guess maybe it always took that long.  Usually I let the updates happen at the middle of the night restart I have scheduled.  I always found everything working just fine in the mornings.

 

Sorry for the alarm, but the 70 minute restart time took me by surprise.

 

One good thing I did learn from all of this was that it may not be necessary to do a reboot every night to prevent lag.  One of my servers ran 12 days with no reboot because I had forgot to turn the nightly reboot back on during a routine manual maintenance I did earlier this month.

 

I may go to a once a week reboot on one server and see how that goes.  If all goes well, I’ll change them all to once a week.

 

Tom

 

 

From: AJLDuarte

Sent: Sunday, June 25, 2017 12:55 PM

Subject: Re: [Opensim-users] What is going on with the latest release ofopensim for OSgrid

 

Hi

                What was the state of folder bin/AssetsCache on those 2 tests?

                According to options, map generation my require full assets  fetch from grid and that can take ages if not cached.

                Of course, make sure FloatSam cache is proper configure. It should no longer be optional, since it is needed and others don’t do the job now.

 

Also seems your t2 is not doing map and not using ubOde ?

                Even with all assets present in cache map is heavy, can take a bit

                ubOde does operations before scene Ready so that ready does mean it a bit more

Other engines do it later so you don’t see it and region Ready really doesn’t mean much.

Ubit

 

 

From: [hidden email] [[hidden email]] On Behalf Of [hidden email]
Sent: Sunday, June 25, 2017 17:32
To: OS-Opensim Users
Subject: [Opensim-users] What is going on with the latest release of opensim for OSgrid

 

I just did a update to the latest release of opensim released by OSgrid and it took 1 hour and 12 minutes from the time the command was give to load before it enabled logins.

 

This is on a 4 by 4 var region that previously took 17 minutes to load.  While that is nothing to write home about, an hour and 12 minutes is totally insane.

 

This region runs on a dedicated server with nothing else running on it at all.  It is a 6 core AMD with 16GB of ram.

 

Using “top” to view cpu usage the most I ever noticed it going to as 16% total with only one core hitting 100% for a few MS every now and then.

 

It looked like it too most og that time generating the 16 tile textures for the grid display, but scripts did take about 11 minutes to get started.  That part did not appear to be significantly different than before.

 

Here are some key log entries so you can see how long each thing is taking.

 

2017-06-25 10:34:15,481 INFO  [OPENSIM MAIN]: configured log4net using default OpenSim.exe.config  <—first log entry

2017-06-25 10:34:25,743 DEBUG [REGION DB]: Loaded 5694 objects using 47381 prims

2017-06-25 10:34:31,207 DEBUG [REGION DB]: Loaded inventory from 3563 objects

2017-06-25 10:34:31,209 INFO  [SCENE]: Loaded 5694 objects from the datastore

2017-06-25 10:34:40,244 DEBUG [WORLD MAP]: Generating map image for TSim 1

                                                                                                                                                        <—big delay

2017-06-25 11:17:53,580 DEBUG [WORLD MAP]: Storing map image c393c67c-eb69-43b1-b6bd-650d06c59e99 for TSim 1

2017-06-25 11:17:55,255 INFO  [SCENE]: Initializing script instances in TSim 1

2017-06-25 11:19:14,586 INFO  [SCENE]: Initialized 6604 script instances in TSim 1

2017-06-25 11:19:14,979 INFO  [ubOde] start processing pending actor operations

                                                                                                                                                        <—big delay

2017-06-25 11:29:23,776 INFO  [XEngine]: Started 50 scripts in TSim 1

                                                                                                                                                        <—big delay but not much different than previous version

2017-06-25 11:45:33,268 INFO  [XEngine]: Completed starting 6583 scripts on TSim 1

2017-06-25 11:45:57,478 DEBUG [RegionReady]: Region "TSim 1" is ready: "server_startup,1,21," on channel -800

 

This is a very limited log entry list as there are hundreds of entries many of them listing bad textures or other things indicating corrupt assets coming from the asset server.

 

This was a cold start where it does have to generate and compile stuff, but it has never taken any where near this long for a region to come online before.

 

No one else has noticed this huge increase in start-up time?

 

I’m running 4 servers, so I tried this same region on my test server, it only has 8gb or ram and a bit slower 6 core AMD cpu.  It took 2 hours on that server.

 

My other 4 by 4 region TSim 2 takes 12 minutes to come online.  This is not the cold start time, which does take longer.

 

2017-06-25 04:02:57,726 INFO  [OPENSIM MAIN]: configured log4net using default OpenSim.exe.config

2017-06-25 04:04:22,724 DEBUG [REGION DB]: Loaded 3068 objects using 33622 prims

2017-06-25 04:03:25,024 INFO  [XEngine]: Initializing scripts in region TSim 2

2017-06-25 04:09:34,146 INFO  [XEngine]: Started 50 scripts in TSim 2

2017-06-25 04:15:45,075 INFO  [XEngine]: Completed starting 4534 scripts on TSim 2

2017-06-25 04:15:45,138 DEBUG [RegionReady]: Region "TSim 2" is ready: "server_startup,1,27," on channel -800

 

 

 

 

 


_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users



_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
-- 
OpenSim: 10 Region Standalone on 0.8.1 Dev
Physics: Open Dynamics Engine
OS: Windows 7 (x64)
CPU: AMD FX 8320 8-Core 3.5 GHz
Memory: 16 GB DDR3
Database: MySQL 5.1.63 (x64)
_______________________________________________ Opensim-users mailing list [hidden email] http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
-- 
OpenSim: 10 Region Standalone on 0.8.1 Dev
Physics: Open Dynamics Engine
OS: Windows 7 (x64)
CPU: AMD FX 8320 8-Core 3.5 GHz
Memory: 16 GB DDR3
Database: MySQL 5.1.63 (x64)

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Loading...