Re: [Opensim-users] [Opensim-dev] some scalability tests...

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [Opensim-users] [Opensim-dev] some scalability tests...

Mic Bowman
final entry on this...

we seem to have found a combination of settings, execution environment, and changes to opensim that give us a reasonably stable experience (there is nothing "normal" about feeding the viewer 100 regions to render so keep it all in perspective, its most likely going to break if you hang out there long enough :-)...

* we got rid of the megaregions. we wanted a "many" region view that we get with megaregons, but we didn't need the multi-region physics. we also found that megaregions next to megaregions have some very painful effects on border crossings especially going west & south. and big megaregions are worse (as far as i could tell, the viewer thinks it doesn't have connections outside its 10x10 range and the sim thinks the viewer already has the connection and then weird stuff happens when the region actually comes in view). instead of the megaregions we hacked opensim to give out all of the regions within 5 coordinates of the current region (which seems to be the max the viewer can handle). that works out to about 100 regions that are visible. the change we made for the test isn't particularly useful in general, but i think we can try to get the viewers far clip distance to set the number of regions to send back.

* 64 bit was causing all kinds of problems with the jpeg decoder (no surprise). given that the only jpeg being decoded was the map image, there might be something in the way the map is created that causes the problems. Regardless... given that every time you opened the map view in the viewer, the region would grab you and not let go (no further TPs, no border crossings, etc) it seemed like we needed a different solution. so now we're running 16 simulators, each 8x8 for a 64 regions per simulator and we're using the 32 bit launcher to force it to stay in 32 bit mode.

* another thing that caused problems at times was the number of mysql connections. increasing the number of simulators increases the number of connections. db contention was causing some bad problems on startup... we haven't dug into the root cause. instead, we've changed to use an alternative mysql connection management solution that uses mysql connection pooling. that seemed to get us past the initial problems (yeah... you can always up the mysql connections... but more efficient use of the connections seems like a good idea). john's working on a patch against the current opensim master for that code.

at any rate... it works. and for proof here's a picture of yellowstone national park from the top of one of the mountains on the eastern side of the park:
http://www.sciencesim.com/wiki/lib/exe/fetch.php/yellowstone.png

(or come see it on the scisim grid.., search for the "geography" regions on the map)

--mic




On Fri, Jan 22, 2010 at 6:01 PM, <[hidden email]> wrote:
+100! Really great that Intel is pushing the performance limits...

Diva
(coming home from an event in the university that had lots of local
industry, including gaming industry, even more convinced that the key
piece that is missing in all of it, for the longest time, is... an open
source MMO platform that *everyone* can explore with)

Kyle wrote:
> Fantastic work Intel!
>
>
>
> *From:* [hidden email]
> [mailto:[hidden email]] *On Behalf Of *Mic Bowman
> *Sent:* Friday, January 22, 2010 7:19 PM
> *To:* [hidden email]; [hidden email]
> *Subject:* Re: [Opensim-dev] some scalability tests...
>
>
>
> Grid mode. Connected to SciSim. Thanks to some help from Brian, we put
> the Yellowstone National Park terrain on it today. I think Brian said it
> works out to about a 1:10 scale. With the far view distance its pretty
> impressive (terrain textures are borked). The performance is terrible,
> but good enough to look around some. Even a completely empty region
> consumes resources and 1000 of them consume a LOT of resources. If you
> come over to sciencesim, look for "geography11 00". I'll probably be
> restarting the regions to get the map tiles updated & i'm not making any
> promises to keep it up very long. But it does make for a very neat
> demonstration and its been a very useful experiment.
>
> --mic
>
> On Fri, Jan 22, 2010 at 3:27 AM, Impalah Shenzhou <[hidden email]
> <mailto:[hidden email]>> wrote:
>
> Just a stupid silly question: standalone or grid mode? If grid: where
> were the ugaim servers? Same machine?
>
> And a comment: 1024 regions? 8 hours for booting? Weird!
>
> 2010/1/22 Mic Bowman <[hidden email] <mailto:[hidden email]>>
>
> this is just fyi... and a very positive comment about how far opensim
> has come in recent months!
>
> as part of sizing the hw requirements for a mirror world project we're
> exploring... we wanted to do some scalability tests on the capacity of
> individual simulators in terms of the total number of regions. we wanted
> baseline numbers that focus on just the most simple region
> configuration: a completely empty region with default terrain. that
> is... this is JUST simulator overhead.
>
> all tests are done on one of our blades... dual proc, quad core with 16G
> ram running 64bit ubuntu. mono 2.6. and the tests are hosting 1024
> regions in a 32x32 grid. the simulator configuration was our standard
> sciencesim config (XEngine, ODE, groups, wind, sun, etc).
>
> the first configuration ran all 1024 regions in *one* simulator. i
> honestly figured this would crash quickly. it didn't. we managed to get
> all 1024 regions created and running well enough to walk around. it did
> take almost 8 hours to start. the first couple regions were created in
> 1-2 seconds each. by the end, it was taking 4-5 minutes per region.
> clearly there is something quadratic in there (stop using linear lists,
> they are evil! :-). but it could have been the mono garbage collector.
> who knows... not sure its worth too much investigation because i can't
> imagine ever running a config like this for real. the simulator did
> crash when we opened the map in the viewer. the crash was caused because
> we ran out of sockets. while it was running, the simulator used just
> over 10G of ram and was running at about 700% CPU utilization (kind of
> scary to see load averages in the 900 range!).
>
> the second configuration ran 16 simulators each with 64 regions. startup
> took about 30 minutes (each of the 16 simulators avoided the quadratic
> "knee" we hit with the one big simulator). consumed about 11G of memory
> and was again consuming essentially all cycles on the machine
> (completely idle regions aren't very idle). all 16 simulators died just
> after startup with a "too many open files" error. not sure what caused
> it, but all of them died loading the same terrain dll as part of some
> wildcard function looking for dlls. that one is an interesting bug.
>
> the final configuration and the one we're shooting for in the short term
> runs 16 simulators, each with an 8x8 megaregion. again startup was
> around 40 minutes. we might be able to cut that time down by starting
> all 16 simulators simultaneously rather than 4 at a time. i really just
> wanted to make sure that some of the spikes we see in startup didn't
> cause failures (some race condition causes all threads to consume
> maximum processor cycles randomly on startup right now). and, well... it
> just worked. i figured the viewer would die horribly (it can't handle
> 250K "real" prims very well) but it survived just fine. turn off far
> clip with 8x8 megaregions providing neighbors and you are "capable" of
> contacting simulators in a 24x24 region range!. the view is pretty cool
> though it seems to not go beyond 12x12. :-) there are a bunch of
> problems (like you can't see the terrain in the region you're standing
> in)... but there are a LOT of little humps of terrain in view. oh... and
> it takes a lot of patience to get everything loaded.
>
> so... the conclusion... this opensim thing is pretty amazing! good work
> everyone.
>
> --mic
>
> _______________________________________________
> Opensim-dev mailing list
> [hidden email] <mailto:[hidden email]>
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
>
>
> _______________________________________________
> Opensim-dev mailing list
> [hidden email] <mailto:[hidden email]>
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
>
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 9.0.730 / Virus Database: 271.1.1/2638 - Release Date: 01/22/10
> 14:33:00
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Opensim-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-dev
_______________________________________________
Opensim-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-dev


_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: [Opensim-users] [Opensim-dev] some scalability tests...

Bob Wellman
Whilst you on the subject of MYSQL connections you may be interested in something I have observed using Opensim 0.6.8. Not something you would see in a high activity grid like OSgrid but easy to see on our low activity PMgrid.
 
MYSQL has a configurable timeout on unused connections (set to 8 hours by default on our Unbuntu system). Usually this is no problem but the Grid server connection to the Regions table is not updated very often as we dont bring down/up any regions every day. So this connection sometimes times out and the MYSQL connection is lost.
When Opensim next goes to use this connection it fails, but doesnt reopen the connection and retry as you might expect. The Regions table just doesnt get the updates anymore. 
 
This also seems to apply to some of the other MYSQL connections. I have seem the User and Messenger ones dissapper in particular. I think this could be the cause of a number of persitance problems which people see intermittently (i.e. when the connection has been lost). 
 
Could someone (I cant progam in C type languages) check all MYSQL connections to ensure that failure results in a reopen and retry attempt.
 
We are currently running with a 1 million second (approx 11 days) max timeout set as a work around, but that solution is far from ideal.
 
 

Date: Tue, 26 Jan 2010 23:58:02 -0800
From: [hidden email]
To: [hidden email]; [hidden email]; [hidden email]
Subject: Re: [Opensim-users] [Opensim-dev] some scalability tests...

final entry on this...

we seem to have found a combination of settings, execution environment, and changes to opensim that give us a reasonably stable experience (there is nothing "normal" about feeding the viewer 100 regions to render so keep it all in perspective, its most likely going to break if you hang out there long enough :-)...

* we got rid of the megaregions. we wanted a "many" region view that we get with megaregons, but we didn't need the multi-region physics. we also found that megaregions next to megaregions have some very painful effects on border crossings especially going west & south. and big megaregions are worse (as far as i could tell, the viewer thinks it doesn't have connections outside its 10x10 range and the sim thinks the viewer already has the connection and then weird stuff happens when the region actually comes in view). instead of the megaregions we hacked opensim to give out all of the regions within 5 coordinates of the current region (which seems to be the max the viewer can handle). that works out to about 100 regions that are visible. the change we made for the test isn't particularly useful in general, but i think we can try to get the viewers far clip distance to set the number of regions to send back.

* 64 bit was causing all kinds of problems with the jpeg decoder (no surprise). given that the only jpeg being decoded was the map image, there might be something in the way the map is created that causes the problems. Regardless... given that every time you opened the map view in the viewer, the region would grab you and not let go (no further TPs, no border crossings, etc) it seemed like we needed a different solution. so now we're running 16 simulators, each 8x8 for a 64 regions per simulator and we're using the 32 bit launcher to force it to stay in 32 bit mode.

* another thing that caused problems at times was the number of mysql connections. increasing the number of simulators increases the number of connections. db contention was causing some bad problems on startup... we haven't dug into the root cause. instead, we've changed to use an alternative mysql connection management solution that uses mysql connection pooling. that seemed to get us past the initial problems (yeah... you can always up the mysql connections... but more efficient use of the connections seems like a good idea). john's working on a patch against the current opensim master for that code.

at any rate... it works. and for proof here's a picture of yellowstone national park from the top of one of the mountains on the eastern side of the park:
http://www.sciencesim.com/wiki/lib/exe/fetch.php/yellowstone.png

(or come see it on the scisim grid.., search for the "geography" regions on the map)

--mic




On Fri, Jan 22, 2010 at 6:01 PM, <[hidden email]> wrote:
+100! Really great that Intel is pushing the performance limits...

Diva
(coming home from an event in the university that had lots of local
industry, including gaming industry, even more convinced that the key
piece that is missing in all of it, for the longest time, is... an open
source MMO platform that *everyone* can explore with)

Kyle wrote:

> Fantastic work Intel!
>
>
>
> *From:* [hidden email]
> [mailto:[hidden email]] *On Behalf Of *Mic Bowman
> *Sent:* Friday, January 22, 2010 7:19 PM
> *To:* [hidden email]; [hidden email]
> *Subject:* Re: [Opensim-dev] some scalability tests...
>
>
>
> Grid mode. Connected to SciSim. Thanks to some help from Brian, we put
> the Yellowstone National Park terrain on it today. I think Brian said it
> works out to about a 1:10 scale. With the far view distance its pretty
> impressive (terrain textures are borked). The performance is terrible,
> but good enough to look around some. Even a completely empty region
> consumes resources and 1000 of them consume a LOT of resources. If you
> come over to sciencesim, look for "geography11 00". I'll probably be
> restarting the regions to get the map tiles updated & i'm not making any
> promises to keep it up very long. But it does make for a very neat
> demonstration and its been a very useful experiment.
>
> --mic
>
> On Fri, Jan 22, 2010 at 3:27 AM, Impalah Shenzhou <[hidden email]
> <mailto:[hidden email]>> wrote:
>
> Just a stupid silly question: standalone or grid mode? If grid: where
> were the ugaim servers? Same machine?
>
> And a comment: 1024 regions? 8 hours for booting? Weird!
>
> 2010/1/22 Mic Bowman <[hidden email] <mailto:[hidden email]>>
>

> this is just fyi... and a very positive comment about how far opensim
> has come in recent months!
>
> as part of sizing the hw requirements for a mirror world project we're
> exploring... we wanted to do some scalability tests on the capacity of
> individual simulators in terms of the total number of regions. we wanted
> baseline numbers that focus on just the most simple region
> configuration: a completely empty region with default terrain. that
> is... this is JUST simulator overhead.
>
> all tests are done on one of our blades... dual proc, quad core with 16G
> ram running 64bit ubuntu. mono 2.6. and the tests are hosting 1024
> regions in a 32x32 grid. the simulator configuration was our standard
> sciencesim config (XEngine, ODE, groups, wind, sun, etc).
>
> the first configuration ran all 1024 regions in *one* simulator. i
> honestly figured this would crash quickly. it didn't. we managed to get
> all 1024 regions created and running well enough to walk around. it did
> take almost 8 hours to start. the first couple regions were created in
> 1-2 seconds each. by the end, it was taking 4-5 minutes per region.
> clearly there is something quadratic in there (stop using linear lists,
> they are evil! :-). but it could have been the mono garbage collector.
> who knows... not sure its worth too much investigation because i can't
> imagine ever running a config like this for real. the simulator did
> crash when we opened the map in the viewer. the crash was caused because
> we ran out of sockets. while it was running, the simulator used just
> over 10G of ram and was running at about 700% CPU utilization (kind of
> scary to see load averages in the 900 range!).
>
> the second configuration ran 16 simulators each with 64 regions. startup
> took about 30 minutes (each of the 16 simulators avoided the quadratic
> "knee" we hit with the one big simulator). consumed about 11G of memory
> and was again consuming essentially all cycles on the machine
> (completely idle regions aren't very idle). all 16 simulators died just
> after startup with a "too many open files" error. not sure what caused
> it, but all of them died loading the same terrain dll as part of some
> wildcard function looking for dlls. that one is an interesting bug.
>
> the final configuration and the one we're shooting for in the short term
> runs 16 simulators, each with an 8x8 megaregion. again startup was
> around 40 minutes. we might be able to cut that time down by starting
> all 16 simulators simultaneously rather than 4 at a time. i really just
> wanted to make sure that some of the spikes we see in startup didn't
> cause failures (some race condition causes all threads to consume
> maximum processor cycles randomly on startup right now). and, well... it
> just worked. i figured the viewer would die horribly (it can't handle
> 250K "real" prims very well) but it survived just fine. turn off far
> clip with 8x8 megaregions providing neighbors and you are "capable" of
> contacting simulators in a 24x24 region range!. the view is pretty cool
> though it seems to not go beyond 12x12. :-) there are a bunch of
> problems (like you can't see the terrain in the region you're standing
> in)... but there are a LOT of little humps of terrain in view. oh... and
> it takes a lot of patience to get everything loaded.
>
> so... the conclusion... this opensim thing is pretty amazing! good work
> everyone.
>
> --mic
>
> _______________________________________________
> Opensim-dev mailing list
> [hidden email] <mailto:[hidden email]>
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
>
>
> _______________________________________________
> Opensim-dev mailing list
> [hidden email] <mailto:[hidden email]>
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
>
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 9.0.730 / Virus Database: 271.1.1/2638 - Release Date: 01/22/10
> 14:33:00
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Opensim-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-dev
_______________________________________________
Opensim-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-dev



Got a cool Hotmail story? Tell us now
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: [Opensim-users] [Opensim-dev] some scalability tests...

Diva Canto
In reply to this post by Mic Bowman
More coolness!

WRT opening up child agents in neighboring regions. Right now there's a
very simple, but too inflexible, rule: the only child agents opened are
those in immediate regions, and they open both the UDP and the CAPs
connections. The inflexibility comes from our fuzzy understanding of how
the viewer works, specifically wrt the events EnableSimulator and
EnableChildAgent -- this last one wasn't being generated by OpenSim
until last March or so. I think we (at least I) know more now than 1
year ago. The idea is to make this more flexible, maybe configurable.


Mic Bowman wrote:

> final entry on this...
>
> we seem to have found a combination of settings, execution environment,
> and changes to opensim that give us a reasonably stable experience
> (there is nothing "normal" about feeding the viewer 100 regions to
> render so keep it all in perspective, its most likely going to break if
> you hang out there long enough :-)...
>
> * we got rid of the megaregions. we wanted a "many" region view that we
> get with megaregons, but we didn't need the multi-region physics. we
> also found that megaregions next to megaregions have some very painful
> effects on border crossings especially going west & south. and big
> megaregions are worse (as far as i could tell, the viewer thinks it
> doesn't have connections outside its 10x10 range and the sim thinks the
> viewer already has the connection and then weird stuff happens when the
> region actually comes in view). instead of the megaregions we hacked
> opensim to give out all of the regions within 5 coordinates of the
> current region (which seems to be the max the viewer can handle). that
> works out to about 100 regions that are visible. the change we made for
> the test isn't particularly useful in general, but i think we can try to
> get the viewers far clip distance to set the number of regions to send
> back.
>
> * 64 bit was causing all kinds of problems with the jpeg decoder (no
> surprise). given that the only jpeg being decoded was the map image,
> there might be something in the way the map is created that causes the
> problems. Regardless... given that every time you opened the map view in
> the viewer, the region would grab you and not let go (no further TPs, no
> border crossings, etc) it seemed like we needed a different solution. so
> now we're running 16 simulators, each 8x8 for a 64 regions per simulator
> and we're using the 32 bit launcher to force it to stay in 32 bit mode.
>
> * another thing that caused problems at times was the number of mysql
> connections. increasing the number of simulators increases the number of
> connections. db contention was causing some bad problems on startup...
> we haven't dug into the root cause. instead, we've changed to use an
> alternative mysql connection management solution that uses mysql
> connection pooling. that seemed to get us past the initial problems
> (yeah... you can always up the mysql connections... but more efficient
> use of the connections seems like a good idea). john's working on a
> patch against the current opensim master for that code.
>
> at any rate... it works. and for proof here's a picture of yellowstone
> national park from the top of one of the mountains on the eastern side
> of the park:
> http://www.sciencesim.com/wiki/lib/exe/fetch.php/yellowstone.png
>
> (or come see it on the scisim grid.., search for the "geography" regions
> on the map)
>
> --mic
>
>
>
>
> On Fri, Jan 22, 2010 at 6:01 PM, <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     +100! Really great that Intel is pushing the performance limits...
>
>     Diva
>     (coming home from an event in the university that had lots of local
>     industry, including gaming industry, even more convinced that the key
>     piece that is missing in all of it, for the longest time, is... an open
>     source MMO platform that *everyone* can explore with)
>
>     Kyle wrote:
>      > Fantastic work Intel!
>      >
>      >
>      >
>      > *From:* [hidden email]
>     <mailto:[hidden email]>
>      > [mailto:[hidden email]
>     <mailto:[hidden email]>] *On Behalf Of *Mic Bowman
>      > *Sent:* Friday, January 22, 2010 7:19 PM
>      > *To:* [hidden email]
>     <mailto:[hidden email]>;
>     [hidden email] <mailto:[hidden email]>
>      > *Subject:* Re: [Opensim-dev] some scalability tests...
>      >
>      >
>      >
>      > Grid mode. Connected to SciSim. Thanks to some help from Brian,
>     we put
>      > the Yellowstone National Park terrain on it today. I think Brian
>     said it
>      > works out to about a 1:10 scale. With the far view distance its
>     pretty
>      > impressive (terrain textures are borked). The performance is
>     terrible,
>      > but good enough to look around some. Even a completely empty region
>      > consumes resources and 1000 of them consume a LOT of resources.
>     If you
>      > come over to sciencesim, look for "geography11 00". I'll probably be
>      > restarting the regions to get the map tiles updated & i'm not
>     making any
>      > promises to keep it up very long. But it does make for a very neat
>      > demonstration and its been a very useful experiment.
>      >
>      > --mic
>      >
>      > On Fri, Jan 22, 2010 at 3:27 AM, Impalah Shenzhou
>     <[hidden email] <mailto:[hidden email]>
>      > <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
>      >
>      > Just a stupid silly question: standalone or grid mode? If grid: where
>      > were the ugaim servers? Same machine?
>      >
>      > And a comment: 1024 regions? 8 hours for booting? Weird!
>      >
>      > 2010/1/22 Mic Bowman <[hidden email]
>     <mailto:[hidden email]> <mailto:[hidden email]
>     <mailto:[hidden email]>>>
>      >
>      > this is just fyi... and a very positive comment about how far opensim
>      > has come in recent months!
>      >
>      > as part of sizing the hw requirements for a mirror world project
>     we're
>      > exploring... we wanted to do some scalability tests on the
>     capacity of
>      > individual simulators in terms of the total number of regions. we
>     wanted
>      > baseline numbers that focus on just the most simple region
>      > configuration: a completely empty region with default terrain. that
>      > is... this is JUST simulator overhead.
>      >
>      > all tests are done on one of our blades... dual proc, quad core
>     with 16G
>      > ram running 64bit ubuntu. mono 2.6. and the tests are hosting 1024
>      > regions in a 32x32 grid. the simulator configuration was our standard
>      > sciencesim config (XEngine, ODE, groups, wind, sun, etc).
>      >
>      > the first configuration ran all 1024 regions in *one* simulator. i
>      > honestly figured this would crash quickly. it didn't. we managed
>     to get
>      > all 1024 regions created and running well enough to walk around.
>     it did
>      > take almost 8 hours to start. the first couple regions were
>     created in
>      > 1-2 seconds each. by the end, it was taking 4-5 minutes per region.
>      > clearly there is something quadratic in there (stop using linear
>     lists,
>      > they are evil! :-). but it could have been the mono garbage
>     collector.
>      > who knows... not sure its worth too much investigation because i
>     can't
>      > imagine ever running a config like this for real. the simulator did
>      > crash when we opened the map in the viewer. the crash was caused
>     because
>      > we ran out of sockets. while it was running, the simulator used just
>      > over 10G of ram and was running at about 700% CPU utilization
>     (kind of
>      > scary to see load averages in the 900 range!).
>      >
>      > the second configuration ran 16 simulators each with 64 regions.
>     startup
>      > took about 30 minutes (each of the 16 simulators avoided the
>     quadratic
>      > "knee" we hit with the one big simulator). consumed about 11G of
>     memory
>      > and was again consuming essentially all cycles on the machine
>      > (completely idle regions aren't very idle). all 16 simulators
>     died just
>      > after startup with a "too many open files" error. not sure what
>     caused
>      > it, but all of them died loading the same terrain dll as part of some
>      > wildcard function looking for dlls. that one is an interesting bug.
>      >
>      > the final configuration and the one we're shooting for in the
>     short term
>      > runs 16 simulators, each with an 8x8 megaregion. again startup was
>      > around 40 minutes. we might be able to cut that time down by starting
>      > all 16 simulators simultaneously rather than 4 at a time. i
>     really just
>      > wanted to make sure that some of the spikes we see in startup didn't
>      > cause failures (some race condition causes all threads to consume
>      > maximum processor cycles randomly on startup right now). and,
>     well... it
>      > just worked. i figured the viewer would die horribly (it can't handle
>      > 250K "real" prims very well) but it survived just fine. turn off far
>      > clip with 8x8 megaregions providing neighbors and you are
>     "capable" of
>      > contacting simulators in a 24x24 region range!. the view is
>     pretty cool
>      > though it seems to not go beyond 12x12. :-) there are a bunch of
>      > problems (like you can't see the terrain in the region you're
>     standing
>      > in)... but there are a LOT of little humps of terrain in view.
>     oh... and
>      > it takes a lot of patience to get everything loaded.
>      >
>      > so... the conclusion... this opensim thing is pretty amazing!
>     good work
>      > everyone.
>      >
>      > --mic
>      >
>      > _______________________________________________
>      > Opensim-dev mailing list
>      > [hidden email]
>     <mailto:[hidden email]>
>     <mailto:[hidden email]
>     <mailto:[hidden email]>>
>      > https://lists.berlios.de/mailman/listinfo/opensim-dev
>      >
>      >
>      >
>      > _______________________________________________
>      > Opensim-dev mailing list
>      > [hidden email]
>     <mailto:[hidden email]>
>     <mailto:[hidden email]
>     <mailto:[hidden email]>>
>      > https://lists.berlios.de/mailman/listinfo/opensim-dev
>      >
>      >
>      >
>      > No virus found in this incoming message.
>      > Checked by AVG - www.avg.com <http://www.avg.com>
>      > Version: 9.0.730 / Virus Database: 271.1.1/2638 - Release Date:
>     01/22/10
>      > 14:33:00
>      >
>      >
>      >
>     ------------------------------------------------------------------------
>      >
>      > _______________________________________________
>      > Opensim-dev mailing list
>      > [hidden email] <mailto:[hidden email]>
>      > https://lists.berlios.de/mailman/listinfo/opensim-dev
>     _______________________________________________
>     Opensim-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.berlios.de/mailman/listinfo/opensim-dev
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Opensim-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-dev
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: [Opensim-users] [Opensim-dev] some scalability tests...

Diva Canto
Err.. that's EstablishAgentCommunication, not EnableChildAgent. Same
initials, different order :-)

[hidden email] wrote:

> More coolness!
>
> WRT opening up child agents in neighboring regions. Right now there's a
> very simple, but too inflexible, rule: the only child agents opened are
> those in immediate regions, and they open both the UDP and the CAPs
> connections. The inflexibility comes from our fuzzy understanding of how
> the viewer works, specifically wrt the events EnableSimulator and
> EnableChildAgent -- this last one wasn't being generated by OpenSim
> until last March or so. I think we (at least I) know more now than 1
> year ago. The idea is to make this more flexible, maybe configurable.
>
>
> Mic Bowman wrote:
>> final entry on this...
>>
>> we seem to have found a combination of settings, execution environment,
>> and changes to opensim that give us a reasonably stable experience
>> (there is nothing "normal" about feeding the viewer 100 regions to
>> render so keep it all in perspective, its most likely going to break if
>> you hang out there long enough :-)...
>>
>> * we got rid of the megaregions. we wanted a "many" region view that we
>> get with megaregons, but we didn't need the multi-region physics. we
>> also found that megaregions next to megaregions have some very painful
>> effects on border crossings especially going west & south. and big
>> megaregions are worse (as far as i could tell, the viewer thinks it
>> doesn't have connections outside its 10x10 range and the sim thinks the
>> viewer already has the connection and then weird stuff happens when the
>> region actually comes in view). instead of the megaregions we hacked
>> opensim to give out all of the regions within 5 coordinates of the
>> current region (which seems to be the max the viewer can handle). that
>> works out to about 100 regions that are visible. the change we made for
>> the test isn't particularly useful in general, but i think we can try to
>> get the viewers far clip distance to set the number of regions to send
>> back.
>>
>> * 64 bit was causing all kinds of problems with the jpeg decoder (no
>> surprise). given that the only jpeg being decoded was the map image,
>> there might be something in the way the map is created that causes the
>> problems. Regardless... given that every time you opened the map view in
>> the viewer, the region would grab you and not let go (no further TPs, no
>> border crossings, etc) it seemed like we needed a different solution. so
>> now we're running 16 simulators, each 8x8 for a 64 regions per simulator
>> and we're using the 32 bit launcher to force it to stay in 32 bit mode.
>>
>> * another thing that caused problems at times was the number of mysql
>> connections. increasing the number of simulators increases the number of
>> connections. db contention was causing some bad problems on startup...
>> we haven't dug into the root cause. instead, we've changed to use an
>> alternative mysql connection management solution that uses mysql
>> connection pooling. that seemed to get us past the initial problems
>> (yeah... you can always up the mysql connections... but more efficient
>> use of the connections seems like a good idea). john's working on a
>> patch against the current opensim master for that code.
>>
>> at any rate... it works. and for proof here's a picture of yellowstone
>> national park from the top of one of the mountains on the eastern side
>> of the park:
>> http://www.sciencesim.com/wiki/lib/exe/fetch.php/yellowstone.png
>>
>> (or come see it on the scisim grid.., search for the "geography" regions
>> on the map)
>>
>> --mic
>>
>>
>>
>>
>> On Fri, Jan 22, 2010 at 6:01 PM, <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>>     +100! Really great that Intel is pushing the performance limits...
>>
>>     Diva
>>     (coming home from an event in the university that had lots of local
>>     industry, including gaming industry, even more convinced that the key
>>     piece that is missing in all of it, for the longest time, is... an open
>>     source MMO platform that *everyone* can explore with)
>>
>>     Kyle wrote:
>>      > Fantastic work Intel!
>>      >
>>      >
>>      >
>>      > *From:* [hidden email]
>>     <mailto:[hidden email]>
>>      > [mailto:[hidden email]
>>     <mailto:[hidden email]>] *On Behalf Of *Mic Bowman
>>      > *Sent:* Friday, January 22, 2010 7:19 PM
>>      > *To:* [hidden email]
>>     <mailto:[hidden email]>;
>>     [hidden email] <mailto:[hidden email]>
>>      > *Subject:* Re: [Opensim-dev] some scalability tests...
>>      >
>>      >
>>      >
>>      > Grid mode. Connected to SciSim. Thanks to some help from Brian,
>>     we put
>>      > the Yellowstone National Park terrain on it today. I think Brian
>>     said it
>>      > works out to about a 1:10 scale. With the far view distance its
>>     pretty
>>      > impressive (terrain textures are borked). The performance is
>>     terrible,
>>      > but good enough to look around some. Even a completely empty region
>>      > consumes resources and 1000 of them consume a LOT of resources.
>>     If you
>>      > come over to sciencesim, look for "geography11 00". I'll probably be
>>      > restarting the regions to get the map tiles updated & i'm not
>>     making any
>>      > promises to keep it up very long. But it does make for a very neat
>>      > demonstration and its been a very useful experiment.
>>      >
>>      > --mic
>>      >
>>      > On Fri, Jan 22, 2010 at 3:27 AM, Impalah Shenzhou
>>     <[hidden email] <mailto:[hidden email]>
>>      > <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
>>      >
>>      > Just a stupid silly question: standalone or grid mode? If grid: where
>>      > were the ugaim servers? Same machine?
>>      >
>>      > And a comment: 1024 regions? 8 hours for booting? Weird!
>>      >
>>      > 2010/1/22 Mic Bowman <[hidden email]
>>     <mailto:[hidden email]> <mailto:[hidden email]
>>     <mailto:[hidden email]>>>
>>      >
>>      > this is just fyi... and a very positive comment about how far opensim
>>      > has come in recent months!
>>      >
>>      > as part of sizing the hw requirements for a mirror world project
>>     we're
>>      > exploring... we wanted to do some scalability tests on the
>>     capacity of
>>      > individual simulators in terms of the total number of regions. we
>>     wanted
>>      > baseline numbers that focus on just the most simple region
>>      > configuration: a completely empty region with default terrain. that
>>      > is... this is JUST simulator overhead.
>>      >
>>      > all tests are done on one of our blades... dual proc, quad core
>>     with 16G
>>      > ram running 64bit ubuntu. mono 2.6. and the tests are hosting 1024
>>      > regions in a 32x32 grid. the simulator configuration was our standard
>>      > sciencesim config (XEngine, ODE, groups, wind, sun, etc).
>>      >
>>      > the first configuration ran all 1024 regions in *one* simulator. i
>>      > honestly figured this would crash quickly. it didn't. we managed
>>     to get
>>      > all 1024 regions created and running well enough to walk around.
>>     it did
>>      > take almost 8 hours to start. the first couple regions were
>>     created in
>>      > 1-2 seconds each. by the end, it was taking 4-5 minutes per region.
>>      > clearly there is something quadratic in there (stop using linear
>>     lists,
>>      > they are evil! :-). but it could have been the mono garbage
>>     collector.
>>      > who knows... not sure its worth too much investigation because i
>>     can't
>>      > imagine ever running a config like this for real. the simulator did
>>      > crash when we opened the map in the viewer. the crash was caused
>>     because
>>      > we ran out of sockets. while it was running, the simulator used just
>>      > over 10G of ram and was running at about 700% CPU utilization
>>     (kind of
>>      > scary to see load averages in the 900 range!).
>>      >
>>      > the second configuration ran 16 simulators each with 64 regions.
>>     startup
>>      > took about 30 minutes (each of the 16 simulators avoided the
>>     quadratic
>>      > "knee" we hit with the one big simulator). consumed about 11G of
>>     memory
>>      > and was again consuming essentially all cycles on the machine
>>      > (completely idle regions aren't very idle). all 16 simulators
>>     died just
>>      > after startup with a "too many open files" error. not sure what
>>     caused
>>      > it, but all of them died loading the same terrain dll as part of some
>>      > wildcard function looking for dlls. that one is an interesting bug.
>>      >
>>      > the final configuration and the one we're shooting for in the
>>     short term
>>      > runs 16 simulators, each with an 8x8 megaregion. again startup was
>>      > around 40 minutes. we might be able to cut that time down by starting
>>      > all 16 simulators simultaneously rather than 4 at a time. i
>>     really just
>>      > wanted to make sure that some of the spikes we see in startup didn't
>>      > cause failures (some race condition causes all threads to consume
>>      > maximum processor cycles randomly on startup right now). and,
>>     well... it
>>      > just worked. i figured the viewer would die horribly (it can't handle
>>      > 250K "real" prims very well) but it survived just fine. turn off far
>>      > clip with 8x8 megaregions providing neighbors and you are
>>     "capable" of
>>      > contacting simulators in a 24x24 region range!. the view is
>>     pretty cool
>>      > though it seems to not go beyond 12x12. :-) there are a bunch of
>>      > problems (like you can't see the terrain in the region you're
>>     standing
>>      > in)... but there are a LOT of little humps of terrain in view.
>>     oh... and
>>      > it takes a lot of patience to get everything loaded.
>>      >
>>      > so... the conclusion... this opensim thing is pretty amazing!
>>     good work
>>      > everyone.
>>      >
>>      > --mic
>>      >
>>      > _______________________________________________
>>      > Opensim-dev mailing list
>>      > [hidden email]
>>     <mailto:[hidden email]>
>>     <mailto:[hidden email]
>>     <mailto:[hidden email]>>
>>      > https://lists.berlios.de/mailman/listinfo/opensim-dev
>>      >
>>      >
>>      >
>>      > _______________________________________________
>>      > Opensim-dev mailing list
>>      > [hidden email]
>>     <mailto:[hidden email]>
>>     <mailto:[hidden email]
>>     <mailto:[hidden email]>>
>>      > https://lists.berlios.de/mailman/listinfo/opensim-dev
>>      >
>>      >
>>      >
>>      > No virus found in this incoming message.
>>      > Checked by AVG - www.avg.com <http://www.avg.com>
>>      > Version: 9.0.730 / Virus Database: 271.1.1/2638 - Release Date:
>>     01/22/10
>>      > 14:33:00
>>      >
>>      >
>>      >
>>     ------------------------------------------------------------------------
>>      >
>>      > _______________________________________________
>>      > Opensim-dev mailing list
>>      > [hidden email] <mailto:[hidden email]>
>>      > https://lists.berlios.de/mailman/listinfo/opensim-dev
>>     _______________________________________________
>>     Opensim-dev mailing list
>>     [hidden email] <mailto:[hidden email]>
>>     https://lists.berlios.de/mailman/listinfo/opensim-dev
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Opensim-dev mailing list
>> [hidden email]
>> https://lists.berlios.de/mailman/listinfo/opensim-dev
> _______________________________________________
> Opensim-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: [Opensim-users] [Opensim-dev] some scalability tests...

Mic Bowman
In reply to this post by Bob Wellman
There are a few of the timeouts in the mysql code that are not checked. I think I sent a patch to Diva a couple weeks ago that fixed one or two of those.

The approach we've taking is different... we dropped most of the connection management code & went back to using the MySQL connection pools. It pretty much solved our problems. Some of the recent changes to master keep the patch from merging & we need to clean that up. If anyone wants to work on it, I'll post it to mantis... Otherwise, it will probably be next week before we finish cleaning up the merge.

--mic


On Wed, Jan 27, 2010 at 2:11 AM, Bob Wellman <[hidden email]> wrote:
Whilst you on the subject of MYSQL connections you may be interested in something I have observed using Opensim 0.6.8. Not something you would see in a high activity grid like OSgrid but easy to see on our low activity PMgrid.
 
MYSQL has a configurable timeout on unused connections (set to 8 hours by default on our Unbuntu system). Usually this is no problem but the Grid server connection to the Regions table is not updated very often as we dont bring down/up any regions every day. So this connection sometimes times out and the MYSQL connection is lost.
When Opensim next goes to use this connection it fails, but doesnt reopen the connection and retry as you might expect. The Regions table just doesnt get the updates anymore. 
 
This also seems to apply to some of the other MYSQL connections. I have seem the User and Messenger ones dissapper in particular. I think this could be the cause of a number of persitance problems which people see intermittently (i.e. when the connection has been lost). 
 
Could someone (I cant progam in C type languages) check all MYSQL connections to ensure that failure results in a reopen and retry attempt.
 
We are currently running with a 1 million second (approx 11 days) max timeout set as a work around, but that solution is far from ideal.
 
 

Date: Tue, 26 Jan 2010 23:58:02 -0800
From: [hidden email]
To: [hidden email]; [hidden email]; [hidden email]
Subject: Re: [Opensim-users] [Opensim-dev] some scalability tests...


final entry on this...

we seem to have found a combination of settings, execution environment, and changes to opensim that give us a reasonably stable experience (there is nothing "normal" about feeding the viewer 100 regions to render so keep it all in perspective, its most likely going to break if you hang out there long enough :-)...

* we got rid of the megaregions. we wanted a "many" region view that we get with megaregons, but we didn't need the multi-region physics. we also found that megaregions next to megaregions have some very painful effects on border crossings especially going west & south. and big megaregions are worse (as far as i could tell, the viewer thinks it doesn't have connections outside its 10x10 range and the sim thinks the viewer already has the connection and then weird stuff happens when the region actually comes in view). instead of the megaregions we hacked opensim to give out all of the regions within 5 coordinates of the current region (which seems to be the max the viewer can handle). that works out to about 100 regions that are visible. the change we made for the test isn't particularly useful in general, but i think we can try to get the viewers far clip distance to set the number of regions to send back.

* 64 bit was causing all kinds of problems with the jpeg decoder (no surprise). given that the only jpeg being decoded was the map image, there might be something in the way the map is created that causes the problems. Regardless... given that every time you opened the map view in the viewer, the region would grab you and not let go (no further TPs, no border crossings, etc) it seemed like we needed a different solution. so now we're running 16 simulators, each 8x8 for a 64 regions per simulator and we're using the 32 bit launcher to force it to stay in 32 bit mode.

* another thing that caused problems at times was the number of mysql connections. increasing the number of simulators increases the number of connections. db contention was causing some bad problems on startup... we haven't dug into the root cause. instead, we've changed to use an alternative mysql connection management solution that uses mysql connection pooling. that seemed to get us past the initial problems (yeah... you can always up the mysql connections... but more efficient use of the connections seems like a good idea). john's working on a patch against the current opensim master for that code.

at any rate... it works. and for proof here's a picture of yellowstone national park from the top of one of the mountains on the eastern side of the park:
http://www.sciencesim.com/wiki/lib/exe/fetch.php/yellowstone.png

(or come see it on the scisim grid.., search for the "geography" regions on the map)

--mic




On Fri, Jan 22, 2010 at 6:01 PM, <[hidden email]> wrote:
+100! Really great that Intel is pushing the performance limits...

Diva
(coming home from an event in the university that had lots of local
industry, including gaming industry, even more convinced that the key
piece that is missing in all of it, for the longest time, is... an open
source MMO platform that *everyone* can explore with)

Kyle wrote:
> Fantastic work Intel!
>
>
>
> *From:* [hidden email]
> [mailto:[hidden email]] *On Behalf Of *Mic Bowman
> *Sent:* Friday, January 22, 2010 7:19 PM
> *To:* [hidden email]; [hidden email]
> *Subject:* Re: [Opensim-dev] some scalability tests...
>
>
>
> Grid mode. Connected to SciSim. Thanks to some help from Brian, we put
> the Yellowstone National Park terrain on it today. I think Brian said it
> works out to about a 1:10 scale. With the far view distance its pretty
> impressive (terrain textures are borked). The performance is terrible,
> but good enough to look around some. Even a completely empty region
> consumes resources and 1000 of them consume a LOT of resources. If you
> come over to sciencesim, look for "geography11 00". I'll probably be
> restarting the regions to get the map tiles updated & i'm not making any
> promises to keep it up very long. But it does make for a very neat
> demonstration and its been a very useful experiment.
>
> --mic
>
> On Fri, Jan 22, 2010 at 3:27 AM, Impalah Shenzhou <[hidden email]
> <mailto:[hidden email]>> wrote:
>
> Just a stupid silly question: standalone or grid mode? If grid: where
> were the ugaim servers? Same machine?
>
> And a comment: 1024 regions? 8 hours for booting? Weird!
>
> 2010/1/22 Mic Bowman <[hidden email] <mailto:[hidden email]>>
>
> this is just fyi... and a very positive comment about how far opensim
> has come in recent months!
>
> as part of sizing the hw requirements for a mirror world project we're
> exploring... we wanted to do some scalability tests on the capacity of
> individual simulators in terms of the total number of regions. we wanted
> baseline numbers that focus on just the most simple region
> configuration: a completely empty region with default terrain. that
> is... this is JUST simulator overhead.
>
> all tests are done on one of our blades... dual proc, quad core with 16G
> ram running 64bit ubuntu. mono 2.6. and the tests are hosting 1024
> regions in a 32x32 grid. the simulator configuration was our standard
> sciencesim config (XEngine, ODE, groups, wind, sun, etc).
>
> the first configuration ran all 1024 regions in *one* simulator. i
> honestly figured this would crash quickly. it didn't. we managed to get
> all 1024 regions created and running well enough to walk around. it did
> take almost 8 hours to start. the first couple regions were created in
> 1-2 seconds each. by the end, it was taking 4-5 minutes per region.
> clearly there is something quadratic in there (stop using linear lists,
> they are evil! :-). but it could have been the mono garbage collector.
> who knows... not sure its worth too much investigation because i can't
> imagine ever running a config like this for real. the simulator did
> crash when we opened the map in the viewer. the crash was caused because
> we ran out of sockets. while it was running, the simulator used just
> over 10G of ram and was running at about 700% CPU utilization (kind of
> scary to see load averages in the 900 range!).
>
> the second configuration ran 16 simulators each with 64 regions. startup
> took about 30 minutes (each of the 16 simulators avoided the quadratic
> "knee" we hit with the one big simulator). consumed about 11G of memory
> and was again consuming essentially all cycles on the machine
> (completely idle regions aren't very idle). all 16 simulators died just
> after startup with a "too many open files" error. not sure what caused
> it, but all of them died loading the same terrain dll as part of some
> wildcard function looking for dlls. that one is an interesting bug.
>
> the final configuration and the one we're shooting for in the short term
> runs 16 simulators, each with an 8x8 megaregion. again startup was
> around 40 minutes. we might be able to cut that time down by starting
> all 16 simulators simultaneously rather than 4 at a time. i really just
> wanted to make sure that some of the spikes we see in startup didn't
> cause failures (some race condition causes all threads to consume
> maximum processor cycles randomly on startup right now). and, well... it
> just worked. i figured the viewer would die horribly (it can't handle
> 250K "real" prims very well) but it survived just fine. turn off far
> clip with 8x8 megaregions providing neighbors and you are "capable" of
> contacting simulators in a 24x24 region range!. the view is pretty cool
> though it seems to not go beyond 12x12. :-) there are a bunch of
> problems (like you can't see the terrain in the region you're standing
> in)... but there are a LOT of little humps of terrain in view. oh... and
> it takes a lot of patience to get everything loaded.
>
> so... the conclusion... this opensim thing is pretty amazing! good work
> everyone.
>
> --mic
>
> _______________________________________________
> Opensim-dev mailing list
> [hidden email] <mailto:[hidden email]>
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
>
>
> _______________________________________________
> Opensim-dev mailing list
> [hidden email] <mailto:[hidden email]>
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
>
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 9.0.730 / Virus Database: 271.1.1/2638 - Release Date: 01/22/10
> 14:33:00
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Opensim-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-dev
_______________________________________________
Opensim-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-dev



Got a cool Hotmail story? Tell us now

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users



_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: [Opensim-users] [Opensim-dev] some scalability tests...

Mic Bowman
In reply to this post by Diva Canto
Getting the fixed size expanded view isn't too difficult (it was maybe 10 lines of code). Getting it tied to far clip setting in the viewer is a bit more interesting. More because I don't know when the agent update packet comes in with the far clip set relative to the time when the child agents are created in all of the neighbor regions. Just needs some more debugging.

--mic



On Wed, Jan 27, 2010 at 7:54 AM, <[hidden email]> wrote:
More coolness!

WRT opening up child agents in neighboring regions. Right now there's a
very simple, but too inflexible, rule: the only child agents opened are
those in immediate regions, and they open both the UDP and the CAPs
connections. The inflexibility comes from our fuzzy understanding of how
the viewer works, specifically wrt the events EnableSimulator and
EnableChildAgent -- this last one wasn't being generated by OpenSim
until last March or so. I think we (at least I) know more now than 1
year ago. The idea is to make this more flexible, maybe configurable.


Mic Bowman wrote:
> final entry on this...
>
> we seem to have found a combination of settings, execution environment,
> and changes to opensim that give us a reasonably stable experience
> (there is nothing "normal" about feeding the viewer 100 regions to
> render so keep it all in perspective, its most likely going to break if
> you hang out there long enough :-)...
>
> * we got rid of the megaregions. we wanted a "many" region view that we
> get with megaregons, but we didn't need the multi-region physics. we
> also found that megaregions next to megaregions have some very painful
> effects on border crossings especially going west & south. and big
> megaregions are worse (as far as i could tell, the viewer thinks it
> doesn't have connections outside its 10x10 range and the sim thinks the
> viewer already has the connection and then weird stuff happens when the
> region actually comes in view). instead of the megaregions we hacked
> opensim to give out all of the regions within 5 coordinates of the
> current region (which seems to be the max the viewer can handle). that
> works out to about 100 regions that are visible. the change we made for
> the test isn't particularly useful in general, but i think we can try to
> get the viewers far clip distance to set the number of regions to send
> back.
>
> * 64 bit was causing all kinds of problems with the jpeg decoder (no
> surprise). given that the only jpeg being decoded was the map image,
> there might be something in the way the map is created that causes the
> problems. Regardless... given that every time you opened the map view in
> the viewer, the region would grab you and not let go (no further TPs, no
> border crossings, etc) it seemed like we needed a different solution. so
> now we're running 16 simulators, each 8x8 for a 64 regions per simulator
> and we're using the 32 bit launcher to force it to stay in 32 bit mode.
>
> * another thing that caused problems at times was the number of mysql
> connections. increasing the number of simulators increases the number of
> connections. db contention was causing some bad problems on startup...
> we haven't dug into the root cause. instead, we've changed to use an
> alternative mysql connection management solution that uses mysql
> connection pooling. that seemed to get us past the initial problems
> (yeah... you can always up the mysql connections... but more efficient
> use of the connections seems like a good idea). john's working on a
> patch against the current opensim master for that code.
>
> at any rate... it works. and for proof here's a picture of yellowstone
> national park from the top of one of the mountains on the eastern side
> of the park:
> http://www.sciencesim.com/wiki/lib/exe/fetch.php/yellowstone.png
>
> (or come see it on the scisim grid.., search for the "geography" regions
> on the map)
>
> --mic
>
>
>
>
> On Fri, Jan 22, 2010 at 6:01 PM, <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     +100! Really great that Intel is pushing the performance limits...
>
>     Diva
>     (coming home from an event in the university that had lots of local
>     industry, including gaming industry, even more convinced that the key
>     piece that is missing in all of it, for the longest time, is... an open
>     source MMO platform that *everyone* can explore with)
>
>     Kyle wrote:
>      > Fantastic work Intel!
>      >
>      >
>      >
>      > *From:* [hidden email]
>     <mailto:[hidden email]>
>      > [mailto:[hidden email]
>     <mailto:[hidden email]>] *On Behalf Of *Mic Bowman
>      > *Sent:* Friday, January 22, 2010 7:19 PM
>      > *To:* [hidden email]
>     <mailto:[hidden email]>;
>     [hidden email] <mailto:[hidden email]>
>      > *Subject:* Re: [Opensim-dev] some scalability tests...
>      >
>      >
>      >
>      > Grid mode. Connected to SciSim. Thanks to some help from Brian,
>     we put
>      > the Yellowstone National Park terrain on it today. I think Brian
>     said it
>      > works out to about a 1:10 scale. With the far view distance its
>     pretty
>      > impressive (terrain textures are borked). The performance is
>     terrible,
>      > but good enough to look around some. Even a completely empty region
>      > consumes resources and 1000 of them consume a LOT of resources.
>     If you
>      > come over to sciencesim, look for "geography11 00". I'll probably be
>      > restarting the regions to get the map tiles updated & i'm not
>     making any
>      > promises to keep it up very long. But it does make for a very neat
>      > demonstration and its been a very useful experiment.
>      >
>      > --mic
>      >
>      > On Fri, Jan 22, 2010 at 3:27 AM, Impalah Shenzhou
>     <[hidden email] <mailto:[hidden email]>
>      > <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
>      >
>      > Just a stupid silly question: standalone or grid mode? If grid: where
>      > were the ugaim servers? Same machine?
>      >
>      > And a comment: 1024 regions? 8 hours for booting? Weird!
>      >
>      > 2010/1/22 Mic Bowman <[hidden email]
>     <mailto:[hidden email]> <mailto:[hidden email]
>     <mailto:[hidden email]>>>
>      >
>      > this is just fyi... and a very positive comment about how far opensim
>      > has come in recent months!
>      >
>      > as part of sizing the hw requirements for a mirror world project
>     we're
>      > exploring... we wanted to do some scalability tests on the
>     capacity of
>      > individual simulators in terms of the total number of regions. we
>     wanted
>      > baseline numbers that focus on just the most simple region
>      > configuration: a completely empty region with default terrain. that
>      > is... this is JUST simulator overhead.
>      >
>      > all tests are done on one of our blades... dual proc, quad core
>     with 16G
>      > ram running 64bit ubuntu. mono 2.6. and the tests are hosting 1024
>      > regions in a 32x32 grid. the simulator configuration was our standard
>      > sciencesim config (XEngine, ODE, groups, wind, sun, etc).
>      >
>      > the first configuration ran all 1024 regions in *one* simulator. i
>      > honestly figured this would crash quickly. it didn't. we managed
>     to get
>      > all 1024 regions created and running well enough to walk around.
>     it did
>      > take almost 8 hours to start. the first couple regions were
>     created in
>      > 1-2 seconds each. by the end, it was taking 4-5 minutes per region.
>      > clearly there is something quadratic in there (stop using linear
>     lists,
>      > they are evil! :-). but it could have been the mono garbage
>     collector.
>      > who knows... not sure its worth too much investigation because i
>     can't
>      > imagine ever running a config like this for real. the simulator did
>      > crash when we opened the map in the viewer. the crash was caused
>     because
>      > we ran out of sockets. while it was running, the simulator used just
>      > over 10G of ram and was running at about 700% CPU utilization
>     (kind of
>      > scary to see load averages in the 900 range!).
>      >
>      > the second configuration ran 16 simulators each with 64 regions.
>     startup
>      > took about 30 minutes (each of the 16 simulators avoided the
>     quadratic
>      > "knee" we hit with the one big simulator). consumed about 11G of
>     memory
>      > and was again consuming essentially all cycles on the machine
>      > (completely idle regions aren't very idle). all 16 simulators
>     died just
>      > after startup with a "too many open files" error. not sure what
>     caused
>      > it, but all of them died loading the same terrain dll as part of some
>      > wildcard function looking for dlls. that one is an interesting bug.
>      >
>      > the final configuration and the one we're shooting for in the
>     short term
>      > runs 16 simulators, each with an 8x8 megaregion. again startup was
>      > around 40 minutes. we might be able to cut that time down by starting
>      > all 16 simulators simultaneously rather than 4 at a time. i
>     really just
>      > wanted to make sure that some of the spikes we see in startup didn't
>      > cause failures (some race condition causes all threads to consume
>      > maximum processor cycles randomly on startup right now). and,
>     well... it
>      > just worked. i figured the viewer would die horribly (it can't handle
>      > 250K "real" prims very well) but it survived just fine. turn off far
>      > clip with 8x8 megaregions providing neighbors and you are
>     "capable" of
>      > contacting simulators in a 24x24 region range!. the view is
>     pretty cool
>      > though it seems to not go beyond 12x12. :-) there are a bunch of
>      > problems (like you can't see the terrain in the region you're
>     standing
>      > in)... but there are a LOT of little humps of terrain in view.
>     oh... and
>      > it takes a lot of patience to get everything loaded.
>      >
>      > so... the conclusion... this opensim thing is pretty amazing!
>     good work
>      > everyone.
>      >
>      > --mic
>      >
>      > _______________________________________________
>      > Opensim-dev mailing list
>      > [hidden email]
>     <mailto:[hidden email]>
>     <mailto:[hidden email]
>     <mailto:[hidden email]>>
>      > https://lists.berlios.de/mailman/listinfo/opensim-dev
>      >
>      >
>      >
>      > _______________________________________________
>      > Opensim-dev mailing list
>      > [hidden email]
>     <mailto:[hidden email]>
>     <mailto:[hidden email]
>     <mailto:[hidden email]>>
>      > https://lists.berlios.de/mailman/listinfo/opensim-dev
>      >
>      >
>      >
>      > No virus found in this incoming message.
>      > Checked by AVG - www.avg.com <http://www.avg.com>
>      > Version: 9.0.730 / Virus Database: 271.1.1/2638 - Release Date:
>     01/22/10
>      > 14:33:00
>      >
>      >
>      >
>     ------------------------------------------------------------------------
>      >
>      > _______________________________________________
>      > Opensim-dev mailing list
>      > [hidden email] <mailto:[hidden email]>
>      > https://lists.berlios.de/mailman/listinfo/opensim-dev
>     _______________________________________________
>     Opensim-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.berlios.de/mailman/listinfo/opensim-dev
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Opensim-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-dev
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users


_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users