Load Testing

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

Load Testing

Robert Klein
It would be nice to do some basic load testing of my OpenSim install. Has anyone developed some method for doing a load test? Getting 40 friends to hit your server at the same time does not count! :)

-Robert
Reply | Threaded
Open this post in threaded view
|

Re: Load Testing

John Sheridan
This may sound kind of goofey but my normal method of load testing
consists of logging in as many clients as humanly possible (usually 2
from hippo with minimal graphics and 15 concurrent copies of Meteabolt)
into one sim / server.  From there I watch the server consoles from a
separate monitor for memory / cpu use while I bump and push the avatars
around using one of the hippo clients.

Thanks, :)

 - John


Robert Klein wrote:
> It would be nice to do some basic load testing of my OpenSim install. Has
> anyone developed some method for doing a load test? Getting 40 friends to
> hit your server at the same time does not count! :)
>
> -Robert
>  


_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Load Testing

Robert Klein
Hey John that is pretty creative! +1 for John!

-Robert

John Sheridan wrote
This may sound kind of goofey but my normal method of load testing
consists of logging in as many clients as humanly possible (usually 2
from hippo with minimal graphics and 15 concurrent copies of Meteabolt)
into one sim / server.  From there I watch the server consoles from a
separate monitor for memory / cpu use while I bump and push the avatars
around using one of the hippo clients.

Thanks, :)

 - John
Reply | Threaded
Open this post in threaded view
|

Re: Load Testing

Mariusz Nowostawski
In reply to this post by Robert Klein
Hi Robert,

If you are serious about the testing, then University of Otago testbed
(www.gni.otago.ac.nz) and World45 (opensim.world45.com) have established
a hardware/software framework for virtual worlds performance testing.
They have computing facilities to run concurrently (within a
semi-isolated LAN) an arbitrary number of concurrent real SL clients (or
any client type of your choice) and run the tests on a number of
hardware architectures (as per your specification).

If you are interested, you would need to drop them your particular
region, specify what clients you would like to use, what the client
activities should be, what server hardware you are interested in testing
and provide the grid configurations. Otago would then run the tests for
you and prepare the report highlighting the encountered bottlenecks and
general behaviour of your setup.  Otago does general profiling, network
evaluations, and scalability studies.

if you are interested, please can contact Melanie Middlemiss for more
details
[hidden email]


--
hth,
Mariusz



Robert Klein wrote:
> It would be nice to do some basic load testing of my OpenSim install. Has
> anyone developed some method for doing a load test? Getting 40 friends to
> hit your server at the same time does not count! :)
>
> -Robert
>  

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Load Testing

Frisby, Adam
I would be really interested in this - is this something they would do as a contribution to us, or is it paid?

If it's paid, does anyone else on this list want to maybe see if we can pool some funds together for testing? (Obviously if we can do it ourselves, great - but it has been a problem in the past simply getting enough users at the same time)

Regards,

Adam

> -----Original Message-----
> From: [hidden email] [mailto:opensim-users-
> [hidden email]] On Behalf Of Mariusz Nowostawski
> Sent: Tuesday, 3 March 2009 7:46 PM
> To: [hidden email]; [hidden email]
> Subject: Re: [Opensim-users] Load Testing
>
> Hi Robert,
>
> If you are serious about the testing, then University of Otago testbed
> (www.gni.otago.ac.nz) and World45 (opensim.world45.com) have
> established
> a hardware/software framework for virtual worlds performance testing.
> They have computing facilities to run concurrently (within a
> semi-isolated LAN) an arbitrary number of concurrent real SL clients
> (or
> any client type of your choice) and run the tests on a number of
> hardware architectures (as per your specification).
>
> If you are interested, you would need to drop them your particular
> region, specify what clients you would like to use, what the client
> activities should be, what server hardware you are interested in
> testing
> and provide the grid configurations. Otago would then run the tests for
> you and prepare the report highlighting the encountered bottlenecks and
> general behaviour of your setup.  Otago does general profiling, network
> evaluations, and scalability studies.
>
> if you are interested, please can contact Melanie Middlemiss for more
> details
> [hidden email]
>
>
> --
> hth,
> Mariusz
>
>
>
> Robert Klein wrote:
> > It would be nice to do some basic load testing of my OpenSim install.
> Has
> > anyone developed some method for doing a load test? Getting 40
> friends to
> > hit your server at the same time does not count! :)
> >
> > -Robert
> >
>
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Load Testing

Dirk Krause
I am not aware of any attempts to get enough users together ...? was this announced on this list? I was sort of waiting in the wings to get all active users of my company together to contribute to such a test, but maybe I just missed it :-/.

Other than that that this seems to be cool solution.

On another thought - I think this could be achieved with Amazons EC2 also. You fire up N instances of AIMs with M instances of TestClient.exe and see what it takes to bring the Amazon Datacenter down :-).

-----Ursprüngliche Nachricht-----
Von: [hidden email] [mailto:[hidden email]] Im Auftrag von Frisby, Adam
Gesendet: Mittwoch, 4. März 2009 07:24
An: [hidden email]; [hidden email]
Betreff: Re: [Opensim-users] Load Testing

I would be really interested in this - is this something they would do as a contribution to us, or is it paid?

If it's paid, does anyone else on this list want to maybe see if we can pool some funds together for testing? (Obviously if we can do it ourselves, great - but it has been a problem in the past simply getting enough users at the same time)

Regards,

Adam

> -----Original Message-----
> From: [hidden email] [mailto:opensim-users-
> [hidden email]] On Behalf Of Mariusz Nowostawski
> Sent: Tuesday, 3 March 2009 7:46 PM
> To: [hidden email]; [hidden email]
> Subject: Re: [Opensim-users] Load Testing
>
> Hi Robert,
>
> If you are serious about the testing, then University of Otago testbed
> (www.gni.otago.ac.nz) and World45 (opensim.world45.com) have
> established
> a hardware/software framework for virtual worlds performance testing.
> They have computing facilities to run concurrently (within a
> semi-isolated LAN) an arbitrary number of concurrent real SL clients
> (or
> any client type of your choice) and run the tests on a number of
> hardware architectures (as per your specification).
>
> If you are interested, you would need to drop them your particular
> region, specify what clients you would like to use, what the client
> activities should be, what server hardware you are interested in
> testing
> and provide the grid configurations. Otago would then run the tests for
> you and prepare the report highlighting the encountered bottlenecks and
> general behaviour of your setup.  Otago does general profiling, network
> evaluations, and scalability studies.
>
> if you are interested, please can contact Melanie Middlemiss for more
> details
> [hidden email]
>
>
> --
> hth,
> Mariusz
>
>
>
> Robert Klein wrote:
> > It would be nice to do some basic load testing of my OpenSim install.
> Has
> > anyone developed some method for doing a load test? Getting 40
> friends to
> > hit your server at the same time does not count! :)
> >
> > -Robert
> >
>
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Load Testing

Dave Coyle
On 2009-03-04 01:33:40 -0500, [hidden email] wrote:
> I am not aware of any attempts to get enough users together ...? was
> this announced on this list? I was sort of waiting in the wings to
> get all active users of my company together to contribute to such a
> test, but maybe I just missed it :-/.

Would there be any interest in a regularly scheduled "come and break
stuff hour" on OSGrid?  Time, victim^H^H^H^H target sim, and desired
activity would be prearranged, and at the appointed time and place
anyone who's interested logs in and we see what happens.  We could
test TPs or region crossings or messaging modules or rezzing objects
or sitting in a boat and sailing around in a circle or capacity of
different hardware combinations.  Wright Plaza seems to get a good
turnout for weekly office hours, but I think everyone goes out of
their way to not do anything to break stuff.  The point of "come and
break stuff hour" would be to do the opposite.

Also, right now the answer to "what hardware do I need to run
OpenSim?" is "it depends".  It would be great if we could test various
combinations of hardware/.NET/Mono builds/etc. and get some more
concrete info on what works, what's mediocre, and what fails.

I haven't been to the "Q&A / Testing" hour on OSGrid in a while, but
when I did attend it was focused on Q&A.

Say, Thursdays @ 0300 UTC?  We could give it a go a couple times and,
depending on turnout, decide if it's worth making a regular thing.
And it's an earlier/later time to try to accomodate people in other
regions/on different schedules.

Anyone interested?

-Coyle
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Load Testing

Tristan-2
I'll go to that, it's at 7 PM my time.

On Wed, Mar 4, 2009 at 8:04 AM, Dave Coyle <[hidden email]> wrote:
On 2009-03-04 01:33:40 -0500, [hidden email] wrote:
> I am not aware of any attempts to get enough users together ...? was
> this announced on this list? I was sort of waiting in the wings to
> get all active users of my company together to contribute to such a
> test, but maybe I just missed it :-/.

Would there be any interest in a regularly scheduled "come and break
stuff hour" on OSGrid?  Time, victim^H^H^H^H target sim, and desired
activity would be prearranged, and at the appointed time and place
anyone who's interested logs in and we see what happens.  We could
test TPs or region crossings or messaging modules or rezzing objects
or sitting in a boat and sailing around in a circle or capacity of
different hardware combinations.  Wright Plaza seems to get a good
turnout for weekly office hours, but I think everyone goes out of
their way to not do anything to break stuff.  The point of "come and
break stuff hour" would be to do the opposite.

Also, right now the answer to "what hardware do I need to run
OpenSim?" is "it depends".  It would be great if we could test various
combinations of hardware/.NET/Mono builds/etc. and get some more
concrete info on what works, what's mediocre, and what fails.

I haven't been to the "Q&A / Testing" hour on OSGrid in a while, but
when I did attend it was focused on Q&A.

Say, Thursdays @ 0300 UTC?  We could give it a go a couple times and,
depending on turnout, decide if it's worth making a regular thing.
And it's an earlier/later time to try to accomodate people in other
regions/on different schedules.

Anyone interested?

-Coyle
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users


_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Load Testing

Charles Krinke-3
In reply to this post by Dave Coyle
Dear Dave:

Originally, Saturday 1900UTC on Wright Plaza was "Test Hour" with just that purpose, but the community has evolved it a bit.

I would say a "Crash Hour" or "Test Hour" again is most appropriate and would encourage you or others so organize it and tell the rest of us where and when.

Charles


From: Dave Coyle <[hidden email]>
To: [hidden email]
Sent: Wednesday, March 4, 2009 8:04:24 AM
Subject: Re: [Opensim-users] Load Testing

On 2009-03-04 01:33:40 -0500, [hidden email] wrote:
> I am not aware of any attempts to get enough users together ...? was
> this announced on this list? I was sort of waiting in the wings to
> get all active users of my company together to contribute to such a
> test, but maybe I just missed it :-/.

Would there be any interest in a regularly scheduled "come and break
stuff hour" on OSGrid?  Time, victim^H^H^H^H target sim, and desired
activity would be prearranged, and at the appointed time and place
anyone who's interested logs in and we see what happens.  We could
test TPs or region crossings or messaging modules or rezzing objects
or sitting in a boat and sailing around in a circle or capacity of
different hardware combinations.  Wright Plaza seems to get a good
turnout for weekly office hours, but I think everyone goes out of
their way to not do anything to break stuff.  The point of "come and
break stuff hour" would be to do the opposite.

Also, right now the answer to "what hardware do I need to run
OpenSim?" is "it depends".  It would be great if we could test various
combinations of hardware/.NET/Mono builds/etc. and get some more
concrete info on what works, what's mediocre, and what fails.

I haven't been to the "Q&A / Testing" hour on OSGrid in a while, but
when I did attend it was focused on Q&A.

Say, Thursdays @ 0300 UTC?  We could give it a go a couple times and,
depending on turnout, decide if it's worth making a regular thing.
And it's an earlier/later time to try to accomodate people in other
regions/on different schedules.

Anyone interested?

-Coyle
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Load Testing

Mic Bowman
In reply to this post by Robert Klein
Robert,

As you can imagine, we're particularly interested in performance.
We've put together a collection of synthetic workloads that isolate
various aspects of the simulator (physics, network, scene complexity).
The tests are largely automated (using the libomv testclient) and can
easily be run with a simulator, a few clients, and a fast network.

Another thing we're doing that you might find interesting is that
we're profiling runs of the various tests to help isolate the
performance bottlenecks. We plan to update the profile information as
we run the various tests against different versions of OpenSim.

All the tests and profile data is available from the ScienceSim wiki:
http://www.sciencesim.com/wiki/doku.php/opensim/start

Finally, we'd really like your feedback on other tests we might add to
the suite.

--mic

C. Mic Bowman, PhD
Principal Engineer, Intel Corp
Virtual World Infrastructure


On Tue, Mar 3, 2009 at 12:24 PM, Robert Klein <[hidden email]> wrote:

>
> It would be nice to do some basic load testing of my OpenSim install. Has
> anyone developed some method for doing a load test? Getting 40 friends to
> hit your server at the same time does not count! :)
>
> -Robert
> --
> View this message in context: http://n2.nabble.com/Load-Testing-tp2417709p2417709.html
> Sent from the opensim-users mailing list archive at Nabble.com.
>
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users
>
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Load Testing

Frisby, Adam
Hey Mic,

Could we somehow get these built as Nant-compatible tests? It may well be worth us setting opensim.org (or another box) to do some automated profiling tests once every few days.

Adam

> -----Original Message-----
> From: [hidden email] [mailto:opensim-users-
> [hidden email]] On Behalf Of Mic Bowman
> Sent: Wednesday, 4 March 2009 8:56 AM
> To: [hidden email]; [hidden email]
> Subject: Re: [Opensim-users] Load Testing
>
> Robert,
>
> As you can imagine, we're particularly interested in performance.
> We've put together a collection of synthetic workloads that isolate
> various aspects of the simulator (physics, network, scene complexity).
> The tests are largely automated (using the libomv testclient) and can
> easily be run with a simulator, a few clients, and a fast network.
>
> Another thing we're doing that you might find interesting is that
> we're profiling runs of the various tests to help isolate the
> performance bottlenecks. We plan to update the profile information as
> we run the various tests against different versions of OpenSim.
>
> All the tests and profile data is available from the ScienceSim wiki:
> http://www.sciencesim.com/wiki/doku.php/opensim/start
>
> Finally, we'd really like your feedback on other tests we might add to
> the suite.
>
> --mic
>
> C. Mic Bowman, PhD
> Principal Engineer, Intel Corp
> Virtual World Infrastructure
>
>
> On Tue, Mar 3, 2009 at 12:24 PM, Robert Klein <[hidden email]>
> wrote:
> >
> > It would be nice to do some basic load testing of my OpenSim install.
> Has
> > anyone developed some method for doing a load test? Getting 40
> friends to
> > hit your server at the same time does not count! :)
> >
> > -Robert
> > --
> > View this message in context: http://n2.nabble.com/Load-Testing-
> tp2417709p2417709.html
> > Sent from the opensim-users mailing list archive at Nabble.com.
> >
> > _______________________________________________
> > Opensim-users mailing list
> > [hidden email]
> > https://lists.berlios.de/mailman/listinfo/opensim-users
> >
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Load Testing

Dirk Krause
In reply to this post by Dave Coyle
I obviously am interested. I just have a problem with the time, UTC 0300 is an awfully un-european time.

-----Ursprüngliche Nachricht-----
Von: [hidden email] [mailto:[hidden email]] Im Auftrag von Dave Coyle
Gesendet: Mittwoch, 4. März 2009 17:04
An: [hidden email]
Betreff: Re: [Opensim-users] Load Testing

On 2009-03-04 01:33:40 -0500, [hidden email] wrote:
> I am not aware of any attempts to get enough users together ...? was
> this announced on this list? I was sort of waiting in the wings to
> get all active users of my company together to contribute to such a
> test, but maybe I just missed it :-/.

Would there be any interest in a regularly scheduled "come and break
stuff hour" on OSGrid?  Time, victim^H^H^H^H target sim, and desired
activity would be prearranged, and at the appointed time and place
anyone who's interested logs in and we see what happens.  We could
test TPs or region crossings or messaging modules or rezzing objects
or sitting in a boat and sailing around in a circle or capacity of
different hardware combinations.  Wright Plaza seems to get a good
turnout for weekly office hours, but I think everyone goes out of
their way to not do anything to break stuff.  The point of "come and
break stuff hour" would be to do the opposite.

Also, right now the answer to "what hardware do I need to run
OpenSim?" is "it depends".  It would be great if we could test various
combinations of hardware/.NET/Mono builds/etc. and get some more
concrete info on what works, what's mediocre, and what fails.

I haven't been to the "Q&A / Testing" hour on OSGrid in a while, but
when I did attend it was focused on Q&A.

Say, Thursdays @ 0300 UTC?  We could give it a go a couple times and,
depending on turnout, decide if it's worth making a regular thing.
And it's an earlier/later time to try to accomodate people in other
regions/on different schedules.

Anyone interested?

-Coyle
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Load Testing

Mic Bowman
In reply to this post by Frisby, Adam
For clarity... what do you mean by "nant-compatible"? If you mean
incorporate the tests into the existing OpenSim test suite... that
would be tough given the number of moving parts. If you mean create
something that could automate execution with a profile build... that's
more likely (whether we drive it through nant or something else). Let
me know more about the behavior you'd like to see.

--mic


On Wed, Mar 4, 2009 at 9:11 AM, Frisby, Adam <[hidden email]> wrote:

> Hey Mic,
>
> Could we somehow get these built as Nant-compatible tests? It may well be worth us setting opensim.org (or another box) to do some automated profiling tests once every few days.
>
> Adam
>
>> -----Original Message-----
>> From: [hidden email] [mailto:opensim-users-
>> [hidden email]] On Behalf Of Mic Bowman
>> Sent: Wednesday, 4 March 2009 8:56 AM
>> To: [hidden email]; [hidden email]
>> Subject: Re: [Opensim-users] Load Testing
>>
>> Robert,
>>
>> As you can imagine, we're particularly interested in performance.
>> We've put together a collection of synthetic workloads that isolate
>> various aspects of the simulator (physics, network, scene complexity).
>> The tests are largely automated (using the libomv testclient) and can
>> easily be run with a simulator, a few clients, and a fast network.
>>
>> Another thing we're doing that you might find interesting is that
>> we're profiling runs of the various tests to help isolate the
>> performance bottlenecks. We plan to update the profile information as
>> we run the various tests against different versions of OpenSim.
>>
>> All the tests and profile data is available from the ScienceSim wiki:
>> http://www.sciencesim.com/wiki/doku.php/opensim/start
>>
>> Finally, we'd really like your feedback on other tests we might add to
>> the suite.
>>
>> --mic
>>
>> C. Mic Bowman, PhD
>> Principal Engineer, Intel Corp
>> Virtual World Infrastructure
>>
>>
>> On Tue, Mar 3, 2009 at 12:24 PM, Robert Klein <[hidden email]>
>> wrote:
>> >
>> > It would be nice to do some basic load testing of my OpenSim install.
>> Has
>> > anyone developed some method for doing a load test? Getting 40
>> friends to
>> > hit your server at the same time does not count! :)
>> >
>> > -Robert
>> > --
>> > View this message in context: http://n2.nabble.com/Load-Testing-
>> tp2417709p2417709.html
>> > Sent from the opensim-users mailing list archive at Nabble.com.
>> >
>> > _______________________________________________
>> > Opensim-users mailing list
>> > [hidden email]
>> > https://lists.berlios.de/mailman/listinfo/opensim-users
>> >
>> _______________________________________________
>> Opensim-users mailing list
>> [hidden email]
>> https://lists.berlios.de/mailman/listinfo/opensim-users
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users
>
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Load Testing

Frisby, Adam
Something we can automate - ideally through our current bamboo install or similar.

Adam

> -----Original Message-----
> From: [hidden email] [mailto:opensim-users-
> [hidden email]] On Behalf Of Mic Bowman
> Sent: Wednesday, 4 March 2009 9:19 AM
> To: [hidden email]
> Subject: Re: [Opensim-users] Load Testing
>
> For clarity... what do you mean by "nant-compatible"? If you mean
> incorporate the tests into the existing OpenSim test suite... that
> would be tough given the number of moving parts. If you mean create
> something that could automate execution with a profile build... that's
> more likely (whether we drive it through nant or something else). Let
> me know more about the behavior you'd like to see.
>
> --mic
>
>
> On Wed, Mar 4, 2009 at 9:11 AM, Frisby, Adam <[hidden email]>
> wrote:
> > Hey Mic,
> >
> > Could we somehow get these built as Nant-compatible tests? It may
> well be worth us setting opensim.org (or another box) to do some
> automated profiling tests once every few days.
> >
> > Adam
> >
> >> -----Original Message-----
> >> From: [hidden email] [mailto:opensim-users-
> >> [hidden email]] On Behalf Of Mic Bowman
> >> Sent: Wednesday, 4 March 2009 8:56 AM
> >> To: [hidden email]; [hidden email]
> >> Subject: Re: [Opensim-users] Load Testing
> >>
> >> Robert,
> >>
> >> As you can imagine, we're particularly interested in performance.
> >> We've put together a collection of synthetic workloads that isolate
> >> various aspects of the simulator (physics, network, scene
> complexity).
> >> The tests are largely automated (using the libomv testclient) and
> can
> >> easily be run with a simulator, a few clients, and a fast network.
> >>
> >> Another thing we're doing that you might find interesting is that
> >> we're profiling runs of the various tests to help isolate the
> >> performance bottlenecks. We plan to update the profile information
> as
> >> we run the various tests against different versions of OpenSim.
> >>
> >> All the tests and profile data is available from the ScienceSim
> wiki:
> >> http://www.sciencesim.com/wiki/doku.php/opensim/start
> >>
> >> Finally, we'd really like your feedback on other tests we might add
> to
> >> the suite.
> >>
> >> --mic
> >>
> >> C. Mic Bowman, PhD
> >> Principal Engineer, Intel Corp
> >> Virtual World Infrastructure
> >>
> >>
> >> On Tue, Mar 3, 2009 at 12:24 PM, Robert Klein <[hidden email]>
> >> wrote:
> >> >
> >> > It would be nice to do some basic load testing of my OpenSim
> install.
> >> Has
> >> > anyone developed some method for doing a load test? Getting 40
> >> friends to
> >> > hit your server at the same time does not count! :)
> >> >
> >> > -Robert
> >> > --
> >> > View this message in context: http://n2.nabble.com/Load-Testing-
> >> tp2417709p2417709.html
> >> > Sent from the opensim-users mailing list archive at Nabble.com.
> >> >
> >> > _______________________________________________
> >> > Opensim-users mailing list
> >> > [hidden email]
> >> > https://lists.berlios.de/mailman/listinfo/opensim-users
> >> >
> >> _______________________________________________
> >> Opensim-users mailing list
> >> [hidden email]
> >> https://lists.berlios.de/mailman/listinfo/opensim-users
> > _______________________________________________
> > Opensim-users mailing list
> > [hidden email]
> > https://lists.berlios.de/mailman/listinfo/opensim-users
> >
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Load Testing

Robert Klein
Wow, I am glad to see all the interest in doing load testing! The site mic mentions is very well done and has some great info so check it out. It sounds like there is a need for both automated and manual testing methods. A good place to start might be a category on the wiki.
Here is the link:
http://opensimulator.org/wiki/Category:Testing  :)

We will need proper scripts for both methods. I am not sure what should be in the automated script, but the manual script could be something the tester could follow step by step no matter what region/simulator they are invited to test. Maybe settle on two time frames that are friendly to all testers? The test window could be two hours to allow enough people to participate.

Here is the page for the manual test script:
http://opensimulator.org/wiki/Manual_Test_Script

There is a nice page on automated testing already on the wiki:
http://opensimulator.org/wiki/Automated_Testing

I am seeing a lot of good ideas on this thread so keep it up and let's hammer some simulators!

-Robert

Frisby, Adam wrote
> >> Robert,
> >>
> >> As you can imagine, we're particularly interested in performance.
> >> We've put together a collection of synthetic workloads that isolate
> >> various aspects of the simulator (physics, network, scene
> complexity).
> >> The tests are largely automated (using the libomv testclient) and
> can
> >> easily be run with a simulator, a few clients, and a fast network.
> >>
> >> Another thing we're doing that you might find interesting is that
> >> we're profiling runs of the various tests to help isolate the
> >> performance bottlenecks. We plan to update the profile information
> as
> >> we run the various tests against different versions of OpenSim.
> >>
> >> All the tests and profile data is available from the ScienceSim
> wiki:
> >> http://www.sciencesim.com/wiki/doku.php/opensim/start
> >>
> >> Finally, we'd really like your feedback on other tests we might add
> to
> >> the suite.
> >>
> >> --mic
> >>
> >> C. Mic Bowman, PhD
> >> Principal Engineer, Intel Corp
> >> Virtual World Infrastructure
Reply | Threaded
Open this post in threaded view
|

Re: Load Testing

Mariusz Nowostawski
In reply to this post by Dave Coyle
Dave Coyle wrote:

> On 2009-03-04 01:33:40 -0500, [hidden email] wrote:
>  
>> I am not aware of any attempts to get enough users together ...? was
>> this announced on this list? I was sort of waiting in the wings to
>> get all active users of my company together to contribute to such a
>> test, but maybe I just missed it :-/.
>>    
>
> Would there be any interest in a regularly scheduled "come and break
> stuff hour" on OSGrid?  Time, victim^H^H^H^H target sim, and desired
> activity would be prearranged, and at the appointed time and place
> anyone who's interested logs in and we see what happens.  We could
> test TPs or region crossings or messaging modules or rezzing objects
> or sitting in a boat and sailing around in a circle or capacity of
> different hardware combinations.  Wright Plaza seems to get a good
> turnout for weekly office hours, but I think everyone goes out of
> their way to not do anything to break stuff.  The point of "come and
> break stuff hour" would be to do the opposite.
>
> Also, right now the answer to "what hardware do I need to run
> OpenSim?" is "it depends".  It would be great if we could test various
> combinations of hardware/.NET/Mono builds/etc. and get some more
> concrete info on what works, what's mediocre, and what fails.
>
> I haven't been to the "Q&A / Testing" hour on OSGrid in a while, but
> when I did attend it was focused on Q&A.
>
> Say, Thursdays @ 0300 UTC?  We could give it a go a couple times and,
> depending on turnout, decide if it's worth making a regular thing.
> And it's an earlier/later time to try to accomodate people in other
> regions/on different schedules.
>
> Anyone interested


Load testing by assembling users in an ad-hoc manner has its place, and
it might be a good way of keeping track of certain server-side behaviour
and user experiences. However, logs from such sessions are limited in
use, because the tests cannot be redone, and no statistics can be
obtained. From our experiences we have observed quite substantial
variations of certain load parameters with no correlation to what was
been actually done in a particular instance. Identifying the causes of
sertain bottlenecks or server behaviour is not easy, and multiple tries
of the same test schema with varying only few parameters are useful in
that regard.


Generally, there are three general types of tests, that people currently
do on OpenSim.
A) ad-hoc gathering of users (as prescribed above), without the ability
to replicate the tests or conditions, and without the ability to draw
statistics on the server behaviour
B) more formal runs with real users (and real user clients) in a
controlled environment, where effectively the same runs can be repeated
over and over again to obtain statistical behaviour and identify
bottlenecks by controlling limited number of factors
C) runs without real clients and without real users, where the inputs to
the system are generated by TestClients that "simulate" real clients -
these can be used as Mic Bowman explained to test various individual
aspects of the server performance and these are ideal for tuning those
isolated modules of the server


Each of these has its place and role, and they all play a complementary
roles. At Otago, we focus primarily on B, where real clients are used to
test the system holistically, and investigate different
cross-dependencies between the server modules. I do not think Type A can
be formalised or automated. As for B, we tried to automate as much as
possible as to make the tests easier to run and manage, but, we are far
from NANT-like automation.  It should be possible to automate Case C,
and conduct multiple runs to obtain statistics in a fully automated way.
We are quite interested in that, and will watch where the developments
are heading.


--
cheers
Mariusz
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Load Testing

Robert Klein
Excellent response Mariusz! There are many ways, as Mariusz points out, to test our simulators. There are formal and informal methods. It really depends on what your goals are for the test. This is generally referred to as "starting with the end in mind". Someone who is just opening up a nightclub on their region may only want an informal "can my region handle 80 avatars while music and video are playing?" Others may need a formal set of replicated tests to manage their server efficiency between certain modules compared to overall bandwidth and RAM usage. Whatever the need, proper load testing is a big help. Maybe start coming up with a set of scripts to fill various needs like a new bare single region, two regions, four regions, party spot, lecture hall, etc and map out some of the formal and informal items each needs to address?

-Robert

Mariusz Nowostawski wrote
Load testing by assembling users in an ad-hoc manner has its place, and
it might be a good way of keeping track of certain server-side behaviour
and user experiences. However, logs from such sessions are limited in
use, because the tests cannot be redone, and no statistics can be
obtained. From our experiences we have observed quite substantial
variations of certain load parameters with no correlation to what was
been actually done in a particular instance. Identifying the causes of
sertain bottlenecks or server behaviour is not easy, and multiple tries
of the same test schema with varying only few parameters are useful in
that regard.


Generally, there are three general types of tests, that people currently
do on OpenSim.
A) ad-hoc gathering of users (as prescribed above), without the ability
to replicate the tests or conditions, and without the ability to draw
statistics on the server behaviour
B) more formal runs with real users (and real user clients) in a
controlled environment, where effectively the same runs can be repeated
over and over again to obtain statistical behaviour and identify
bottlenecks by controlling limited number of factors
C) runs without real clients and without real users, where the inputs to
the system are generated by TestClients that "simulate" real clients -
these can be used as Mic Bowman explained to test various individual
aspects of the server performance and these are ideal for tuning those
isolated modules of the server


Each of these has its place and role, and they all play a complementary
roles. At Otago, we focus primarily on B, where real clients are used to
test the system holistically, and investigate different
cross-dependencies between the server modules. I do not think Type A can
be formalised or automated. As for B, we tried to automate as much as
possible as to make the tests easier to run and manage, but, we are far
from NANT-like automation.  It should be possible to automate Case C,
and conduct multiple runs to obtain statistics in a fully automated way.
We are quite interested in that, and will watch where the developments
are heading.


--
cheers
Mariusz
_______________________________________________
Opensim-users mailing list
Opensim-users@lists.berlios.de
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Load Testing

Dave Coyle
In reply to this post by Mariusz Nowostawski
On 2009-03-04 22:58:11 -0500, [hidden email] wrote:
> Load testing by assembling users in an ad-hoc manner has its place,
> and it might be a good way of keeping track of certain server-side
> behaviour and user experiences.

And that's the intent.  Obviously having a different number of avs do
different things on different OpenSim revs running on different
boxes running different operating systems connected to different
networks and attached to a public grid each week isn't going to
produce a list of "x causes y" results.  That's not what I'm after,
with this meetup, anyway.

Every day someone asks on #opensim "what kind of hardware do I need to
run OpenSim?", and the answer is always "it depends" and/or "here's
what I run, try that".  There aren't enough published data points for
someone to even begin to guestimate where to start.  Yes, OpenSim is
changing rapidly, and what happily runs it today may crash and set
fire tomorrow.  But some people would still find the data useful.

There's also a social aspect to it.  Formal, reproducable testing
under controlled conditions has its place.  As does hosting 15 people
from around the world on your own little piece of this wild frontier.

-Coyle
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Load Testing

Jeroen van Veen
Cacti/netsnmp?
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users