Ubuntu and magic mini-trampolines (crash problem)

classic Classic list List threaded Threaded
28 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Ubuntu and magic mini-trampolines (crash problem)

John Hopkin
I've just upgraded Mono from 2.1 to 2.2 under Ubuntu Hardy, using Dr
Scofield's script from here:

http://xyzzyxyzzy.net/2009/02/17/updated-mono-build-script-mono-22/

and all went smoothly with that.

But now, when I run OpenSim.exe, it loads OK at first, but then I get
the error:

<start paste>
Region (root) # 0x18 0x0 0x0 0x55 0x8b 0xec 0x57 0x56 0x83 0xec 0x20
0x8b 0x75 0x8 0x83 0xec 0x8 0x56 0x68 0x10 0x75 0xe8 0x9c 0x8b 0x5
0x10 0x75 0xe8 0x9c 0xff 0x50 0x34
**
ERROR:mini-trampolines.c:122:mono_magic_trampoline: assertion failed:
(vt)
Stacktrace:

  at log4net.LogManager.WrapLogger (log4net.Core.ILogger) <0xffffffff>
  at log4net.LogManager.WrapLogger (log4net.Core.ILogger) <0x0001d>
<end paste>

and a long stacktrace.  The server crashes, of course.  Presumably,
the references to magic mini-trampolines are within the code; I don't
have any scripts or objects that I'm aware of like that.

Vital stats:

* OpenSim revision 9205 (binary package from OSGrid)
* Ubuntu Intrepid, kernel 2.6.27-11-generic
* Mono run with MONO_THREADS_PER_CPU set to 2000
* 7 regions run by one OpenSim.exe
*  P4/3.6GHz, 2GB RAM, more than adequate (or were under Mono 2.1)
* UGAIM through OSGrid
* MySQL 14.12

I'm completely at a loss as to what could be causing this.  Does
anyone have any ideas?
--
John Hopkin

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu and magic mini-trampolines (crash problem)

Snoopy Pfeffer
I have experienced the same when MONO_THREADS_PER_CPU is set to a very high number. 500 works for me.

  Snoopy Pfeffer


From: John Hopkin <[hidden email]>
To: [hidden email]
Sent: Thursday, April 23, 2009 1:50:37 AM
Subject: [Opensim-users] Ubuntu and magic mini-trampolines (crash problem)

I've just upgraded Mono from 2.1 to 2.2 under Ubuntu Hardy, using Dr
Scofield's script from here:

http://xyzzyxyzzy.net/2009/02/17/updated-mono-build-script-mono-22/

and all went smoothly with that.

But now, when I run OpenSim.exe, it loads OK at first, but then I get
the error:

<start paste>
Region (root) # 0x18 0x0 0x0 0x55 0x8b 0xec 0x57 0x56 0x83 0xec 0x20
0x8b 0x75 0x8 0x83 0xec 0x8 0x56 0x68 0x10 0x75 0xe8 0x9c 0x8b 0x5
0x10 0x75 0xe8 0x9c 0xff 0x50 0x34
**
ERROR:mini-trampolines.c:122:mono_magic_trampoline: assertion failed:
(vt)
Stacktrace:

  at log4net.LogManager.WrapLogger (log4net.Core.ILogger) <0xffffffff>
  at log4net.LogManager.WrapLogger (log4net.Core.ILogger) <0x0001d>
<end paste>

and a long stacktrace.  The server crashes, of course.  Presumably,
the references to magic mini-trampolines are within the code; I don't
have any scripts or objects that I'm aware of like that.

Vital stats:

* OpenSim revision 9205 (binary package from OSGrid)
* Ubuntu Intrepid, kernel 2.6.27-11-generic
* Mono run with MONO_THREADS_PER_CPU set to 2000
* 7 regions run by one OpenSim.exe
*  P4/3.6GHz, 2GB RAM, more than adequate (or were under Mono 2.1)
* UGAIM through OSGrid
* MySQL 14.12

I'm completely at a loss as to what could be causing this.  Does
anyone have any ideas?
--
John Hopkin

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users


_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu and magic mini-trampolines (crash problem)

John Hopkin
Thanks.  I'll try that.  At the moment, it's working OK - it seems to
be intermittent, whereas before it was during each bootup.  If/when it
happens again, I'll drop that value.

John

Snoopy Pfeffer wrote:

>I have experienced the same when MONO_THREADS_PER_CPU is set to a very high number. 500 works for me.
>
>  Snoopy Pfeffer
>
>
>
>
>________________________________
>From: John Hopkin <[hidden email]>
>To: [hidden email]
>Sent: Thursday, April 23, 2009 1:50:37 AM
>Subject: [Opensim-users] Ubuntu and magic mini-trampolines (crash problem)
>
>I've just upgraded Mono from 2.1 to 2.2 under Ubuntu Hardy, using Dr
>Scofield's script from here:
>
>http://xyzzyxyzzy.net/2009/02/17/updated-mono-build-script-mono-22/
>
>and all went smoothly with that.
>
>But now, when I run OpenSim.exe, it loads OK at first, but then I get
>the error:
>
><start paste>
>Region (root) # 0x18 0x0 0x0 0x55 0x8b 0xec 0x57 0x56 0x83 0xec 0x20
>0x8b 0x75 0x8 0x83 0xec 0x8 0x56 0x68 0x10 0x75 0xe8 0x9c 0x8b 0x5
>0x10 0x75 0xe8 0x9c 0xff 0x50 0x34
>**
>ERROR:mini-trampolines.c:122:mono_magic_trampoline: assertion failed:
>(vt)
>Stacktrace:
>
>  at log4net.LogManager.WrapLogger (log4net.Core.ILogger) <0xffffffff>
>  at log4net.LogManager.WrapLogger (log4net.Core.ILogger) <0x0001d>
><end paste>
>
>and a long stacktrace.  The server crashes, of course.  Presumably,
>the references to magic mini-trampolines are within the code; I don't
>have any scripts or objects that I'm aware of like that.
>
>Vital stats:
>
>* OpenSim revision 9205 (binary package from OSGrid)
>* Ubuntu Intrepid, kernel 2.6.27-11-generic
>* Mono run with MONO_THREADS_PER_CPU set to 2000
>* 7 regions run by one OpenSim.exe
>*  P4/3.6GHz, 2GB RAM, more than adequate (or were under Mono 2.1)
>* UGAIM through OSGrid
>* MySQL 14.12
>
>I'm completely at a loss as to what could be causing this.  Does
>anyone have any ideas?
--
John Hopkin

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu and magic mini-trampolines (crash problem)

Mic Bowman
there is a mantis for this (3237)...

i've been going through the various script engine calls to log4net and
commenting them out to get rid of the problem. two in particular
seemed to help. i'll put up another patch on that mantis in the next
couple days.

the problem comes and goes. i've tried various combinations of mono
threads, debug/release builds, and even rebuilt log4net. the more
scripts i have in a region that is restarting, the more likely the
problem occurs. there is clearly a race condition somewhere in the
mono 2.2 and 2.4 internals which opensim is tweaking.

--mic


On Wed, Apr 22, 2009 at 6:09 PM, John Hopkin
<[hidden email]> wrote:

> Thanks.  I'll try that.  At the moment, it's working OK - it seems to
> be intermittent, whereas before it was during each bootup.  If/when it
> happens again, I'll drop that value.
>
> John
>
> Snoopy Pfeffer wrote:
>
>>I have experienced the same when MONO_THREADS_PER_CPU is set to a very high number. 500 works for me.
>>
>>  Snoopy Pfeffer
>>
>>
>>
>>
>>________________________________
>>From: John Hopkin <[hidden email]>
>>To: [hidden email]
>>Sent: Thursday, April 23, 2009 1:50:37 AM
>>Subject: [Opensim-users] Ubuntu and magic mini-trampolines (crash problem)
>>
>>I've just upgraded Mono from 2.1 to 2.2 under Ubuntu Hardy, using Dr
>>Scofield's script from here:
>>
>>http://xyzzyxyzzy.net/2009/02/17/updated-mono-build-script-mono-22/
>>
>>and all went smoothly with that.
>>
>>But now, when I run OpenSim.exe, it loads OK at first, but then I get
>>the error:
>>
>><start paste>
>>Region (root) # 0x18 0x0 0x0 0x55 0x8b 0xec 0x57 0x56 0x83 0xec 0x20
>>0x8b 0x75 0x8 0x83 0xec 0x8 0x56 0x68 0x10 0x75 0xe8 0x9c 0x8b 0x5
>>0x10 0x75 0xe8 0x9c 0xff 0x50 0x34
>>**
>>ERROR:mini-trampolines.c:122:mono_magic_trampoline: assertion failed:
>>(vt)
>>Stacktrace:
>>
>>  at log4net.LogManager.WrapLogger (log4net.Core.ILogger) <0xffffffff>
>>  at log4net.LogManager.WrapLogger (log4net.Core.ILogger) <0x0001d>
>><end paste>
>>
>>and a long stacktrace.  The server crashes, of course.  Presumably,
>>the references to magic mini-trampolines are within the code; I don't
>>have any scripts or objects that I'm aware of like that.
>>
>>Vital stats:
>>
>>* OpenSim revision 9205 (binary package from OSGrid)
>>* Ubuntu Intrepid, kernel 2.6.27-11-generic
>>* Mono run with MONO_THREADS_PER_CPU set to 2000
>>* 7 regions run by one OpenSim.exe
>>*  P4/3.6GHz, 2GB RAM, more than adequate (or were under Mono 2.1)
>>* UGAIM through OSGrid
>>* MySQL 14.12
>>
>>I'm completely at a loss as to what could be causing this.  Does
>>anyone have any ideas?
> --
> John Hopkin
>
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users
>
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu and magic mini-trampolines (crash problem)

Frisby, Adam
We frequently run into Mono issues - we really should have an active dialog with those guys since I suspect we're probably at the forefront of things relying on Mono.

Adam

> -----Original Message-----
> From: [hidden email] [mailto:opensim-users-
> [hidden email]] On Behalf Of Mic Bowman
> Sent: Wednesday, 22 April 2009 9:47 PM
> To: [hidden email]
> Subject: Re: [Opensim-users] Ubuntu and magic mini-trampolines (crash
> problem)
>
> there is a mantis for this (3237)...
>
> i've been going through the various script engine calls to log4net and
> commenting them out to get rid of the problem. two in particular
> seemed to help. i'll put up another patch on that mantis in the next
> couple days.
>
> the problem comes and goes. i've tried various combinations of mono
> threads, debug/release builds, and even rebuilt log4net. the more
> scripts i have in a region that is restarting, the more likely the
> problem occurs. there is clearly a race condition somewhere in the
> mono 2.2 and 2.4 internals which opensim is tweaking.
>
> --mic
>
>
> On Wed, Apr 22, 2009 at 6:09 PM, John Hopkin
> <[hidden email]> wrote:
> > Thanks.  I'll try that.  At the moment, it's working OK - it seems to
> > be intermittent, whereas before it was during each bootup.  If/when
> it
> > happens again, I'll drop that value.
> >
> > John
> >
> > Snoopy Pfeffer wrote:
> >
> >>I have experienced the same when MONO_THREADS_PER_CPU is set to a
> very high number. 500 works for me.
> >>
> >>  Snoopy Pfeffer
> >>
> >>
> >>
> >>
> >>________________________________
> >>From: John Hopkin <[hidden email]>
> >>To: [hidden email]
> >>Sent: Thursday, April 23, 2009 1:50:37 AM
> >>Subject: [Opensim-users] Ubuntu and magic mini-trampolines (crash
> problem)
> >>
> >>I've just upgraded Mono from 2.1 to 2.2 under Ubuntu Hardy, using Dr
> >>Scofield's script from here:
> >>
> >>http://xyzzyxyzzy.net/2009/02/17/updated-mono-build-script-mono-22/
> >>
> >>and all went smoothly with that.
> >>
> >>But now, when I run OpenSim.exe, it loads OK at first, but then I get
> >>the error:
> >>
> >><start paste>
> >>Region (root) # 0x18 0x0 0x0 0x55 0x8b 0xec 0x57 0x56 0x83 0xec 0x20
> >>0x8b 0x75 0x8 0x83 0xec 0x8 0x56 0x68 0x10 0x75 0xe8 0x9c 0x8b 0x5
> >>0x10 0x75 0xe8 0x9c 0xff 0x50 0x34
> >>**
> >>ERROR:mini-trampolines.c:122:mono_magic_trampoline: assertion failed:
> >>(vt)
> >>Stacktrace:
> >>
> >>  at log4net.LogManager.WrapLogger (log4net.Core.ILogger)
> <0xffffffff>
> >>  at log4net.LogManager.WrapLogger (log4net.Core.ILogger) <0x0001d>
> >><end paste>
> >>
> >>and a long stacktrace.  The server crashes, of course.  Presumably,
> >>the references to magic mini-trampolines are within the code; I don't
> >>have any scripts or objects that I'm aware of like that.
> >>
> >>Vital stats:
> >>
> >>* OpenSim revision 9205 (binary package from OSGrid)
> >>* Ubuntu Intrepid, kernel 2.6.27-11-generic
> >>* Mono run with MONO_THREADS_PER_CPU set to 2000
> >>* 7 regions run by one OpenSim.exe
> >>*  P4/3.6GHz, 2GB RAM, more than adequate (or were under Mono 2.1)
> >>* UGAIM through OSGrid
> >>* MySQL 14.12
> >>
> >>I'm completely at a loss as to what could be causing this.  Does
> >>anyone have any ideas?
> > --
> > John Hopkin
> >
> > _______________________________________________
> > Opensim-users mailing list
> > [hidden email]
> > https://lists.berlios.de/mailman/listinfo/opensim-users
> >
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu and magic mini-trampolines (crash problem)

Dr Scofield
In reply to this post by John Hopkin
John Hopkin wrote:
> I've just upgraded Mono from 2.1 to 2.2 under Ubuntu Hardy, using Dr
> Scofield's script from here:
>
> http://xyzzyxyzzy.net/2009/02/17/updated-mono-build-script-mono-22/

i've found that 2.2 has major problems and have since gone back to 2.0.1.

i'll add a comment to the blog, sorry about not doing it earlier.

        DrS/dirk

>
> and all went smoothly with that.
>
> But now, when I run OpenSim.exe, it loads OK at first, but then I get
> the error:
>
> <start paste>
> Region (root) # 0x18 0x0 0x0 0x55 0x8b 0xec 0x57 0x56 0x83 0xec 0x20
> 0x8b 0x75 0x8 0x83 0xec 0x8 0x56 0x68 0x10 0x75 0xe8 0x9c 0x8b 0x5
> 0x10 0x75 0xe8 0x9c 0xff 0x50 0x34
> **
> ERROR:mini-trampolines.c:122:mono_magic_trampoline: assertion failed:
> (vt)
> Stacktrace:
>
>   at log4net.LogManager.WrapLogger (log4net.Core.ILogger) <0xffffffff>
>   at log4net.LogManager.WrapLogger (log4net.Core.ILogger) <0x0001d>
> <end paste>
>
> and a long stacktrace.  The server crashes, of course.  Presumably,
> the references to magic mini-trampolines are within the code; I don't
> have any scripts or objects that I'm aware of like that.
>
> Vital stats:
>
> * OpenSim revision 9205 (binary package from OSGrid)
> * Ubuntu Intrepid, kernel 2.6.27-11-generic
> * Mono run with MONO_THREADS_PER_CPU set to 2000
> * 7 regions run by one OpenSim.exe
> *  P4/3.6GHz, 2GB RAM, more than adequate (or were under Mono 2.1)
> * UGAIM through OSGrid
> * MySQL 14.12
>
> I'm completely at a loss as to what could be causing this.  Does
> anyone have any ideas?


--
dr dirk husemann ---- virtual worlds research ---- ibm zurich research lab
SL: dr scofield ---- [hidden email] ---- http://xyzzyxyzzy.net/
RL: [hidden email] - +41 44 724 8573 - http://www.zurich.ibm.com/~hud/
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu and magic mini-trampolines (crash problem)

John Hopkin
In reply to this post by Mic Bowman
Thanks, Mic.  I've put a monitor on that mantis, and I'll upgrade when
the patch comes through.  It's not a killer problem - if I can get the
sim up, which can take a couple of tries, it seems to stay up pretty
well, with only one crash so far during normal running, which was
during a border crossing.

John

Mic Bowman wrote:

>there is a mantis for this (3237)...
>
>i've been going through the various script engine calls to log4net and
>commenting them out to get rid of the problem. two in particular
>seemed to help. i'll put up another patch on that mantis in the next
>couple days.
>
>the problem comes and goes. i've tried various combinations of mono
>threads, debug/release builds, and even rebuilt log4net. the more
>scripts i have in a region that is restarting, the more likely the
>problem occurs. there is clearly a race condition somewhere in the
>mono 2.2 and 2.4 internals which opensim is tweaking.
>
>--mic
>
>
>On Wed, Apr 22, 2009 at 6:09 PM, John Hopkin
><[hidden email]> wrote:
>> Thanks.  I'll try that.  At the moment, it's working OK - it seems to
>> be intermittent, whereas before it was during each bootup.  If/when it
>> happens again, I'll drop that value.
>>
>> John
>>
>> Snoopy Pfeffer wrote:
>>
>>>I have experienced the same when MONO_THREADS_PER_CPU is set to a very high number. 500 works for me.
>>>
>>>  Snoopy Pfeffer
>>>
>>>
>>>
>>>
>>>________________________________
>>>From: John Hopkin <[hidden email]>
>>>To: [hidden email]
>>>Sent: Thursday, April 23, 2009 1:50:37 AM
>>>Subject: [Opensim-users] Ubuntu and magic mini-trampolines (crash problem)
>>>
>>>I've just upgraded Mono from 2.1 to 2.2 under Ubuntu Hardy, using Dr
>>>Scofield's script from here:
>>>
>>>http://xyzzyxyzzy.net/2009/02/17/updated-mono-build-script-mono-22/
>>>
>>>and all went smoothly with that.
>>>
>>>But now, when I run OpenSim.exe, it loads OK at first, but then I get
>>>the error:
>>>
>>><start paste>
>>>Region (root) # 0x18 0x0 0x0 0x55 0x8b 0xec 0x57 0x56 0x83 0xec 0x20
>>>0x8b 0x75 0x8 0x83 0xec 0x8 0x56 0x68 0x10 0x75 0xe8 0x9c 0x8b 0x5
>>>0x10 0x75 0xe8 0x9c 0xff 0x50 0x34
>>>**
>>>ERROR:mini-trampolines.c:122:mono_magic_trampoline: assertion failed:
>>>(vt)
>>>Stacktrace:
>>>
>>>  at log4net.LogManager.WrapLogger (log4net.Core.ILogger) <0xffffffff>
>>>  at log4net.LogManager.WrapLogger (log4net.Core.ILogger) <0x0001d>
>>><end paste>
>>>
>>>and a long stacktrace.  The server crashes, of course.  Presumably,
>>>the references to magic mini-trampolines are within the code; I don't
>>>have any scripts or objects that I'm aware of like that.
>>>
>>>Vital stats:
>>>
>>>* OpenSim revision 9205 (binary package from OSGrid)
>>>* Ubuntu Intrepid, kernel 2.6.27-11-generic
>>>* Mono run with MONO_THREADS_PER_CPU set to 2000
>>>* 7 regions run by one OpenSim.exe
>>>*  P4/3.6GHz, 2GB RAM, more than adequate (or were under Mono 2.1)
>>>* UGAIM through OSGrid
>>>* MySQL 14.12
>>>
>>>I'm completely at a loss as to what could be causing this.  Does
>>>anyone have any ideas?
>> --
>> John Hopkin
>>
>> _______________________________________________
>> Opensim-users mailing list
>> [hidden email]
>> https://lists.berlios.de/mailman/listinfo/opensim-users
>>
--
John Hopkin

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu and magic mini-trampolines (crash problem)

John Hopkin
In reply to this post by Dr Scofield
Dr Scofield wrote:

>John Hopkin wrote:
>> I've just upgraded Mono from 2.1 to 2.2 under Ubuntu Hardy, using Dr
>> Scofield's script from here:
>>
>> http://xyzzyxyzzy.net/2009/02/17/updated-mono-build-script-mono-22/
>
>i've found that 2.2 has major problems and have since gone back to 2.0.1.
>
>i'll add a comment to the blog, sorry about not doing it earlier.

I'd really, really like to stay on 2.2 if possible - under 2.1, the
setup had recently become so slow and temperamental it was practically
unusable, which is why I tried the upgrade.  With 2.2, this problem
aside, there've been none of the performance issues, and generally
it's a *lot* smoother than it has been for months.
--
John Hopkin

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu and magic mini-trampolines (crash problem)

Sacha Magne
Not sure it will help in your case, but you can try
http://www.mono-project.com/AOT

It seems to give some good results, but we just test that since yesterday.

Sacha


On Thu, Apr 23, 2009 at 2:53 PM, John Hopkin <[hidden email]> wrote:
Dr Scofield wrote:

>John Hopkin wrote:
>> I've just upgraded Mono from 2.1 to 2.2 under Ubuntu Hardy, using Dr
>> Scofield's script from here:
>>
>> http://xyzzyxyzzy.net/2009/02/17/updated-mono-build-script-mono-22/
>
>i've found that 2.2 has major problems and have since gone back to 2.0.1.
>
>i'll add a comment to the blog, sorry about not doing it earlier.

I'd really, really like to stay on 2.2 if possible - under 2.1, the
setup had recently become so slow and temperamental it was practically
unusable, which is why I tried the upgrade.  With 2.2, this problem
aside, there've been none of the performance issues, and generally
it's a *lot* smoother than it has been for months.
--
John Hopkin

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users



--

http://K-grid.com
Just be cause it's Kool

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu and magic mini-trampolines (crash problem)

BlueWall Slade
In reply to this post by John Hopkin
I tested the 2.2/2.4 waters and regressed back to 2.0.1 . But, I did find that the 2.2/2.4 works much better if you can keep some headroom in RAM. Things seem to take a turn for the worse when you start swapping. Maybe the GC has a hard time dealing with cached pages?

Thanks!
BlueWall

On Thu, Apr 23, 2009 at 8:53 AM, John Hopkin <[hidden email]> wrote:
Dr Scofield wrote:

>John Hopkin wrote:
>> I've just upgraded Mono from 2.1 to 2.2 under Ubuntu Hardy, using Dr
>> Scofield's script from here:
>>
>> http://xyzzyxyzzy.net/2009/02/17/updated-mono-build-script-mono-22/
>
>i've found that 2.2 has major problems and have since gone back to 2.0.1.
>
>i'll add a comment to the blog, sorry about not doing it earlier.

I'd really, really like to stay on 2.2 if possible - under 2.1, the
setup had recently become so slow and temperamental it was practically
unusable, which is why I tried the upgrade.  With 2.2, this problem
aside, there've been none of the performance issues, and generally
it's a *lot* smoother than it has been for months.
--
John Hopkin

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users


_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu and magic mini-trampolines (crash problem)

Mic Bowman
GC may also be a problem. Though my experience with 2.4 at least is
that memory utilization is very good and in this case we have... "more
than enough" memory. :-)

Also... the consistency with which the problem occurs in log4net
methods called from within an executing script suggests that its not
just garbage collection. I'm not sufficiently mono-smart to root cause
this... so I'm looking for bandaids (which in this case amounts to
commenting out log4net calls in the call stack that caused the
problem) that work consistently. If someone with mono-smarts would
like to suggest some other ways to get data on this, I'm more than
willing to try something different.

--mic

On Thu, Apr 23, 2009 at 6:09 AM, BlueWall Slade
<[hidden email]> wrote:

> I tested the 2.2/2.4 waters and regressed back to 2.0.1 . But, I did find
> that the 2.2/2.4 works much better if you can keep some headroom in RAM.
> Things seem to take a turn for the worse when you start swapping. Maybe the
> GC has a hard time dealing with cached pages?
>
> Thanks!
> BlueWall
>
> On Thu, Apr 23, 2009 at 8:53 AM, John Hopkin <[hidden email]>
> wrote:
>>
>> Dr Scofield wrote:
>>
>> >John Hopkin wrote:
>> >> I've just upgraded Mono from 2.1 to 2.2 under Ubuntu Hardy, using Dr
>> >> Scofield's script from here:
>> >>
>> >> http://xyzzyxyzzy.net/2009/02/17/updated-mono-build-script-mono-22/
>> >
>> >i've found that 2.2 has major problems and have since gone back to 2.0.1.
>> >
>> >i'll add a comment to the blog, sorry about not doing it earlier.
>>
>> I'd really, really like to stay on 2.2 if possible - under 2.1, the
>> setup had recently become so slow and temperamental it was practically
>> unusable, which is why I tried the upgrade.  With 2.2, this problem
>> aside, there've been none of the performance issues, and generally
>> it's a *lot* smoother than it has been for months.
>> --
>> John Hopkin
>>
>> _______________________________________________
>> Opensim-users mailing list
>> [hidden email]
>> https://lists.berlios.de/mailman/listinfo/opensim-users
>
>
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users
>
>
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu and magic mini-trampolines (crash problem)

John Hopkin
To be more accurate, with 7 regions up, it's using just around 1GB out
of the 2GB of memory, same as 2.1 was.  That's with only a couple of
users, though, which is all I need.  I'll keep an eye on that, and see
what memory it's been using in the case of another crash.

One thing worth noting is that CPU usage is a *lot* lower under 2.2,
including when the server's just ticking over with no users.

Just observations - thanks to all for your input on this.

 John

Mic Bowman wrote:

>GC may also be a problem. Though my experience with 2.4 at least is
>that memory utilization is very good and in this case we have... "more
>than enough" memory. :-)
>
>Also... the consistency with which the problem occurs in log4net
>methods called from within an executing script suggests that its not
>just garbage collection. I'm not sufficiently mono-smart to root cause
>this... so I'm looking for bandaids (which in this case amounts to
>commenting out log4net calls in the call stack that caused the
>problem) that work consistently. If someone with mono-smarts would
>like to suggest some other ways to get data on this, I'm more than
>willing to try something different.
>
>--mic
>
>On Thu, Apr 23, 2009 at 6:09 AM, BlueWall Slade
><[hidden email]> wrote:
>> I tested the 2.2/2.4 waters and regressed back to 2.0.1 . But, I did find
>> that the 2.2/2.4 works much better if you can keep some headroom in RAM.
>> Things seem to take a turn for the worse when you start swapping. Maybe the
>> GC has a hard time dealing with cached pages?
>>
>> Thanks!
>> BlueWall
>>
>> On Thu, Apr 23, 2009 at 8:53 AM, John Hopkin <[hidden email]>
>> wrote:
>>>
>>> Dr Scofield wrote:
>>>
>>> >John Hopkin wrote:
>>> >> I've just upgraded Mono from 2.1 to 2.2 under Ubuntu Hardy, using Dr
>>> >> Scofield's script from here:
>>> >>
>>> >> http://xyzzyxyzzy.net/2009/02/17/updated-mono-build-script-mono-22/
>>> >
>>> >i've found that 2.2 has major problems and have since gone back to 2.0.1.
>>> >
>>> >i'll add a comment to the blog, sorry about not doing it earlier.
>>>
>>> I'd really, really like to stay on 2.2 if possible - under 2.1, the
>>> setup had recently become so slow and temperamental it was practically
>>> unusable, which is why I tried the upgrade.  With 2.2, this problem
>>> aside, there've been none of the performance issues, and generally
>>> it's a *lot* smoother than it has been for months.
>>> --
>>> John Hopkin
>>>
>>> _______________________________________________
>>> Opensim-users mailing list
>>> [hidden email]
>>> https://lists.berlios.de/mailman/listinfo/opensim-users
>>
>>
>> _______________________________________________
>> Opensim-users mailing list
>> [hidden email]
>> https://lists.berlios.de/mailman/listinfo/opensim-users
>>
>>
--
John Hopkin

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu and magic mini-trampolines (crash problem)

John Hopkin
In reply to this post by Sacha Magne
That looks very interesting indeed - I'll set aside some time to try
that out.  Thanks for the pointer.

Sacha Magne wrote:

>Not sure it will help in your case, but you can try
>http://www.mono-project.com/AOT
>
>It seems to give some good results, but we just test that since yesterday.
>
>Sacha
>
>
>On Thu, Apr 23, 2009 at 2:53 PM, John Hopkin
><[hidden email]>wrote:
>
>> Dr Scofield wrote:
>>
>> >John Hopkin wrote:
>> >> I've just upgraded Mono from 2.1 to 2.2 under Ubuntu Hardy, using Dr
>> >> Scofield's script from here:
>> >>
>> >> http://xyzzyxyzzy.net/2009/02/17/updated-mono-build-script-mono-22/
>> >
>> >i've found that 2.2 has major problems and have since gone back to 2.0.1.
>> >
>> >i'll add a comment to the blog, sorry about not doing it earlier.
>>
>> I'd really, really like to stay on 2.2 if possible - under 2.1, the
>> setup had recently become so slow and temperamental it was practically
>> unusable, which is why I tried the upgrade.  With 2.2, this problem
>> aside, there've been none of the performance issues, and generally
>> it's a *lot* smoother than it has been for months.
>> --
>> John Hopkin
>>
>> _______________________________________________
>> Opensim-users mailing list
>> [hidden email]
>> https://lists.berlios.de/mailman/listinfo/opensim-users
>>
--
John Hopkin

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu and magic mini-trampolines (crash problem)

Robert Klein
While we are on the subject, does anyone have a foolproof way to completely and cleanly remove mono on Ubuntu 8.10? I had 1.9.1 installed and eveything worked fine then installed 2.1 and opensim would not compile correctly so I tried to remove mono 2.1 and go back to 1.9.1 now nothing compiles and I would like to just start from scratch with mono. Thanks!

-Robert

John Hopkin wrote
That looks very interesting indeed - I'll set aside some time to try
that out.  Thanks for the pointer.

Sacha Magne wrote:

>Not sure it will help in your case, but you can try
>http://www.mono-project.com/AOT
>
>It seems to give some good results, but we just test that since yesterday.
>
>Sacha
>
>
>On Thu, Apr 23, 2009 at 2:53 PM, John Hopkin
><opensim@jfhopkin.karoo.co.uk>wrote:
>
>> Dr Scofield wrote:
>>
>> >John Hopkin wrote:
>> >> I've just upgraded Mono from 2.1 to 2.2 under Ubuntu Hardy, using Dr
>> >> Scofield's script from here:
>> >>
>> >> http://xyzzyxyzzy.net/2009/02/17/updated-mono-build-script-mono-22/
>> >
>> >i've found that 2.2 has major problems and have since gone back to 2.0.1.
>> >
>> >i'll add a comment to the blog, sorry about not doing it earlier.
>>
>> I'd really, really like to stay on 2.2 if possible - under 2.1, the
>> setup had recently become so slow and temperamental it was practically
>> unusable, which is why I tried the upgrade.  With 2.2, this problem
>> aside, there've been none of the performance issues, and generally
>> it's a *lot* smoother than it has been for months.
>> --
>> John Hopkin
>>
>> _______________________________________________
>> Opensim-users mailing list
>> Opensim-users@lists.berlios.de
>> https://lists.berlios.de/mailman/listinfo/opensim-users
>>
--
John Hopkin

_______________________________________________
Opensim-users mailing list
Opensim-users@lists.berlios.de
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu and magic mini-trampolines (crash problem)

Jürgen Hamel
On Thu, 23 Apr 2009 19:11:20 -0700 (PDT)
Robert Klein <[hidden email]> wrote:

>
> While we are on the subject, does anyone have a foolproof way to completely
> and cleanly remove mono on Ubuntu 8.10? I had 1.9.1 installed and eveything
> worked fine then installed 2.1 and opensim would not compile correctly so I
> tried to remove mono 2.1 and go back to 1.9.1 now nothing compiles and I
> would like to just start from scratch with mono. Thanks!
>
> -Robert
>
I know, mono is a pain in the ass, and I have only for this opensim
mono installed. So try this:
open a terminal, write this  dpkg -l "*mono*" | grep ^ii
then try to remove then apt-get remove.

Hope it works for you.

bye
Jürgen

--
Cyrus-Computer GmbH Linux Server Support Jürgen Hamel
Cuon - Warenwirtschaft mit Linux  http://www.cuon.org

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users

signature.asc (204 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu and magic mini-trampolines (crash problem)

Sean Dague-2
In reply to this post by Mic Bowman
Mic Bowman wrote:
> there is a mantis for this (3237)...
>
> i've been going through the various script engine calls to log4net and
> commenting them out to get rid of the problem. two in particular
> seemed to help. i'll put up another patch on that mantis in the next
> couple days.

Is this because the script engine is sharing a log4net instance?  I
noticed it's the only module that does that, and in the past that was a
known gotcha.

        -Sean

--
Sean Dague / Neas Bade
[hidden email]
http://dague.net



_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users

signature.asc (260 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu and magic mini-trampolines (crash problem)

Sean Dague-2
In reply to this post by John Hopkin
John Hopkin wrote:

> Dr Scofield wrote:
>
>> John Hopkin wrote:
>>> I've just upgraded Mono from 2.1 to 2.2 under Ubuntu Hardy, using Dr
>>> Scofield's script from here:
>>>
>>> http://xyzzyxyzzy.net/2009/02/17/updated-mono-build-script-mono-22/
>> i've found that 2.2 has major problems and have since gone back to 2.0.1.
>>
>> i'll add a comment to the blog, sorry about not doing it earlier.
>
> I'd really, really like to stay on 2.2 if possible - under 2.1, the
> setup had recently become so slow and temperamental it was practically
> unusable, which is why I tried the upgrade.  With 2.2, this problem
> aside, there've been none of the performance issues, and generally
> it's a *lot* smoother than it has been for months.
For what it is worth, 2.2 is provably wrong on Linux 64bit.  It fails 5
unit tests in a way that it shows it isn't respecting byte fields.  2.4
does not seem to have this issue.

One of my goals with panda is to actually get us building OpenSim across
multiple release versions of mono (plus maybe upstream trunk), so we can
see where issues might be different between different revisions.

        -Sean

--
Sean Dague / Neas Bade
[hidden email]
http://dague.net



_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users

signature.asc (260 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu and magic mini-trampolines (crash problem)

Sean Dague-2
In reply to this post by Mic Bowman
Mic Bowman wrote:

> GC may also be a problem. Though my experience with 2.4 at least is
> that memory utilization is very good and in this case we have... "more
> than enough" memory. :-)
>
> Also... the consistency with which the problem occurs in log4net
> methods called from within an executing script suggests that its not
> just garbage collection. I'm not sufficiently mono-smart to root cause
> this... so I'm looking for bandaids (which in this case amounts to
> commenting out log4net calls in the call stack that caused the
> problem) that work consistently. If someone with mono-smarts would
> like to suggest some other ways to get data on this, I'm more than
> willing to try something different.
Can you comment out any passing around of a log4net instance, and get a
static readonly in the class that it's used?  I really think that
passing that instance is the cause of a bunch of this pain.

        -Sean

--
Sean Dague / Neas Bade
[hidden email]
http://dague.net



_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users

signature.asc (260 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu and magic mini-trampolines (crash problem)

John Hopkin
In reply to this post by John Hopkin
I'm still getting the problem in r9323, although OpenSim.exe now
freezes rather than returning to the command line, and needs "kill -9"
to terminate it.

John Hopkin wrote:

>Thanks, Mic.  I've put a monitor on that mantis, and I'll upgrade when
>the patch comes through.  It's not a killer problem - if I can get the
>sim up, which can take a couple of tries, it seems to stay up pretty
>well, with only one crash so far during normal running, which was
>during a border crossing.
>
>John
>
>Mic Bowman wrote:
>
>>there is a mantis for this (3237)...
>>
>>i've been going through the various script engine calls to log4net and
>>commenting them out to get rid of the problem. two in particular
>>seemed to help. i'll put up another patch on that mantis in the next
>>couple days.
>>
>>the problem comes and goes. i've tried various combinations of mono
>>threads, debug/release builds, and even rebuilt log4net. the more
>>scripts i have in a region that is restarting, the more likely the
>>problem occurs. there is clearly a race condition somewhere in the
>>mono 2.2 and 2.4 internals which opensim is tweaking.
>>
>>--mic
>>
>>
>>On Wed, Apr 22, 2009 at 6:09 PM, John Hopkin
>><[hidden email]> wrote:
>>> Thanks.  I'll try that.  At the moment, it's working OK - it seems to
>>> be intermittent, whereas before it was during each bootup.  If/when it
>>> happens again, I'll drop that value.
>>>
>>> John
>>>
>>> Snoopy Pfeffer wrote:
>>>
>>>>I have experienced the same when MONO_THREADS_PER_CPU is set to a very high number. 500 works for me.
>>>>
>>>>  Snoopy Pfeffer
>>>>
>>>>
>>>>
>>>>
>>>>________________________________
>>>>From: John Hopkin <[hidden email]>
>>>>To: [hidden email]
>>>>Sent: Thursday, April 23, 2009 1:50:37 AM
>>>>Subject: [Opensim-users] Ubuntu and magic mini-trampolines (crash problem)
>>>>
>>>>I've just upgraded Mono from 2.1 to 2.2 under Ubuntu Hardy, using Dr
>>>>Scofield's script from here:
>>>>
>>>>http://xyzzyxyzzy.net/2009/02/17/updated-mono-build-script-mono-22/
>>>>
>>>>and all went smoothly with that.
>>>>
>>>>But now, when I run OpenSim.exe, it loads OK at first, but then I get
>>>>the error:
>>>>
>>>><start paste>
>>>>Region (root) # 0x18 0x0 0x0 0x55 0x8b 0xec 0x57 0x56 0x83 0xec 0x20
>>>>0x8b 0x75 0x8 0x83 0xec 0x8 0x56 0x68 0x10 0x75 0xe8 0x9c 0x8b 0x5
>>>>0x10 0x75 0xe8 0x9c 0xff 0x50 0x34
>>>>**
>>>>ERROR:mini-trampolines.c:122:mono_magic_trampoline: assertion failed:
>>>>(vt)
>>>>Stacktrace:
>>>>
>>>>  at log4net.LogManager.WrapLogger (log4net.Core.ILogger) <0xffffffff>
>>>>  at log4net.LogManager.WrapLogger (log4net.Core.ILogger) <0x0001d>
>>>><end paste>
>>>>
>>>>and a long stacktrace.  The server crashes, of course.  Presumably,
>>>>the references to magic mini-trampolines are within the code; I don't
>>>>have any scripts or objects that I'm aware of like that.
>>>>
>>>>Vital stats:
>>>>
>>>>* OpenSim revision 9205 (binary package from OSGrid)
>>>>* Ubuntu Intrepid, kernel 2.6.27-11-generic
>>>>* Mono run with MONO_THREADS_PER_CPU set to 2000
>>>>* 7 regions run by one OpenSim.exe
>>>>*  P4/3.6GHz, 2GB RAM, more than adequate (or were under Mono 2.1)
>>>>* UGAIM through OSGrid
>>>>* MySQL 14.12
>>>>
>>>>I'm completely at a loss as to what could be causing this.  Does
>>>>anyone have any ideas?
>>> --
>>> John Hopkin
>>>
>>> _______________________________________________
>>> Opensim-users mailing list
>>> [hidden email]
>>> https://lists.berlios.de/mailman/listinfo/opensim-users
>>>
--
John Hopkin

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Ubuntu and magic mini-trampolines (crash problem)

John Hopkin
In reply to this post by Sean Dague-2
Sean Dague wrote:

>John Hopkin wrote:
>> Dr Scofield wrote:
>>
>>> John Hopkin wrote:
>>>> I've just upgraded Mono from 2.1 to 2.2 under Ubuntu Hardy, using Dr
>>>> Scofield's script from here:
>>>>
>>>> http://xyzzyxyzzy.net/2009/02/17/updated-mono-build-script-mono-22/
>>> i've found that 2.2 has major problems and have since gone back to 2.0.1.
>>>
>>> i'll add a comment to the blog, sorry about not doing it earlier.
>>
>> I'd really, really like to stay on 2.2 if possible - under 2.1, the
>> setup had recently become so slow and temperamental it was practically
>> unusable, which is why I tried the upgrade.  With 2.2, this problem
>> aside, there've been none of the performance issues, and generally
>> it's a *lot* smoother than it has been for months.
>
>For what it is worth, 2.2 is provably wrong on Linux 64bit.  It fails 5
>unit tests in a way that it shows it isn't respecting byte fields.  2.4
>does not seem to have this issue.

Thanks.  I'm actually on 32-bit (Intrepid, not Hardy - don't know why
I said that), but it's worth noting.
--
John Hopkin

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
12