"Save OAR" stopping since r9700

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

"Save OAR" stopping since r9700

John Hopkin
Since I upgraded from r9561 to r9700, two out of seven regions don't
back up using "save oar".  Even when they're the only region running,
those two have the same symptom: after a while, the archiver appears
to stop writing to the file, as if it's given up.

Looking at the files with tar, they're fine for many MB, then the
archive ends prematurely.  There's nothing on the screen or in the log
to indicate a problem.  This problem persists, and only on the same
two regions - other regions back up fine.

Platform is Ubuntu Linux (Jaunty), Mono 2.2, MySQL 5.0; all seven
regions run under one instance of OpenSim.exe.

Any ideas?  Thanks in advance.
--
John Hopkin

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: "Save OAR" stopping since r9700

Dr Scofield
John Hopkin wrote:

> Since I upgraded from r9561 to r9700, two out of seven regions don't
> back up using "save oar".  Even when they're the only region running,
> those two have the same symptom: after a while, the archiver appears
> to stop writing to the file, as if it's given up.
>
> Looking at the files with tar, they're fine for many MB, then the
> archive ends prematurely.  There's nothing on the screen or in the log
> to indicate a problem.  This problem persists, and only on the same
> two regions - other regions back up fine.
>
> Platform is Ubuntu Linux (Jaunty), Mono 2.2, MySQL 5.0; all seven
> regions run under one instance of OpenSim.exe.
>  
my not actually be the issue, but mono 2.2 has a bad reputation. 2.0.1
seems to be quite stable, 2.4 is also not too bad.

    dirk
> Any ideas?  Thanks in advance.
>  


--
dr dirk husemann ---- math & computer science ---- ibm zurich research lab
RL: [hidden email] - +41 44 724 8573 - http://www.zurich.ibm.com/~hud/ 
SL: [hidden email] --------------------- http://xyzzyxyzzy.net/

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: "Save OAR" stopping since r9700

BlueWall Slade
I have seen oar failures on Linux (Mono 2.4/64 Bit and Mono 2.5/32 Bit environments). And across several versions of OpenSim on several different regions. My experience is that any region, in these environments, with more than trivial content fails to complete the backup process. I am continuing to investigate. Any other information would be most helpful.

Thanks,
BlueWall

On Thu, May 28, 2009 at 3:25 PM, dr scofield <[hidden email]> wrote:
John Hopkin wrote:
> Since I upgraded from r9561 to r9700, two out of seven regions don't
> back up using "save oar".  Even when they're the only region running,
> those two have the same symptom: after a while, the archiver appears
> to stop writing to the file, as if it's given up.
>
> Looking at the files with tar, they're fine for many MB, then the
> archive ends prematurely.  There's nothing on the screen or in the log
> to indicate a problem.  This problem persists, and only on the same
> two regions - other regions back up fine.
>
> Platform is Ubuntu Linux (Jaunty), Mono 2.2, MySQL 5.0; all seven
> regions run under one instance of OpenSim.exe.
>
my not actually be the issue, but mono 2.2 has a bad reputation. 2.0.1
seems to be quite stable, 2.4 is also not too bad.

   dirk
> Any ideas?  Thanks in advance.
>


--
dr dirk husemann ---- math & computer science ---- ibm zurich research lab
RL: [hidden email] - +41 44 724 8573 - http://www.zurich.ibm.com/~hud/
SL: [hidden email] --------------------- http://xyzzyxyzzy.net/

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users


_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: "Save OAR" stopping since r9700

Dr Scofield
BlueWall Slade wrote:
> I have seen oar failures on Linux (Mono 2.4/64 Bit and Mono 2.5/32 Bit
> environments). And across several versions of OpenSim on several
> different regions. My experience is that any region, in these
> environments, with more than trivial content fails to complete the
> backup process. I am continuing to investigate. Any other information
> would be most helpful.

hmm. we use it regularly with fairly complex regions and haven't had an issue
like that so far.

        dirk

>
> Thanks,
> BlueWall
>
> On Thu, May 28, 2009 at 3:25 PM, dr scofield <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     John Hopkin wrote:
>     > Since I upgraded from r9561 to r9700, two out of seven regions don't
>     > back up using "save oar".  Even when they're the only region running,
>     > those two have the same symptom: after a while, the archiver appears
>     > to stop writing to the file, as if it's given up.
>     >
>     > Looking at the files with tar, they're fine for many MB, then the
>     > archive ends prematurely.  There's nothing on the screen or in the log
>     > to indicate a problem.  This problem persists, and only on the same
>     > two regions - other regions back up fine.
>     >
>     > Platform is Ubuntu Linux (Jaunty), Mono 2.2, MySQL 5.0; all seven
>     > regions run under one instance of OpenSim.exe.
>     >
>     my not actually be the issue, but mono 2.2 has a bad reputation. 2.0.1
>     seems to be quite stable, 2.4 is also not too bad.
>
>        dirk
>     > Any ideas?  Thanks in advance.
>     >
>
>
>     --
>     dr dirk husemann ---- math & computer science ---- ibm zurich
>     research lab
>     RL: [hidden email] <mailto:[hidden email]> - +41 44 724 8573
>     - http://www.zurich.ibm.com/~hud/ <http://www.zurich.ibm.com/%7Ehud/>
>     SL: [hidden email] <mailto:[hidden email]>
>     --------------------- http://xyzzyxyzzy.net/
>
>     _______________________________________________
>     Opensim-users mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.berlios.de/mailman/listinfo/opensim-users
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users


--
dr dirk husemann ---- virtual worlds research ---- ibm zurich research lab
SL: dr scofield ---- [hidden email] ---- http://xyzzyxyzzy.net/
RL: [hidden email] - +41 44 724 8573 - http://www.zurich.ibm.com/~hud/
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: "Save OAR" stopping since r9700

Ralf Haifisch
In reply to this post by John Hopkin
I noticed this with a version slightly before 9700...  something around 9661
?

Restore of did fail sometimes...

Cheers,
Ralf
----------------------------------------------------------------------

Message: 1
Date: Fri, 29 May 2009 10:28:38 +0200
From: Dr Scofield <[hidden email]>
Subject: Re: [Opensim-users] "Save OAR" stopping since r9700
To: [hidden email]
Message-ID: <[hidden email]>
Content-Type: text/plain; charset=ISO-8859-15

BlueWall Slade wrote:
> I have seen oar failures on Linux (Mono 2.4/64 Bit and Mono 2.5/32 Bit
> environments). And across several versions of OpenSim on several
> different regions. My experience is that any region, in these
> environments, with more than trivial content fails to complete the
> backup process. I am continuing to investigate. Any other information
> would be most helpful.

hmm. we use it regularly with fairly complex regions and haven't had an
issue
like that so far.

        dirk

>
> Thanks,
> BlueWall
>
> On Thu, May 28, 2009 at 3:25 PM, dr scofield <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     John Hopkin wrote:
>     > Since I upgraded from r9561 to r9700, two out of seven regions don't
>     > back up using "save oar".  Even when they're the only region
running,
>     > those two have the same symptom: after a while, the archiver appears
>     > to stop writing to the file, as if it's given up.
>     >
>     > Looking at the files with tar, they're fine for many MB, then the
>     > archive ends prematurely.  There's nothing on the screen or in the
log

>     > to indicate a problem.  This problem persists, and only on the same
>     > two regions - other regions back up fine.
>     >
>     > Platform is Ubuntu Linux (Jaunty), Mono 2.2, MySQL 5.0; all seven
>     > regions run under one instance of OpenSim.exe.
>     >
>     my not actually be the issue, but mono 2.2 has a bad reputation. 2.0.1
>     seems to be quite stable, 2.4 is also not too bad.
>
>        dirk
>     > Any ideas?  Thanks in advance.
>     >
>
>
>     --
>     dr dirk husemann ---- math & computer science ---- ibm zurich
>     research lab
>     RL: [hidden email] <mailto:[hidden email]> - +41 44 724 8573
>     - http://www.zurich.ibm.com/~hud/ <http://www.zurich.ibm.com/%7Ehud/>
>     SL: [hidden email] <mailto:[hidden email]>
>     --------------------- http://xyzzyxyzzy.net/
>
>     _______________________________________________
>     Opensim-users mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.berlios.de/mailman/listinfo/opensim-users
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users


_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: "Save OAR" stopping since r9700

John Hopkin
In reply to this post by BlueWall Slade
Interesting - thanks (to both of you).

I'm not what other information is relevant, but here's some:

1. For prims counts, they are two of the largest regions among the
seven, marked with asterisks here:

2889
2717*
 858*
 526
 257
 195
   3

2. In total data size, they're the two of largest, as measured by the
size of the last good backup:

140M*
 31M
 23M*
  7M
  5M
  4M
  0.5M

3. Regarding terrain, as well as location on the grid, they're not
unusual; although they're adjacent to each other, all seven are
located contiguously and contain a mix of elevations.

4. They could well be the regions with the most scripts; I don't know
how I can find that out without a manual audit in-world.

There are four other regions, not normally loaded, used occasionally
as sandboxes and storage areas.  I don't yet know if they're affected
by the bug.

Hope this gives some idea, and maybe helps.  If you want to know
anything else, please ask.

John

BlueWall Slade wrote:

>I have seen oar failures on Linux (Mono 2.4/64 Bit and Mono 2.5/32 Bit
>environments). And across several versions of OpenSim on several different
>regions. My experience is that any region, in these environments, with more
>than trivial content fails to complete the backup process. I am continuing
>to investigate. Any other information would be most helpful.
>
>Thanks,
>BlueWall
>
>On Thu, May 28, 2009 at 3:25 PM, dr scofield <[hidden email]>wrote:
>
>> John Hopkin wrote:
>> > Since I upgraded from r9561 to r9700, two out of seven regions don't
>> > back up using "save oar".  Even when they're the only region running,
>> > those two have the same symptom: after a while, the archiver appears
>> > to stop writing to the file, as if it's given up.
>> >
>> > Looking at the files with tar, they're fine for many MB, then the
>> > archive ends prematurely.  There's nothing on the screen or in the log
>> > to indicate a problem.  This problem persists, and only on the same
>> > two regions - other regions back up fine.
>> >
>> > Platform is Ubuntu Linux (Jaunty), Mono 2.2, MySQL 5.0; all seven
>> > regions run under one instance of OpenSim.exe.
>> >
>> my not actually be the issue, but mono 2.2 has a bad reputation. 2.0.1
>> seems to be quite stable, 2.4 is also not too bad.
>>
>>    dirk
>> > Any ideas?  Thanks in advance.
>> >
>>
>>
>> --
>> dr dirk husemann ---- math & computer science ---- ibm zurich research lab
>> RL: [hidden email] - +41 44 724 8573 - http://www.zurich.ibm.com/~hud/<http://www.zurich.ibm.com/%7Ehud/>
>> SL: [hidden email] --------------------- http://xyzzyxyzzy.net/
>>
>> _______________________________________________
>> Opensim-users mailing list
>> [hidden email]
>> https://lists.berlios.de/mailman/listinfo/opensim-users
>>
--
John Hopkin

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: "Save OAR" stopping since r9700

justincc
The data that we (the developers) really need is the exact revision between r9561 to r9700 on which this started
failing.  I suggest a divide an conquer approach (see if the problem occurs on r9641, if not, then try r9670, etc.).
Then that information should be put in a mantis (or an existing one, in which case I'll need to know the number).

John Hopkin wrote:

> Interesting - thanks (to both of you).
>
> I'm not what other information is relevant, but here's some:
>
> 1. For prims counts, they are two of the largest regions among the
> seven, marked with asterisks here:
>
> 2889
> 2717*
>  858*
>  526
>  257
>  195
>    3
>
> 2. In total data size, they're the two of largest, as measured by the
> size of the last good backup:
>
> 140M*
>  31M
>  23M*
>   7M
>   5M
>   4M
>   0.5M
>
> 3. Regarding terrain, as well as location on the grid, they're not
> unusual; although they're adjacent to each other, all seven are
> located contiguously and contain a mix of elevations.
>
> 4. They could well be the regions with the most scripts; I don't know
> how I can find that out without a manual audit in-world.
>
> There are four other regions, not normally loaded, used occasionally
> as sandboxes and storage areas.  I don't yet know if they're affected
> by the bug.
>
> Hope this gives some idea, and maybe helps.  If you want to know
> anything else, please ask.
>
> John
>
> BlueWall Slade wrote:
>
>> I have seen oar failures on Linux (Mono 2.4/64 Bit and Mono 2.5/32 Bit
>> environments). And across several versions of OpenSim on several different
>> regions. My experience is that any region, in these environments, with more
>> than trivial content fails to complete the backup process. I am continuing
>> to investigate. Any other information would be most helpful.
>>
>> Thanks,
>> BlueWall
>>
>> On Thu, May 28, 2009 at 3:25 PM, dr scofield <[hidden email]>wrote:
>>
>>> John Hopkin wrote:
>>>> Since I upgraded from r9561 to r9700, two out of seven regions don't
>>>> back up using "save oar".  Even when they're the only region running,
>>>> those two have the same symptom: after a while, the archiver appears
>>>> to stop writing to the file, as if it's given up.
>>>>
>>>> Looking at the files with tar, they're fine for many MB, then the
>>>> archive ends prematurely.  There's nothing on the screen or in the log
>>>> to indicate a problem.  This problem persists, and only on the same
>>>> two regions - other regions back up fine.
>>>>
>>>> Platform is Ubuntu Linux (Jaunty), Mono 2.2, MySQL 5.0; all seven
>>>> regions run under one instance of OpenSim.exe.
>>>>
>>> my not actually be the issue, but mono 2.2 has a bad reputation. 2.0.1
>>> seems to be quite stable, 2.4 is also not too bad.
>>>
>>>    dirk
>>>> Any ideas?  Thanks in advance.
>>>>
>>>
>>> --
>>> dr dirk husemann ---- math & computer science ---- ibm zurich research lab
>>> RL: [hidden email] - +41 44 724 8573 - http://www.zurich.ibm.com/~hud/<http://www.zurich.ibm.com/%7Ehud/>
>>> SL: [hidden email] --------------------- http://xyzzyxyzzy.net/
>>>
>>> _______________________________________________
>>> Opensim-users mailing list
>>> [hidden email]
>>> https://lists.berlios.de/mailman/listinfo/opensim-users
>>>


--
justincc
Justin Clark-Casey
http://justincc.wordpress.com
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: "Save OAR" stopping since r9700

Charles Krinke-3
I tried with r9573 this morning to do a "save oar <oarFileName> and here is my observation:

First on ckTai which is essentially a blank region.
Region (ckTai) # save oar ckTai0529
18:16:09 - [ARCHIVER]: Writing archive for region ckTai to ckTai0529
18:16:09 - [ARCHIVER]: 0 scene objects to serialize requiring save of 0 assets
18:16:09 - [ARCHIVER]: AssetsRequest executed looking for 0 assets
18:16:09 - [ARCHIVER]: Creating archive file.  This may take some time.
18:16:09 - [ARCHIVER]: Added control file to archive.
18:16:09 - [ARCHIVER]: Added region settings to archive.
18:16:10 - [ARCHIVER]: Added terrain information to archive.
18:16:10 - [ARCHIVER]: Added scene objects to archive.
18:16:10 - [ARCHIVER]: Wrote out OpenSimulator archive for ckTai
Region (ckTai) # show version
Version: OpenSimulator Server  0.6.4.9573  (interface version 3)

Then on Europa which has pushing 1000 prims. What is interesting about Europa is that the first line (in bold) printed out immediately and then there is no response to console. The line below is 40 minutes later and there is still no console response.


Region (Europa) # save oar Europa0529
18:18:03 - [ARCHIVER]: Writing archive for region Europa to Europa0529
18:31:43 - DIAGNOSTICS

Time now is 5/29/2009 6:31:43 PM
Server has been running since Tuesday, 5/19/2009 2:31:42 AM
That is an elapsed time of 10.16:00:00.4956310

ASSET STATISTICS
Asset cache contains        1 assets
Latest asset request time after cache miss: 0s
Blocked client requests for missing textures: 0
Asset service request failures: 0

CONNECTION STATISTICS
Abnormal client thread terminations: 0

INVENTORY STATISTICS
Initial inventory caching failures: 0

FRAME STATISTICS
Dilatn  SimFPS  PhyFPS  AgntUp  RootAg  ChldAg  Prims   AtvPrm  AtvScr  ScrLPS
  0.00       0     0.0     0.0       0       1      48       1       0       0






From: Justin Clark-Casey <[hidden email]>
To: [hidden email]
Sent: Friday, May 29, 2009 9:01:37 AM
Subject: Re: [Opensim-users] "Save OAR" stopping since r9700

The data that we (the developers) really need is the exact revision between r9561 to r9700 on which this started
failing.  I suggest a divide an conquer approach (see if the problem occurs on r9641, if not, then try r9670, etc.).
Then that information should be put in a mantis (or an existing one, in which case I'll need to know the number).

John Hopkin wrote:

> Interesting - thanks (to both of you).
>
> I'm not what other information is relevant, but here's some:
>
> 1. For prims counts, they are two of the largest regions among the
> seven, marked with asterisks here:
>
> 2889
> 2717*
>  858*
>  526
>  257
>  195
>    3
>
> 2. In total data size, they're the two of largest, as measured by the
> size of the last good backup:
>
> 140M*
>  31M
>  23M*
>  7M
>  5M
>  4M
>  0.5M
>
> 3. Regarding terrain, as well as location on the grid, they're not
> unusual; although they're adjacent to each other, all seven are
> located contiguously and contain a mix of elevations.
>
> 4. They could well be the regions with the most scripts; I don't know
> how I can find that out without a manual audit in-world.
>
> There are four other regions, not normally loaded, used occasionally
> as sandboxes and storage areas.  I don't yet know if they're affected
> by the bug.
>
> Hope this gives some idea, and maybe helps.  If you want to know
> anything else, please ask.
>
> John
>
> BlueWall Slade wrote:
>
>> I have seen oar failures on Linux (Mono 2.4/64 Bit and Mono 2.5/32 Bit
>> environments). And across several versions of OpenSim on several different
>> regions. My experience is that any region, in these environments, with more
>> than trivial content fails to complete the backup process. I am continuing
>> to investigate. Any other information would be most helpful.
>>
>> Thanks,
>> BlueWall
>>
>> On Thu, May 28, 2009 at 3:25 PM, dr scofield <[hidden email]>wrote:
>>
>>> John Hopkin wrote:
>>>> Since I upgraded from r9561 to r9700, two out of seven regions don't
>>>> back up using "save oar".  Even when they're the only region running,
>>>> those two have the same symptom: after a while, the archiver appears
>>>> to stop writing to the file, as if it's given up.
>>>>
>>>> Looking at the files with tar, they're fine for many MB, then the
>>>> archive ends prematurely.  There's nothing on the screen or in the log
>>>> to indicate a problem.  This problem persists, and only on the same
>>>> two regions - other regions back up fine.
>>>>
>>>> Platform is Ubuntu Linux (Jaunty), Mono 2.2, MySQL 5.0; all seven
>>>> regions run under one instance of OpenSim.exe.
>>>>
>>> my not actually be the issue, but mono 2.2 has a bad reputation. 2.0.1
>>> seems to be quite stable, 2.4 is also not too bad.
>>>
>>>    dirk
>>>> Any ideas?  Thanks in advance.
>>>>
>>>
>>> --
>>> dr dirk husemann ---- math & computer science ---- ibm zurich research lab
>>> RL: [hidden email] - +41 44 724 8573 - http://www.zurich.ibm.com/~hud/<http://www.zurich.ibm.com/%7Ehud/>
>>> SL: [hidden email] --------------------- http://xyzzyxyzzy.net/
>>>
>>> _______________________________________________
>>> Opensim-users mailing list
>>> [hidden email]
>>> https://lists.berlios.de/mailman/listinfo/opensim-users
>>>


--
justincc
Justin Clark-Casey
http://justincc.wordpress.com
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: "Save OAR" stopping since r9700

justincc
The most helpful thing is if you could go backwards from r9573 on Europa until the problem goes away.  And then slowly
inch forwards until it reappears, so we know which revision number the issue is associated with.

Charles Krinke wrote:

> I tried with r9573 this morning to do a "save oar <oarFileName> and here
> is my observation:
>
> First on ckTai which is essentially a blank region.
> Region (ckTai) # save oar ckTai0529
> 18:16:09 - [ARCHIVER]: Writing archive for region ckTai to ckTai0529
> 18:16:09 - [ARCHIVER]: 0 scene objects to serialize requiring save of 0
> assets
> 18:16:09 - [ARCHIVER]: AssetsRequest executed looking for 0 assets
> 18:16:09 - [ARCHIVER]: Creating archive file.  This may take some time.
> 18:16:09 - [ARCHIVER]: Added control file to archive.
> 18:16:09 - [ARCHIVER]: Added region settings to archive.
> 18:16:10 - [ARCHIVER]: Added terrain information to archive.
> 18:16:10 - [ARCHIVER]: Added scene objects to archive.
> 18:16:10 - [ARCHIVER]: Wrote out OpenSimulator archive for ckTai
> Region (ckTai) # show version
> Version: OpenSimulator Server  0.6.4.9573  (interface version 3)
>
> Then on Europa which has pushing 1000 prims. What is interesting about
> Europa is that the first line (in bold) printed out immediately and then
> there is no response to console. The line below is 40 minutes later and
> there is still no console response.
>
>
> Region (Europa) # save oar Europa0529
> 18:18:03 - [ARCHIVER]: Writing archive for region Europa to Europa0529
> 18:31:43 - DIAGNOSTICS
>
> Time now is 5/29/2009 6:31:43 PM
> Server has been running since Tuesday, 5/19/2009 2:31:42 AM
> That is an elapsed time of 10.16:00:00.4956310
>
> ASSET STATISTICS
> Asset cache contains        1 assets
> Latest asset request time after cache miss: 0s
> Blocked client requests for missing textures: 0
> Asset service request failures: 0
>
> CONNECTION STATISTICS
> Abnormal client thread terminations: 0
>
> INVENTORY STATISTICS
> Initial inventory caching failures: 0
>
> FRAME STATISTICS
> Dilatn  SimFPS  PhyFPS  AgntUp  RootAg  ChldAg  Prims   AtvPrm  AtvScr  
> ScrLPS
>   0.00       0     0.0     0.0       0       1      48       1      
> 0       0
>
>
>
>
>
> ------------------------------------------------------------------------
> *From:* Justin Clark-Casey <[hidden email]>
> *To:* [hidden email]
> *Sent:* Friday, May 29, 2009 9:01:37 AM
> *Subject:* Re: [Opensim-users] "Save OAR" stopping since r9700
>
> The data that we (the developers) really need is the exact revision
> between r9561 to r9700 on which this started
> failing.  I suggest a divide an conquer approach (see if the problem
> occurs on r9641, if not, then try r9670, etc.).
> Then that information should be put in a mantis (or an existing one, in
> which case I'll need to know the number).
>
> John Hopkin wrote:
>  > Interesting - thanks (to both of you).
>  >
>  > I'm not what other information is relevant, but here's some:
>  >
>  > 1. For prims counts, they are two of the largest regions among the
>  > seven, marked with asterisks here:
>  >
>  > 2889
>  > 2717*
>  >  858*
>  >  526
>  >  257
>  >  195
>  >    3
>  >
>  > 2. In total data size, they're the two of largest, as measured by the
>  > size of the last good backup:
>  >
>  > 140M*
>  >  31M
>  >  23M*
>  >  7M
>  >  5M
>  >  4M
>  >  0.5M
>  >
>  > 3. Regarding terrain, as well as location on the grid, they're not
>  > unusual; although they're adjacent to each other, all seven are
>  > located contiguously and contain a mix of elevations.
>  >
>  > 4. They could well be the regions with the most scripts; I don't know
>  > how I can find that out without a manual audit in-world.
>  >
>  > There are four other regions, not normally loaded, used occasionally
>  > as sandboxes and storage areas.  I don't yet know if they're affected
>  > by the bug.
>  >
>  > Hope this gives some idea, and maybe helps.  If you want to know
>  > anything else, please ask.
>  >
>  > John
>  >
>  > BlueWall Slade wrote:
>  >
>  >> I have seen oar failures on Linux (Mono 2.4/64 Bit and Mono 2.5/32 Bit
>  >> environments). And across several versions of OpenSim on several
> different
>  >> regions. My experience is that any region, in these environments,
> with more
>  >> than trivial content fails to complete the backup process. I am
> continuing
>  >> to investigate. Any other information would be most helpful.
>  >>
>  >> Thanks,
>  >> BlueWall
>  >>
>  >> On Thu, May 28, 2009 at 3:25 PM, dr scofield
> <[hidden email] <mailto:[hidden email]>>wrote:
>  >>
>  >>> John Hopkin wrote:
>  >>>> Since I upgraded from r9561 to r9700, two out of seven regions don't
>  >>>> back up using "save oar".  Even when they're the only region running,
>  >>>> those two have the same symptom: after a while, the archiver appears
>  >>>> to stop writing to the file, as if it's given up.
>  >>>>
>  >>>> Looking at the files with tar, they're fine for many MB, then the
>  >>>> archive ends prematurely.  There's nothing on the screen or in the log
>  >>>> to indicate a problem.  This problem persists, and only on the same
>  >>>> two regions - other regions back up fine.
>  >>>>
>  >>>> Platform is Ubuntu Linux (Jaunty), Mono 2.2, MySQL 5.0; all seven
>  >>>> regions run under one instance of OpenSim.exe.
>  >>>>
>  >>> my not actually be the issue, but mono 2.2 has a bad reputation. 2.0.1
>  >>> seems to be quite stable, 2.4 is also not too bad.
>  >>>
>  >>>    dirk
>  >>>> Any ideas?  Thanks in advance.
>  >>>>
>  >>>
>  >>> --
>  >>> dr dirk husemann ---- math & computer science ---- ibm zurich
> research lab
>  >>> RL: [hidden email] <mailto:[hidden email]> - +41 44 724
> 8573 - http://www.zurich.ibm.com/~hud/ 
> <http://www.zurich.ibm.com/%7Ehud/><http://www.zurich.ibm.com/%7Ehud/>
>  >>> SL: [hidden email] <mailto:[hidden email]>
> --------------------- http://xyzzyxyzzy.net/
>  >>>
>  >>> _______________________________________________
>  >>> Opensim-users mailing list
>  >>> [hidden email] <mailto:[hidden email]>
>  >>> https://lists.berlios.de/mailman/listinfo/opensim-users
>  >>>
>
>
> --
> justincc
> Justin Clark-Casey
> http://justincc.wordpress.com
> _______________________________________________
> Opensim-users mailing list
> [hidden email] <mailto:[hidden email]>
> https://lists.berlios.de/mailman/listinfo/opensim-users
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users


--
justincc
Justin Clark-Casey
http://justincc.wordpress.com
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: "Save OAR" stopping since r9700

BlueWall Slade
Justin - I am setting up a local grid on 9395 and working up from there.

Thanks!
BlueWall

On Fri, May 29, 2009 at 3:25 PM, Justin Clark-Casey <[hidden email]> wrote:
The most helpful thing is if you could go backwards from r9573 on Europa until the problem goes away.  And then slowly
inch forwards until it reappears, so we know which revision number the issue is associated with.

Charles Krinke wrote:
> I tried with r9573 this morning to do a "save oar <oarFileName> and here
> is my observation:
>
> First on ckTai which is essentially a blank region.
> Region (ckTai) # save oar ckTai0529
> 18:16:09 - [ARCHIVER]: Writing archive for region ckTai to ckTai0529
> 18:16:09 - [ARCHIVER]: 0 scene objects to serialize requiring save of 0
> assets
> 18:16:09 - [ARCHIVER]: AssetsRequest executed looking for 0 assets
> 18:16:09 - [ARCHIVER]: Creating archive file.  This may take some time.
> 18:16:09 - [ARCHIVER]: Added control file to archive.
> 18:16:09 - [ARCHIVER]: Added region settings to archive.
> 18:16:10 - [ARCHIVER]: Added terrain information to archive.
> 18:16:10 - [ARCHIVER]: Added scene objects to archive.
> 18:16:10 - [ARCHIVER]: Wrote out OpenSimulator archive for ckTai
> Region (ckTai) # show version
> Version: OpenSimulator Server  0.6.4.9573  (interface version 3)
>
> Then on Europa which has pushing 1000 prims. What is interesting about
> Europa is that the first line (in bold) printed out immediately and then
> there is no response to console. The line below is 40 minutes later and
> there is still no console response.
>
>
> Region (Europa) # save oar Europa0529
> 18:18:03 - [ARCHIVER]: Writing archive for region Europa to Europa0529
> 18:31:43 - DIAGNOSTICS
>
> Time now is 5/29/2009 6:31:43 PM
> Server has been running since Tuesday, 5/19/2009 2:31:42 AM
> That is an elapsed time of 10.16:00:00.4956310
>
> ASSET STATISTICS
> Asset cache contains        1 assets
> Latest asset request time after cache miss: 0s
> Blocked client requests for missing textures: 0
> Asset service request failures: 0
>
> CONNECTION STATISTICS
> Abnormal client thread terminations: 0
>
> INVENTORY STATISTICS
> Initial inventory caching failures: 0
>
> FRAME STATISTICS
> Dilatn  SimFPS  PhyFPS  AgntUp  RootAg  ChldAg  Prims   AtvPrm  AtvScr
> ScrLPS
>   0.00       0     0.0     0.0       0       1      48       1
> 0       0
>
>
>
>
>
> ------------------------------------------------------------------------
> *From:* Justin Clark-Casey <[hidden email]>
> *To:* [hidden email]
> *Sent:* Friday, May 29, 2009 9:01:37 AM
> *Subject:* Re: [Opensim-users] "Save OAR" stopping since r9700
>
> The data that we (the developers) really need is the exact revision
> between r9561 to r9700 on which this started
> failing.  I suggest a divide an conquer approach (see if the problem
> occurs on r9641, if not, then try r9670, etc.).
> Then that information should be put in a mantis (or an existing one, in
> which case I'll need to know the number).
>
> John Hopkin wrote:
>  > Interesting - thanks (to both of you).
>  >
>  > I'm not what other information is relevant, but here's some:
>  >
>  > 1. For prims counts, they are two of the largest regions among the
>  > seven, marked with asterisks here:
>  >
>  > 2889
>  > 2717*
>  >  858*
>  >  526
>  >  257
>  >  195
>  >    3
>  >
>  > 2. In total data size, they're the two of largest, as measured by the
>  > size of the last good backup:
>  >
>  > 140M*
>  >  31M
>  >  23M*
>  >  7M
>  >  5M
>  >  4M
>  >  0.5M
>  >
>  > 3. Regarding terrain, as well as location on the grid, they're not
>  > unusual; although they're adjacent to each other, all seven are
>  > located contiguously and contain a mix of elevations.
>  >
>  > 4. They could well be the regions with the most scripts; I don't know
>  > how I can find that out without a manual audit in-world.
>  >
>  > There are four other regions, not normally loaded, used occasionally
>  > as sandboxes and storage areas.  I don't yet know if they're affected
>  > by the bug.
>  >
>  > Hope this gives some idea, and maybe helps.  If you want to know
>  > anything else, please ask.
>  >
>  > John
>  >
>  > BlueWall Slade wrote:
>  >
>  >> I have seen oar failures on Linux (Mono 2.4/64 Bit and Mono 2.5/32 Bit
>  >> environments). And across several versions of OpenSim on several
> different
>  >> regions. My experience is that any region, in these environments,
> with more
>  >> than trivial content fails to complete the backup process. I am
> continuing
>  >> to investigate. Any other information would be most helpful.
>  >>
>  >> Thanks,
>  >> BlueWall
>  >>
>  >> On Thu, May 28, 2009 at 3:25 PM, dr scofield
> <[hidden email] <mailto:[hidden email]>>wrote:
>  >>
>  >>> John Hopkin wrote:
>  >>>> Since I upgraded from r9561 to r9700, two out of seven regions don't
>  >>>> back up using "save oar".  Even when they're the only region running,
>  >>>> those two have the same symptom: after a while, the archiver appears
>  >>>> to stop writing to the file, as if it's given up.
>  >>>>
>  >>>> Looking at the files with tar, they're fine for many MB, then the
>  >>>> archive ends prematurely.  There's nothing on the screen or in the log
>  >>>> to indicate a problem.  This problem persists, and only on the same
>  >>>> two regions - other regions back up fine.
>  >>>>
>  >>>> Platform is Ubuntu Linux (Jaunty), Mono 2.2, MySQL 5.0; all seven
>  >>>> regions run under one instance of OpenSim.exe.
>  >>>>
>  >>> my not actually be the issue, but mono 2.2 has a bad reputation. 2.0.1
>  >>> seems to be quite stable, 2.4 is also not too bad.
>  >>>
>  >>>    dirk
>  >>>> Any ideas?  Thanks in advance.
>  >>>>
>  >>>
>  >>> --
>  >>> dr dirk husemann ---- math & computer science ---- ibm zurich
> research lab
>  >>> RL: [hidden email] <mailto:[hidden email]> - +41 44 724
> <http://www.zurich.ibm.com/%7Ehud/><http://www.zurich.ibm.com/%7Ehud/>
>  >>> SL: [hidden email] <mailto:[hidden email]>
> --------------------- http://xyzzyxyzzy.net/
>  >>>
>  >>> _______________________________________________
>  >>> Opensim-users mailing list
>  >>> [hidden email] <mailto:[hidden email]>
>  >>> https://lists.berlios.de/mailman/listinfo/opensim-users
>  >>>
>
>
> --
> justincc
> Justin Clark-Casey
> http://justincc.wordpress.com
> _______________________________________________
> Opensim-users mailing list
> [hidden email] <mailto:[hidden email]>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users


--
justincc
Justin Clark-Casey
http://justincc.wordpress.com
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users


_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: "Save OAR" stopping since r9700

BlueWall Slade
Rev 9603
http://opensimulator.org/mantis/view.php?id=3741

On Fri, May 29, 2009 at 3:45 PM, BlueWall Slade <[hidden email]> wrote:
Justin - I am setting http://opensimulator.org/mantis/view.php?id=3741up a local grid on 9395 and working up from there.

Thanks!
BlueWall


On Fri, May 29, 2009 at 3:25 PM, Justin Clark-Casey <[hidden email]> wrote:
The most helpful thing is if you could go backwards from r9573 on Europa until the problem goes away.  And then slowly
inch forwards until it reappears, so we know which revision number the issue is associated with.

Charles Krinke wrote:
> I tried with r9573 this morning to do a "save oar <oarFileName> and here
> is my observation:
>
> First on ckTai which is essentially a blank region.
> Region (ckTai) # save oar ckTai0529
> 18:16:09 - [ARCHIVER]: Writing archive for region ckTai to ckTai0529
> 18:16:09 - [ARCHIVER]: 0 scene objects to serialize requiring save of 0
> assets
> 18:16:09 - [ARCHIVER]: AssetsRequest executed looking for 0 assets
> 18:16:09 - [ARCHIVER]: Creating archive file.  This may take some time.
> 18:16:09 - [ARCHIVER]: Added control file to archive.
> 18:16:09 - [ARCHIVER]: Added region settings to archive.
> 18:16:10 - [ARCHIVER]: Added terrain information to archive.
> 18:16:10 - [ARCHIVER]: Added scene objects to archive.
> 18:16:10 - [ARCHIVER]: Wrote out OpenSimulator archive for ckTai
> Region (ckTai) # show version
> Version: OpenSimulator Server  0.6.4.9573  (interface version 3)
>
> Then on Europa which has pushing 1000 prims. What is interesting about
> Europa is that the first line (in bold) printed out immediately and then
> there is no response to console. The line below is 40 minutes later and
> there is still no console response.
>
>
> Region (Europa) # save oar Europa0529
> 18:18:03 - [ARCHIVER]: Writing archive for region Europa to Europa0529
> 18:31:43 - DIAGNOSTICS
>
> Time now is 5/29/2009 6:31:43 PM
> Server has been running since Tuesday, 5/19/2009 2:31:42 AM
> That is an elapsed time of 10.16:00:00.4956310
>
> ASSET STATISTICS
> Asset cache contains        1 assets
> Latest asset request time after cache miss: 0s
> Blocked client requests for missing textures: 0
> Asset service request failures: 0
>
> CONNECTION STATISTICS
> Abnormal client thread terminations: 0
>
> INVENTORY STATISTICS
> Initial inventory caching failures: 0
>
> FRAME STATISTICS
> Dilatn  SimFPS  PhyFPS  AgntUp  RootAg  ChldAg  Prims   AtvPrm  AtvScr
> ScrLPS
>   0.00       0     0.0     0.0       0       1      48       1
> 0       0
>
>
>
>
>
> ------------------------------------------------------------------------
> *From:* Justin Clark-Casey <[hidden email]>
> *To:* [hidden email]
> *Sent:* Friday, May 29, 2009 9:01:37 AM
> *Subject:* Re: [Opensim-users] "Save OAR" stopping since r9700
>
> The data that we (the developers) really need is the exact revision
> between r9561 to r9700 on which this started
> failing.  I suggest a divide an conquer approach (see if the problem
> occurs on r9641, if not, then try r9670, etc.).
> Then that information should be put in a mantis (or an existing one, in
> which case I'll need to know the number).
>
> John Hopkin wrote:
>  > Interesting - thanks (to both of you).
>  >
>  > I'm not what other information is relevant, but here's some:
>  >
>  > 1. For prims counts, they are two of the largest regions among the
>  > seven, marked with asterisks here:
>  >
>  > 2889
>  > 2717*
>  >  858*
>  >  526
>  >  257
>  >  195
>  >    3
>  >
>  > 2. In total data size, they're the two of largest, as measured by the
>  > size of the last good backup:
>  >
>  > 140M*
>  >  31M
>  >  23M*
>  >  7M
>  >  5M
>  >  4M
>  >  0.5M
>  >
>  > 3. Regarding terrain, as well as location on the grid, they're not
>  > unusual; although they're adjacent to each other, all seven are
>  > located contiguously and contain a mix of elevations.
>  >
>  > 4. They could well be the regions with the most scripts; I don't know
>  > how I can find that out without a manual audit in-world.
>  >
>  > There are four other regions, not normally loaded, used occasionally
>  > as sandboxes and storage areas.  I don't yet know if they're affected
>  > by the bug.
>  >
>  > Hope this gives some idea, and maybe helps.  If you want to know
>  > anything else, please ask.
>  >
>  > John
>  >
>  > BlueWall Slade wrote:
>  >
>  >> I have seen oar failures on Linux (Mono 2.4/64 Bit and Mono 2.5/32 Bit
>  >> environments). And across several versions of OpenSim on several
> different
>  >> regions. My experience is that any region, in these environments,
> with more
>  >> than trivial content fails to complete the backup process. I am
> continuing
>  >> to investigate. Any other information would be most helpful.
>  >>
>  >> Thanks,
>  >> BlueWall
>  >>
>  >> On Thu, May 28, 2009 at 3:25 PM, dr scofield
> <[hidden email] <mailto:[hidden email]>>wrote:
>  >>
>  >>> John Hopkin wrote:
>  >>>> Since I upgraded from r9561 to r9700, two out of seven regions don't
>  >>>> back up using "save oar".  Even when they're the only region running,
>  >>>> those two have the same symptom: after a while, the archiver appears
>  >>>> to stop writing to the file, as if it's given up.
>  >>>>
>  >>>> Looking at the files with tar, they're fine for many MB, then the
>  >>>> archive ends prematurely.  There's nothing on the screen or in the log
>  >>>> to indicate a problem.  This problem persists, and only on the same
>  >>>> two regions - other regions back up fine.
>  >>>>
>  >>>> Platform is Ubuntu Linux (Jaunty), Mono 2.2, MySQL 5.0; all seven
>  >>>> regions run under one instance of OpenSim.exe.
>  >>>>
>  >>> my not actually be the issue, but mono 2.2 has a bad reputation. 2.0.1
>  >>> seems to be quite stable, 2.4 is also not too bad.
>  >>>
>  >>>    dirk
>  >>>> Any ideas?  Thanks in advance.
>  >>>>
>  >>>
>  >>> --
>  >>> dr dirk husemann ---- math & computer science ---- ibm zurich
> research lab
>  >>> RL: [hidden email] <mailto:[hidden email]> - +41 44 724
> <http://www.zurich.ibm.com/%7Ehud/><http://www.zurich.ibm.com/%7Ehud/>
>  >>> SL: [hidden email] <mailto:[hidden email]>
> --------------------- http://xyzzyxyzzy.net/
>  >>>
>  >>> _______________________________________________
>  >>> Opensim-users mailing list
>  >>> [hidden email] <mailto:[hidden email]>
>  >>> https://lists.berlios.de/mailman/listinfo/opensim-users
>  >>>
>
>
> --
> justincc
> Justin Clark-Casey
> http://justincc.wordpress.com
> _______________________________________________
> Opensim-users mailing list
> [hidden email] <mailto:[hidden email]>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users


--
justincc
Justin Clark-Casey
http://justincc.wordpress.com
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users



_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: "Save OAR" stopping since r9700

justincc
BlueWall Slade wrote:
> Rev 9603
> http://opensimulator.org/mantis/view.php?id=3741

Thanks very much Bluewall.  Ironically, r9603 doesn't seem to have any implications for oar saving, but in general this
is the kind of data that I need.

The problem might be alleviated in r9753.  Further feedback in that Mantis would be appreciated.

--
justincc
Justin Clark-Casey
http://justincc.wordpress.com
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users