OAR not receiving asset data in time

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

OAR not receiving asset data in time

John Hopkin
Finding this a lot while doing "save OAR" (text in <> is variable):

00:10:27 - [ARCHIVER]: Asset service failed to return information
about <n> requested assets
00:10:32 - [ARCHIVER]: No information about asset <UUID> received
[repeated <n> times]
00:10:32 - [ARCHIVER]: OAR save aborted.
[...]
00:10:53 - [ARCHIVER]: Received information about asset <UUID> after
archive save abortion.  Ignoring.
[repeated <n> times]

So, I'll get several "No information" lines, followed by "OAR save
aborted" and, a few seconds later, several "received information"
messages.

The OAR checks as complete with gzip, but I don't know if it's
complete or not. When performing several "save OAR"s in a command
script, it's not clear which region is affected, or even that there
has been an error at all unless one looks for it.

This could lead to incomplete saves being archived. Obviously, this is
dangerous.

On rerunning the "save oars" during the same OpenSim.exe session, I
get the message:

[00:29:35] - [ARCHIVER]: Could not find asset <UUID>

for each of the above lines . This time, there is no "OAR save
aborted" message; presumably, the OAR worked.

Any ideas?

OpenSim v0.7.1 8731c2b, OS is Ubuntu Meerkat latest; database is MySQL
14.14;  Mono is 2.6.7; physics is Meshmerizer/ODE.
--
John Hopkin

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: OAR not receiving asset data in time

justincc
OAR save times out after 60 seconds if it doesn't receive all the assets it's asked for.  This is to prevent completely
or very unresponsive servers from keeping the file open indefinitely.

The errors you're seeing suggest that the assets not being delivered are missing anyway, otherwise they would get into
the subsequent save.

On 21/10/10 00:35, John Hopkin wrote:

> Finding this a lot while doing "save OAR" (text in<>  is variable):
>
> 00:10:27 - [ARCHIVER]: Asset service failed to return information
> about<n>  requested assets
> 00:10:32 - [ARCHIVER]: No information about asset<UUID>  received
> [repeated<n>  times]
> 00:10:32 - [ARCHIVER]: OAR save aborted.
> [...]
> 00:10:53 - [ARCHIVER]: Received information about asset<UUID>  after
> archive save abortion.  Ignoring.
> [repeated<n>  times]
>
> So, I'll get several "No information" lines, followed by "OAR save
> aborted" and, a few seconds later, several "received information"
> messages.
>
> The OAR checks as complete with gzip, but I don't know if it's
> complete or not. When performing several "save OAR"s in a command
> script, it's not clear which region is affected, or even that there
> has been an error at all unless one looks for it.
>
> This could lead to incomplete saves being archived. Obviously, this is
> dangerous.
>
> On rerunning the "save oars" during the same OpenSim.exe session, I
> get the message:
>
> [00:29:35] - [ARCHIVER]: Could not find asset<UUID>
>
> for each of the above lines . This time, there is no "OAR save
> aborted" message; presumably, the OAR worked.
>
> Any ideas?
>
> OpenSim v0.7.1 8731c2b, OS is Ubuntu Meerkat latest; database is MySQL
> 14.14;  Mono is 2.6.7; physics is Meshmerizer/ODE.


--
Justin Clark-Casey (justincc)
http://justincc.org
http://twitter.com/justincc
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: OAR not receiving asset data in time

John Hopkin
Thanks, Justin. Do you happen to know if the first save will be
reliable (other than the missing assets)? It's that word "aborted"
that worries me.

And how can I go about removing references to the missing assets? Is
this a case of hacking the MySQL database?

John

Justin Clark-Casey wrote:

> OAR save times out after 60 seconds if it doesn't receive all the assets it's asked for.  This is to prevent completely
> or very unresponsive servers from keeping the file open indefinitely.
>
> The errors you're seeing suggest that the assets not being delivered are missing anyway, otherwise they would get into
> the subsequent save.
>
> On 21/10/10 00:35, John Hopkin wrote:
> > Finding this a lot while doing "save OAR" (text in<>  is variable):
> >
> > 00:10:27 - [ARCHIVER]: Asset service failed to return information
> > about<n>  requested assets
> > 00:10:32 - [ARCHIVER]: No information about asset<UUID>  received
> > [repeated<n>  times]
> > 00:10:32 - [ARCHIVER]: OAR save aborted.
> > [...]
> > 00:10:53 - [ARCHIVER]: Received information about asset<UUID>  after
> > archive save abortion.  Ignoring.
> > [repeated<n>  times]
> >
> > So, I'll get several "No information" lines, followed by "OAR save
> > aborted" and, a few seconds later, several "received information"
> > messages.
> >
> > The OAR checks as complete with gzip, but I don't know if it's
> > complete or not. When performing several "save OAR"s in a command
> > script, it's not clear which region is affected, or even that there
> > has been an error at all unless one looks for it.
> >
> > This could lead to incomplete saves being archived. Obviously, this is
> > dangerous.
> >
> > On rerunning the "save oars" during the same OpenSim.exe session, I
> > get the message:
> >
> > [00:29:35] - [ARCHIVER]: Could not find asset<UUID>
> >
> > for each of the above lines . This time, there is no "OAR save
> > aborted" message; presumably, the OAR worked.
> >
> > Any ideas?
> >
> > OpenSim v0.7.1 8731c2b, OS is Ubuntu Meerkat latest; database is MySQL
> > 14.14;  Mono is 2.6.7; physics is Meshmerizer/ODE.
--
John Hopkin

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: OAR not receiving asset data in time

justincc
On 21/10/10 23:57, John Hopkin wrote:
> Thanks, Justin. Do you happen to know if the first save will be
> reliable (other than the missing assets)? It's that word "aborted"
> that worries me.

You're right to be worried - the first OAR will not be usable.  I have added some text to the error message to make that
clear.

>
> And how can I go about removing references to the missing assets? Is
> this a case of hacking the MySQL database?

No, unfortunately it's more complicated than that.  The missing assets could be referenced anywhere (within serialized
objects, asset metadata or even hardcoded within scripts).

In principle, you could untar the successfully saved OAR and grep all the files for the missing asset UUIDs.  That would
highlight where they are being referenced.

But really, the missing asset problem should not be causing your initial OAR abort.  The asset service should return
notification of missing assets as quickly as it returns actual assets.  Are you using a stock asset service?  Also, how
many assets actually end up being saved - is it a very populated region?

>
> John
>
> Justin Clark-Casey wrote:
>
>> OAR save times out after 60 seconds if it doesn't receive all the assets it's asked for.  This is to prevent completely
>> or very unresponsive servers from keeping the file open indefinitely.
>>
>> The errors you're seeing suggest that the assets not being delivered are missing anyway, otherwise they would get into
>> the subsequent save.
>>
>> On 21/10/10 00:35, John Hopkin wrote:
>>> Finding this a lot while doing "save OAR" (text in<>   is variable):
>>>
>>> 00:10:27 - [ARCHIVER]: Asset service failed to return information
>>> about<n>   requested assets
>>> 00:10:32 - [ARCHIVER]: No information about asset<UUID>   received
>>> [repeated<n>   times]
>>> 00:10:32 - [ARCHIVER]: OAR save aborted.
>>> [...]
>>> 00:10:53 - [ARCHIVER]: Received information about asset<UUID>   after
>>> archive save abortion.  Ignoring.
>>> [repeated<n>   times]
>>>
>>> So, I'll get several "No information" lines, followed by "OAR save
>>> aborted" and, a few seconds later, several "received information"
>>> messages.
>>>
>>> The OAR checks as complete with gzip, but I don't know if it's
>>> complete or not. When performing several "save OAR"s in a command
>>> script, it's not clear which region is affected, or even that there
>>> has been an error at all unless one looks for it.
>>>
>>> This could lead to incomplete saves being archived. Obviously, this is
>>> dangerous.
>>>
>>> On rerunning the "save oars" during the same OpenSim.exe session, I
>>> get the message:
>>>
>>> [00:29:35] - [ARCHIVER]: Could not find asset<UUID>
>>>
>>> for each of the above lines . This time, there is no "OAR save
>>> aborted" message; presumably, the OAR worked.
>>>
>>> Any ideas?
>>>
>>> OpenSim v0.7.1 8731c2b, OS is Ubuntu Meerkat latest; database is MySQL
>>> 14.14;  Mono is 2.6.7; physics is Meshmerizer/ODE.


--
Justin Clark-Casey (justincc)
http://justincc.org
http://twitter.com/justincc
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: OAR not receiving asset data in time

John Hopkin
Justin Clark-Casey wrote:

> On 21/10/10 23:57, John Hopkin wrote:
> > Thanks, Justin. Do you happen to know if the first save will be
> > reliable (other than the missing assets)? It's that word "aborted"
> > that worries me.
>
> You're right to be worried - the first OAR will not be usable.  I have added some text to the error message to make that
> clear.

Thanks.

> > And how can I go about removing references to the missing assets? Is
> > this a case of hacking the MySQL database?
>
> No, unfortunately it's more complicated than that.  The missing assets could be referenced anywhere (within serialized
> objects, asset metadata or even hardcoded within scripts).
>
> In principle, you could untar the successfully saved OAR and grep all the files for the missing asset UUIDs.  That would
> highlight where they are being referenced.
>
> But really, the missing asset problem should not be causing your initial OAR abort.  The asset service should return
> notification of missing assets as quickly as it returns actual assets.  Are you using a stock asset service?  Also, how
> many assets actually end up being saved - is it a very populated region?

The asset server is OSGrid's. One of the regions affected has around
8100 assets, the other about 230. I think there's another one, but I'm
not sure which of the 13 I have it is.

> > Justin Clark-Casey wrote:
> >
> >> OAR save times out after 60 seconds if it doesn't receive all the assets it's asked for.  This is to prevent completely
> >> or very unresponsive servers from keeping the file open indefinitely.
> >>
> >> The errors you're seeing suggest that the assets not being delivered are missing anyway, otherwise they would get into
> >> the subsequent save.
> >>
> >> On 21/10/10 00:35, John Hopkin wrote:
> >>> Finding this a lot while doing "save OAR" (text in<>   is variable):
> >>>
> >>> 00:10:27 - [ARCHIVER]: Asset service failed to return information
> >>> about<n>   requested assets
> >>> 00:10:32 - [ARCHIVER]: No information about asset<UUID>   received
> >>> [repeated<n>   times]
> >>> 00:10:32 - [ARCHIVER]: OAR save aborted.
> >>> [...]
> >>> 00:10:53 - [ARCHIVER]: Received information about asset<UUID>   after
> >>> archive save abortion.  Ignoring.
> >>> [repeated<n>   times]
> >>>
> >>> So, I'll get several "No information" lines, followed by "OAR save
> >>> aborted" and, a few seconds later, several "received information"
> >>> messages.
> >>>
> >>> The OAR checks as complete with gzip, but I don't know if it's
> >>> complete or not. When performing several "save OAR"s in a command
> >>> script, it's not clear which region is affected, or even that there
> >>> has been an error at all unless one looks for it.
> >>>
> >>> This could lead to incomplete saves being archived. Obviously, this is
> >>> dangerous.
> >>>
> >>> On rerunning the "save oars" during the same OpenSim.exe session, I
> >>> get the message:
> >>>
> >>> [00:29:35] - [ARCHIVER]: Could not find asset<UUID>
> >>>
> >>> for each of the above lines . This time, there is no "OAR save
> >>> aborted" message; presumably, the OAR worked.
> >>>
> >>> Any ideas?
> >>>
> >>> OpenSim v0.7.1 8731c2b, OS is Ubuntu Meerkat latest; database is MySQL
> >>> 14.14;  Mono is 2.6.7; physics is Meshmerizer/ODE.
--
John Hopkin

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: OAR not receiving asset data in time

John Hopkin
In reply to this post by John Hopkin
John Hopkin wrote:

> Finding this a lot while doing "save OAR" (text in <> is variable):
>
> 00:10:27 - [ARCHIVER]: Asset service failed to return information
> about <n> requested assets
> 00:10:32 - [ARCHIVER]: No information about asset <UUID> received
> [repeated <n> times]
> 00:10:32 - [ARCHIVER]: OAR save aborted.
> [...]
> 00:10:53 - [ARCHIVER]: Received information about asset <UUID> after
> archive save abortion.  Ignoring.
> [repeated <n> times]
>
> So, I'll get several "No information" lines, followed by "OAR save
> aborted" and, a few seconds later, several "received information"
> messages.
>
> The OAR checks as complete with gzip, but I don't know if it's
> complete or not. When performing several "save OAR"s in a command
> script, it's not clear which region is affected, or even that there
> has been an error at all unless one looks for it.
>
> This could lead to incomplete saves being archived. Obviously, this is
> dangerous.
>
> On rerunning the "save oars" during the same OpenSim.exe session, I
> get the message:
>
> [00:29:35] - [ARCHIVER]: Could not find asset <UUID>
>
> for each of the above lines . This time, there is no "OAR save
> aborted" message; presumably, the OAR worked.
>
> Any ideas?
>
> OpenSim v0.7.1 8731c2b, OS is Ubuntu Meerkat latest; database is MySQL
> 14.14;  Mono is 2.6.7; physics is Meshmerizer/ODE.

Now on OpenSim 80f9022 , otherwise as before, and getting several
thousand of these errors on another set of moderately-populated (in
asset terms) regions. Re-running the "save oars" in the same
OpenSim.exe session appeared to work fine.
--
John Hopkin

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: OAR not receiving asset data in time

justincc
I have a TODO item to improve the way that "save oar" fetches asset data.  Unfortunately, it's rather low down the list
so until then double saving to get things properly into the cache is the workaround.

On 28/10/10 17:48, John Hopkin wrote:

> John Hopkin wrote:
>
>> Finding this a lot while doing "save OAR" (text in<>  is variable):
>>
>> 00:10:27 - [ARCHIVER]: Asset service failed to return information
>> about<n>  requested assets
>> 00:10:32 - [ARCHIVER]: No information about asset<UUID>  received
>> [repeated<n>  times]
>> 00:10:32 - [ARCHIVER]: OAR save aborted.
>> [...]
>> 00:10:53 - [ARCHIVER]: Received information about asset<UUID>  after
>> archive save abortion.  Ignoring.
>> [repeated<n>  times]
>>
>> So, I'll get several "No information" lines, followed by "OAR save
>> aborted" and, a few seconds later, several "received information"
>> messages.
>>
>> The OAR checks as complete with gzip, but I don't know if it's
>> complete or not. When performing several "save OAR"s in a command
>> script, it's not clear which region is affected, or even that there
>> has been an error at all unless one looks for it.
>>
>> This could lead to incomplete saves being archived. Obviously, this is
>> dangerous.
>>
>> On rerunning the "save oars" during the same OpenSim.exe session, I
>> get the message:
>>
>> [00:29:35] - [ARCHIVER]: Could not find asset<UUID>
>>
>> for each of the above lines . This time, there is no "OAR save
>> aborted" message; presumably, the OAR worked.
>>
>> Any ideas?
>>
>> OpenSim v0.7.1 8731c2b, OS is Ubuntu Meerkat latest; database is MySQL
>> 14.14;  Mono is 2.6.7; physics is Meshmerizer/ODE.
>
> Now on OpenSim 80f9022 , otherwise as before, and getting several
> thousand of these errors on another set of moderately-populated (in
> asset terms) regions. Re-running the "save oars" in the same
> OpenSim.exe session appeared to work fine.


--
Justin Clark-Casey (justincc)
http://justincc.org
http://twitter.com/justincc
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: OAR not receiving asset data in time

John Hopkin
OK - thanks for your help, Justin. I'll do that.

Justin Clark-Casey wrote:

> I have a TODO item to improve the way that "save oar" fetches asset data.  Unfortunately, it's rather low down the list
> so until then double saving to get things properly into the cache is the workaround.
>
> On 28/10/10 17:48, John Hopkin wrote:
> > John Hopkin wrote:
> >
> >> Finding this a lot while doing "save OAR" (text in<>  is variable):
> >>
> >> 00:10:27 - [ARCHIVER]: Asset service failed to return information
> >> about<n>  requested assets
> >> 00:10:32 - [ARCHIVER]: No information about asset<UUID>  received
> >> [repeated<n>  times]
> >> 00:10:32 - [ARCHIVER]: OAR save aborted.
> >> [...]
> >> 00:10:53 - [ARCHIVER]: Received information about asset<UUID>  after
> >> archive save abortion.  Ignoring.
> >> [repeated<n>  times]
> >>
> >> So, I'll get several "No information" lines, followed by "OAR save
> >> aborted" and, a few seconds later, several "received information"
> >> messages.
> >>
> >> The OAR checks as complete with gzip, but I don't know if it's
> >> complete or not. When performing several "save OAR"s in a command
> >> script, it's not clear which region is affected, or even that there
> >> has been an error at all unless one looks for it.
> >>
> >> This could lead to incomplete saves being archived. Obviously, this is
> >> dangerous.
> >>
> >> On rerunning the "save oars" during the same OpenSim.exe session, I
> >> get the message:
> >>
> >> [00:29:35] - [ARCHIVER]: Could not find asset<UUID>
> >>
> >> for each of the above lines . This time, there is no "OAR save
> >> aborted" message; presumably, the OAR worked.
> >>
> >> Any ideas?
> >>
> >> OpenSim v0.7.1 8731c2b, OS is Ubuntu Meerkat latest; database is MySQL
> >> 14.14;  Mono is 2.6.7; physics is Meshmerizer/ODE.
> >
> > Now on OpenSim 80f9022 , otherwise as before, and getting several
> > thousand of these errors on another set of moderately-populated (in
> > asset terms) regions. Re-running the "save oars" in the same
> > OpenSim.exe session appeared to work fine.
--
John Hopkin

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users