Approach to get fresh content for an OpenSim Grid on New Servers

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Approach to get fresh content for an OpenSim Grid on New Servers

aiaustin
Our OpenSim grid has been running continuously since version 0.4 from
September 2007... and has probably by now accumulated a LOT of stuff
that has not been garbage collected.

We are changing to some new more powerful core servers in the next
few weeks, and I had an idea of how to do the upgrade that MIGHT get
rid of the clutter while still keeping in place the region UUIDs (for
web map servers, etc), the avatar UUIDs for permissions and .ini
files that refer to these, and even perhaps the UUIDs of key assets
in case they are in scripts.

but i am assuming that the IARs and OARs work by trying to reuse the
UUIDs if they are not already use din the data base.

Will this process work..

1. Save each region as an OAR and the avatar inventories we wish to
carry over as an IAR

2. Create a completely fresh data base - empty.

3. Create the avatars we want to preserve using the ption to set a
specific UUID for them

4. Start up with the original region.ini files which have the old UUID in them

5. restore the OARS to the relevant regions, and the IARs to the
relevant avatars.

6. Then fingers crossed we have a smaller neater, no clutter version
of what we have now?

Or is that just plain silly and we have to take over more, or dump to
SQL and restore the data base as is?

Any comments from devs or those with experience of a migration would
be appreciated before we try this if its going to be a waste of time.  Ai

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Approach to get fresh content for an OpenSim Grid on New Servers

Jeff Kelley
At 3:39 PM +0100 4/11/13, Ai Austin wrote:

>Or is that just plain silly and we have to take over more, or dump to SQL
>and restore the data base as is?

Some gridmasters proceed this way, although for my (very) small grid,
I prefer to purge the database from what I know to be garbage (my
thousands old versions of scripts for example). It runs from version
0.5 on the same database.

OAR will preserve assets UUID, but not prims UUID (scripts may rely
on hard-coded IDs, osMessageObject and such). Also, HG owners will be
zapped and replaced by the region master avatar.


-- Jeff

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Approach to get fresh content for an OpenSim Grid on New Servers

Chris M
In reply to this post by aiaustin
Hi Ai, It has been my experience that when I take the approach of saving
and reloading OAR/IAR files to a new fresh database it usually does
result in a smaller assets table (as compared to the previous database)
but I suppose the space gain all depends on how many unreferenced assets
there were on the previous database.

As far as I can tell, IAR and OAR saves work by only collecting assets
that are referenced on the regions that have had OARs made from, and any
assets that are referenced in the path specified for IARs that are
created from user inventories. Any thing left over will be gone once a
new empty database is created.

I am not 100% certain since I am only going by what I have observed in
the process of saving/loading archives and transferring to a new
database so anyone please by all means correct me if I'm not entirely
accurate here :)


On 4/11/2013 9:39 AM, Ai Austin wrote:

> Our OpenSim grid has been running continuously since version 0.4 from
> September 2007... and has probably by now accumulated a LOT of stuff
> that has not been garbage collected.
>
> We are changing to some new more powerful core servers in the next few
> weeks, and I had an idea of how to do the upgrade that MIGHT get rid
> of the clutter while still keeping in place the region UUIDs (for web
> map servers, etc), the avatar UUIDs for permissions and .ini files
> that refer to these, and even perhaps the UUIDs of key assets in case
> they are in scripts.
>
> but i am assuming that the IARs and OARs work by trying to reuse the
> UUIDs if they are not already use din the data base.
>
> Will this process work..
>
> 1. Save each region as an OAR and the avatar inventories we wish to
> carry over as an IAR
>
> 2. Create a completely fresh data base - empty.
>
> 3. Create the avatars we want to preserve using the ption to set a
> specific UUID for them
>
> 4. Start up with the original region.ini files which have the old UUID
> in them
>
> 5. restore the OARS to the relevant regions, and the IARs to the
> relevant avatars.
>
> 6. Then fingers crossed we have a smaller neater, no clutter version
> of what we have now?
>
> Or is that just plain silly and we have to take over more, or dump to
> SQL and restore the data base as is?
>
> Any comments from devs or those with experience of a migration would
> be appreciated before we try this if its going to be a waste of time.  Ai
>
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users


--
OpenSim: 10 Region Standalone on 0.7.6 Dev
Physics: Open Dynamics Engine
OS: Windows 7 (x64)
CPU: AMD Phenom II X4 840 3.2 GHz
Memory: 11 GB DDR3
Database: MySQL 5.1.63 (x64)

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Approach to get fresh content for an OpenSim Grid on New Servers

Jeff Kelley
At 6:46 AM -0500 4/12/13, Chris wrote:

>As far as I can tell, IAR and OAR saves work by only collecting
>assets that are referenced on the regions that have had OARs made
>from, and any assets that are referenced in the path specified for
>IARs that are created from user inventories

... and any asset referenced in an object (SceneObjectGroup blob)
referenced in the path specified for IARs (they may point to assets
outside the path), and recursively (objects containing objects).

I've never checked for recursive objects, however.

-- jeff

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Approach to get fresh content for an OpenSim Grid on New Servers

Chris M
On 4/12/2013 9:58 AM, Jeff Kelley wrote:

> At 6:46 AM -0500 4/12/13, Chris wrote:
>
>> As far as I can tell, IAR and OAR saves work by only collecting
>> assets that are referenced on the regions that have had OARs made
>> from, and any assets that are referenced in the path specified for
>> IARs that are created from user inventories
>
> ... and any asset referenced in an object (SceneObjectGroup blob)
> referenced in the path specified for IARs (they may point to assets
> outside the path), and recursively (objects containing objects).
>
> I've never checked for recursive objects, however.
>
> -- jeff
>
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users

Yes, objects as well. I thought that was implied when I mentioned assets
in the inventory path but thank you for clarifying Jeff :) As far as I
can tell it does also pick up assets recursively although how deeply
nested I am unsure of; I have not tested more than a few objects deep.

--
OpenSim: 10 Region Standalone on 0.7.6 Dev
Physics: Open Dynamics Engine
OS: Windows 7 (x64)
CPU: AMD Phenom II X4 840 3.2 GHz
Memory: 11 GB DDR3
Database: MySQL 5.1.63 (x64)

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Approach to get fresh content for an OpenSim Grid on New Servers

Jeff Kelley
At 4:20 PM -0500 4/12/13, Chris wrote:

>  As far as I can tell it does also pick up assets recursively although how
>  deeply nested I am unsure of; I have not tested more than a few objects deep.

Well, there is no recursivity, strictly speaking. "Nested" is more
accurate. An object in itself is stramlined in the XML (not a
reference pointing to self). Recursivity is achieved in-world  with
the "rez and give a copy of self" trick.

I'm searching a way to clean the database more deeply than with SQL queries.
That implies parsing SceneObjectGroups, as well as the binary UUIDs
in primShapes:Texture. Not a simple task.


-- Jeff
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Approach to get fresh content for an OpenSim Grid on New Servers

Melanie
Hi,

this requires a sweep and mark approach.

Create a table that parallels your asset table. It should contain
the asset ID and a date field which would default to the asset's
creation date.

Next you need to iterate over all rezzed objects in all sims and
collect the UUID of any used textures and the UUIDs of all inventory
items. Keep those that refer to objects separately, as well as those
that refer to scripts and wearables. Also make a list of all
notecards. Don't forget to include ground textures and parcel
textures. Mark all textures and wearables with the current date.

Then, similarly iterate all user inventories. Mark all UUIDs of
simple objects like wearables and textures, then add the UUIDs of
the objects, wearables, notecards and scripts to the UUID lists
created in the first step. Mark all "simple" (non-object) assets
with the current date.

Next, iterate all notecards. Extract all attached (embedded) objects
and add the wearables and objects and other notecards to the lists
above. Mark all simple assets found. Repeat until no more notecards
are left.

Finally, iterate all object assets, retrieving and parsing them to
glean the same info as you obtained from the sim. Mark textures and
wearables and also add wearables, objects and scripts to the above
lists. Mark all object assets with the current date.

Repeat until no more objects are left in the object list.

If notecards were collected, go back to iterating the notecards.

Last, iterate wearables (extract texture ids and mark them) and
scripts (regex-match UUID-like things and if they are a known asset,
mark them).

Depending on the grid size, this should leave you anywhere from a
few days to a few months on. Assuming the sweep & mark took less
than 6 months, all UUID in the secondary table that have a date
stamp OLDER than 6 months are unreferenced and the assets they name
can be deleted.

Since old assets are static, limiting future scans to only look at
things changed after the last scan was done will allow you to keep
this dated list and use it for further purging, dramatically
shortening the next sweeping run.

A lot of this code is present in the flotsam cache and the
archivers. It's a matter of putting it together properly and filling
in a few tricky spots.

Melanie

On 13/04/2013 02:40, Jeff Kelley wrote:

> At 4:20 PM -0500 4/12/13, Chris wrote:
>
>>  As far as I can tell it does also pick up assets recursively although how
>>  deeply nested I am unsure of; I have not tested more than a few objects deep.
>
> Well, there is no recursivity, strictly speaking. "Nested" is more
> accurate. An object in itself is stramlined in the XML (not a
> reference pointing to self). Recursivity is achieved in-world  with
> the "rez and give a copy of self" trick.
>
> I'm searching a way to clean the database more deeply than with SQL queries.
> That implies parsing SceneObjectGroups, as well as the binary UUIDs
> in primShapes:Texture. Not a simple task.
>
>
> -- Jeff
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users
>
>
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Approach to get fresh content for an OpenSim Grid on New Servers

Dahlia Trimble
You might want to also regex-search all notecard text for embedded UUIDs that match any assets.


On Fri, Apr 12, 2013 at 6:34 PM, Melanie <[hidden email]> wrote:
Hi,

this requires a sweep and mark approach.

Create a table that parallels your asset table. It should contain
the asset ID and a date field which would default to the asset's
creation date.

Next you need to iterate over all rezzed objects in all sims and
collect the UUID of any used textures and the UUIDs of all inventory
items. Keep those that refer to objects separately, as well as those
that refer to scripts and wearables. Also make a list of all
notecards. Don't forget to include ground textures and parcel
textures. Mark all textures and wearables with the current date.

Then, similarly iterate all user inventories. Mark all UUIDs of
simple objects like wearables and textures, then add the UUIDs of
the objects, wearables, notecards and scripts to the UUID lists
created in the first step. Mark all "simple" (non-object) assets
with the current date.

Next, iterate all notecards. Extract all attached (embedded) objects
and add the wearables and objects and other notecards to the lists
above. Mark all simple assets found. Repeat until no more notecards
are left.

Finally, iterate all object assets, retrieving and parsing them to
glean the same info as you obtained from the sim. Mark textures and
wearables and also add wearables, objects and scripts to the above
lists. Mark all object assets with the current date.

Repeat until no more objects are left in the object list.

If notecards were collected, go back to iterating the notecards.

Last, iterate wearables (extract texture ids and mark them) and
scripts (regex-match UUID-like things and if they are a known asset,
mark them).

Depending on the grid size, this should leave you anywhere from a
few days to a few months on. Assuming the sweep & mark took less
than 6 months, all UUID in the secondary table that have a date
stamp OLDER than 6 months are unreferenced and the assets they name
can be deleted.

Since old assets are static, limiting future scans to only look at
things changed after the last scan was done will allow you to keep
this dated list and use it for further purging, dramatically
shortening the next sweeping run.

A lot of this code is present in the flotsam cache and the
archivers. It's a matter of putting it together properly and filling
in a few tricky spots.

Melanie

On 13/04/2013 02:40, Jeff Kelley wrote:
> At 4:20 PM -0500 4/12/13, Chris wrote:
>
>>  As far as I can tell it does also pick up assets recursively although how
>>  deeply nested I am unsure of; I have not tested more than a few objects deep.
>
> Well, there is no recursivity, strictly speaking. "Nested" is more
> accurate. An object in itself is stramlined in the XML (not a
> reference pointing to self). Recursivity is achieved in-world  with
> the "rez and give a copy of self" trick.
>
> I'm searching a way to clean the database more deeply than with SQL queries.
> That implies parsing SceneObjectGroups, as well as the binary UUIDs
> in primShapes:Texture. Not a simple task.
>
>
> -- Jeff
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users
>
>
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users


_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Approach to get fresh content for an OpenSim Grid on New Servers

aiaustin
In reply to this post by aiaustin
Thanks to everyone for their inputs on this.  Sound like its worth
giving it a try as the data base seems to have grown a lot with many
mesh tests and lots of sandbox work at one stage a few years ago.

The discussion between Jeff and Melanie on a deep clean approach to
tidy up a data base is also interesting.  As the opportunity to clean
out and restore from OARS/IARs is not really applicable to larger
grids.  Openvue only has a few (less than 20 active) "builder"
registered avatars, and works mostly be allowing hypergrid travel
from OSGrid and other grids for "normal" users.

Thanks, Ai

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Approach to get fresh content for an OpenSim Grid on New Servers

Jeff Kelley
In reply to this post by Melanie
At 3:34 AM +0200 4/13/13, Melanie wrote:

>A lot of this code is present in the flotsam cache and the
>archivers. It's a matter of putting it together properly and filling
>in a few tricky spots.

Very informative post. Thanks Melanie.

I archive for further reference.


-- Jeff
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Approach to get fresh content for an OpenSim Grid on New Servers

justincc
Yes, just to be clear, the archivers (and HG and parts of Flotsam since they uses the same scan code) will pick up all
assets referenced by objects within objects to any level of nesting.  It also scans scripts, wearables, etc. for UUID
strings.  This is why there are a number of 'possibly missing assets' that are reported on an OAR save - these are often
false positives and the code now exists to make this clearer.

It struck me that there might be a small hole in OAR saving since the assets for 'parcel snapshots' are not currently
being saved.  If so, this will need to be corrected.

On 13/04/13 17:33, Jeff Kelley wrote:

> At 3:34 AM +0200 4/13/13, Melanie wrote:
>
>> A lot of this code is present in the flotsam cache and the
>> archivers. It's a matter of putting it together properly and filling
>> in a few tricky spots.
>
> Very informative post. Thanks Melanie.
>
> I archive for further reference.
>
>
> -- Jeff
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users
>


--
Justin Clark-Casey (justincc)
OSVW Consulting
http://justincc.org
http://twitter.com/justincc
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Approach to get fresh content for an OpenSim Grid on New Servers

aiaustin
In reply to this post by aiaustin
Sounds good folks.. and given I am moving to a new server if all hell
breaks out I can revert to our original setup and find another
route.  So I will test this over the next month or so (first core
server machine for new Openvue grid is on order) and report back on
my experience.

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users