asset cleaner

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

asset cleaner

Philippe Scoffoni
Hello,

Does anyone have a script to clear the asset ? I'm using a grid with 9
regions. And the assets is becoming very big.

On the wiki
http://opensimulator.org/wiki/Performance
there's a reference to
the asset cleaner from Wizardry and Steamworks
But the web site seems to be down

Thanks for your help

Philippe
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: asset cleaner

Melanie
This question comes up about every 6 months on average.

No.

There is no reasonable way to clean assets. Assets will just grow
infinitely. Hard disks are cheap and everyone who intends to run any
kind of region or grid where content remains stable needs to take
the growth of assets into account.

Melanie

On 29/03/2014 21:38, Philippe Scoffoni wrote:

> Hello,
>
> Does anyone have a script to clear the asset ? I'm using a grid with 9
> regions. And the assets is becoming very big.
>
> On the wiki
> http://opensimulator.org/wiki/Performance
> there's a reference to
> the asset cleaner from Wizardry and Steamworks
> But the web site seems to be down
>
> Thanks for your help
>
> Philippe
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users
>
>
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: asset cleaner

Dahlia Trimble
Such a script would need to traverse every object in every region, and all nested objects in each object's inventory. It would need to parse all objects, scripts, and notecards for anything resembling a UUID, and keep a list of all of the UUIDs found. It would then need to do a similar procedure for all entries in each user's inventory, and all nested objects, notecards, etc. found there. Assuming this is a complete list, such a script could compare it against all assets in the asset server for the grid, and then archive any assets not in the list.

There's probably a few possibilities where asset UUIDs could be stored that I did not mention above, and I'm not sure if hypergrid could complicate things. Given that, I agree with Melanie: "Hard disks are cheap"


On Sat, Mar 29, 2014 at 2:19 PM, Melanie <[hidden email]> wrote:
This question comes up about every 6 months on average.

No.

There is no reasonable way to clean assets. Assets will just grow
infinitely. Hard disks are cheap and everyone who intends to run any
kind of region or grid where content remains stable needs to take
the growth of assets into account.

Melanie

On 29/03/2014 21:38, Philippe Scoffoni wrote:
> Hello,
>
> Does anyone have a script to clear the asset ? I'm using a grid with 9
> regions. And the assets is becoming very big.
>
> On the wiki
> http://opensimulator.org/wiki/Performance
> there's a reference to
> the asset cleaner from Wizardry and Steamworks
> But the web site seems to be down
>
> Thanks for your help
>
> Philippe
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users
>
>
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users


_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: asset cleaner

Mic Bowman
most of the code to enumerate assets is in the archiver. 
just need to oar up every region. iar up all inventories. and check last access times... 
simple matter of programming... :-)



On Sat, Mar 29, 2014 at 2:55 PM, Dahlia Trimble <[hidden email]> wrote:
Such a script would need to traverse every object in every region, and all nested objects in each object's inventory. It would need to parse all objects, scripts, and notecards for anything resembling a UUID, and keep a list of all of the UUIDs found. It would then need to do a similar procedure for all entries in each user's inventory, and all nested objects, notecards, etc. found there. Assuming this is a complete list, such a script could compare it against all assets in the asset server for the grid, and then archive any assets not in the list.

There's probably a few possibilities where asset UUIDs could be stored that I did not mention above, and I'm not sure if hypergrid could complicate things. Given that, I agree with Melanie: "Hard disks are cheap"


On Sat, Mar 29, 2014 at 2:19 PM, Melanie <[hidden email]> wrote:
This question comes up about every 6 months on average.

No.

There is no reasonable way to clean assets. Assets will just grow
infinitely. Hard disks are cheap and everyone who intends to run any
kind of region or grid where content remains stable needs to take
the growth of assets into account.

Melanie

On 29/03/2014 21:38, Philippe Scoffoni wrote:
> Hello,
>
> Does anyone have a script to clear the asset ? I'm using a grid with 9
> regions. And the assets is becoming very big.
>
> On the wiki
> http://opensimulator.org/wiki/Performance
> there's a reference to
> the asset cleaner from Wizardry and Steamworks
> But the web site seems to be down
>
> Thanks for your help
>
> Philippe
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users
>
>
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users


_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users


_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: asset cleaner

M.E. Verhagen
the only way to clean up the database is to do a rebuild. Save everything in iar and oar files and restore them in a new database.

Or buy yourself a 3tb harddisk

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: asset cleaner

Dahlia Trimble
Not sure about you but I usually have broken links in outfit folders whenever I restore iar files.


On Sun, Mar 30, 2014 at 1:50 AM, M.E. Verhagen <[hidden email]> wrote:
the only way to clean up the database is to do a rebuild. Save everything in iar and oar files and restore them in a new database.

Or buy yourself a 3tb harddisk

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users


_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: asset cleaner

Fleep Tuque
In reply to this post by M.E. Verhagen
That sounds terrifying.  Has anyone actually done this on a large or old grid successfully?

- Fleep

On Sun, Mar 30, 2014 at 4:50 AM, M.E. Verhagen <[hidden email]> wrote:
the only way to clean up the database is to do a rebuild. Save everything in iar and oar files and restore them in a new database.

Or buy yourself a 3tb harddisk

_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users


_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: asset cleaner

justincc
As I think Mic has proposed before, I believe it's possible to clean up assets in a distributed grid if you track access
times.

Essentially, periodically when as asset is fetched from the asset service (e.g. every 6 months), the access time is
updated.  Then you reap assets which have an access time older than 3 years, for instance.

Off the top of my head, this would rely on the following conditions:

1) Regions periodically clear their asset caches and preferably fetch all assets referenced in the scene.  This periodic
fetch is somewhat like taking an OAR, which should also fetch all assets in the scene.  An alternative to fetch would be
a more inexpensive asset ping.  If regions cache indefinitely then asset access times may go stale.  Of course, this
reduces cache efficiency.

2) One periodically sweeps through each users inventory and pings/fetches associated assets.  This is what fetching an
IAR effectively does.  This could be done over a fairly long period without disrupting grid operations.

There is always going to be some risk in asset removal but I think that it would be negligible if these conditions were met.

Unfortunately, the existing asset database plugins, at least the MySQL time, have an access_time field but for some
reason never updates it, making it completely useless.  The deduplicating XAssetService does implement this and works
perfectly (I've been using it for more than a year now) but I haven't yet fully thought out the process for finally
migrating from the existing asset service.  Of course, deduplicating also significantly reduces asset db bloat.

On 30/03/14 13:48, Fleep Tuque wrote:

> That sounds terrifying.  Has anyone actually done this on a large or old grid successfully?
>
> - Fleep
>
> On Sun, Mar 30, 2014 at 4:50 AM, M.E. Verhagen <[hidden email] <mailto:[hidden email]>> wrote:
>
>     the only way to clean up the database is to do a rebuild. Save everything in iar and oar files and restore them in a
>     new database.
>
>     Or buy yourself a 3tb harddisk
>
>     _______________________________________________
>     Opensim-users mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.berlios.de/mailman/listinfo/opensim-users
>
>
>
>
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users
>


--
Justin Clark-Casey (justincc)
OSVW Consulting
http://justincc.org
http://twitter.com/justincc
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: asset cleaner

justincc
In reply to this post by Dahlia Trimble
Yes, this is a known bug which I simply haven't had time to fix.  I don't think that IAR save is properly following
links, though it might also be that it's not properly reconstructing them on restore.

On 30/03/14 10:42, Dahlia Trimble wrote:

> Not sure about you but I usually have broken links in outfit folders whenever I restore iar files.
>
>
> On Sun, Mar 30, 2014 at 1:50 AM, M.E. Verhagen <[hidden email] <mailto:[hidden email]>> wrote:
>
>     the only way to clean up the database is to do a rebuild. Save everything in iar and oar files and restore them in a
>     new database.
>
>     Or buy yourself a 3tb harddisk
>
>     _______________________________________________
>     Opensim-users mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.berlios.de/mailman/listinfo/opensim-users
>
>
>
>
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-users
>


--
Justin Clark-Casey (justincc)
OSVW Consulting
http://justincc.org
http://twitter.com/justincc
_______________________________________________
Opensim-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-users