Are unused assets deleted? (Fred Beckhusen)

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Are unused assets deleted? (Fred Beckhusen)

Fred Beckhusen
1. Are unused assets deleted? (Ethan A. Gardener)

Dumping and reloading breaks all Firestorm links to Outfits.

Take a look at thi web page at Wizardry and Steamworks
(grimore.org/opensim/database/asset_cleaner) for a possible way to clean
a db

It uses PHP with OARS and IARS to make lists of what to keep, then
deletes the rest.

Fred

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Are unused assets deleted? (Fred Beckhusen)

Melanie
There isn't a real reason to expend that sort of time and effort on cleaning assets. First off, for assets named in scripts, a heuristic is used. It can't detect all assets with certainty and it will also have false positives. Therefore, some things may break an some unused assets will be kept. However, and that is the more important part, even a large commercial grid operating for years accumulates assets that still easily fit on a modern hard disk. Furthermore, if one disk fills up another can be added. So, for many grids I know and certainly for myself, the mantra is "disk space is cheap" and that way there isn't a risk of breaking users' creations. Deduplicating asset systems like fsassets make this even more relevant, because most large assets are meshes and textures that are uploade by many people, but are actually identical. Deduplicating is a massive help there. At the end of the day, there is only one way to remove a prim, even one without contents, completely. If you rez a prim and then delete it, it will be taken off the region, serialized and put into an asset to be put into your trash. So that is not the solution. In fact, the only way to remove a prim from a region without creating an asset is llDie(). It creates no copy. - Melanie ---- On Sun, 12 Aug 2018 16:46:15 +0000 Fred Beckhusen <[hidden email]> wrote ---- 1. Are unused assets deleted? (Ethan A. Gardener) Dumping and reloading breaks all Firestorm links to Outfits. Take a look at thi web page at Wizardry and Steamworks (grimore.org/opensim/database/asset_cleaner) for a possible way to clean a db It uses PHP with OARS and IARS to make lists of what to keep, then deletes the rest. Fred _______________________________________________ Opensim-users mailing list [hidden email] http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Are unused assets deleted? (Fred Beckhusen)

Ethan Gardener
Thank you for your reply, melanie.  Reasoning on the problem, I've come to agree that deleting unused assets is an extremely difficult problem, perhaps not worth solving.

However, everything I've ever learned about databases says that it's a bad idea to just look at the cost of the disks.  The cost of *accessing* the data (with acceptable performance) goes up exponentially with the size of the dataset.  There is a solution which I think practically returns the problem to just counting the cost of the disks:  Cache the asset server.  The actual asset server which never deletes anything can then be very slow, with all regularly used assets served from the much faster cache.  

I'm almost certain Second Life implemented such a cache years ago.  I remember when I logged in there after a long absence, things which I'd made or which were old-style took minutes to rez.  I don't ever remember seeing such an effect in InWorldz, so I think it's possible InWorldz didn't have such a cache.  

Does OpenSim cache the asset server?  I'm thinking it doesn't, because some grid techs will tell you the first thing they did for their new grid was rewrite the asset server.  Of course, maybe it does now but didn't when they were getting started.  I'd be interested to know.

On Sun, Aug 12, 2018, at 6:08 PM, Melanie wrote:

> There isn't a real reason to expend that sort of time and effort on
> cleaning assets. First off, for assets named in scripts, a heuristic is
> used. It can't detect all assets with certainty and it will also have
> false positives. Therefore, some things may break an some unused assets
> will be kept. However, and that is the more important part, even a large
> commercial grid operating for years accumulates assets that still easily
> fit on a modern hard disk. Furthermore, if one disk fills up another can
> be added. So, for many grids I know and certainly for myself, the mantra
> is "disk space is cheap" and that way there isn't a risk of breaking
> users' creations. Deduplicating asset systems like fsassets make this
> even more relevant, because most large assets are meshes and textures
> that are uploade by many people, but are actually identical.
> Deduplicating is a massive help there. At the end of the day, there is
> only one way to remove a prim, even one without contents, completely. If
> you rez a prim and then delete it, it will be taken off the region,
> serialized and put into an asset to be put into your trash. So that is
> not the solution. In fact, the only way to remove a prim from a region
> without creating an asset is llDie(). It creates no copy. - Melanie ----
> On Sun, 12 Aug 2018 16:46:15 +0000 Fred Beckhusen <[hidden email]>
> wrote ---- 1. Are unused assets deleted? (Ethan A. Gardener) Dumping and
> reloading breaks all Firestorm links to Outfits. Take a look at thi web
> page at Wizardry and Steamworks (grimore.org/opensim/database/
> asset_cleaner) for a possible way to clean a db It uses PHP with OARS
> and IARS to make lists of what to keep, then deletes the rest. Fred
> _______________________________________________ Opensim-users mailing
> list [hidden email]
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users


--
The lyf so short, the craft so long to lerne. -- Chaucer
_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Are unused assets deleted? (Fred Beckhusen)

Haravikk

> On 26 Aug 2018, at 15:05, Ethan Gardener <[hidden email]> wrote:
>
> However, everything I've ever learned about databases says that it's a bad idea to just look at the cost of the disks.  The cost of *accessing* the data (with acceptable performance) goes up exponentially with the size of the dataset.

This shouldn't be the case; any properly indexed database should have roughly constant, or at worst logarithmic, access times for data. You should only really see big performance drops if the number of assets requests increases beyond what the hardware can cope with (i.e- too many users online simultaneously).

> There is a solution which I think practically returns the problem to just counting the cost of the disks:  Cache the asset server.  The actual asset server which never deletes anything can then be very slow, with all regularly used assets served from the much faster cache.  
>
> I'm almost certain Second Life implemented such a cache years ago.  I remember when I logged in there after a long absence, things which I'd made or which were old-style took minutes to rez.  I don't ever remember seeing such an effect in InWorldz, so I think it's possible InWorldz didn't have such a cache.

Most (all?) external asset requests are now served to the viewer via HTTP, so you could use Squid proxy or similar to cache those requests before they touch your asset server, it might even be possible to use a reverse proxy such as Cloudflare to do this for you, though I've never tried it (you might need to use page rules to force it to cache asset URLs).

- Haravikk
_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Are unused assets deleted? (Fred Beckhusen)

Ethan Gardener
On Mon, Aug 27, 2018, at 8:06 AM, Haravikk wrote:
>
> > On 26 Aug 2018, at 15:05, Ethan Gardener <[hidden email]> wrote:
> >
> > However, everything I've ever learned about databases says that it's a bad idea to just look at the cost of the disks.  The cost of *accessing* the data (with acceptable performance) goes up exponentially with the size of the dataset.
>
> This shouldn't be the case; any properly indexed database should have
> roughly constant, or at worst logarithmic, access times for data.

*sigh* "Logarithmic" is another way of saying "exponential'.  It is not a trivial matter unless the data set is comparatively small.  In asking about these things, I'm particularly thinking of InWorldz, where asset data was growing at a terabyte a month.  Not small.  

> You
> should only really see big performance drops if the number of assets
> requests increases beyond what the hardware can cope with (i.e- too many
> users online simultaneously).
>
> > There is a solution which I think practically returns the problem to just counting the cost of the disks:  Cache the asset server.  The actual asset server which never deletes anything can then be very slow, with all regularly used assets served from the much faster cache.  
> >
> > I'm almost certain Second Life implemented such a cache years ago.  I remember when I logged in there after a long absence, things which I'd made or which were old-style took minutes to rez.  I don't ever remember seeing such an effect in InWorldz, so I think it's possible InWorldz didn't have such a cache.
>
> Most (all?) external asset requests are now served to the viewer via
> HTTP, so you could use Squid proxy or similar to cache those requests
> before they touch your asset server, it might even be possible to use a
> reverse proxy such as Cloudflare to do this for you, though I've never
> tried it (you might need to use page rules to force it to cache asset
> URLs).

I'm sure this would help, but wouldn't it miss the case of large vehicles making sim crossings?  A vehicle may be non-physical and not phantom, so the sim needs to know what shape it is and the shape may be complex.  In any case, sim crossings seem to be the first thing to go as a grid grows.
_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Are unused assets deleted? (Fred Beckhusen)

Haravikk

> On 27 Aug 2018, at 09:57, Ethan Gardener <[hidden email] <mailto:[hidden email]>> wrote:
>
> On Mon, Aug 27, 2018, at 8:06 AM, Haravikk wrote:
>>
>>> On 26 Aug 2018, at 15:05, Ethan Gardener <[hidden email] <mailto:[hidden email]>> wrote:
>>>
>>> However, everything I've ever learned about databases says that it's a bad idea to just look at the cost of the disks.  The cost of *accessing* the data (with acceptable performance) goes up exponentially with the size of the dataset.
>>
>> This shouldn't be the case; any properly indexed database should have
>> roughly constant, or at worst logarithmic, access times for data.
>
> *sigh* "Logarithmic" is another way of saying "exponential'.  It is not a trivial matter unless the data set is comparatively small.  In asking about these things, I'm particularly thinking of InWorldz, where asset data was growing at a terabyte a month.  Not small.  

Logarithmic complexity isn't the same as exponential complexity, in fact it's the opposite; logarithmic complexity plateaus as the search space increases, meaning it effectively becomes constant time, an algorithm with exponential complexity would grow by a power of the search space size (e.g- the search time might quadruple each time the search space doubles in size). See here:
https://en.wikipedia.org/wiki/Logarithmic_growth <https://en.wikipedia.org/wiki/Logarithmic_growth>
https://en.wikipedia.org/wiki/Exponential_growth <https://en.wikipedia.org/wiki/Exponential_growth>

A logarithmic algorithm is usually one that partitions the search space, for example a binary search which divides the search space in two with each iteration until only the target result (or nothing) remains. Some databases do use such algorithms but usually for intermediate operations that asset lookups shouldn't normally require.

For the vast majority of asset lookups (should be all of them, but it may depend on database used) a lookup will be constant time, e.g- using a form of hash-table to immediately find (or not) a request GUID. The amount of data stored shouldn't have a big impact on such lookups, the only case where you might notice a difference is if previously the entire index fit within RAM, but after growing no longer does (so has to come from disk), but most database engines are optimised on the assumption that you'll never have enough RAM to rely on keeping a whole index in memory anyway.

>> Most (all?) external asset requests are now served to the viewer via
>> HTTP, so you could use Squid proxy or similar to cache those requests
>> before they touch your asset server, it might even be possible to use a
>> reverse proxy such as Cloudflare to do this for you, though I've never
>> tried it (you might need to use page rules to force it to cache asset
>> URLs).
>
> I'm sure this would help, but wouldn't it miss the case of large vehicles making sim crossings?  A vehicle may be non-physical and not phantom, so the sim needs to know what shape it is and the shape may be complex.  In any case, sim crossings seem to be the first thing to go as a grid grows.

I'm not overly familiar with how OpenSim handles inter-region asset exchanges; but with regions belonging to the same grid there should be no need to send the asset to the asset server, it should either be sent directly to the destination region, or be retained by the origin region on failure. An increase in failures/laggy crossings would normally be an issue of a slow or overloaded connection between regions, or the destination region itself being overloaded.

In cases where a region does send an asset to the asset server a cache wouldn't help anyway, as you'd be storing a new asset (or overwriting an existing one, in the case of worn attachments). But this again wouldn't be an issue with the size of the asset server, but the amount of traffic being sent to it.
_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users