Re: Googlebot trying OpenSim /agent/<uuid> ?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Googlebot trying OpenSim /agent/<uuid> ?

aiaustin
I mentioned before that occasionally we see red "error" level
messages on our OpenSim.exe consoles of this format...

[AGENT HANDLER]: method GET not supported in agent message
/agent/e24a9015-f5ca-452b-8c95-d32e34cb9d64/, (caller is 66.249.64.223)

The UUID is an actual one for an avatar on our grid...

The caller is a Google Bot...

Name:    crawl-66-249-64-223.googlebot.com
Address:  66.249.64.223

It looks like its a simple attempt by Google to index pages it find
that mention this string in things like OpenSim mantis issue details
or comments, OpenSim mailing list web page archives, etc.

Should such things be red errors or just warnings?

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Googlebot trying OpenSim /agent/<uuid> ?

Trinity
may i suggest making a agents.txt available to feed to the google monster


On Mon, Aug 18, 2014 at 2:49 AM, Ai Austin <[hidden email]> wrote:
I mentioned before that occasionally we see red "error" level messages on our OpenSim.exe consoles of this format...

[AGENT HANDLER]: method GET not supported in agent message /agent/e24a9015-f5ca-452b-8c95-d32e34cb9d64/, (caller is 66.249.64.223)

The UUID is an actual one for an avatar on our grid...

The caller is a Google Bot...

Name:    crawl-66-249-64-223.googlebot.com
Address:  <a href="tel:66.249.64.223" value="+16624964223" target="_blank">66.249.64.223

It looks like its a simple attempt by Google to index pages it find that mention this string in things like OpenSim mantis issue details or comments, OpenSim mailing list web page archives, etc.

Should such things be red errors or just warnings?

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users


_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Googlebot trying OpenSim /agent/<uuid> ?

Melanie
Google is not supposed to index Opensim REST endpoints. Google's IP
should be blocked there, it's not an indexable web page.
Else, maybe Opensim should provide robots.txt with a generall "go
away" in it.

Melanie

On 18/08/2014 15:17, Trinity wrote:

> may i suggest making a agents.txt available to feed to the google monster
>
>
> On Mon, Aug 18, 2014 at 2:49 AM, Ai Austin <[hidden email]> wrote:
>
>> I mentioned before that occasionally we see red "error" level messages on
>> our OpenSim.exe consoles of this format...
>>
>> [AGENT HANDLER]: method GET not supported in agent message
>> /agent/e24a9015-f5ca-452b-8c95-d32e34cb9d64/, (caller is 66.249.64.223)
>>
>> The UUID is an actual one for an avatar on our grid...
>>
>> The caller is a Google Bot...
>>
>> Name:    crawl-66-249-64-223.googlebot.com
>> Address:  66.249.64.223
>>
>> It looks like its a simple attempt by Google to index pages it find that
>> mention this string in things like OpenSim mantis issue details or
>> comments, OpenSim mailing list web page archives, etc.
>>
>> Should such things be red errors or just warnings?
>>
>> _______________________________________________
>> Opensim-users mailing list
>> [hidden email]
>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>>
>
>
>
> _______________________________________________
> Opensim-users mailing list
> [hidden email]
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Googlebot trying OpenSim /agent/<uuid> ?

aiaustin
In reply to this post by aiaustin

>From: Trinity <[hidden email]>
>may i suggest making a agents.txt available to feed to the google monster


>From: Melanie <[hidden email]>
>Google is not supposed to index Opensim REST endpoints. Google's IP
>should be blocked there, it's not an indexable web page.
>Else, maybe Opensim should provide robots.txt with a generall "go
>away" in it.


Good idea Melanie and Trinity.  I assume robots.txt would need to be
incorporated in the distribution and dev code/resources as it will
not be something we can just add into a directory is it?

We get these on agent/<uuid> and object/<uuid> for entries that have
shown up in mantis issue texts and OpenSim mailing list archives.



_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Googlebot trying OpenSim /agent/<uuid> ?

aiaustin
In reply to this post by aiaustin
I assume something like this as robots.txt would do the job... but
the base HTTP support level would need to be changed to admit this?

User-agent: *
Disallow: /agent
Disallow: /object

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Googlebot trying OpenSim /agent/<uuid> ?

aiaustin
In reply to this post by aiaustin
Or perhaps the robots.txt should really just by default not want any
Google indexing at server level...

# go away
User-agent: *
Disallow: /

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users