Need help with a serious LAG problem

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Need help with a serious LAG problem

tringate
I am at a loss for where to look for a serious LAG problem on more than one of my regions.
I run 3 servers, each run individual instances for the regions they host.
Normally things run fairly good but for the past couple of weeks LAG has become a real problem.
Today with three avatars on one region we could not move.  I rebooted the server to restart all the regions on it with no help at all for the LAG problem.
I then deleted the region,, dropped the database, and restarted it to make it a bump in the ocean kind of region.
The three of us returned, and now we could move.  Prior to doing this I turned off scripts and in fact all scripts did stop.  That had no effect on the region at all.
I then restored the original region with a database restore and went to the region alone.  Everything was just fine.  I then asked one of the people to return and as soon as that avatar started to arrive with the white puff of smoke I was frozen and could not move.  After about five minutes, and she has rezzed, we once again could both move about.  We asked the third person to come back and at that point all avatars were frozen in place and it took almost 10 minutes before we could move at all.  Unfortunately once we started to move we could not stop.
 
I decided to use the return objects function in “about land” to clear the region and it took forever for that function to even display the lines where you could return objects.  I finally succeeded in returning all the objects, but that did not solve the lag problem.  The region now did not have a single prime on it.
 
It was then I decided to look at the log file and see if anything there might tell me what was going on.
 
I see hundreds of the following types of messages in the log file as each avatar arrived.
 
2015-03-17 17:47:20,867 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #24386, type AgentCachedTexture from lila stone
 
Page after page of them.  This tells me something must be going wrong with my internet but what?  Everything else works just fine.  Both the other things running on the servers, and the many devices we have in our home using the internet.  We have Verizon FIOS 75/75 so we have plenty of speed.  My opensim ran perfectly on my old roadrunner service back a few years ago and that was a 3mb service.  Hard to say what speed I had on the outbound packets.
 
I need help looking for what is causing this problem but do not know where to start.
 
Other symptoms of the problem are when you type into chat and it takes 20-30 seconds to appear in the chat window.  I see many timeouts with communications to OSgrid as well.
 
Today the LAG was not just on my regions, I went to wright plaza but could never actually arrive, the viewer simply closed down as I arrived and I saw a brief glimpse of the plaza.  I went to some other plaza, can’t remember its name and could hardly move there either.
 
Other people are not having this problem it appears, so it is focused on anything that is using my internet connection I believe.  I have not changed anything for over two years, no hardware, or configuration settings.
 
HELP what do I look for?
 
Oh yes, the servers are 8 core AMD machines with 8GB ram with only 35% of memory used.  Looking at “top” they are not even loafing let alone busy.
Right now it says the server I have the really bad region running on is 97% idle.
 
If someone wants to experience it for themselves, go to Tranquil Bay, it is the worst of the land regions I am running and the one we worked with today.  It has around 7,000 prims on it.  That is less than most of my regions.
 
Tom
 
 
 

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Need help with a serious LAG problem

Dahlia Trimble
I went there, everything loaded fairly quickly and I didn't notice any lag except for a breif moment when I opened the map.

On Tue, Mar 17, 2015 at 5:04 PM, Thomas Ringate <[hidden email]> wrote:
I am at a loss for where to look for a serious LAG problem on more than one of my regions.
I run 3 servers, each run individual instances for the regions they host.
Normally things run fairly good but for the past couple of weeks LAG has become a real problem.
Today with three avatars on one region we could not move.  I rebooted the server to restart all the regions on it with no help at all for the LAG problem.
I then deleted the region,, dropped the database, and restarted it to make it a bump in the ocean kind of region.
The three of us returned, and now we could move.  Prior to doing this I turned off scripts and in fact all scripts did stop.  That had no effect on the region at all.
I then restored the original region with a database restore and went to the region alone.  Everything was just fine.  I then asked one of the people to return and as soon as that avatar started to arrive with the white puff of smoke I was frozen and could not move.  After about five minutes, and she has rezzed, we once again could both move about.  We asked the third person to come back and at that point all avatars were frozen in place and it took almost 10 minutes before we could move at all.  Unfortunately once we started to move we could not stop.
 
I decided to use the return objects function in “about land” to clear the region and it took forever for that function to even display the lines where you could return objects.  I finally succeeded in returning all the objects, but that did not solve the lag problem.  The region now did not have a single prime on it.
 
It was then I decided to look at the log file and see if anything there might tell me what was going on.
 
I see hundreds of the following types of messages in the log file as each avatar arrived.
 
2015-03-17 17:47:20,867 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #24386, type AgentCachedTexture from lila stone
 
Page after page of them.  This tells me something must be going wrong with my internet but what?  Everything else works just fine.  Both the other things running on the servers, and the many devices we have in our home using the internet.  We have Verizon FIOS 75/75 so we have plenty of speed.  My opensim ran perfectly on my old roadrunner service back a few years ago and that was a 3mb service.  Hard to say what speed I had on the outbound packets.
 
I need help looking for what is causing this problem but do not know where to start.
 
Other symptoms of the problem are when you type into chat and it takes 20-30 seconds to appear in the chat window.  I see many timeouts with communications to OSgrid as well.
 
Today the LAG was not just on my regions, I went to wright plaza but could never actually arrive, the viewer simply closed down as I arrived and I saw a brief glimpse of the plaza.  I went to some other plaza, can’t remember its name and could hardly move there either.
 
Other people are not having this problem it appears, so it is focused on anything that is using my internet connection I believe.  I have not changed anything for over two years, no hardware, or configuration settings.
 
HELP what do I look for?
 
Oh yes, the servers are 8 core AMD machines with 8GB ram with only 35% of memory used.  Looking at “top” they are not even loafing let alone busy.
Right now it says the server I have the really bad region running on is 97% idle.
 
If someone wants to experience it for themselves, go to Tranquil Bay, it is the worst of the land regions I am running and the one we worked with today.  It has around 7,000 prims on it.  That is less than most of my regions.
 
Tom
 
 
 

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users



_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Need help with a serious LAG problem

tringate
I see you logged into the region and the only errors I saw in the log were these:
2015-03-17 20:26:20,450 WARN  - OpenSim.Region.CoreModules.World.WorldMap.WorldMapModule [WORLD MAP]: Bad send on GetMapItems Error: ConnectFailure (Connection refused)
2015-03-17 20:26:20,450 WARN  - OpenSim.Region.CoreModules.World.WorldMap.WorldMapModule [WORLD MAP]: Blacklisted http://tsim.us.to:9266/MAP/MapItems/11046793326821120
2015-03-17 20:26:22,484 WARN  - OpenSim.Region.CoreModules.World.WorldMap.WorldMapModule [WORLD MAP]: Bad send on GetMapItems Error: ConnectFailure (Connection refused)
2015-03-17 20:26:22,485 WARN  - OpenSim.Region.CoreModules.World.WorldMap.WorldMapModule [WORLD MAP]: Blacklisted http://198.27.81.182:54300/MAP/MapItems/11045693815192320
2015-03-17 20:26:22,486 DEBUG - OpenSim.Region.CoreModules.World.WorldMap.WorldMapModule [WORLD MAP MODULE] RequestMapBlocks min=<10051,9980>, max=<10051,9980>, flag=10002, cntFound=0
2015-03-17 20:26:22,610 WARN  - OpenSim.Region.CoreModules.World.WorldMap.WorldMapModule [WORLD MAP]: Bad send on GetMapItems Error: ConnectFailure (Connection refused)
2015-03-17 20:26:22,610 WARN  - OpenSim.Region.CoreModules.World.WorldMap.WorldMapModule [WORLD MAP]: Bad send on GetMapItems Error: ConnectFailure (Connection refused)
2015-03-17 20:26:22,610 WARN  - OpenSim.Region.CoreModules.World.WorldMap.WorldMapModule [WORLD MAP]: Blacklisted http://198.27.80.157:7626/MAP/MapItems/11043494791936256
2015-03-17 20:26:22,610 WARN  - OpenSim.Region.CoreModules.World.WorldMap.WorldMapModule [WORLD MAP]: Blacklisted http://198.27.81.182:54300/MAP/MapItems/11043494791936768
2015-03-17 20:26:22,650 WARN  - OpenSim.Region.CoreModules.World.WorldMap.WorldMapModule [WORLD MAP]: Bad send on GetMapItems Error: ConnectFailure (Connection refused)
2015-03-17 20:26:22,650 WARN  - OpenSim.Region.CoreModules.World.WorldMap.WorldMapModule [WORLD MAP]: Blacklisted http://tsim.us.to:9265/MAP/MapItems/11046793326821376
2015-03-17 20:26:22,958 WARN  - OpenSim.Region.CoreModules.World.WorldMap.WorldMapModule [WORLD MAP]: Bad send on GetMapItems Error: ConnectFailure (Connection refused)
2015-03-17 20:26:22,958 WARN  - OpenSim.Region.CoreModules.World.WorldMap.WorldMapModule [WORLD MAP]: Blacklisted http://198.27.80.157:7626/MAP/MapItems/11045693815191808
2015-03-17 20:26:22,990 WARN  - OpenSim.Region.CoreModules.World.WorldMap.WorldMapModule [WORLD MAP]: Bad send on GetMapItems Error: ConnectFailure (Connection refused)
2015-03-17 20:26:22,990 WARN  - OpenSim.Region.CoreModules.World.WorldMap.WorldMapModule [WORLD MAP]: Blacklisted http://tsim.us.to:9267/MAP/MapItems/11047892838448896
2015-03-17 20:26:23,690 WARN  - OpenSim.Region.CoreModules.World.WorldMap.WorldMapModule [WORLD MAP]: Bad send on GetMapItems Error: ConnectFailure (Connection refused)
2015-03-17 20:26:23,690 WARN  - OpenSim.Region.CoreModules.World.WorldMap.WorldMapModule [WORLD MAP]: Blacklisted http://198.27.80.157:7626/MAP/MapItems/11047892838447360
2015-03-17 20:26:24,948 DEBUG - OpenSim.Region.CoreModules.World.WorldMap.WorldMapModule [WORLD MAP]: Remote sim does not have the stated region. Blacklisting.
2015-03-17 20:26:25,530 DEBUG - OpenSim.Region.CoreModules.World.WorldMap.WorldMapModule [WORLD MAP]: Remote sim does not have the stated region. Blacklisting.
2015-03-17 20:26:28,078 DEBUG - OpenSim.Region.CoreModules.World.WorldMap.WorldMapModule [WORLD MAP]: Remote sim does not have the stated region. Blacklisting.
2015-03-17 20:26:30,162 DEBUG - OpenSim.Region.CoreModules.World.WorldMap.WorldMapModule [WORLD MAP]: Remote sim does not have the stated region. Blacklisting.
 
I think there needs to be more than one avatar on the region for the problem to show itself.
 
I will go there again right now if you can give it another try.
 
Tom
 
Sent: Tuesday, March 17, 2015 8:29 PM
Subject: Re: [Opensim-users] Need help with a serious LAG problem
 
I went there, everything loaded fairly quickly and I didn't notice any lag except for a breif moment when I opened the map.
 
On Tue, Mar 17, 2015 at 5:04 PM, Thomas Ringate <[hidden email]> wrote:
I am at a loss for where to look for a serious LAG problem on more than one of my regions.
I run 3 servers, each run individual instances for the regions they host.
Normally things run fairly good but for the past couple of weeks LAG has become a real problem.
Today with three avatars on one region we could not move.  I rebooted the server to restart all the regions on it with no help at all for the LAG problem.
I then deleted the region,, dropped the database, and restarted it to make it a bump in the ocean kind of region.
The three of us returned, and now we could move.  Prior to doing this I turned off scripts and in fact all scripts did stop.  That had no effect on the region at all.
I then restored the original region with a database restore and went to the region alone.  Everything was just fine.  I then asked one of the people to return and as soon as that avatar started to arrive with the white puff of smoke I was frozen and could not move.  After about five minutes, and she has rezzed, we once again could both move about.  We asked the third person to come back and at that point all avatars were frozen in place and it took almost 10 minutes before we could move at all.  Unfortunately once we started to move we could not stop.
 
I decided to use the return objects function in “about land” to clear the region and it took forever for that function to even display the lines where you could return objects.  I finally succeeded in returning all the objects, but that did not solve the lag problem.  The region now did not have a single prime on it.
 
It was then I decided to look at the log file and see if anything there might tell me what was going on.
 
I see hundreds of the following types of messages in the log file as each avatar arrived.
 
2015-03-17 17:47:20,867 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #24386, type AgentCachedTexture from lila stone
 
Page after page of them.  This tells me something must be going wrong with my internet but what?  Everything else works just fine.  Both the other things running on the servers, and the many devices we have in our home using the internet.  We have Verizon FIOS 75/75 so we have plenty of speed.  My opensim ran perfectly on my old roadrunner service back a few years ago and that was a 3mb service.  Hard to say what speed I had on the outbound packets.
 
I need help looking for what is causing this problem but do not know where to start.
 
Other symptoms of the problem are when you type into chat and it takes 20-30 seconds to appear in the chat window.  I see many timeouts with communications to OSgrid as well.
 
Today the LAG was not just on my regions, I went to wright plaza but could never actually arrive, the viewer simply closed down as I arrived and I saw a brief glimpse of the plaza.  I went to some other plaza, can’t remember its name and could hardly move there either.
 
Other people are not having this problem it appears, so it is focused on anything that is using my internet connection I believe.  I have not changed anything for over two years, no hardware, or configuration settings.
 
HELP what do I look for?
 
Oh yes, the servers are 8 core AMD machines with 8GB ram with only 35% of memory used.  Looking at “top” they are not even loafing let alone busy.
Right now it says the server I have the really bad region running on is 97% idle.
 
If someone wants to experience it for themselves, go to Tranquil Bay, it is the worst of the land regions I am running and the one we worked with today.  It has around 7,000 prims on it.  That is less than most of my regions.
 
Tom
 
 
 

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users

 


_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Need help with a serious LAG problem

tringate
In reply to this post by Dahlia Trimble
Just had 6 avatars on the region and when the lag hit and I could not move the log had a bunch of these entries.  Eventually everyone could finally move after several minutes.  I watched the idle time on the server and it never went below 95% so the server never had a busy time.  I am stumped.  It looks like my acks to inbound messages are not getting to the sending node so they send the same packets over and over again.  Just guessing at that.
 
 
2015-03-17 20:45:15,896 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #551, type RequestObjectPropertiesFamily from Michellie sky
2015-03-17 20:45:16,024 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #6717, type AgentAnimation from Sydney Idealist
2015-03-17 20:45:16,046 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #6718, type AgentAnimation from Sydney Idealist
2015-03-17 20:45:16,902 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1491, type AgentAnimation from Rob Stone
2015-03-17 20:45:16,969 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1496, type TransferRequest from Rob Stone
2015-03-17 20:45:17,079 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1502, type RequestObjectPropertiesFamily from Rob Stone
2015-03-17 20:45:17,091 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1503, type AvatarPropertiesRequest from Rob Stone
2015-03-17 20:45:17,121 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1504, type GenericMessage from Rob Stone
2015-03-17 20:45:17,587 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #6728, type AgentAnimation from Slow Putzo
2015-03-17 20:45:17,587 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #6729, type AgentAnimation from Slow Putzo
2015-03-17 20:45:17,762 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #73, type AgentAnimation from shaun marsh
2015-03-17 20:45:17,763 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #74, type TransferRequest from shaun marsh
2015-03-17 20:45:17,811 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1272, type RequestObjectPropertiesFamily from Rob Stone
2015-03-17 20:45:17,813 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #4987, type ImprovedInstantMessage from Jessie Campbell
2015-03-17 20:45:17,815 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1280, type RequestObjectPropertiesFamily from Rob Stone
2015-03-17 20:45:17,993 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #5128, type ImprovedInstantMessage from Jessie Campbell
2015-03-17 20:45:19,275 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1297, type AgentAnimation from Rob Stone
2015-03-17 20:45:19,306 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1298, type AgentAnimation from Rob Stone
2015-03-17 20:45:19,425 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #625, type AgentAnimation from Michellie sky
2015-03-17 20:45:19,841 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1301, type GenericMessage from Rob Stone
2015-03-17 20:45:20,657 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #571, type AgentAnimation from Michellie sky
2015-03-17 20:45:21,822 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #5157, type AgentAnimation from Jessie Campbell
2015-03-17 20:45:21,822 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #5158, type AgentAnimation from Jessie Campbell
2015-03-17 20:45:22,015 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1631, type AgentAnimation from Rob Stone
2015-03-17 20:45:22,046 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #5164, type ImprovedInstantMessage from Jessie Campbell
2015-03-17 20:45:22,314 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #6870, type AgentAnimation from Slow Putzo
2015-03-17 20:45:22,315 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #6871, type ChatFromViewer from Slow Putzo
2015-03-17 20:45:22,329 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #6872, type AgentAnimation from Slow Putzo
2015-03-17 20:45:22,330 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #6873, type ChatFromViewer from Slow Putzo
2015-03-17 20:45:22,805 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #639, type AgentAnimation from Michellie sky
2015-03-17 20:45:22,847 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1654, type RequestObjectPropertiesFamily from Rob Stone
2015-03-17 20:45:23,301 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1672, type RequestObjectPropertiesFamily from Rob Stone
2015-03-17 20:45:23,665 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #645, type RequestObjectPropertiesFamily from Michellie sky
2015-03-17 20:45:24,064 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1712, type RequestObjectPropertiesFamily from Rob Stone
2015-03-17 20:45:24,096 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #5188, type ImprovedInstantMessage from Jessie Campbell
2015-03-17 20:45:24,246 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1724, type RequestObjectPropertiesFamily from Rob Stone
2015-03-17 20:45:24,547 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #6621, type AgentAnimation from Sydney Idealist
2015-03-17 20:45:24,893 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #6914, type ParcelPropertiesRequest from Slow Putzo
 
Sent: Tuesday, March 17, 2015 8:29 PM
Subject: Re: [Opensim-users] Need help with a serious LAG problem
 
I went there, everything loaded fairly quickly and I didn't notice any lag except for a breif moment when I opened the map.
 
On Tue, Mar 17, 2015 at 5:04 PM, Thomas Ringate <[hidden email]> wrote:
I am at a loss for where to look for a serious LAG problem on more than one of my regions.
I run 3 servers, each run individual instances for the regions they host.
Normally things run fairly good but for the past couple of weeks LAG has become a real problem.
Today with three avatars on one region we could not move.  I rebooted the server to restart all the regions on it with no help at all for the LAG problem.
I then deleted the region,, dropped the database, and restarted it to make it a bump in the ocean kind of region.
The three of us returned, and now we could move.  Prior to doing this I turned off scripts and in fact all scripts did stop.  That had no effect on the region at all.
I then restored the original region with a database restore and went to the region alone.  Everything was just fine.  I then asked one of the people to return and as soon as that avatar started to arrive with the white puff of smoke I was frozen and could not move.  After about five minutes, and she has rezzed, we once again could both move about.  We asked the third person to come back and at that point all avatars were frozen in place and it took almost 10 minutes before we could move at all.  Unfortunately once we started to move we could not stop.
 
I decided to use the return objects function in “about land” to clear the region and it took forever for that function to even display the lines where you could return objects.  I finally succeeded in returning all the objects, but that did not solve the lag problem.  The region now did not have a single prime on it.
 
It was then I decided to look at the log file and see if anything there might tell me what was going on.
 
I see hundreds of the following types of messages in the log file as each avatar arrived.
 
2015-03-17 17:47:20,867 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #24386, type AgentCachedTexture from lila stone
 
Page after page of them.  This tells me something must be going wrong with my internet but what?  Everything else works just fine.  Both the other things running on the servers, and the many devices we have in our home using the internet.  We have Verizon FIOS 75/75 so we have plenty of speed.  My opensim ran perfectly on my old roadrunner service back a few years ago and that was a 3mb service.  Hard to say what speed I had on the outbound packets.
 
I need help looking for what is causing this problem but do not know where to start.
 
Other symptoms of the problem are when you type into chat and it takes 20-30 seconds to appear in the chat window.  I see many timeouts with communications to OSgrid as well.
 
Today the LAG was not just on my regions, I went to wright plaza but could never actually arrive, the viewer simply closed down as I arrived and I saw a brief glimpse of the plaza.  I went to some other plaza, can’t remember its name and could hardly move there either.
 
Other people are not having this problem it appears, so it is focused on anything that is using my internet connection I believe.  I have not changed anything for over two years, no hardware, or configuration settings.
 
HELP what do I look for?
 
Oh yes, the servers are 8 core AMD machines with 8GB ram with only 35% of memory used.  Looking at “top” they are not even loafing let alone busy.
Right now it says the server I have the really bad region running on is 97% idle.
 
If someone wants to experience it for themselves, go to Tranquil Bay, it is the worst of the land regions I am running and the one we worked with today.  It has around 7,000 prims on it.  That is less than most of my regions.
 
Tom
 
 
 

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users

 


_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Need help with a serious LAG problem

Dahlia Trimble
All I can say is it *looks like* a lot of packet loss or network dropouts. It probably could be something else but I've not seen logs like that with properly functioning networks.

On Tue, Mar 17, 2015 at 6:16 PM, Thomas Ringate <[hidden email]> wrote:
Just had 6 avatars on the region and when the lag hit and I could not move the log had a bunch of these entries.  Eventually everyone could finally move after several minutes.  I watched the idle time on the server and it never went below 95% so the server never had a busy time.  I am stumped.  It looks like my acks to inbound messages are not getting to the sending node so they send the same packets over and over again.  Just guessing at that.
 
 
2015-03-17 20:45:15,896 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #551, type RequestObjectPropertiesFamily from Michellie sky
2015-03-17 20:45:16,024 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #6717, type AgentAnimation from Sydney Idealist
2015-03-17 20:45:16,046 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #6718, type AgentAnimation from Sydney Idealist
2015-03-17 20:45:16,902 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1491, type AgentAnimation from Rob Stone
2015-03-17 20:45:16,969 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1496, type TransferRequest from Rob Stone
2015-03-17 20:45:17,079 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1502, type RequestObjectPropertiesFamily from Rob Stone
2015-03-17 20:45:17,091 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1503, type AvatarPropertiesRequest from Rob Stone
2015-03-17 20:45:17,121 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1504, type GenericMessage from Rob Stone
2015-03-17 20:45:17,587 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #6728, type AgentAnimation from Slow Putzo
2015-03-17 20:45:17,587 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #6729, type AgentAnimation from Slow Putzo
2015-03-17 20:45:17,762 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #73, type AgentAnimation from shaun marsh
2015-03-17 20:45:17,763 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #74, type TransferRequest from shaun marsh
2015-03-17 20:45:17,811 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1272, type RequestObjectPropertiesFamily from Rob Stone
2015-03-17 20:45:17,813 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #4987, type ImprovedInstantMessage from Jessie Campbell
2015-03-17 20:45:17,815 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1280, type RequestObjectPropertiesFamily from Rob Stone
2015-03-17 20:45:17,993 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #5128, type ImprovedInstantMessage from Jessie Campbell
2015-03-17 20:45:19,275 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1297, type AgentAnimation from Rob Stone
2015-03-17 20:45:19,306 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1298, type AgentAnimation from Rob Stone
2015-03-17 20:45:19,425 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #625, type AgentAnimation from Michellie sky
2015-03-17 20:45:19,841 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1301, type GenericMessage from Rob Stone
2015-03-17 20:45:20,657 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #571, type AgentAnimation from Michellie sky
2015-03-17 20:45:21,822 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #5157, type AgentAnimation from Jessie Campbell
2015-03-17 20:45:21,822 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #5158, type AgentAnimation from Jessie Campbell
2015-03-17 20:45:22,015 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1631, type AgentAnimation from Rob Stone
2015-03-17 20:45:22,046 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #5164, type ImprovedInstantMessage from Jessie Campbell
2015-03-17 20:45:22,314 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #6870, type AgentAnimation from Slow Putzo
2015-03-17 20:45:22,315 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #6871, type ChatFromViewer from Slow Putzo
2015-03-17 20:45:22,329 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #6872, type AgentAnimation from Slow Putzo
2015-03-17 20:45:22,330 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #6873, type ChatFromViewer from Slow Putzo
2015-03-17 20:45:22,805 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #639, type AgentAnimation from Michellie sky
2015-03-17 20:45:22,847 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1654, type RequestObjectPropertiesFamily from Rob Stone
2015-03-17 20:45:23,301 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1672, type RequestObjectPropertiesFamily from Rob Stone
2015-03-17 20:45:23,665 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #645, type RequestObjectPropertiesFamily from Michellie sky
2015-03-17 20:45:24,064 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1712, type RequestObjectPropertiesFamily from Rob Stone
2015-03-17 20:45:24,096 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #5188, type ImprovedInstantMessage from Jessie Campbell
2015-03-17 20:45:24,246 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1724, type RequestObjectPropertiesFamily from Rob Stone
2015-03-17 20:45:24,547 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #6621, type AgentAnimation from Sydney Idealist
2015-03-17 20:45:24,893 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #6914, type ParcelPropertiesRequest from Slow Putzo
 
Sent: Tuesday, March 17, 2015 8:29 PM
Subject: Re: [Opensim-users] Need help with a serious LAG problem
 
I went there, everything loaded fairly quickly and I didn't notice any lag except for a breif moment when I opened the map.
 
On Tue, Mar 17, 2015 at 5:04 PM, Thomas Ringate <[hidden email]> wrote:
I am at a loss for where to look for a serious LAG problem on more than one of my regions.
I run 3 servers, each run individual instances for the regions they host.
Normally things run fairly good but for the past couple of weeks LAG has become a real problem.
Today with three avatars on one region we could not move.  I rebooted the server to restart all the regions on it with no help at all for the LAG problem.
I then deleted the region,, dropped the database, and restarted it to make it a bump in the ocean kind of region.
The three of us returned, and now we could move.  Prior to doing this I turned off scripts and in fact all scripts did stop.  That had no effect on the region at all.
I then restored the original region with a database restore and went to the region alone.  Everything was just fine.  I then asked one of the people to return and as soon as that avatar started to arrive with the white puff of smoke I was frozen and could not move.  After about five minutes, and she has rezzed, we once again could both move about.  We asked the third person to come back and at that point all avatars were frozen in place and it took almost 10 minutes before we could move at all.  Unfortunately once we started to move we could not stop.
 
I decided to use the return objects function in “about land” to clear the region and it took forever for that function to even display the lines where you could return objects.  I finally succeeded in returning all the objects, but that did not solve the lag problem.  The region now did not have a single prime on it.
 
It was then I decided to look at the log file and see if anything there might tell me what was going on.
 
I see hundreds of the following types of messages in the log file as each avatar arrived.
 
2015-03-17 17:47:20,867 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #24386, type AgentCachedTexture from lila stone
 
Page after page of them.  This tells me something must be going wrong with my internet but what?  Everything else works just fine.  Both the other things running on the servers, and the many devices we have in our home using the internet.  We have Verizon FIOS 75/75 so we have plenty of speed.  My opensim ran perfectly on my old roadrunner service back a few years ago and that was a 3mb service.  Hard to say what speed I had on the outbound packets.
 
I need help looking for what is causing this problem but do not know where to start.
 
Other symptoms of the problem are when you type into chat and it takes 20-30 seconds to appear in the chat window.  I see many timeouts with communications to OSgrid as well.
 
Today the LAG was not just on my regions, I went to wright plaza but could never actually arrive, the viewer simply closed down as I arrived and I saw a brief glimpse of the plaza.  I went to some other plaza, can’t remember its name and could hardly move there either.
 
Other people are not having this problem it appears, so it is focused on anything that is using my internet connection I believe.  I have not changed anything for over two years, no hardware, or configuration settings.
 
HELP what do I look for?
 
Oh yes, the servers are 8 core AMD machines with 8GB ram with only 35% of memory used.  Looking at “top” they are not even loafing let alone busy.
Right now it says the server I have the really bad region running on is 97% idle.
 
If someone wants to experience it for themselves, go to Tranquil Bay, it is the worst of the land regions I am running and the one we worked with today.  It has around 7,000 prims on it.  That is less than most of my regions.
 
Tom
 
 
 

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users

 


_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users


_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users



_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users