Unusual MySql Timeouts - Ferd Frederix

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Unusual MySql Timeouts - Ferd Frederix

Fred Beckhusen
A user of stock Opensim (Robust or Standalone) 0.9.0 or 0.8.20 is having
unusual MySQL timeouts that I cannot explain or fix.  The timeout
happens when the BackupWorker thread kicks in to flush the data to
disk.  So it also happens when I manually type in 'backup' and it
happens when I type 'quit', too, both also trigger the backup.

Mysql:  She is running Mysql 5.5 on a dedicated server.  The
max_allowed_packet = 128M, and the database passes all mysqlupdate and
mysqlrepairs on mysql, robust and opensim databases with no errors.

The prims or item it fails on seem to be random.  As a result of the
backup thread aborting, items are always missing on the region after the
restart.   Nothing goes missing unless she shuts down and restarts.

One region loses the same part of the same house each time it reboots,
another region lose more than half the prims. This is somewhat
consistent.  But the same oar loads on other machines and works fine. It
is something with this particular machine.

Originally it seems to be due to a bad hard disk, but that has since
been replaced.  It happens on Server 2016, and on Windows 10.  The
server has 32 GB of RAM, an I7 CPU

The error will occur with just one of many regions when loaded with an
oar. The oars are in the 30K-45,000 prim range.

Any ideas on what to do?

16:53:52 - Timeout in threadfunc 13094 (STP:Util:40) (BackupWorker (Town
of Haven))    at System.Net.UnsafeNclNativeMethods.OSSOCK.recv(IntPtr
socketHandle, Byte* pinnedBuffer, Int32 len, SocketFlags socketFlags)
    at System.Net.UnsafeNclNativeMethods.OSSOCK.recv(IntPtr
socketHandle, Byte* pinnedBuffer, Int32 len, SocketFlags socketFlags)
    at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset,
Int32 size, SocketFlags socketFlags, SocketError& errorCode)
    at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32
offset, Int32 size)
    at MySql.Data.Common.MyNetworkStream.Read(Byte[] buffer, Int32
offset, Int32 count)
    at MySql.Data.MySqlClient.TimedStream.Read(Byte[] buffer, Int32
offset, Int32 count)
    at System.IO.BufferedStream.Read(Byte[] array, Int32 offset, Int32
count)
    at MySql.Data.MySqlClient.MySqlStream.ReadFully(Stream stream,
Byte[] buffer, Int32 offset, Int32 count)
    at MySql.Data.MySqlClient.MySqlStream.LoadPacket()
    at MySql.Data.MySqlClient.MySqlStream.ReadPacket()
    at MySql.Data.MySqlClient.NativeDriver.GetResult(Int32& affectedRow,
Int64& insertedId)
    at MySql.Data.MySqlClient.Driver.NextResult(Int32 statementId,
Boolean force)
    at MySql.Data.MySqlClient.MySqlDataReader.NextResult()
    at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader(CommandBehavior
behavior)
    at MySql.Data.MySqlClient.MySqlCommand.ExecuteNonQuery()
    at
OpenSim.Data.MySQL.MySQLSimulationData.ExecuteNonQuery(MySqlCommand c)
in
c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Data\MySQL\MySQLSimulationData.cs:line
116
    at
OpenSim.Data.MySQL.MySQLSimulationData.StoreObject(SceneObjectGroup obj,
UUID regionUUID) in
c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Data\MySQL\MySQLSimulationData.cs:line
262
    at
OpenSim.Services.SimulationService.SimulationDataService.StoreObject(SceneObjectGroup
obj, UUID regionUUID) in
c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Services\SimulationService\SimulationDataService.cs:line
84
    at
OpenSim.Region.Framework.Scenes.SceneObjectGroup.ProcessBackup(ISimulationDataService
datastore, Boolean forcedBackup) in
c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Region\Framework\Scenes\SceneObjectGroup.cs:line
2544
    at
OpenSim.Region.Framework.Scenes.EventManager.TriggerOnBackup(ISimulationDataService
dstore, Boolean forced) in
c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Region\Framework\Scenes\EventManager.cs:line
1410
    at OpenSim.Region.Framework.Scenes.Scene.Backup(Boolean forced) in
c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Region\Framework\Scenes\Scene.cs:line
1983
    at
OpenSim.Region.Framework.Scenes.Scene.<UpdateStorageBackup>b__1f(Object
o) in
c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Region\Framework\Scenes\Scene.cs:line
1946
    at
OpenSim.Framework.Util.<>c__DisplayClass9.<FireAndForget>b__2(Object o)
in
c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Framework\Util.cs:line
2675
    at OpenSim.Framework.Util.<FireAndForget>b__3(WaitCallback cb,
Object o) in
c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Framework\Util.cs:line
2761
    at
Amib.Threading.Internal.WorkItemsGroupBase.<>c__DisplayClass7`2.<QueueWorkItem>b__6(Object
state) in c:\Opensim\Opensim 0.9.0
release\opensim\ThirdParty\SmartThreadPool\WorkItemsGroupBase.cs:line 333
    at Amib.Threading.Internal.WorkItem.ExecuteWorkItem() in
c:\Opensim\Opensim 0.9.0
release\opensim\ThirdParty\SmartThreadPool\WorkItem.cs:line 379
    at Amib.Threading.Internal.WorkItem.Execute() in c:\Opensim\Opensim
0.9.0 release\opensim\ThirdParty\SmartThreadPool\WorkItem.cs:line 312
    at Amib.Threading.SmartThreadPool.ExecuteWorkItem(WorkItem workItem)
in c:\Opensim\Opensim 0.9.0
release\opensim\ThirdParty\SmartThreadPool\SmartThreadPool.cs:line 882
    at Amib.Threading.SmartThreadPool.ProcessQueuedItems() in
c:\Opensim\Opensim 0.9.0
release\opensim\ThirdParty\SmartThreadPool\SmartThreadPool.cs:line 822
    at System.Threading.ExecutionContext.RunInternal(ExecutionContext
executionContext, ContextCallback callback, Object state, Boolean
preserveSyncCtx)
    at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state, Boolean
preserveSyncCtx)
    at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state)
    at System.Threading.ThreadHelper.ThreadStart()

After the above warning,  it throws an error on items like this:

13:51:49 - [REGION DB]: MySQL error in ExecuteNonQuery: Thread was being
aborted.
13:51:49 - [SCENE]: Storing of salon-StylistCenter Stand&Drawers v1,
15f57bde-610b-43a9-97a1-946b31a72273 in Town of Haven failed with
exception Thread was being aborted. at
MySql.Data.Common.MyNetworkStream.HandleOrRethrowException(Exception e)
at MySql.Data.Common.MyNetworkStream............

Fred Beckhusen/Ferd Frederix
www.outworldz.com
[hidden email]

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Unusual MySql Timeouts - Ferd Frederix

AJLDuarte
Possible the damm threads timeout on threadpool
It is taking minutes to do the backup no?
( well in never seen that issue there..
Ubit


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Fred
Sent: Tuesday, January 16, 2018 05:05
To: [hidden email]
Subject: [Opensim-users] Unusual MySql Timeouts - Ferd Frederix

A user of stock Opensim (Robust or Standalone) 0.9.0 or 0.8.20 is having unusual MySQL timeouts that I cannot explain or fix.  The timeout happens when the BackupWorker thread kicks in to flush the data to disk.  So it also happens when I manually type in 'backup' and it happens when I type 'quit', too, both also trigger the backup.

Mysql:  She is running Mysql 5.5 on a dedicated server.  The max_allowed_packet = 128M, and the database passes all mysqlupdate and mysqlrepairs on mysql, robust and opensim databases with no errors.

The prims or item it fails on seem to be random.  As a result of the backup thread aborting, items are always missing on the region after the restart.   Nothing goes missing unless she shuts down and restarts.

One region loses the same part of the same house each time it reboots, another region lose more than half the prims. This is somewhat consistent.  But the same oar loads on other machines and works fine. It is something with this particular machine.

Originally it seems to be due to a bad hard disk, but that has since been replaced.  It happens on Server 2016, and on Windows 10.  The server has 32 GB of RAM, an I7 CPU

The error will occur with just one of many regions when loaded with an oar. The oars are in the 30K-45,000 prim range.

Any ideas on what to do?

16:53:52 - Timeout in threadfunc 13094 (STP:Util:40) (BackupWorker (Town of Haven))    at System.Net.UnsafeNclNativeMethods.OSSOCK.recv(IntPtr
socketHandle, Byte* pinnedBuffer, Int32 len, SocketFlags socketFlags)
    at System.Net.UnsafeNclNativeMethods.OSSOCK.recv(IntPtr
socketHandle, Byte* pinnedBuffer, Int32 len, SocketFlags socketFlags)
    at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset,
Int32 size, SocketFlags socketFlags, SocketError& errorCode)
    at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
    at MySql.Data.Common.MyNetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count)
    at MySql.Data.MySqlClient.TimedStream.Read(Byte[] buffer, Int32 offset, Int32 count)
    at System.IO.BufferedStream.Read(Byte[] array, Int32 offset, Int32
count)
    at MySql.Data.MySqlClient.MySqlStream.ReadFully(Stream stream, Byte[] buffer, Int32 offset, Int32 count)
    at MySql.Data.MySqlClient.MySqlStream.LoadPacket()
    at MySql.Data.MySqlClient.MySqlStream.ReadPacket()
    at MySql.Data.MySqlClient.NativeDriver.GetResult(Int32& affectedRow, Int64& insertedId)
    at MySql.Data.MySqlClient.Driver.NextResult(Int32 statementId, Boolean force)
    at MySql.Data.MySqlClient.MySqlDataReader.NextResult()
    at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader(CommandBehavior
behavior)
    at MySql.Data.MySqlClient.MySqlCommand.ExecuteNonQuery()
    at
OpenSim.Data.MySQL.MySQLSimulationData.ExecuteNonQuery(MySqlCommand c) in c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Data\MySQL\MySQLSimulationData.cs:line
116
    at
OpenSim.Data.MySQL.MySQLSimulationData.StoreObject(SceneObjectGroup obj, UUID regionUUID) in c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Data\MySQL\MySQLSimulationData.cs:line
262
    at
OpenSim.Services.SimulationService.SimulationDataService.StoreObject(SceneObjectGroup
obj, UUID regionUUID) in
c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Services\SimulationService\SimulationDataService.cs:line
84
    at
OpenSim.Region.Framework.Scenes.SceneObjectGroup.ProcessBackup(ISimulationDataService
datastore, Boolean forcedBackup) in
c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Region\Framework\Scenes\SceneObjectGroup.cs:line
2544
    at
OpenSim.Region.Framework.Scenes.EventManager.TriggerOnBackup(ISimulationDataService
dstore, Boolean forced) in
c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Region\Framework\Scenes\EventManager.cs:line
1410
    at OpenSim.Region.Framework.Scenes.Scene.Backup(Boolean forced) in c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Region\Framework\Scenes\Scene.cs:line
1983
    at
OpenSim.Region.Framework.Scenes.Scene.<UpdateStorageBackup>b__1f(Object
o) in
c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Region\Framework\Scenes\Scene.cs:line
1946
    at
OpenSim.Framework.Util.<>c__DisplayClass9.<FireAndForget>b__2(Object o) in c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Framework\Util.cs:line
2675
    at OpenSim.Framework.Util.<FireAndForget>b__3(WaitCallback cb, Object o) in c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Framework\Util.cs:line
2761
    at
Amib.Threading.Internal.WorkItemsGroupBase.<>c__DisplayClass7`2.<QueueWorkItem>b__6(Object
state) in c:\Opensim\Opensim 0.9.0
release\opensim\ThirdParty\SmartThreadPool\WorkItemsGroupBase.cs:line 333
    at Amib.Threading.Internal.WorkItem.ExecuteWorkItem() in c:\Opensim\Opensim 0.9.0 release\opensim\ThirdParty\SmartThreadPool\WorkItem.cs:line 379
    at Amib.Threading.Internal.WorkItem.Execute() in c:\Opensim\Opensim 0.9.0 release\opensim\ThirdParty\SmartThreadPool\WorkItem.cs:line 312
    at Amib.Threading.SmartThreadPool.ExecuteWorkItem(WorkItem workItem) in c:\Opensim\Opensim 0.9.0 release\opensim\ThirdParty\SmartThreadPool\SmartThreadPool.cs:line 882
    at Amib.Threading.SmartThreadPool.ProcessQueuedItems() in c:\Opensim\Opensim 0.9.0 release\opensim\ThirdParty\SmartThreadPool\SmartThreadPool.cs:line 822
    at System.Threading.ExecutionContext.RunInternal(ExecutionContext
executionContext, ContextCallback callback, Object state, Boolean
preserveSyncCtx)
    at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state, Boolean
preserveSyncCtx)
    at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state)
    at System.Threading.ThreadHelper.ThreadStart()

After the above warning,  it throws an error on items like this:

13:51:49 - [REGION DB]: MySQL error in ExecuteNonQuery: Thread was being aborted.
13:51:49 - [SCENE]: Storing of salon-StylistCenter Stand&Drawers v1,
15f57bde-610b-43a9-97a1-946b31a72273 in Town of Haven failed with exception Thread was being aborted. at MySql.Data.Common.MyNetworkStream.HandleOrRethrowException(Exception e) at MySql.Data.Common.MyNetworkStream............

Fred Beckhusen/Ferd Frederix
www.outworldz.com
[hidden email]

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
Reply | Threaded
Open this post in threaded view
|

Re: Unusual MySql Timeouts - Ferd Frederix

AJLDuarte
In reply to this post by Fred Beckhusen
Hi.
        Just removed on master the timeout on BackupWorker
Ubit


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Fred
Sent: Tuesday, January 16, 2018 05:05
To: [hidden email]
Subject: [Opensim-users] Unusual MySql Timeouts - Ferd Frederix

A user of stock Opensim (Robust or Standalone) 0.9.0 or 0.8.20 is having unusual MySQL timeouts that I cannot explain or fix.  The timeout happens when the BackupWorker thread kicks in to flush the data to disk.  So it also happens when I manually type in 'backup' and it happens when I type 'quit', too, both also trigger the backup.

Mysql:  She is running Mysql 5.5 on a dedicated server.  The max_allowed_packet = 128M, and the database passes all mysqlupdate and mysqlrepairs on mysql, robust and opensim databases with no errors.

The prims or item it fails on seem to be random.  As a result of the backup thread aborting, items are always missing on the region after the restart.   Nothing goes missing unless she shuts down and restarts.

One region loses the same part of the same house each time it reboots, another region lose more than half the prims. This is somewhat consistent.  But the same oar loads on other machines and works fine. It is something with this particular machine.

Originally it seems to be due to a bad hard disk, but that has since been replaced.  It happens on Server 2016, and on Windows 10.  The server has 32 GB of RAM, an I7 CPU

The error will occur with just one of many regions when loaded with an oar. The oars are in the 30K-45,000 prim range.

Any ideas on what to do?

16:53:52 - Timeout in threadfunc 13094 (STP:Util:40) (BackupWorker (Town of Haven))    at System.Net.UnsafeNclNativeMethods.OSSOCK.recv(IntPtr
socketHandle, Byte* pinnedBuffer, Int32 len, SocketFlags socketFlags)
    at System.Net.UnsafeNclNativeMethods.OSSOCK.recv(IntPtr
socketHandle, Byte* pinnedBuffer, Int32 len, SocketFlags socketFlags)
    at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset,
Int32 size, SocketFlags socketFlags, SocketError& errorCode)
    at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
    at MySql.Data.Common.MyNetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count)
    at MySql.Data.MySqlClient.TimedStream.Read(Byte[] buffer, Int32 offset, Int32 count)
    at System.IO.BufferedStream.Read(Byte[] array, Int32 offset, Int32
count)
    at MySql.Data.MySqlClient.MySqlStream.ReadFully(Stream stream, Byte[] buffer, Int32 offset, Int32 count)
    at MySql.Data.MySqlClient.MySqlStream.LoadPacket()
    at MySql.Data.MySqlClient.MySqlStream.ReadPacket()
    at MySql.Data.MySqlClient.NativeDriver.GetResult(Int32& affectedRow, Int64& insertedId)
    at MySql.Data.MySqlClient.Driver.NextResult(Int32 statementId, Boolean force)
    at MySql.Data.MySqlClient.MySqlDataReader.NextResult()
    at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader(CommandBehavior
behavior)
    at MySql.Data.MySqlClient.MySqlCommand.ExecuteNonQuery()
    at
OpenSim.Data.MySQL.MySQLSimulationData.ExecuteNonQuery(MySqlCommand c) in c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Data\MySQL\MySQLSimulationData.cs:line
116
    at
OpenSim.Data.MySQL.MySQLSimulationData.StoreObject(SceneObjectGroup obj, UUID regionUUID) in c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Data\MySQL\MySQLSimulationData.cs:line
262
    at
OpenSim.Services.SimulationService.SimulationDataService.StoreObject(SceneObjectGroup
obj, UUID regionUUID) in
c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Services\SimulationService\SimulationDataService.cs:line
84
    at
OpenSim.Region.Framework.Scenes.SceneObjectGroup.ProcessBackup(ISimulationDataService
datastore, Boolean forcedBackup) in
c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Region\Framework\Scenes\SceneObjectGroup.cs:line
2544
    at
OpenSim.Region.Framework.Scenes.EventManager.TriggerOnBackup(ISimulationDataService
dstore, Boolean forced) in
c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Region\Framework\Scenes\EventManager.cs:line
1410
    at OpenSim.Region.Framework.Scenes.Scene.Backup(Boolean forced) in c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Region\Framework\Scenes\Scene.cs:line
1983
    at
OpenSim.Region.Framework.Scenes.Scene.<UpdateStorageBackup>b__1f(Object
o) in
c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Region\Framework\Scenes\Scene.cs:line
1946
    at
OpenSim.Framework.Util.<>c__DisplayClass9.<FireAndForget>b__2(Object o) in c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Framework\Util.cs:line
2675
    at OpenSim.Framework.Util.<FireAndForget>b__3(WaitCallback cb, Object o) in c:\Opensim\Outworldz-Source\OutworldzFiles\opensim\OpenSim\Framework\Util.cs:line
2761
    at
Amib.Threading.Internal.WorkItemsGroupBase.<>c__DisplayClass7`2.<QueueWorkItem>b__6(Object
state) in c:\Opensim\Opensim 0.9.0
release\opensim\ThirdParty\SmartThreadPool\WorkItemsGroupBase.cs:line 333
    at Amib.Threading.Internal.WorkItem.ExecuteWorkItem() in c:\Opensim\Opensim 0.9.0 release\opensim\ThirdParty\SmartThreadPool\WorkItem.cs:line 379
    at Amib.Threading.Internal.WorkItem.Execute() in c:\Opensim\Opensim 0.9.0 release\opensim\ThirdParty\SmartThreadPool\WorkItem.cs:line 312
    at Amib.Threading.SmartThreadPool.ExecuteWorkItem(WorkItem workItem) in c:\Opensim\Opensim 0.9.0 release\opensim\ThirdParty\SmartThreadPool\SmartThreadPool.cs:line 882
    at Amib.Threading.SmartThreadPool.ProcessQueuedItems() in c:\Opensim\Opensim 0.9.0 release\opensim\ThirdParty\SmartThreadPool\SmartThreadPool.cs:line 822
    at System.Threading.ExecutionContext.RunInternal(ExecutionContext
executionContext, ContextCallback callback, Object state, Boolean
preserveSyncCtx)
    at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state, Boolean
preserveSyncCtx)
    at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state)
    at System.Threading.ThreadHelper.ThreadStart()

After the above warning,  it throws an error on items like this:

13:51:49 - [REGION DB]: MySQL error in ExecuteNonQuery: Thread was being aborted.
13:51:49 - [SCENE]: Storing of salon-StylistCenter Stand&Drawers v1,
15f57bde-610b-43a9-97a1-946b31a72273 in Town of Haven failed with exception Thread was being aborted. at MySql.Data.Common.MyNetworkStream.HandleOrRethrowException(Exception e) at MySql.Data.Common.MyNetworkStream............

Fred Beckhusen/Ferd Frederix
www.outworldz.com
[hidden email]

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users

_______________________________________________
Opensim-users mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users