Vague world crash
Moderators: Community Managers, Developers
- John Adams
- Retired
- Posts: 4581
- Joined: Wed Aug 28, 2013 9:40 am
- Location: Phoenix, AZ.
- Contact:
Vague world crash
Second time I've seen this in as many days. World crashing on character creation, in the most unhelpful place.
Code
Here: (*thread_args->func)(thread_args->arg);
Console:
[quote]11:07:55.670 I UDP New client connected from 46.166.188.232:61011
11:07:56.236 D Ability Success: 11 starting abilities set for character_id 648
11:07:56.237 D Char Successfully loaded 11 starter abilities.
11:07:56.238 D Char Adding default hotbar icons for race: 16, class: 4 for character id: 648
11:07:56.795 D Char Success: 6 starting hotbar icons set for character_id 648
11:07:56.796 D Char Successfully loaded 6 starter hotbar icons.
11:07:57.158 D Ability Success: 22 starting diplomacy cards set for char_id 648
11:07:57.159 W Guilds TODO: Adding new character to default guild (id: 1)
11:07:57.502 D Guilds Chiefswabee has been added to the guild member map.
11:07:57.503 D LS Character 'Chiefswabee' created with character ID 648[/quote]
Not much else to go on, except the memcpy ASM
Sure hope char create isn't broken with the recent code. I'm sure everyone is testing creates, especially when it comes to the new Attributes assignments?
Code
Code: Select all
/**
* Helper function to actually run the thread.
*
* This function is what is called by the actual thread API. This in turn will
* call the function pointer setup earlier in the custom \link ThreadArgs \unlink
* structure. After the thread finishes it will clean itself up.
*
* @param arg The thread arguments which include the function to run and any
* real arguments to pass onto the thread.
*/
static THREAD_RETURN_TYPE ThreadRun(void *arg) {
ThreadArgs *thread_args = (ThreadArgs *)arg;
(*thread_args->func)(thread_args->arg);
free(thread_args);
THREAD_EXIT();
}
Console:
[quote]11:07:55.670 I UDP New client connected from 46.166.188.232:61011
11:07:56.236 D Ability Success: 11 starting abilities set for character_id 648
11:07:56.237 D Char Successfully loaded 11 starter abilities.
11:07:56.238 D Char Adding default hotbar icons for race: 16, class: 4 for character id: 648
11:07:56.795 D Char Success: 6 starting hotbar icons set for character_id 648
11:07:56.796 D Char Successfully loaded 6 starter hotbar icons.
11:07:57.158 D Ability Success: 22 starting diplomacy cards set for char_id 648
11:07:57.159 W Guilds TODO: Adding new character to default guild (id: 1)
11:07:57.502 D Guilds Chiefswabee has been added to the guild member map.
11:07:57.503 D LS Character 'Chiefswabee' created with character ID 648[/quote]
Not much else to go on, except the memcpy ASM
Code: Select all
rep movsd ;N - move all of our dwords
Re: Vague world crash
That is definitely an unhelpful crash. I am, more then I'd like, creating new characters. I'm not really changing the code that loads/saves the attributes in the DB though. Most of my changes are in the manipulation of attributes.
I'm guessing this was a seg fault? If so it sounds like we have some memory corruption going on somehow/somwhere.
I'm guessing this was a seg fault? If so it sounds like we have some memory corruption going on somehow/somwhere.
- John Adams
- Retired
- Posts: 4581
- Joined: Wed Aug 28, 2013 9:40 am
- Location: Phoenix, AZ.
- Contact:
Re: Vague world crash
This is now happening daily, and not about character creation -- apparently random. And no one is around to reset the server, so our testers are out of luck until I notice.
This IS in the recent commits, so I will revert NT until I do not see it anymore, since it doesn't happen to anyone locally.
Stack was the same (unhelpful)
Console
[quote]17:40:51.863 D Combat Processing 54 Damage
17:40:53.849 D Combat Processing 6 Damage
17:40:54.629 D Ability Calling lua script for spell result
17:40:54.631 D Combat Processing 54 Damage
17:40:54.634 D Combat Player Doctor Wily killed Deepdark Lithovore, database id 1000095.
17:40:54.640 D Item Adding item 1413888 (Yukionna Chainmail) with unique_id 0 to pawn inventory.
17:40:54.766 D NPC Deepdark Lithovore Exiting Combat
17:40:54.934 D Char Regenerating HPs for Doctor Wily by 18.
17:40:57.371 D Item Sending Pawn Items to Loot Window
17:40:57.372 D Item Loot::SendToClient: entity_low: 44984, entity_high: 44984
17:40:58.341 D Item Loot::ClientLootItem: unique_id: 0
17:40:58.342 D Char Adding item #1413888 to character.
17:40:58.359 D Item Loot::ClientLootItem: Added item id 1413888 to character!
17:40:58.360 D Item Loot::ClientLootItem: Removed item id 1413888, unique_id 0 from pawn.
17:40:58.361 D Item Loot::ClientLootItem: Removing unique id 0 from loot window.
17:40:58.362 D Item Pawn Inventory Empty--closing loot window.
17:40:58.363 D Item Loot::ClientLootItem: Pawn Inventory Empty--removing visual.
17:40:59.971 D Char Regenerating HPs for Doctor Wily by 18.
17:41:18.105 D Chunk In ChunkServer::CheckSpawnRespawns: Calling master spawn list to respawn spawn.
17:41:58.854 D Chunk In ChunkServer::CheckSpawnRespawns: Calling master spawn list to respawn spawn.
17:43:17.035 I Chunk Shutdown timer expired. Shutting down Tar Janashir.
17:43:46.071 I Chunk Shutdown timer expired. Shutting down Jathred's Twist.
17:45:30.583 D Char Saved successful for character 'Doctor Wily' (650)
17:45:30.600 D Char Saved successful for character 'Vurgood' (59)[/quote]
This is happening in a level of C++ that I have no knowledge, and need people who understand this stuff to offer suggestions. This is a critical issue, and therefore is the highest priority to resolve.
This IS in the recent commits, so I will revert NT until I do not see it anymore, since it doesn't happen to anyone locally.
Stack was the same (unhelpful)
Console
[quote]17:40:51.863 D Combat Processing 54 Damage
17:40:53.849 D Combat Processing 6 Damage
17:40:54.629 D Ability Calling lua script for spell result
17:40:54.631 D Combat Processing 54 Damage
17:40:54.634 D Combat Player Doctor Wily killed Deepdark Lithovore, database id 1000095.
17:40:54.640 D Item Adding item 1413888 (Yukionna Chainmail) with unique_id 0 to pawn inventory.
17:40:54.766 D NPC Deepdark Lithovore Exiting Combat
17:40:54.934 D Char Regenerating HPs for Doctor Wily by 18.
17:40:57.371 D Item Sending Pawn Items to Loot Window
17:40:57.372 D Item Loot::SendToClient: entity_low: 44984, entity_high: 44984
17:40:58.341 D Item Loot::ClientLootItem: unique_id: 0
17:40:58.342 D Char Adding item #1413888 to character.
17:40:58.359 D Item Loot::ClientLootItem: Added item id 1413888 to character!
17:40:58.360 D Item Loot::ClientLootItem: Removed item id 1413888, unique_id 0 from pawn.
17:40:58.361 D Item Loot::ClientLootItem: Removing unique id 0 from loot window.
17:40:58.362 D Item Pawn Inventory Empty--closing loot window.
17:40:58.363 D Item Loot::ClientLootItem: Pawn Inventory Empty--removing visual.
17:40:59.971 D Char Regenerating HPs for Doctor Wily by 18.
17:41:18.105 D Chunk In ChunkServer::CheckSpawnRespawns: Calling master spawn list to respawn spawn.
17:41:58.854 D Chunk In ChunkServer::CheckSpawnRespawns: Calling master spawn list to respawn spawn.
17:43:17.035 I Chunk Shutdown timer expired. Shutting down Tar Janashir.
17:43:46.071 I Chunk Shutdown timer expired. Shutting down Jathred's Twist.
17:45:30.583 D Char Saved successful for character 'Doctor Wily' (650)
17:45:30.600 D Char Saved successful for character 'Vurgood' (59)[/quote]
This is happening in a level of C++ that I have no knowledge, and need people who understand this stuff to offer suggestions. This is a critical issue, and therefore is the highest priority to resolve.
Re: Vague world crash
Do you have any stack traces at all? Without one there really isn't anything I can do at all. At least with a stack trace I have a list of functions to try to trace the issue through.
Re: Vague world crash
I probally wouldn't revert just yet if you do it's going to be even harder to track down. I get alerts in chrome when the status of the server changes but i can only really keep a eye on it when I'm at work.
Re: Vague world crash
If there is some sort of ping packet that I can send then I can probably add NT to my nagios monitoring and send emails...
- John Adams
- Retired
- Posts: 4581
- Joined: Wed Aug 28, 2013 9:40 am
- Location: Phoenix, AZ.
- Contact:
Re: Vague world crash
[quote="Xinux"]I probally wouldn't revert just yet if you do it's going to be even harder to track down. I get alerts in chrome when the status of the server changes but i can only really keep a eye on it when I'm at work.[/quote]
The stack is 2 lines. Something about asm memcpy and ThreadRun, and one leads to the asm I posted up top and the other leads to the Code block I posted up top. There is no Usable stack, or I would have posted it.
There's nothing to trace, so reverting allows me to see what commit it came in.
The stack is 2 lines. Something about asm memcpy and ThreadRun, and one leads to the asm I posted up top and the other leads to the Code block I posted up top. There is no Usable stack, or I would have posted it.
There's nothing to trace, so reverting allows me to see what commit it came in.
- John Adams
- Retired
- Posts: 4581
- Joined: Wed Aug 28, 2013 9:40 am
- Location: Phoenix, AZ.
- Contact:
Re: Vague world crash
I'm reporting a bug in ThreadRun, and this is the latest commit? Maybe someone should look into this, though I'm pretty sure I saw this crash before this no-stack crash showed up.
[quote]Revision: 1501
Author: Ratief
Date: Tuesday, August 18, 2015 6:01:57 PM
Message:
-Moved all of the regen timers into a single function and a separate thread.
-Added class point regeneration.
-Fixed a couple of bugs in the attributes system.
----
Modified : /trunk/src/world/ChunkServer.h
Modified : /trunk/src/world/ChunkServer_core.cpp
Modified : /trunk/src/world/ChunkServer_process.cpp
Modified : /trunk/src/world/ChunkServer_spawn_looping.cpp
Modified : /trunk/src/world/WorldCharacter.cpp
Modified : /trunk/src/world/WorldCharacter.h
Modified : /trunk/src/world/WorldCharacter_Attributes.cpp[/quote]
[quote]Revision: 1501
Author: Ratief
Date: Tuesday, August 18, 2015 6:01:57 PM
Message:
-Moved all of the regen timers into a single function and a separate thread.
-Added class point regeneration.
-Fixed a couple of bugs in the attributes system.
----
Modified : /trunk/src/world/ChunkServer.h
Modified : /trunk/src/world/ChunkServer_core.cpp
Modified : /trunk/src/world/ChunkServer_process.cpp
Modified : /trunk/src/world/ChunkServer_spawn_looping.cpp
Modified : /trunk/src/world/WorldCharacter.cpp
Modified : /trunk/src/world/WorldCharacter.h
Modified : /trunk/src/world/WorldCharacter_Attributes.cpp[/quote]
Re: Vague world crash
I am using the ThreadStart function (which I think uses the ThreadRun function). I don't see how my usage could cause the crash that you are seeing though. I'm not doing any strange function pointer stuff, I'm just passing the function to be run. If it was crashing within my function that would be something else. I guess I could add more error checking though.
I will mention that I did have some strange crashes as I was working on this. The cause seemed to be that I needed to do a full clean/build for the code to pick up some of the changes to the thread function. I assumed that was because the thread function was static at that point (I changed that though) and wasn't getting relinked correctly.
Edit: I have been running my server for almost a day now trying to get it to crash like that, but so far no luck. I'm still trying though.
I will mention that I did have some strange crashes as I was working on this. The cause seemed to be that I needed to do a full clean/build for the code to pick up some of the changes to the thread function. I assumed that was because the thread function was static at that point (I changed that though) and wasn't getting relinked correctly.
Edit: I have been running my server for almost a day now trying to get it to crash like that, but so far no luck. I'm still trying though.
- John Adams
- Retired
- Posts: 4581
- Joined: Wed Aug 28, 2013 9:40 am
- Location: Phoenix, AZ.
- Contact:
Re: Vague world crash
New Telon is a Windows system, debugging via VS2012, and (because I want to eliminate all possible reasons) I 100% of the time do a Clean/Rebuild Solution during every update. I also do a make clean before any Linux updates.
A cliche' I've heard at my job every day for 20 years: "I don't see how there could ever be a crash in my code".
We'll get it, but this one is dastardly because it is giving us zero info to go on.
A cliche' I've heard at my job every day for 20 years: "I don't see how there could ever be a crash in my code".
We'll get it, but this one is dastardly because it is giving us zero info to go on.