ResendPackets()

Post by **Lokked** » Wed Jan 14, 2015 1:46 am

Take what about to say as a theory. I've resolved quite a few of the recent issues around crashing while researching this. The commits will have to wait for tomorrow, but let's share knowledge anyways.

First, the last few crashes at GetClient() (this is usually a mutex crash, but if you walk the stack, you'll run into GetClient()) are because we get the chunk from character->GetCurrentChunk() but we don't check that chunk is valid. This happens because if a users client crashes or they are DCd during chunking or login, they will have a chunk client, but not necessarily a chunk. Checking for chunk and skipping the loop or current scope will resolve this.

Second, I think we're missing something packet/opcode wise around out of sync packets and the re-sending of these. I believe OpCode 0x0400 means something like Acknowledge Resent Packets or Wrong Resent Packets Sent.

I know the VGClient is not accepting our attempt to respond to 0x0011 (Out of Order Packet, either this or 0x0019, not at PC), because when I purposefully out of sync a single packet, my particular Client object is stuck in an infinite loop of try to resend packets, sometimes the VGClient is quiet about it: it does not respond with "garbage"; sometimes it is noisy: it responds with 0x0400s and all sorts of nonsense. While the client is re-senting packets forever, it is not sending ANY other packets. This is why nothing seems to work while this is happening. This infinite loop continues until all 1000 spots in the Deque are filled and some server logic disconnects the user.

Couple other things I noticed with this 0x0400 packet and the other "garbage" that sometimes accompanies it:
The garbage always starts with an unknown uint16, and I think this is the sequence number that the client is missing. The next 2 bytes are always the same: 00 11 (which happens to be our Out of Sync OpCode, but in the wrong place in the packet. The next 2 bytes are unknown to me.
Sometimes one of these packets may be very large, but the same pattern repeats: 2 bytes, 00 11, then 2-4 bytes. I think these are either Acks to the Resent packets bunched together.

I ran out of energy tonight but will follow up with this theory tomorrow.

Oh, why is this happening now? I believe this has ALWAYS been happening, and is responsible for nearly all the random CTDs we've encountered that weren't bugged World Heartbeat related. This is as random as you get, it just takes 1 missed packet, something which will NEVER happen on local dev, but can happen every so often over WAN. Previously, the client would just be silently disconnected. Now we see more logging spam.

Post by **John Adams** » Wed Jan 14, 2015 7:45 am

Brilliant analysis and theorizing. You have my attention. Maybe Xinux can provide insight into the packet flow as well, though I am sure you get that by now yourself.

One other thing I had thought of last night, in a sorta "duh" moment; why are we not asking these disco victims to log into NT using the collector? Would that help diagnose what THEY are seeing? And secondly, is there anything we can add to the world code (or outside, wire shark?) that can capture a days worth of logging packets to World, which we can then compare to collected logs to get a better idea of the communication breakdowns?

Personally, I'd rather see World able to diagnose itself but I fear adding a shit-ton of TRACE loggers to our code will only slow things down more. By this I mean a packet dump to an actual file, not console, during a full day of tracing. Nothing we'd do forever, although I think the real servers kept extensive logs of what was going on above our basic info, debug and errors.

Thoughts?

Post by **Xinux** » Wed Jan 14, 2015 10:49 am

FYI i updated the packet collector to spit out more information and also updated the opcodes if you want to use it to try and get some collects on this.

VGPacketCollector.rar: (101.09 KiB) Downloaded 149 times

Post by **Xinux** » Wed Jan 14, 2015 11:31 am

Ok here is what i'm seeing so far.

See this post

viewtopic.php?f=22&t=1180&p=9535#p9535

Volt · Post by **Volt** » Wed Jan 14, 2015 11:37 am

Lokked/all: Great writeup and great thinking! Sounds very reasonable. Is in line with why some players (i.e. locations/computers) have more issues than others.

Post by **Xinux** » Wed Jan 14, 2015 12:21 pm

Ok this is better i swapped to WPE Pro to do the collect.

Good response before out of order packet.