Page 1 of 1

Dev Chat: 04/11/2014 - Multi-Chunk Support

Posted: Sat Apr 12, 2014 11:01 am
by John Adams
Some discussion about how to handle more than 1 running ChunkServer at at time, and deciding (at least for now) to Keep It Simple and just go with 1 ip:port combo for multiple chunkservers. Later we can implement "clustering".

2nd post is about visuals.

[quote][19:54] <@Xen> So scat, about multiple chunkserver's. For the port number, is it okay to just increment it each time a new one is added to the list? Is there anyway to check if the port is already bound?
[19:55] <@Scatman> there is a way to check if it's bound but i think i have a socket option enabled to not care
[19:55] <@Scatman> i think we should keep them all on the same port right now
[19:57] <@Xen> Hmm, so how does that work? Does each chunk know how to handle the clients connected to it?
[19:59] <@Scatman> gimme a sec to bring the project because up i have the memory of a goldfish. but i thought world server determined where the chunk was?
[20:00] <@Scatman> however, i am not familiar with chunk -> chunk communications
[20:03] <@Xen> When you say "where", what do you mean?
[20:05] <@Xen> Also, each chunkserver has a UDP attached to it and processes packets (at least how I understand it). If two chunk servers are running on the same port, then which one processes my clients packets that are being sent to that port?
[20:11] <@Scatman> xen i see what you're saying
[20:12] <@Scatman> do you know if the client disconnects from world once it's connected to a chunk?
[20:13] <@Xen> No idea.
[20:14] <@Scatman> so i designed it like how SOE was designed. but you're right. i might not be suitable for us.
[20:14] <@Xen> But I know the chunk port is sent to the client. I see this in my vanguard client log: "Log: Connecting to LVSVGO-01-14.vanguardsoh.net:10137 (chunk_n13_n9)"
[20:14] <@Scatman> yup
[20:16] <@Xen> A different log with a different chunk: Log: Connecting to LVSVGO-01-09.vanguardsoh.net:10138 (chunk_n26_1)
[20:16] <@Scatman> yeah it 00-14 (or maybe more)
[20:17] <@Xen> In the same log as the (chunk_n13_n9), I crossed a zoneline and this: "Log: Connecting to LVSVGO-01-03.vanguardsoh.net:10135 (chunk_n12_n9)"
[20:17] <@Scatman> i like the design where each chunk server is (or could be) on a different server but i'm trying to figure out how to make all chunk servers hit the same IP/port
[20:17] <@Xen> No idea if each chunk it tied to an individual port or not
[20:18] <@Scatman> they are. each chunkserver (is literally a server) can be on it's on box
[20:18] <@Xen> Yeah it would be kind of cool to be able to host different chunks on different servers
[20:18] <@Xen> That's how you basically have it setup now
[20:18] <@Scatman> yep
[20:19] <@Xen> And I'm fine with that, except I'm not sure how to find valid ports to bind to. For now I was just going to store a port variable in CHunkServerList and increment it each time a new chunk is added
[20:19] <@Scatman> i know. i overlooked that
[20:19] <@Scatman> thinking...
[20:19] <@Xen> Is there a problem with that?
[20:20] <@Xen> There are around 320 chunks. Can't we just say you have to have ports 10000 - 10340 free?
[20:20] <@Xen> Or let them define the first port number and we increment from that
[20:21] <@Scatman> if we do it that way, we just have to let our admins know that they need to open port x to x+n where n is the number of chunkservers :/
[20:21] <@Xen> Use that rule stuff John put in
[20:21] <@Scatman> give me the night to think about it
[20:22] <@Scatman> and research it
[20:22] <@Xen> Another option would be to have them all on the same port but keep a list of which client is in which chunk
[20:22] <@Scatman> i was thinking that too
[20:22] <@Xen> Then in the CHunkServer->Process() function forward it to the right one
[20:23] <@Scatman> our goal would be to let this run like SOE and let each chunkserver run on a different physical box. that'd be sweet
[20:24] <@Xen> Isn't that possible with the code you have now?
[20:24] <@Scatman> it is. absoutely
[20:24] <~john> Scat, I PM'd you extracted ClientLog's from a complete session. if it will help determine service connections.
[20:24] <@Xen> So what are we thinking about then?
[20:24] <@Scatman> how to solve the problem of an admin running multiple chunkservers on one box
[20:25] <@Scatman> unless we open up 300 ports?
[20:25] <@Xen> I say we just let them define the low end of the port range and we increment. Or we let them define a set of ranges (ie: 10000-10100, 12000-12400)
[20:26] <@Scatman> right.
[20:26] <@Scatman> we could round robin them too
[20:26] <@Xen> As chunkServers close, we dump those ports back into the "free" pool for the new chunkserver
[20:26] <@Scatman> or balance load 'em
[20:26] <@Scatman> john. is that how eq2emu works?
[20:27] <@Scatman> wasn't each zoneserver it's own UDP server?
[20:27] <@Scatman> i lost their source heh
[20:28] <~john> EQ2Emu is all in EQ2World.exe, no separate zoneserver exe's. so it's all under 1 port (incoming).
[20:28] <~john> EQEmu though uses 2 options, Dynamic Ports (reused) and Static Ports.
[20:29] <~john> My EQEmu server for example has 20 ports dynamically allocated to zoneservers. 20 max run at once. then, it's full. No more zones can start. but, they are always idle on a lonely server. player logs in, the server finds the 1st avalilable dynamic port and uses it.
[20:29] <~john> thoughts: First, no one is going to run 320 chunkservers. Ever. not on 1 box, not on a farm :D
[20:30] <~john> if we can utilize the round-robin or dynamic approach, we only need 'x' number defined (up to the admin).
[20:31] <~john> btw, Scat... recall my post about "Clustering"? :p
[20:31] <@Scatman> i thought by clustering you meant 1 chunkserver per ip :P which is what i implemented lol
[20:32] <@Scatman> just in their own exes
[20:32] <~john> yeah that's what we talked about.
[20:33] <~john> but it would be 1 physical box, with say 10 zones running.
[20:33] <~john> EQEmu would require me to open 10 ports to that LAN IP, by it's design. EQ2Emu has no design of this nature :D It's 1 port into World, and that's it.
[20:34] <@Scatman> right. but eq2emu also has no design of clustering
[20:34] <~john> right.
[20:34] <@Scatman> well i want both
[20:35] <~john> You guys know I am down with whatever you come up with, I just ask that you consider; VGEmu will be the ONLY place to play VG - it **might** actually get busy (cheers) so that's why I brought up load balancing on day 1.
[20:36] <@Scatman> yah :)
[20:36] <~john> (well, I brought it up before the sunset announcement, but I must have known it was coming!! lol)
[20:37] <@Scatman> haha
[20:38] <@Xen> Can we just tie and ip/port to the chunks table?
[20:38] <@Xen> chunk 1-10 go to ....54:10000
[20:38] <@Xen> chunk 5 ....55:10000
[20:38] <@Xen> At least allow that as an option
[20:39] <@Xen> Have a chunk_clustering table
[20:39] <@Scatman> that could work
[20:40] <@Scatman> but that would also limit us to a specific chunk per ip
[20:40] <@Scatman> maybe i'm thinking too crazy but i'd really like the ability to load balance
[20:40] <@Xen> So your saying have a chunk be split across two servers
[20:40] <@Scatman> no
[20:41] <@Xen> Oh nm, you mean server A is getting hit hard, so start spawning chunkservers on server B
[20:42] <@Scatman> basically it would start a new chunk server on a physical box [1-n] which the least amount of people on it
[20:42] <@Scatman> or a VM, whatever
[20:43] <@Xen> You'd probably need a central ChunkServer manager running then
[20:43] <@Scatman> that's what worldserver is for
[20:45] <@Scatman> we need to figure out how to handle every single chunk on 1 ip/port combo
[20:45] <@Scatman> or just on 1 ip
[20:45] <@Scatman> and open up a bunch of ports like xen said
[20:46] <~john> well, the thing troubling me about telling Admins to open 320 ports, because players could POSSIBLY go to chunks 1-320... that's a wide-ass gap in their universe. I wouldn't do it.[/quote]

Dev Chat: 04/11/2014 - Seeing across zone lines

Posted: Sat Apr 12, 2014 11:02 am
by John Adams
Seeing NPCs across zone lines, including a VG Live test.

Conclusion: Again, keeping it simple. We'll just focus on getting a chunk running, with spawns in it, and deal with how they look across zone lines later.

[quote][20:48] <@Xen> However, remember we are still going to have that issue where you can "see" across zone lines
[20:48] <~john> yep.
[20:48] <~john> i keep thinking of VGOEmu as 1 giant zone. maybe that's wrong.
[20:48] <~john> broken into little pieces for management.
[20:49] <@Xen> Yeah but the question is how to you break up the pieces
[20:50] <@Xen> Have we actually had someone verify that you can see across zone lines?
[20:51] <~john> Yes, me.
[20:51] <@Scatman> when you cross "chunk" lines, isn't a new packet sent with the ip/port and all?
[20:51] <~john> I was just standing in 1 chunk, looking at the zone line, and saw a spider walking around in the next chunk over. I did not get aggro til I crossed though.
[20:52] <~john> VG is an Open World, really. So you can cross a zone line anywhere along it's border. You do not get surprised by attacking monsters you did not see 2 secs before
[20:53] <@Xen> Yes scat
[20:53] <@Xen> Hmm so it seems all logic is done within chunk
[20:54] <@Scatman> if you're in Chunk A, and a spider is in chunk B, can you attack each other without going in each other's chunk?
[20:54] <@Scatman> or is it all visual?
[20:54] <@Xen> No idea on that
[20:55] <@Scatman> or at least aggro each other?
[20:55] <@Scatman> or cause the spider to come into chunk A?
[20:55] <@Xen> Sound like john was saying you don't get aggro until you are in the same chunk
[20:56] <@Scatman> so if that's the case, it's visual
[20:56] <@Scatman> and chunks handle their own logic, like you said
[20:56] <@Xen> So we could keep a global list of mobs in "world"
[20:57] <~john> no i do not believe there is interaction across the chunk. let me log in and check for you.
[20:57] <@Xen> the question is how do we send that data between servers
[20:58] <@Xen> So EQ2 emu is all hosted on 1 box huh?
[20:58] <~john> yep
[20:58] <@Scatman> a global list of mobs wouldn;t be a bad idea
[20:58] <@Scatman> we'd just have to say that all chunk servers need access to the same db
[20:59] <@Xen> Sure, but that global list only exists in the "world" program space
[20:59] <@Scatman> or we have a database server too (a nice to have later maybe)
[20:59] <@Scatman> right
[20:59] <@Xen> So the chunk servers will have to request that from the world server
[20:59] <@Scatman> login and world communicate through tcp. we could do the same for world->chunk
[21:00] <@Xen> Is that going to be a big performance issue?
[21:00] <@Xen> Say a zone has 500+ mobs. Isn't that going to take some time to send over tcp?
[21:00] <@Scatman> it could. you're more familiar with unreal than i am
[21:01] <@Xen> What would handle mob and player movement? Chunkserver or WOrld?
[21:01] <@Scatman> chunkserver i'd assume
[21:01] <@Scatman> i need to make a VG live account
[21:02] <@Xen> It will need to communicate back to world server when things move around
[21:02] <@Xen> Since that it holding the globabl list

*** Verifying Live functionality ***
[21:02] <~john> I am in chunk A looking into chunk B, and with the mouse, targetted a spider in another chunnk. it's too far away to attack.
[21:02] <@Scatman> so if you see a spider in chunk B. and you zone into chunk B. is it sitting right in the same spot?
[21:03] <@Scatman> if their chunkservers are on different IPs...they're either sending all close mobs to the client
[21:03] <@Xen> I think you are right though, we are going to have to think this through
[21:03] <@Scatman> or simulating it
[21:05] <@Scatman> if i was in chunk A, near chunk B, it wouldn't be rocket science to send mobs 10meters (random measurement) to the client that are in chunk B
[21:06] <~john> ok, i just shot the spider and dragged it back to Chunk A, and it's there, but I cannot attack it.
[21:06] <~john> but the entity followed me across the chunk line like it wasn't there. and hates me.
[21:06] <@Scatman> interesting
[21:06] <@Scatman> i wonder if VG even has it figured out :P
[21:06] <~john> and my client distance indicator thinks it's 40' away
[21:07] <~john> so I dragged him back into HIS chunk, and I can attack him now. BUG!
[21:07] <@Xen> lol
[21:07] <@Scatman> so. it seems like that mob is tagged, logically, to that chunk?
[21:07] <~john> hmm, literally. (giant spider joke)
[21:08] <~john> could have been a fluke. i'll pull another one.
[21:09] <~john> yeah, you cannot attack once dragged across apparently.
[21:09] <~john> OOR
[21:09] <~john> soI was 1/2 right. I do remember being chased, but was so scared I kept running and never figured out they couldn't attack me anyway :D
[21:09] <@Scatman> heh
[21:09] <@Xen> Just saw this on Vanguard forums from May 2013:
[21:10] <@Xen> While not the same symptoms I suspect this bug might be related:
[21:10] <@Xen> When carrying aggro across a chunk line, the mob will continue to chase but behaves as if it is a vanity pet rather than a mob. It follows you and although it can be targetted as an offensive target, it does not attack and neither can it be attacked. Once you have crossed the chunk line your character is put into non-combat mode. If you feign death the mob will just stand next to you. The mobs
[21:10] <@Xen> can still be leashed though.
[21:10] <@Xen> So mobs are pretty much controlled by their zone
[21:10] <@Scatman> yeah


[21:11] <@Scatman> i say we keep it clean. don't even show mobs across chunk lines
[21:11] <@Scatman> if you can't implement the entire thing, don't show your ankles!
[21:11] <~john> problem with that is the surprise factor.
[21:11] <@Xen> Or we show them but if someone targets them and attacks, we just display a message
[21:12] <@Xen> So we don't need a global list, each chunk can host it's own list
[21:13] <~john> what if just by the nature of the engine, you plop a NPC in 1 chunk and go to another, maybe you can just see it?
[21:13] <~john> i mean the Unreal stuff, not our code.
[21:13] <@Xen> We could do some checking based on location and radius to query the specific adjacent chunk(s) to get viewable mobs
[21:14] <@Xen> I don't believe that works john. When you enter a new chunk all hte channels are cleared
[21:14] <@Scatman> yeah i'm not sure how unreal's coords work, but that would be really cool john
[21:14] <~john> oh. even if "they" have all their stuff already spawned in a running chunk, and all you are doing is entering it?
[21:15] <@Xen> Well the way live is implemented they resend new crap on already used channels when you zone
[21:15] <@Xen> I think that's because the client is loading a new "level"
[21:15] <@Xen> So it clears everything
[21:15] <~john> that makes sense.
[21:16] <@Xen> However the client is aware of adjacent zones
[21:16] <@Scatman> so i wonder what coordinates those mobs would get across chunk lines?
[21:16] <~john> how about this then, if we know where to spawn an NPC, if I am in Chunk A approaching the zone line, is it possible to just pop that NPC in a "buffer zone"? maybe that's how they do it.
[21:17] <@Scatman> the buffer zone being in chunk b?
[21:17] <@Xen> Well as you run around I'm assuming they spawn "actors" in channels, although I can't confirm that
[21:18] <@Xen> But I get what you're saying I think
[21:19] <@Xen> We have a "load" radius that is bigger than the chunk
[21:20] <~john> right. i just don't want to be TOO different here unless we have no choice. Some little noob will book it into Trengel Keep and soon as he chunks over, a level 35 appears and 1-shots him :)
[21:20] <@Xen> When you enter chunk A, it sends all the mobs in the chunk + any mobs within a buffer of the surround 8 chunks.
[21:20] <@Xen> Well so that's interesting, how do they handle player movement?
[21:21] <~john> now, your challenge will be to put that buffered NPC exactly where my client saw it before I zoned :D
[21:22] <~john> I know we'll figure it out. this is the kinda stuff we need the live servers for. thankfully they are still there.
[21:22] <@Xen> Or what if we don't allow NPC's in the buffer area of each chunk?
[21:22] <@Scatman> yeah but what if me and you are running across chunk A into chunk B at the same time?
[21:22] <@Scatman> do they figure out who to aggro before we get there?
[21:22] <@Scatman> or whoever zones first?
[21:23] <@Xen> I would think whoever is in the chunk when the aggro check happens
[21:23] <@Xen> If you are both there then it's random
[21:24] <~john> Xen, that might be the case in VG too. The spiders were definitely OOR. and I ran a good distance trying to find one that was in range.
[21:24] <~john> if it's me and Scat, always attack the Halfling first.
[21:25] <@Xen> So essentially it would be like Scat said, we just don't show NPC's across zonelines :P
[21:25] <@Xen> Players would be a different story
[21:25] <@Xen> But I'm not a PvPer so I don't care =)
[21:26] <@Scatman> i say we start simple
[21:26] <@Scatman> don't show them across zone lines
[21:26] <@Scatman> is it really a game changer?
[21:27] <@Xen> THe question is, will that be hard to change later?
[21:27] <@Xen> Isn't all of this determining where we hold lists of npc's and players?
[21:28] <@Scatman> it would be become of matter of where we're storing the NPCs
[21:28] <@Scatman> i think getting us into zones and stuff is more important
[21:28] <@Xen> I agree
[21:28] <@Scatman> so let's consider this
[21:29] <@Scatman> we get the game working great
[21:29] <~john> it is.
[21:29] <@Scatman> all npcs, quests, etc
[21:29] <@Scatman> after that, we can play around with showing npcs across zones on our own time
[21:29] <@Scatman> i dont think we'd need vanguard live arouind to figure that out
[21:29] <~john> cocky! I like it!
[21:29] * Xen claps
[21:30] <@Scatman> right? at this pont it's between us and unreal
[21:30] <@Xen> Ok so sounds like we figured out that each chunkserver will have it's own list of NPC's to keep track of
[21:30] <@Scatman> yes[/quote]