Dev Chat: 04/11/2014 - Multi-Chunk Support
Posted: Sat Apr 12, 2014 11:01 am
Some discussion about how to handle more than 1 running ChunkServer at at time, and deciding (at least for now) to Keep It Simple and just go with 1 ip:port combo for multiple chunkservers. Later we can implement "clustering".
2nd post is about visuals.
[quote][19:54] <@Xen> So scat, about multiple chunkserver's. For the port number, is it okay to just increment it each time a new one is added to the list? Is there anyway to check if the port is already bound?
[19:55] <@Scatman> there is a way to check if it's bound but i think i have a socket option enabled to not care
[19:55] <@Scatman> i think we should keep them all on the same port right now
[19:57] <@Xen> Hmm, so how does that work? Does each chunk know how to handle the clients connected to it?
[19:59] <@Scatman> gimme a sec to bring the project because up i have the memory of a goldfish. but i thought world server determined where the chunk was?
[20:00] <@Scatman> however, i am not familiar with chunk -> chunk communications
[20:03] <@Xen> When you say "where", what do you mean?
[20:05] <@Xen> Also, each chunkserver has a UDP attached to it and processes packets (at least how I understand it). If two chunk servers are running on the same port, then which one processes my clients packets that are being sent to that port?
[20:11] <@Scatman> xen i see what you're saying
[20:12] <@Scatman> do you know if the client disconnects from world once it's connected to a chunk?
[20:13] <@Xen> No idea.
[20:14] <@Scatman> so i designed it like how SOE was designed. but you're right. i might not be suitable for us.
[20:14] <@Xen> But I know the chunk port is sent to the client. I see this in my vanguard client log: "Log: Connecting to LVSVGO-01-14.vanguardsoh.net:10137 (chunk_n13_n9)"
[20:14] <@Scatman> yup
[20:16] <@Xen> A different log with a different chunk: Log: Connecting to LVSVGO-01-09.vanguardsoh.net:10138 (chunk_n26_1)
[20:16] <@Scatman> yeah it 00-14 (or maybe more)
[20:17] <@Xen> In the same log as the (chunk_n13_n9), I crossed a zoneline and this: "Log: Connecting to LVSVGO-01-03.vanguardsoh.net:10135 (chunk_n12_n9)"
[20:17] <@Scatman> i like the design where each chunk server is (or could be) on a different server but i'm trying to figure out how to make all chunk servers hit the same IP/port
[20:17] <@Xen> No idea if each chunk it tied to an individual port or not
[20:18] <@Scatman> they are. each chunkserver (is literally a server) can be on it's on box
[20:18] <@Xen> Yeah it would be kind of cool to be able to host different chunks on different servers
[20:18] <@Xen> That's how you basically have it setup now
[20:18] <@Scatman> yep
[20:19] <@Xen> And I'm fine with that, except I'm not sure how to find valid ports to bind to. For now I was just going to store a port variable in CHunkServerList and increment it each time a new chunk is added
[20:19] <@Scatman> i know. i overlooked that
[20:19] <@Scatman> thinking...
[20:19] <@Xen> Is there a problem with that?
[20:20] <@Xen> There are around 320 chunks. Can't we just say you have to have ports 10000 - 10340 free?
[20:20] <@Xen> Or let them define the first port number and we increment from that
[20:21] <@Scatman> if we do it that way, we just have to let our admins know that they need to open port x to x+n where n is the number of chunkservers :/
[20:21] <@Xen> Use that rule stuff John put in
[20:21] <@Scatman> give me the night to think about it
[20:22] <@Scatman> and research it
[20:22] <@Xen> Another option would be to have them all on the same port but keep a list of which client is in which chunk
[20:22] <@Scatman> i was thinking that too
[20:22] <@Xen> Then in the CHunkServer->Process() function forward it to the right one
[20:23] <@Scatman> our goal would be to let this run like SOE and let each chunkserver run on a different physical box. that'd be sweet
[20:24] <@Xen> Isn't that possible with the code you have now?
[20:24] <@Scatman> it is. absoutely
[20:24] <~john> Scat, I PM'd you extracted ClientLog's from a complete session. if it will help determine service connections.
[20:24] <@Xen> So what are we thinking about then?
[20:24] <@Scatman> how to solve the problem of an admin running multiple chunkservers on one box
[20:25] <@Scatman> unless we open up 300 ports?
[20:25] <@Xen> I say we just let them define the low end of the port range and we increment. Or we let them define a set of ranges (ie: 10000-10100, 12000-12400)
[20:26] <@Scatman> right.
[20:26] <@Scatman> we could round robin them too
[20:26] <@Xen> As chunkServers close, we dump those ports back into the "free" pool for the new chunkserver
[20:26] <@Scatman> or balance load 'em
[20:26] <@Scatman> john. is that how eq2emu works?
[20:27] <@Scatman> wasn't each zoneserver it's own UDP server?
[20:27] <@Scatman> i lost their source heh
[20:28] <~john> EQ2Emu is all in EQ2World.exe, no separate zoneserver exe's. so it's all under 1 port (incoming).
[20:28] <~john> EQEmu though uses 2 options, Dynamic Ports (reused) and Static Ports.
[20:29] <~john> My EQEmu server for example has 20 ports dynamically allocated to zoneservers. 20 max run at once. then, it's full. No more zones can start. but, they are always idle on a lonely server. player logs in, the server finds the 1st avalilable dynamic port and uses it.
[20:29] <~john> thoughts: First, no one is going to run 320 chunkservers. Ever. not on 1 box, not on a farm :D
[20:30] <~john> if we can utilize the round-robin or dynamic approach, we only need 'x' number defined (up to the admin).
[20:31] <~john> btw, Scat... recall my post about "Clustering"? :p
[20:31] <@Scatman> i thought by clustering you meant 1 chunkserver per ip :P which is what i implemented lol
[20:32] <@Scatman> just in their own exes
[20:32] <~john> yeah that's what we talked about.
[20:33] <~john> but it would be 1 physical box, with say 10 zones running.
[20:33] <~john> EQEmu would require me to open 10 ports to that LAN IP, by it's design. EQ2Emu has no design of this nature :D It's 1 port into World, and that's it.
[20:34] <@Scatman> right. but eq2emu also has no design of clustering
[20:34] <~john> right.
[20:34] <@Scatman> well i want both
[20:35] <~john> You guys know I am down with whatever you come up with, I just ask that you consider; VGEmu will be the ONLY place to play VG - it **might** actually get busy (cheers) so that's why I brought up load balancing on day 1.
[20:36] <@Scatman> yah :)
[20:36] <~john> (well, I brought it up before the sunset announcement, but I must have known it was coming!! lol)
[20:37] <@Scatman> haha
[20:38] <@Xen> Can we just tie and ip/port to the chunks table?
[20:38] <@Xen> chunk 1-10 go to ....54:10000
[20:38] <@Xen> chunk 5 ....55:10000
[20:38] <@Xen> At least allow that as an option
[20:39] <@Xen> Have a chunk_clustering table
[20:39] <@Scatman> that could work
[20:40] <@Scatman> but that would also limit us to a specific chunk per ip
[20:40] <@Scatman> maybe i'm thinking too crazy but i'd really like the ability to load balance
[20:40] <@Xen> So your saying have a chunk be split across two servers
[20:40] <@Scatman> no
[20:41] <@Xen> Oh nm, you mean server A is getting hit hard, so start spawning chunkservers on server B
[20:42] <@Scatman> basically it would start a new chunk server on a physical box [1-n] which the least amount of people on it
[20:42] <@Scatman> or a VM, whatever
[20:43] <@Xen> You'd probably need a central ChunkServer manager running then
[20:43] <@Scatman> that's what worldserver is for
[20:45] <@Scatman> we need to figure out how to handle every single chunk on 1 ip/port combo
[20:45] <@Scatman> or just on 1 ip
[20:45] <@Scatman> and open up a bunch of ports like xen said
[20:46] <~john> well, the thing troubling me about telling Admins to open 320 ports, because players could POSSIBLY go to chunks 1-320... that's a wide-ass gap in their universe. I wouldn't do it.[/quote]
2nd post is about visuals.
[quote][19:54] <@Xen> So scat, about multiple chunkserver's. For the port number, is it okay to just increment it each time a new one is added to the list? Is there anyway to check if the port is already bound?
[19:55] <@Scatman> there is a way to check if it's bound but i think i have a socket option enabled to not care
[19:55] <@Scatman> i think we should keep them all on the same port right now
[19:57] <@Xen> Hmm, so how does that work? Does each chunk know how to handle the clients connected to it?
[19:59] <@Scatman> gimme a sec to bring the project because up i have the memory of a goldfish. but i thought world server determined where the chunk was?
[20:00] <@Scatman> however, i am not familiar with chunk -> chunk communications
[20:03] <@Xen> When you say "where", what do you mean?
[20:05] <@Xen> Also, each chunkserver has a UDP attached to it and processes packets (at least how I understand it). If two chunk servers are running on the same port, then which one processes my clients packets that are being sent to that port?
[20:11] <@Scatman> xen i see what you're saying
[20:12] <@Scatman> do you know if the client disconnects from world once it's connected to a chunk?
[20:13] <@Xen> No idea.
[20:14] <@Scatman> so i designed it like how SOE was designed. but you're right. i might not be suitable for us.
[20:14] <@Xen> But I know the chunk port is sent to the client. I see this in my vanguard client log: "Log: Connecting to LVSVGO-01-14.vanguardsoh.net:10137 (chunk_n13_n9)"
[20:14] <@Scatman> yup
[20:16] <@Xen> A different log with a different chunk: Log: Connecting to LVSVGO-01-09.vanguardsoh.net:10138 (chunk_n26_1)
[20:16] <@Scatman> yeah it 00-14 (or maybe more)
[20:17] <@Xen> In the same log as the (chunk_n13_n9), I crossed a zoneline and this: "Log: Connecting to LVSVGO-01-03.vanguardsoh.net:10135 (chunk_n12_n9)"
[20:17] <@Scatman> i like the design where each chunk server is (or could be) on a different server but i'm trying to figure out how to make all chunk servers hit the same IP/port
[20:17] <@Xen> No idea if each chunk it tied to an individual port or not
[20:18] <@Scatman> they are. each chunkserver (is literally a server) can be on it's on box
[20:18] <@Xen> Yeah it would be kind of cool to be able to host different chunks on different servers
[20:18] <@Xen> That's how you basically have it setup now
[20:18] <@Scatman> yep
[20:19] <@Xen> And I'm fine with that, except I'm not sure how to find valid ports to bind to. For now I was just going to store a port variable in CHunkServerList and increment it each time a new chunk is added
[20:19] <@Scatman> i know. i overlooked that
[20:19] <@Scatman> thinking...
[20:19] <@Xen> Is there a problem with that?
[20:20] <@Xen> There are around 320 chunks. Can't we just say you have to have ports 10000 - 10340 free?
[20:20] <@Xen> Or let them define the first port number and we increment from that
[20:21] <@Scatman> if we do it that way, we just have to let our admins know that they need to open port x to x+n where n is the number of chunkservers :/
[20:21] <@Xen> Use that rule stuff John put in
[20:21] <@Scatman> give me the night to think about it
[20:22] <@Scatman> and research it
[20:22] <@Xen> Another option would be to have them all on the same port but keep a list of which client is in which chunk
[20:22] <@Scatman> i was thinking that too
[20:22] <@Xen> Then in the CHunkServer->Process() function forward it to the right one
[20:23] <@Scatman> our goal would be to let this run like SOE and let each chunkserver run on a different physical box. that'd be sweet
[20:24] <@Xen> Isn't that possible with the code you have now?
[20:24] <@Scatman> it is. absoutely
[20:24] <~john> Scat, I PM'd you extracted ClientLog's from a complete session. if it will help determine service connections.
[20:24] <@Xen> So what are we thinking about then?
[20:24] <@Scatman> how to solve the problem of an admin running multiple chunkservers on one box
[20:25] <@Scatman> unless we open up 300 ports?
[20:25] <@Xen> I say we just let them define the low end of the port range and we increment. Or we let them define a set of ranges (ie: 10000-10100, 12000-12400)
[20:26] <@Scatman> right.
[20:26] <@Scatman> we could round robin them too
[20:26] <@Xen> As chunkServers close, we dump those ports back into the "free" pool for the new chunkserver
[20:26] <@Scatman> or balance load 'em
[20:26] <@Scatman> john. is that how eq2emu works?
[20:27] <@Scatman> wasn't each zoneserver it's own UDP server?
[20:27] <@Scatman> i lost their source heh
[20:28] <~john> EQ2Emu is all in EQ2World.exe, no separate zoneserver exe's. so it's all under 1 port (incoming).
[20:28] <~john> EQEmu though uses 2 options, Dynamic Ports (reused) and Static Ports.
[20:29] <~john> My EQEmu server for example has 20 ports dynamically allocated to zoneservers. 20 max run at once. then, it's full. No more zones can start. but, they are always idle on a lonely server. player logs in, the server finds the 1st avalilable dynamic port and uses it.
[20:29] <~john> thoughts: First, no one is going to run 320 chunkservers. Ever. not on 1 box, not on a farm :D
[20:30] <~john> if we can utilize the round-robin or dynamic approach, we only need 'x' number defined (up to the admin).
[20:31] <~john> btw, Scat... recall my post about "Clustering"? :p
[20:31] <@Scatman> i thought by clustering you meant 1 chunkserver per ip :P which is what i implemented lol
[20:32] <@Scatman> just in their own exes
[20:32] <~john> yeah that's what we talked about.
[20:33] <~john> but it would be 1 physical box, with say 10 zones running.
[20:33] <~john> EQEmu would require me to open 10 ports to that LAN IP, by it's design. EQ2Emu has no design of this nature :D It's 1 port into World, and that's it.
[20:34] <@Scatman> right. but eq2emu also has no design of clustering
[20:34] <~john> right.
[20:34] <@Scatman> well i want both
[20:35] <~john> You guys know I am down with whatever you come up with, I just ask that you consider; VGEmu will be the ONLY place to play VG - it **might** actually get busy (cheers) so that's why I brought up load balancing on day 1.
[20:36] <@Scatman> yah :)
[20:36] <~john> (well, I brought it up before the sunset announcement, but I must have known it was coming!! lol)
[20:37] <@Scatman> haha
[20:38] <@Xen> Can we just tie and ip/port to the chunks table?
[20:38] <@Xen> chunk 1-10 go to ....54:10000
[20:38] <@Xen> chunk 5 ....55:10000
[20:38] <@Xen> At least allow that as an option
[20:39] <@Xen> Have a chunk_clustering table
[20:39] <@Scatman> that could work
[20:40] <@Scatman> but that would also limit us to a specific chunk per ip
[20:40] <@Scatman> maybe i'm thinking too crazy but i'd really like the ability to load balance
[20:40] <@Xen> So your saying have a chunk be split across two servers
[20:40] <@Scatman> no
[20:41] <@Xen> Oh nm, you mean server A is getting hit hard, so start spawning chunkservers on server B
[20:42] <@Scatman> basically it would start a new chunk server on a physical box [1-n] which the least amount of people on it
[20:42] <@Scatman> or a VM, whatever
[20:43] <@Xen> You'd probably need a central ChunkServer manager running then
[20:43] <@Scatman> that's what worldserver is for
[20:45] <@Scatman> we need to figure out how to handle every single chunk on 1 ip/port combo
[20:45] <@Scatman> or just on 1 ip
[20:45] <@Scatman> and open up a bunch of ports like xen said
[20:46] <~john> well, the thing troubling me about telling Admins to open 320 ports, because players could POSSIBLY go to chunks 1-320... that's a wide-ass gap in their universe. I wouldn't do it.[/quote]