Test Server Character Copy

Discussion in 'Test Server' started by Wigin, Nov 15, 2013.

  1. Wigin Developer

    We are currently running a character copy from a 2 day old Live backup to the Test Server. This should get every(slightly outdated) character onto test from live. This may take a few hours but then everyone will have their characters to use on Test Server. We will try and keep the Test Server Unlocked all weekend.

    If anyone would like to test any of Cave of Wonders raids this weekend it will be up but if everything goes smoothly we will begin some official raid testing for Cave of Wonders next week.

    The BBO changes are also fully on Test server. All of the hour long class buffs should be replaced with BBO buffs.

    If you have an APW all in one buff casting it should remove it from your book and give a smart BBO buff that will scale to the level of your target.

    If you have a Pantheon buff it will be converted to a buff for your full raid.

    We also fixed the only known crash issue with new performance updates please post if you find any problems.
  2. Lavyndar Well-Known Member

    What about classes that don't have all in ones? Pallies, for example.

    By Pantheon buff do you mean the level 55 all in one? Because the one you get for the PotA entrance quest is the same as the APW one, and I can't think of any other Panteon buffs.
  3. Wigin Developer

    Paladins get them both if they have the Pantheon one.
  4. TLoch Developer

    The full live server DB migrate is running into some snags. We're continuing to work on it to get it in your hands as soon as possible. We hate how long it's taking just as much as you are and are doing everything we can to get it done and done right as soon as possible. We'll keep you updated on the progress.
  5. TLoch Developer

    Won't be able to complete the database migration tonight. We're going to continue working tomorrow morning on the issue.
  6. Cummings Active Member

    It's not the dev's fault guys. Seriously, this is all Management. Wigin and Tloch and co. are doing what they've been told to do. You wanna bash people? Start at the top.
    Gresteh and Leavwiz like this.
  7. TLoch Developer

    Since it's Monday, it's pretty obvious we weren't able to get the test server ready to unlock over the weekend. We apologize for that.

    We're working on getting the problem solved and test back up with a full Live character database copy. This process is important to us to hammer down really well, since it will be the process we use for migrating the Live character database to the new database system.
    Ghostchild and Kimja like this.
  8. Kimja Active Member

    This is the DB update that's going to fix a lot of load-related issues in raids?

    If so, I'm really looking forward to this one.

    Cheers.
  9. Wigin Developer

    When we parsed server performance during raids DB wait times were almost the entire cause of lag and the rewrite should fix most everything related to DB wait times.

    Since CoW is a multiple encounter raid dungeon and even a single raid can cause lag it would have been a lag fest without the DB rewrite. That is why even though we have been finished with the first wing of CoW for a while we haven't pushed it live because it wouldn't perform horribly if 3 or 4 raids tried to do it at the same time.

    Our code team has finished their part of the DB rewrite we need to wait for other teams to convert our character files and servers over to the new system and that process is taking a long time for various reasons. As soon as they finish their part of the conversion we should start moving ahead very quickly.
    Ghostchild and Kimja like this.
  10. Thretosix Active Member


    I'm not sure why you think there will even be 3 to 4 raids going on at the same time with players bleeding out at the moment, I'm not even sure there are 3 to 4 raids going on now at prime time during any night of the week. It looks like guilds are being consolidated at this point. The game needs players coming in so we have more new guilds and feed the current guilds. This new content isn't going to be for new people coming in, it is for the end game players who have been waiting for years. We tell you we like apples and you give us oranges. What makes you think this strategy is working? The bleeding players, the feedback from forum posts, the gold membership cancellations?
    Leavwiz likes this.
  11. Leavwiz Well-Known Member

    unfortunately very true
  12. dravid Well-Known Member

    If there's more than 40 people in a chunk right now, things slow down to a grinding halt for most raids. e.g. Renton Keep and Tharridon and people just rifting in to get diplos. I share the sentiment with most posters that is not the answer we want.

    But unfortunately, without some form of a performance remedy, I personally feel it is very unlikely that two guilds can share a chunk for raiding. Remember Tomb of Lord Tsang where we had do coordinate our burn phases when we engaged Sparkles and there was another guild on IWG or Shendu.

    If they feel Database is the main bottleneck then as much I hate waiting as every other person I really see no way around this.
    Ghostchild likes this.
  13. Wigin Developer

    Basically every time the game hitches in a raid we see it related to the DB. If we tried to release CoW before we got the performance update it would have been an atrocious experience if two or three raids tried to go at the same time.We know of a couple other performance updates that we want to do after the DB but the DB upgrades should eliminate the vast majority of hitching.

    The good news is the the code and design work is done for the first wing of CoW and for the past weeks our only reason to not push content and performance is waiting for our new DB to be configured and characters converted. As soon as other teams finish their work we can move forward much quicker giving content much faster then players have seen before.

    We know players are starved for content and we want to give you good content that you can play and enjoy without hitching.
  14. Lavyndar Well-Known Member

    Despite all the doom and gloom, there ARE still multiple raid guilds with clashing raid schedules and all of them will want to be doing CoW. Please do us a favor and shard CoW in the same way as APW was. Even if it's only 2 or 3 shards and/or you have to shut down some APW shards.

    It's NOT about performance. More than one raid trying to do the same content in the same zone is just a terrible experience from a gameplay point of view.
  15. Ghostchild Active Member

    That is a bold statement Wigin, I only hope you can pull things off without a hitch.
  16. Shal Active Member

    I C WAT U DID THAR
    Ghostchild likes this.
  17. TLoch Developer

    The test servers should be unlocking in a few minutes. I'm currently on my Dev character (TLoch, the ranger) and everything is working fine. So in a few short minutes, everybody else should be able to log in as well.

    A few things I want to mention. First, known issues, and places that may need additional testing.
    1. Corpses can save incorrectly if a player dies (creating a corpse) and then picks up all corpses the player may have had. Any further corpses will not save correctly. This has already been fixed internally will be pushed out sometime this week, along with other bug fixes.
    2. There is an issue where buffs are sometimes still appearing on the client's buff bar, while they don't actually have that buff. Dying seems to be a good way of causing the bug.
    3. The MotD can't currently be set.
    4. The targeting system saw a small rewrite, mostly involving converting static arrays to exception-safe arrays. Abilities, especially AoE abilities, need to see more testing by a larger testing pool.
    5. The Attached Abilities (buffs) rewrite could us testing as well.
    6. We are also concerned with any data that may have been converted incorrectly. Most especially, true/false values (such as whether your helmet/cloak/hood were hidden or not). Please keep in mind that these characters are copies from last week and some data may have changed.
    Second, a little bit about the database migration process, and why this has taken so long. As all of your know, Vanguard was originally created by Sigil, so a lot of the technology isn't in line with what SOE typically uses. The databases were among those technologies. The way Vanguard is set up on Live currently, the server will actually wait for a database callback, to verify that information got saved, or what have you. Now, we save asynchronously, meaning the server sends out a save and keeps moving on with whatever the server was doing. Raiding characters have a lot going on. Between changing buffs, refresh timers, item refresh timers, broken gear, health updates, position updates, and all the stuff that raiders have (multiplied by 24, of course), the database would sometimes take half a second to save a single character (doesn't sound like much, but in game programming half a second is a LONG time). That was even after we had been very selective about how much we saved (saving only differences) and how often.

    So there are some major differences in the database we used to use and the one we're now going to be using. Those databases don't always use the same datatypes, and some of them were peskier to convert than others. Then, every time we wanted to test it, we were testing it on hundreds of thousands of characters and needing to verify that the data all got to it's destination correctly. So that's why there has been such a delay. Trust me when I say, it will be worth it.

    Lastly, I want to talk about what this step means for Vanguard's development. The network performance update earlier this year was the first of a few broad brush strokes. It was a big thing we knew would clear up lots of the code and make it easier for us to find and address performance problems. In fact, the network rewrite, with it's performance logging, is what clued us into the database being a problem. Now that we've wiped away another 'big one' with the player database, it gives us a clearer picture of what needs work and where things can not only be improved for performance, but also improved for the player experience. Nothing is ever in a vacuum in programming, and changing one thing usually means changing a bunch of other things. While changing those other things, we've been taking the time to clean them up, reduce redundant functions, or simplify overly complicated logic. This allows us to better implement future features and new mechanics into the game. We're focusing on 'quality of life' for both the players and the developers, which will allow us to develop content more rapidly, as Wigin already indicated.

    This doesn't mean we're done with our mops and shovels. There's still more performance improvements to make. We'll get to those when we can, but we know it's been awhile since players have gotten to dig their teeth into new content, and that's what we'll be doing for the next few months -- content and features.


    TL;DR, get out there and test.
    Dosi, Gresteh, Leavwiz and 3 others like this.
  18. Shal Active Member

    thanks!

    how about putting cow into 2-3 separate shards? as lavy said, its not the performance we're worried about but fighting over 3-4 other guilds in the same area over a limited number of targets. not to mention less low level alts hiding around raid targets watching other guild attempts
  19. Kimja Active Member

    Mass Tuurgin shows up in general abilities when learned and not adventuring abilities.
    Sustained abilities (like necro's Bloodfeast) remains running even after you toggle the ability off. You have to manually click off the buff or type in /cancelbuff.

    edited: meat/orb have cast times again.
  20. Wigin Developer

    We are doing a quick build and it should fix the Tuurgin specific buffs.

    Meat/Orb Cast times will be removed in this build as well the Test DB was missing some of the Data from live patches.

    We will investigate what is happening with Bloodfeast. I imagine it is the same issue that Tloch called out in the patch notes but we will make sure to double check it.

Share This Page