Page 1 of 2

Upgrades and Outages

Posted: Sat Feb 14, 2015 9:00 am
by John Adams
All,

The hardware upgrades are here and in place, so I will be performing migrations and maintenance throughout the weekend. This spells outages for all services intermittently. I'll try to keep it to a minimum, so if you get errors or Page Not Found's, don't panic. It's just me, tweakin'

Re: Upgrades and Outages

Posted: Sat Feb 14, 2015 6:56 pm
by shargash
Did you do the hearts and valentines? Or was that something automatic in the client?

Re: Upgrades and Outages

Posted: Mon Feb 16, 2015 7:44 pm
by John Adams
Attn: SVN Users

Sometime in the afternoon (PST) over the next few days, I will be migrating our SVN Repositories to their new server, and upgrading VisualSVN in the process. One of 2 things will happen; 1) Everything will go swimmingly and the new service will start up and read our existing repos (I have 27 repos) without a hitch, or 2) catastrophe will strike and all data will be lost.



Just kidding with #2. While anything is possible, the upgrade should go flawless, and I always have backups and SVN dumps, just in case. Devs can commit until the moment the service is shut down, so don't worry about synchronizing with me. The change-over shouldn't take a few minutes (hopes).

Re: Upgrades and Outages

Posted: Tue Feb 17, 2015 7:57 pm
by John Adams
The SVN migration is complete, almost flawless For those hitting the new server, the only thing to note is you'll need to accept the new self-signed certificate when you first hit it.

Please report any issues immediately.

Re: Upgrades and Outages

Posted: Wed Feb 18, 2015 6:00 pm
by John Adams
The last machine to be replaced is this webserver. It is the one most noticeable as having damage, as the sites are growing slower every week, so before things get much worse, I need to move to the new webserver in the new cluster. I have everything configured and ready to go, I am just vacillating over whether or not to use the convoluted, clunky, over-engineered ISPConfig 3 CP, or just go native Apache (like we currently have). There's some advantages to the CP, like single-point management - I just hate the product, even if it's free.

I'll probably just go with apache and set up load balancing web heads just cuz it's cool. Plus, I have to move 50 databases over, and I am not looking forward to that task.

Re: Upgrades and Outages

Posted: Wed Feb 18, 2015 10:06 pm
by Scatman
HTTP 2.0 is finished and on the way! I don't think apache supports it *yet* but it will and it'll be a lot faster with multiplexing and the other features!

Re: Upgrades and Outages

Posted: Sat Feb 21, 2015 11:03 am
by John Adams
Since this is my only "active" community I will only post this here... but we're going down for a few hours while I (painstakingly) move all of my websites to a new host, so we can ditch this dying VM.

Should take a couple of hours, pending disaster.

Re: Upgrades and Outages

Posted: Sat Feb 21, 2015 2:02 pm
by John Adams
Okay, I think I got the bugs worked out. It is [color=#FF0000:30r7pwtc][b:30r7pwtc]imperative[/b:30r7pwtc][/color:30r7pwtc] that you report [u:30r7pwtc]ANY[/u:30r7pwtc] page or module that reports an error. Thanks for your patience.

Re: Upgrades and Outages

Posted: Sat Feb 21, 2015 4:23 pm
by Odinzz
none so far, but i must say its super frigging fast now holy crap!

Re: Upgrades and Outages

Posted: Sat Feb 21, 2015 6:10 pm
by John Adams
I do have to share the one scare I had today.


Was just telling Jabantiz, that it's taken me almost 6 hours straight to try and recover the EQ2Emulators Forums database.... yes, the entire thing. All their posts, were gone. The bad spot on the drive sat right under their Posts table, of course. Almost 29,000 posts on that site spanning almost 9 years. I panicked.

After trying every thing I knew, and searching the stupid ass google/blogger/armchair "experts" on stackoverflow for hours, I finally walked away for a moment... when it suddenly hit me; what if I read the Posts table in chunks, and skipped over the bad stuff?

So I started doing "insert into new_table select * from old_table limit 0,1000" until I hit the bad spots. I got about 9,000 rows into the corrupt table. Then, it started looking bleak again. 10,000, 11,000, and so on... all bad.

Of course I had a backup, duh. But it was from the last time I moved everything (bad JA, bad!) which was Aug 2014. So I loaded up the backup. Low and behold, there were over 28,000 posts in that table... so worst come to worst, EQ2 would only lose their latest posts. That was not good enough for me!

So I had another brainstorm... when there's a bad "spot" on a drive, it's just a spot right? So, I saw the highest record ID in the backup and went to the corrupted table and did a DESC sort on that ID and what do ya know? 175 posts was all that was missing. And the corrupted table let me have them.

Peace and Tranquility has been restored to [url=http://eq2emulator.net:2h88eags]EQ2Emulator[/url:2h88eags].

F*#k I need a beer.