View RSS Feed

The Silph Scope

What's the matter, PokeGym?

Rating: 3 votes, 2.33 average.
Well, that was awkward. The Gym went offline during 2011 U.S. Nationals.

Doh!

I guess it had been down all Saturday, but I didn't hear about it until I was leaving the hall. That was a mercy since I was judging and couldn't have done anything anyway. So, instead of going to the Mutant Madness Draft™, I ordered room service (nice club sandwich, but too much mayonnaise) and Wi-Fi, and went to work on the Gym.

I made a phone call home to make sure that the following usual suspects were not at fault:
  1. Power Outage
  2. Pudge the Cat

Pudge is very fond of chewing on low voltage wires and had brought the Gym offline twice (and cost numerous guests headphone and cellphone cords - yes we warn them, they just... sigh).

As my wife rebooted the hardware (again and again), she spent about half an hour reading me linux boot screens from the console (she gets some kind of award for that). It was clear that sometimes the boot sequence wasn't event making it to the BIOS, sometimes it was but got hung right there, other times it would boot all the way up, the go offline after a few seconds. Mostly, we just got a dark console after cycling power.

OK, it had to be powered off and left off. What to do to prevent all the folks looking for the PokeGym from getting dead air? What about access to the Compendium?

I have another small server, so I changed some DNS entries, changed the mail routing, brought up a new webserver on that host, pulled down a copy of the Compendium LvX from our online backup, wrote a small page linking to the copy of the Compendium and our Facebook page, tested this setup, and went off the Mutant event about 1am Saturday.

I didn't get home until Tuesday about 2am. Straight to bed then off to work. The next night testing made it clear there was a hardware problem. Best I could tell is we had a blown electrolytic capacitor in the memory circuit. Google search brought that fault up for this motherboard... and when I looked for it on the 'mobo', sure enough, there was its little bulging self. Nuts, a 25˘ part, probably. Capacitor failure rate is an Internet scandal among motherboard manufacturers, as it turns out.

After bouncing a couple ideas of Chairman Kaga I determined a Fry's run was in order -- to get a whole new machine (except for the disks). You see, while you can still find a couple small operators selling motherboards compatible with the older 2009 CPU sockets & memory speeds, I didn't know what was damaged, if anything, by the wildly varying voltage caused by the bad capacitor. Since I'm on the way to vacation, no time to troubleshoot other burned out parts, or to go back to the store for replacements. Might as well go for a new machine now and try to use the old parts in a backup/test machine later.

As a result of the failure, there have been some improvements. We've improved from 2.6->3.6 Ghz and 4->6 cores on the chip. The Gym hasn't been slow or bottlenecky since we got the last hardware upgrade, but this future proofs us a bit.

I may look into how to have a standby machine - maybe even a cloud instance - in case this happens in future. Meantime, the new motherboard has all solid state capacitors.
Categories
The Plateau

Comments

  1. mysterioustrainer's Avatar
    Lol... I always thought I was standard policy that the Gym remains down during Nationals and Worlds because of the bulk of Team Compendium and the moderator staff is usually at these events. But I was getting curious when the boards did not go back online until yesterday.
  2. P_A's Avatar
    Awkward? Ummm, yeah probably somewhat, but hey, life doesn't necessarily crap on you when it's convenient now does it? LOL. We got through the pokegym withdrawal symptoms after a bit. Glad to hear it's looking better for the experience though. You rock, Kim.
  3. Prime's Avatar
    This is a great story. I wish blog posts could be seen on the front page. I'm sure others would be interested in reading this.

    Go PokeGym!
  4. bulbasnore's Avatar
    Quote Originally Posted by Prime
    This is a great story. I wish blog posts could be seen on the front page. I'm sure others would be interested in reading this.
    Tried to get the blog module to work on the front page, but it gives an error. I'll keep it on the list for v4.
  5. chrataxe's Avatar
    Lol, funny how life works.