What's the matter, PokeGym?

Well, that was awkward. The Gym went offline during 2011 U.S. Nationals.

Doh!

I guess it had been down all Saturday, but I didn't hear about it until I was leaving the hall. That was a mercy since I was judging and couldn't have done anything anyway. So, instead of going to the Mutant Madness Draft™, I ordered room service (nice club sandwich, but too much mayonnaise) and Wi-Fi, and went to work on the Gym.

I made a phone call home to make sure that the following usual suspects were not at fault:

1. Power Outage
2. Pudge the Cat

Pudge is very fond of chewing on low voltage wires and had brought the Gym offline twice (and cost numerous guests headphone and cellphone cords - yes we warn them, they just... sigh).

As my wife rebooted the hardware (again and again), she spent about half an hour reading me linux boot screens from the console (she gets some kind of award for that). It was clear that sometimes the boot sequence wasn't event making it to the BIOS, sometimes it was but got hung right there, other times it would boot all the way up, the go offline after a few seconds. Mostly, we just got a dark console after cycling power.

OK, it had to be powered off and left off. What to do to prevent all the folks looking for the PokeGym from getting dead air? What about access to the Compendium?

I have another small server, so I changed some DNS entries, changed the mail routing, brought up a new webserver on that host, pulled down a copy of the Compendium LvX from our online backup, wrote a small page linking to the copy of the Compendium and our Facebook page, tested this setup, and went off the Mutant event about 1am Saturday.

I didn't get home until Tuesday about 2am. Straight to bed then off to work. The next night testing made it clear there was a hardware problem. Best I could tell is we had a blown electrolytic capacitor in the memory circuit. Google search brought that fault up for this motherboard... and when I looked for it on the 'mobo', sure enough, there was its little bulging self. Nuts, a 25¢ part, probably. Capacitor failure rate is an Internet scandal among motherboard manufacturers, as it turns out.

After bouncing a couple ideas of Chairman Kaga I determined a Fry's run was in order -- to get a whole new machine (except for the disks). You see, while you can still find a couple small operators selling motherboards compatible with the older 2009 CPU sockets & memory speeds, I didn't know what was damaged, if anything, by the wildly varying voltage caused by the bad capacitor. Since I'm on the way to vacation, no time to troubleshoot other burned out parts, or to go back to the store for replacements. Might as well go for a new machine now and try to use the old parts in a backup/test machine later.

As a result of the failure, there have been some improvements. We've improved from 2.6->3.6 Ghz and 4->6 cores on the chip. The Gym hasn't been slow or bottlenecky since we got the last hardware upgrade, but this future proofs us a bit.

I may look into how to have a standby machine - maybe even a cloud instance - in case this happens in future. Meantime, the new motherboard has all solid state capacitors.