McWetboy's Outage Central

Site slowdown

mcwetboy — Sat, 01 Mar 2008 01:22:18 +0000

Some kind of issue at DreamHost: my sites now load at a crawl (if at all). Hopefully this won’t take long to resolve, but the panel isn’t loading either, so I can’t share my traceroute.

Update, 8:40 PM: Their web panel is still down, and presumably so are other sites/servers, but mine seem to be functioning more or less properly at the moment.

All sites down

mcwetboy — Mon, 18 Feb 2008 14:14:04 +0000

All sites have been down since just before 9 a.m. EST.

Update #1: My sites came back, sort of, after about an hour, but now my MySQL databases are not responding (which makes blogging impossible), and server loads are stratospheric:

07:08:33 up 32 min, 1 user, load average: 327.42, 301.77, 223.48

Just heard back from a DreamHost customer support agent who seemed to think everything was fine. Shit.

Update #2: Nearly two hours later, and the incompetent retards still haven’t fixed anything.

Update #3: The server was rebooted two hours ago. Since that time, load averages have regularly exceeded 300 (anything above 10 is a crisis, I’ve been told) and I haven’t been able to do anything with my Movable Type install. I also haven’t heard a motherfucking peep from the DreamHost asstards since the first, hydrocephalic one in which they said everything was hunky-dorky, hyuck hyuck.

Update #4: Three hours in, still no MySQL access. Server loads creeping downwards slowly — to around 360! (They spiked at around 485.) Still no further word from DreamHost.

Update #5: I was able to publish a single entry on The Map Room. What normally takes seconds took at least 23 minutes before terminating in a server error. I don’t think all pages were built. Still no further word from DreamHost (12:07 p.m.).

Update #6: Looks like we’re done. Server loads down to 9.08 (1-minute average), MT rebuilds back to normal.

E-mail password issue

mcwetboy — Tue, 27 Nov 2007 02:29:14 +0000

My main e-mail accounts — my personal account and my Map Room account — are inaccessible: the server will not accept the correct passwords. I’ve notified DreamHost, but you can still contact me via my Gmail account in the meantime.

Update, 9:45 PM: My e-mail is back.

Database down due to filer problems

mcwetboy — Sat, 14 Jul 2007 15:40:55 +0000

An issue has arisen in the wake of this morning’s server move: “Seems we hit a slight bump with the move last night. Users on a particular filer currently can’t see their sites or their users due to this. We are in the process of fixing this right now.”

As a result, my web sites are available, but my Movable Type database is inaccessible: I can’t post new entries at this time. This is probably quite temporary so no big deal. Though there is one potential issue arising from this move that may complicate my Movable Type-powered blogs — I won’t know, however, until the database is back online. (Good thing I made backups last night.)

Update, 1:37 PM: DreamHost is reporting the issue resolved but I still can’t access the database.

Update, 1:46 PM: The database is back and functioning normally.

Scheduled outage due to server move

mcwetboy — Mon, 09 Jul 2007 23:57:54 +0000

DreamHost is moving my server to its other datacenter on Friday evening local time, beginning at 11 PM PDT (2 AM EDT Saturday morning). The move will take my sites and e-mail temporarily offline: “All webservers, mail servers, file servers, and MySQL servers [...] will be unreachable during the move, which we expect to take approximately 5 hours.”

The impact on my North American readers will be minimal, since this is taking place overnight, but my European friends will not be able to reach my sites Saturday morning.

Update, 7/14 at 7:00 AM: The move is now under way; my sites are not available.

Update, 7:15 AM: And we’re back.

Brief PHP outage

mcwetboy — Mon, 18 Jun 2007 23:45:03 +0000

Most of my sites were unavailable for a short period of time just now, as a result of what I assume was a PHP-related problem: static HTML pages continued to be served, but anything involving PHP — which I use on almost all my sites –threw a 500 Internal Server Error.

Update, 7:50 PM: From DreamHost:

Some of you may have noticed your site is giving a 500 Internal Server Error. This is due to a package upgrade we are currently performing. We tested it last week on one of our clusters and it went without incident, so we’re not sure why it’s giving problems today. We are working on resolving this issue ASAP.

(I love it when tech support says “we’re not sure.”)

Update #2, 7:58 PM: More on the DreamHost status blog: package upgrades = momentarily broken PHP.

Scheduled outage

mcwetboy — Tue, 13 Mar 2007 01:21:24 +0000

My server is on the cluster affected by the relocation by DreamHost of two file servers between 1:00 AM and 3:00 AM EDT Wednesday morning. As a result, my sites will likely be unavailable for some time during that period. (Given DreamHost’s luck lately, it might be longer, if problems arise during the move.)

Hopefully the move will improve my server’s performance: I’ve been observing somewhat-to-unacceptably high load averages for the past week or so.

Intermittent problems

mcwetboy — Mon, 26 Feb 2007 15:04:13 +0000

DreamHost seems to be having trouble cleaning up after yesterday’s planned outage; my sites have been down periodically — usually for only minutes at a time — since then. Hopefully things will stabilize shortly.

Planned outage on Sunday

mcwetboy — Fri, 23 Feb 2007 04:09:18 +0000

A planned power outage for emergency maintenance purposes will render all sites and e-mail unavailable for several hours early Sunday morning or thereabouts.

Update, 2/25 at 9:48 AM: My sites were unavailable until 9:30 AM EST this morning — about 2½ hours later than expected, due to a core router issue.

All sites down

mcwetboy — Wed, 17 Jan 2007 01:30:40 +0000

My server has apparently gone blooey sometime within the last half hour or so: at the moment, none of my web sites is currently reachable. I’ve contacted DreamHost about it; let’s see how long they take to fix it.

Update, 9:13 PM: Everything just came back online. The server had some NFS issues, and then didn’t want to reboot, but the technician power-cycled it and now everything appears to be back to normal. Total downtime was a bit more than an hour.