Sites down; changing server on Sunday

Friday, September 29, 2006 am30 8:55 am

The server on which my sites are hosted is reporting abnormally high load averages, which is what usually happens when my sites are unavailable, which they have been for the past half hour.

During DreamHost’s network problems, they moved me to my current server a month ago. Since then, I have been experiencing short outages associated with load-average spikes: the outages last about 10-20 minutes, and occur between one and three times a day. (The current outage has lasted longer and appears to be much more severe.)

It turns out that I should not have been moved to that server, which seems to be even less stable than my previous server. So we’re moving again — on Sunday afternoon (2 PM PDT, 5 PM EDT). The move should take an hour or so to complete, and there may be some brief downtime associated with it. Hopefully, though, it’ll put an end to the periodic outages I’ve been facing for the past month: they’ve been short enough not to cause serious problems with my sites, but frequent enough to be annoying.

As for the current outage, it seems to be abating at the moment (8:55 AM EDT).

Update, Oct. 1 at 11:40 PM EDT: The server move did not take place as scheduled. DreamHost apparently had some trouble today (I was out and about so missed the opportunity to freak out about it) and I suspect I got lost in the shuffle. I’m still raising hell about it.

Oct. 1 at 11:48 PM EDT: No sooner had I written the above … DreamHost has initiated the server move. It should be on a new server within the hour, if all goes well.

Scheduled downtime redux

Thursday, September 21, 2006 pm30 10:13 pm

DreamHost is moving filers tonight between 1 AM EDT (10 PM PDT) and 4 AM EDT (1 AM PDT). My sites may be down for a brief period (they say around 30 minutes) during that time.

Scheduled downtime

Wednesday, September 13, 2006 pm30 10:50 pm

DreamHost has announced that additional downtime for maintenance is scheduled for tonight at 1 AM EDT (10 PM PDT). I anticipate that my sites will be unavailable for a short while around that time.

Another major outage

Wednesday, September 13, 2006 am30 11:34 am

My sites have been down for the last 15 minutes; DreamHost’s web panel is also down, so it’s probably a significant outage this time. Sigh.

Update, 11:40 AM: Everything is now back online. That one took everything on DreamHost offline for a bit.

11:45 AM: Back on the roller-coaster. My sites are down again.

11:50 AM: Up again.

12:55 PM: Everything continues to work well. DreamHost reports that a router “hiccup” was to blame.

Network upgrades unsuccessful; sites down again

Tuesday, September 12, 2006 pm30 7:33 pm

DreamHost reports that their router upgrades weren’t as successful as they’d hoped, and indeed problems continue: my sites have been down for the last 10 minutes or so (since around 7:20 PM). I’ve had periodic outages for the past week or so of a few minutes’ duration; hopefully it’s one of these, rather than a clusterfuck like the one we faced on Friday night.

Update, 7:40 PM: Back after an outage of 20 minutes or so. Hopefully it’ll stay up for a while.

Sites are down again

Friday, September 8, 2006 pm30 9:03 pm

My sites are down again. DreamHost seems to be facing a major outage, including both their control panel (so much for reporting outages) and their off-site status blog. (My patience has been exemplary.)

Update, 9:25 PM: From DreamHost’s support desk, via e-mail:

We are currently experiencing a network problem that’s effecting [sic] all traffic to our hosting servers. This is causing mail, web and mysql access to slow to a crawl. In some cases, some servers are crashing in response to the bottleneck.

Our administration team is currently working on the problem, and should have it fixed soon. We apologize for any inconvenience this may cause!

I managed to report the server downtime via the panel during a brief moment it was open, but I think they’re aware of the problem.

Monday night they will be doing some serious network upgrades which promise to put an end to the problems that started nearly two months ago.

10:00 PM: My sites have now been down for more than an hour. No idea when they’ll be back.

10:38 PM: My sites came back online about 10 minutes ago.

10:39 PM: Correction — they were up for less than 10 minutes and are down again. Fuck.

11:04 PM: And we’re back again — at least for now. (Crosses fingers.)

11:20 PM: That didn’t take long. Down again.

Sept. 9 at 12:10 AM: Sites came back just after midnight; not sure how long they’ll last this time.

12:17 AM: Load averages skyrocket immediately after the sites come back online — from 17.45 at 12:03 to 181.88 just now — which suggests why the available period is only 10-15 minutes or so each time. Or at least how.

12:24 AM: No surprise, then, that the sites no longer load.

12:47 AM: Except for a few periods of less than 15 minutes (mentioned above), my sites have been gone for around four hours now.

12:48 AM: No sooner than … they’re back now. I have no faith at this point that it will be for long.

12:57 AM: And down again. Damn it.

8:00 AM: It looks like the worst of it may be over. I was able to post an update to The Map Room and it rebuilt rather quickly. Let’s hope that this more or less holds through the weekend.


Monday, September 4, 2006 pm30 12:32 pm

The server itself is quite speedy, and load averages are nominal, but Movable Type is very slow this morning. New posts and page rebuilds are taking a while, and some commenters are posting twice because of the long delay. Be advised that the first submission did go through, though it may take an awfully long time to appear at the moment.

Update, 9:20 PM: It improved considerably by late afternoon.

Update, Sept. 5 at 7:10 AM: DreamHost reports load issues on the server on which the MySQL database is being hosted. They’re moving services to other servers to balance the load and straighten things out.