PDA

View Full Version : What do they DO to WoW servers on Tuesdays?


Gary Robson
10-31-2006, 03:06 PM
This is a GQ, but it's about online gaming, so I figured it belonged in Cafe Society. Mods, feel free to move it around as you please.

Every Tuesday morning, all U.S. World of Warcraft servers are taken down for maintenance for six hours (5:00am to 11:00am Pacific). I've spent many years working in computers and networking, and I've never seen anything like this. With 187 servers ("Realms" in WoW parlance), why don't they take them down in rotation, so that players with alts on multiple servers can play pretty much any time?

And what kind of server maintenance takes six hours a week, during which the whole server has to be down? What are they actually DOING?

Ethilrist
10-31-2006, 03:18 PM
It's not so much that they're doing something as reserving a big chunk of time in all the crack addict's lives so that if they do have a five-hour upgrade to install, they can do so without people screaming bloody murder.

Doing rolling updates might be a more customer-service oriented way of doing things, but it would put extra stress on the servers everybody moves to while the update is being performed.

What they're actually doing: Probably rebooting, clearing off some colossal log files onto DVD backups, installing a zillion fiddly little patches for stuff you'd never notice in game, and doing a detailed scan of the PC data to see if they can tell whether or not people are cheating and, if so, how.

Darwin's Finch
10-31-2006, 03:23 PM
They are also doing the weekly honor calculations during that time. It may be easier to just do it for all servers all at once than to do it piecemeal. And who knows, once the expansion comes out it may not be necessary to take them down for as long, what with the revamping of the whole honor system.

RickJay
10-31-2006, 05:17 PM
They're regreasing the murlocs.

Fucking murlocs.

Sage Rat
10-31-2006, 05:28 PM
Personally, (and I say this since this isn't GQ) I prefer this to the CoH system of being down for an hour every single day. Once a week on Wednesday, during work hours, just ain't bad.

lisacurl
10-31-2006, 05:32 PM
My personal theory is that Blizzard's stacks of cash piles up so quickly that they need that time to move the money out of the server rooms.

Jayn_Newell
10-31-2006, 05:32 PM
And who knows, once the expansion comes out it may not be necessary to take them down for as long, what with the revamping of the whole honor system.

Don't count on it. They've been doing this for as long as the game has been live, long before the honour system was put into place. I think they just use the downtime as a good time to insert things like that--all patches are released on Tuesdays as well.

I suspect the main reason is what Ethilrist said--if they do rolling restarts then people who would normally play on a server that is down will create an alt on one that's up, causing extra stress. You see this already on days when not all the servers come up at the same time.

sandra_nz
10-31-2006, 05:34 PM
I think one of the reasons they don't do rolling updates is that whenever a server goes down unexpectedly, players on that server immediate create new characters on other servers and play them until their server is back up again.

So if you had rolling updates, you'd constantly be having new characters created on new servers, filling up those server quotas yet potentially only ever being used for 3 hours.

The Controvert
10-31-2006, 05:43 PM
How is that a problem? Don't they make enough money to provide equipment to support the server load even if people hop over?

Revenant Threshold
10-31-2006, 05:45 PM
How is that a problem? Don't they make enough money to provide equipment to support the server load even if people hop over? Probably. But why spend that extra money if they don't need to?

sandra_nz
10-31-2006, 05:55 PM
It's not about the server load on the server they've hopped on to, but the limit to the number of characters that can play on a server before they have to open a new server.

So if people are constantly creating new characters, they'll be constantly opening new servers which is going to impact on their profit (if the existing servers are not being fully utilised due to people starting and not continuing with a character).

Gary Robson
10-31-2006, 06:01 PM
They're regreasing the murlocs.

Fucking murlocs.Best answer yet, although lisacurl's is good, too.

I see Ethilrist's point about reserving the time for when they do need a 5-hour chunk (it's been 6 hours lately, BTW), but that just seems like an excessive amount of time. If the patches have been tested properly, they could be installed on a mirror and switched in a few minutes. A restart of the WoW server software probably takes long enough to swap out the log files while it's happening (create new files, swap the pointers, and then back up the old ones at your leisure). I fail to understand why the honor calculations and cheat detection can't happen while the server is up.

As for increased loads during rotating server outages, I don't buy it. There were 187 servers in the U.S. last time I counted. There are 168 hours in a week. Even if the maintenance needs to be 5 hours (which I don't believe), then they'd never need to have more than 5 or 6 servers down at a time. If every single person who was going to play on the downed server switched to another, it would be an increased load of under 3%. They certainly ought to have an extra 3% load capacity.

Scupper
10-31-2006, 06:11 PM
It's even weirder that they do it all in one chunk because of just how many physical boxes we're probably talking about: You enter an instance, you are jumping onto a different server. You go from Kalimdor to the Eastern Kingdoms, you are on a different server. You enter a battleground, you are on a different server.

I have no idea how many servers there are per realm, but I'm fairly confident that it's at least "several."

What I can't understand is how it isn't actually more difficult for Blizzard to get all their maintenance done on so many servers in the same damn maintenance window.

Which leads me, in a roundabout fashion, to suspect that the amount of "maintenance" that blizz does in these gaps is probably minimal. What they probably do is swap the main server for the mirror, take the mirror offline, and do maintenance on THAT. It may even be that they have three sets of hardware per realm and rotate them.

Sierra Indigo
10-31-2006, 06:37 PM
They're regreasing the murlocs.

Fucking murlocs.

QFT

Jayn_Newell
10-31-2006, 07:05 PM
As for increased loads during rotating server outages, I don't buy it. There were 187 servers in the U.S. last time I counted. There are 168 hours in a week. Even if the maintenance needs to be 5 hours (which I don't believe), then they'd never need to have more than 5 or 6 servers down at a time. If every single person who was going to play on the downed server switched to another, it would be an increased load of under 3%. They certainly ought to have an extra 3% load capacity.

That's a bit of a bad way of looking at it. They're not going to have servers going down in the evening if they can avoid it, because that's prime playing time. The reason they do it Tuesday mornings is because there aren't very many people playing at that time of day. If even one server went down during the evenings (this could screw raiding guilds a bit--even if it's the same night every week that's one less night avaliable to raid, and some nights are better than others depending on people's schedules) Blizzard would get an earful. It would probably be better to think of it in terms of there being seven days in a week, and that assuming they would take a server offline on a weekend. Although you do have a point, in the mornings the load is light enough that the other servers could probably handle it.

drachillix
10-31-2006, 07:57 PM
Which leads me, in a roundabout fashion, to suspect that the amount of "maintenance" that blizz does in these gaps is probably minimal. What they probably do is swap the main server for the mirror, take the mirror offline, and do maintenance on THAT. It may even be that they have three sets of hardware per realm and rotate them.

From what I know of clustering (which isn't much) this sounds like a fairly reasonable explanation.

Dropping a set of servers from a cluster would just reduce capacity a bit and could be done during off peak hours. Of course loading a patch and have it propagate across all servers in probably hundreds of clusters could take a while. In addition, I'm sure that there is a wide variety of hardware involved and it gives them some time to resolve any quirks that develop secondary to any hardware incompatibilities that may crop up during the patching.

SenorBeef
10-31-2006, 10:41 PM
It's even weirder that they do it all in one chunk because of just how many physical boxes we're probably talking about: You enter an instance, you are jumping onto a different server. You go from Kalimdor to the Eastern Kingdoms, you are on a different server. You enter a battleground, you are on a different server.

This might actually help to explain it, rather than make it more perplexing.

If they set it up so that servers specialize in, say, a certain group of instances, then that server could be running all instances for multiple realms. In that case, in order to take down that server, you have to disable multiple realms at the same time. It's probably easiest just to take the whole network down in that case.

It's probably technically feasible for them to create a rolling system, and all that - but if they're only disrupting a small amount of people who play on tuesday mornings (they probably did a statistical analysis to figure out the least disruptive time), they're just going with the easy thing that affects few people and offers them flexibility in the future than working out a complex rolling restart system that would benfit a small group of people.

congodwarf
10-31-2006, 10:41 PM
I started playing WoW in July, after becoming bored with CoH. Exactly one week after I started playing, my work started giving me Tuesdays off. I was so damn annoyed. All I wanted to do was spend the time trying to catch up to my boyfriend since he gets to play while I'm at work.
Just recently, I started getting mostly Mondays and Thursdays off. I don't have to deal with the stupid maintenance any more. BUT, my condo association has a set schedule for doing lawn maintenance and Monday is my turn for mowing. If it's not one thing, it's another. I just can't win. :D

shijinn
11-01-2006, 01:08 AM
don't forget they add another two hours downtime during patch days, plus the inevitable extra downtime post patch on select realms.


They're regreasing the murlocs.

Fucking murlocs.

Critical1
11-01-2006, 04:42 AM
No idea how many "servers" make up a "realm" but in everquest it was along the lines of 70 or more. each individual server only controled 3 or so zones.
I would guess that with Wow its al hell of alot less considering the basic differences in the 2 games. I wouldnt be surprised to learn that a wow realm was only a few linked servers, under 10 would be my bet.

jacquilynne
11-01-2006, 09:51 AM
Another factor that has to be considered for creating single instance outages rather than rolling ones throughout the week is the actual staffload required to do the work. It's likely that 'take all servers down' 'patch all servers' 'restart all servers' takes pretty close to the same amount of human time as 'take this server down' 'patch this server' 'restart this server'. If they do that separately throughout the week for each group of servers, they've just massively increased their staff costs to watch over the process.

The Octagon
11-01-2006, 10:07 AM
I am not a WOW player, but a regularly scheduled downtime makes more sense to me than a rolling update. I would be fairly miffed if I turned on the computer at a random time during the week, ready to play, and found there was an update and I couldn't log on. Scheduling them as they do, folks will not need to be startled by a heads up each Monday night, they'll get used to it, and find something else to do during the downtime, rather than getting mad about being kicked off halfway through a quest or whatever at X o clock on an Xday. I used to play a MUD, and I would sometimes log on a minute before the "5:00 minutes to one hour shutdown" prompt started going. Now that sucked.

lisacurl
11-01-2006, 10:50 AM
The Octagon's comments got me thinking. I'm a sysadmin and the regular Tuesday downtime in WoW never surprised me. It just makes sense. We have a regular maintenance window for our servers where our users know that we have the "right" to take them down to do such things as patch the OS or whatnot. Granted, we don't do this weekly, but we could. Just applying security patches to the OS, rebooting, doing honor calcs, reindexing or doing whatever maintenance they need to do to their databases, and getting a secure clean backup... it's possible they could use a lot of their time even on a non-patch week.

Gary Robson
11-02-2006, 07:46 PM
QFT??? What does QFT mean?

I am not a WOW player, but a regularly scheduled downtime makes more sense to me than a rolling update. I would be fairly miffed if I turned on the computer at a random time during the week, ready to play, and found there was an update and I couldn't log on.I don't think anybody was recommending random rolling downtime. I was talking about a regular schedule where these three realms are down Sunday from midnight to 4:00am; the next four are Sunday from 4:00am to 8:00am; and so on through the 187 U.S. realms, thus giving the best of both worlds: a schedule everybody can count on and the ability to still play the game during the scheduled downtime of your favorite realm.

The Octagon's comments got me thinking. I'm a sysadmin and the regular Tuesday downtime in WoW never surprised me. It just makes sense. We have a regular maintenance window for our servers where our users know that we have the "right" to take them down to do such things as patch the OS or whatnot.Last time I ran a significant network (about 10 years ago), our users expected 100% uptime. We had a scheduled maintenance window twice a year, and we did it in a couple of hours on Sunday night. Why would you plan for 5 hours of downtime a week (3% downtime)--AND do it during the regular workday instead of the middle of the night?

jayjay
11-02-2006, 07:59 PM
Scheduling them as they do, folks will not need to be startled by a heads up each Monday night, they'll get used to it, and find something else to do during the downtime

From experience, that something appears to be "bitch about the downtime on the WoW forums and livejournal community".

What does QFT mean?

Quoted For Truth (another WoW forums coinage)

do it during the regular workday instead of the middle of the night?

Because during the regular workday is the gaming equivalent of the middle of the night. And the middle of the night is the gaming equivalent of rush hour.

SenorBeef
11-02-2006, 08:12 PM
Don't they do the updates from like 4am-10am? That's not exactly gaming primetime.

jacquilynne
11-02-2006, 08:16 PM
Last time I ran a significant network (about 10 years ago), our users expected 100% uptime. We had a scheduled maintenance window twice a year, and we did it in a couple of hours on Sunday night. Why would you plan for 5 hours of downtime a week (3% downtime)--AND do it during the regular workday instead of the middle of the night?

Because paying people to do upgrades in the middle of the night is expensive? Especially is the middle of the night isn't the time of day when your usage is the lowest? It's a game, not a business application.

lisacurl
11-02-2006, 09:40 PM
Last time I ran a significant network (about 10 years ago), our users expected 100% uptime. We had a scheduled maintenance window twice a year, and we did it in a couple of hours on Sunday night. Why would you plan for 5 hours of downtime a week (3% downtime)--AND do it during the regular workday instead of the middle of the night?

As others have pointed out, what's true for a business application and environment is not necessarily - one might say emphatically not true for a gaming environment. And what has been true for other gaming environments won't necessarily be true for WoW, because there's never been a game with a player base this huge before.

The middle of the school and business day IS the best time to do maintenance. On my server, the west coasters and Alaskans might not log on to start playing until after 10pm central, the Aussies even later than that. We have raiding guilds for Aussies that start their regular runs at 11pm server time or later.

I also have to say this about the difference between a business customer and a WoW player. Blizzard really isn't losing any sleep over whether a bunch of players are getting their panties in a twist over a few hours downtime a week. They've reached heights of success with this game that are unimaginable, and they know very well that users have zero recourse if they don't like the maintenance schedule.

Best Topics: yahtzee bonus chips cutthroat rules lysol kills mold magnum cartridges volunteers firefighters brenda and eddie maru ship ameriprise theme song tinted green patel pronunciation swimming in boxers moonshine fuel simpsons plagiarismo large ejaculatory volume gumption etymology dnd warhammer coin wrappers staples origin of zipperhead does silver tarnish black title swallowed aa battery quentin dean poe public domain cap'n chronic brownswager meat saloon sounds jewish cossacks selling fur coat football pool tiebreaker skag urban dictionary redeeming coupons for cash value how does little caesars work butane lighter refill home depot do employers verify education rib bones safe for dogs someone stole my package ups the frugal gourmet jeff smith how to turn brownie mix into cake ralph williams car dealer michael jackson voice change button button twilight zone johnny cash eating cake what happens when you mix ammonia with bleach songs with french lyrics what was possible for a sharecropper who made money during a growing season? how much is a piano tuning what does bx stand for in medical terms what is the boiling point of isopropyl alcohol how to make thioacetone using tv as computer monitor screen cut off should i turn off electric water heater if water is off so it shall be written so it shall be done drain opener sulfuric acid 100 years is a century what is 1000 years throat pain only at night has there ever been a good dictator ultimate soldier challenge wiki car undercoating worth it air mattress new york what year would i be born in if i was 23 olive drab vs ranger green blue with two yellow stripes