Most of the current status can be found here.

So I played a lot of games of Demigod online today. And the issues mentioned in the Friday status report were ones I experienced occasionally.  I didn’t experience many problems with the exception of people getting kicked out of a Pantheon game. But I’ve gone through the responses here and forwarded them on to both Stardock’s NetDev team and GPG after doing my own analysis.

One of the things I’ve concluded we’re going to need to do to make things better quicker is to set up proxy servers for people who simply can’t directly connect.  So the player would connect to us and then to that user. In theory, it should add relatively little lag but it would dramatically ease the pain some users, especially those overseas, are having.

The plan is to bring the team together tomorrow (Sunday) to finish up what we had on Friday which should help a lot with skirmish and pantheon.

One request I do have for the community: While we know statistically that multiplayer is working in most cases, there are a lot of people who cannot connect. Please refrain from telling users that they need to learn how to use their router or whatever. The fact is, people shouldn’t have to configure their routers to play a video game. We do ask those who are having problems to have as much patience as they can muster. Both Stardock and GPG are doing their best to make the multiplayer experience good for everyone.

Anyone familiar with Stardock can also tell you that we support our games – for years. We’re on the job.

I see reports of "bad connectivity".  But please understand, when we play the game with people online it works perfectly so we need to try to understand specifically what you're seeing.

So when you say "I can't get into a multiplayer custom game" for instance, what specifically do you mean?

Can you bring up the connectivity dialog? What does it say? 

That kind of thing.  The more you tell us, the quicker we can resolve it. 

See http://forums.demigodthegame.com/347467 for where we're at.

Thanks!

Well, what a dramatic week it’s been. The teams at Stardock and GPG have been burning the midnight oil this week.

As those of you who have the game can already see, the server issues are gone. We’ve recreated a duplicate of the server infrastructure we had but dedicated to users who have the most recent version of the game and a valid CD key (serial #).

Based on the logs, we are seeing lots of games being played on-line now.  Yay. Average game has approximately 4.7 humans in it which is a good sign.

Some clarifications

I’ve seen a lot of news articles this week and a lot of confusion about what occurred this week. The issue isn’t terribly complicated.

Ars Technica had a good article that describes what happened.  But still, a lot of people seem to think warez users are able to play multiplayer games. No, they can’t.  Even the retail box has a serial # in it that users have to use and be validated to play online.  What brought down servers was a lot more benign than that.  It was the HTTPS requests to inform users if there was a new version along with checking the community features for info (friends lists, chat channels, etc.) and things like that.  Things like that are pretty piddly. It’s only when you get a ton of users doing that at the same time that it becomes a problem as we saw.

But here’s the thing: While piracy is annoying, you can’t blame piracy for this problem. Let’s face it, there’s plenty of data out there about how many pirated games are being played. We should have looked at that.  We assumed since Sins of a Solar Empire and Galactic Civilizations, both of which sold extremely well and got great reviews, that the # of pirated copies of Demigod in use would probably be in the same ballpark, maybe twice as much.  But had we looked at what other publishers have said, we would have known that it’s not unusual for there to be hundreds of thousands of warez copies in use. And if we had, we could have simply had the retail version not have any HTTP calls in it and instead just had an update button on the main menu to check for updates and voila, problem solved.

The second misconception is the argument that because Demigod’s retail version is heavily pirated that it costs massive sales. But that, again, puts the blame on the wrong parties.  If you want to talk about the horrible multiplayer experience on launch day, well, that’s our fault because of what I said above.  If you want to say that the horrible day 1 multiplayer experience resulted in negative game reviews which will seriously damage the game’s sales then I say again, that’s our fault too because of what I said above OR we could have just sent out the review copies on release day (Tuesday) and reviewers wouldn’t have had it until Thursday by which point the problem had largely been resolved and the review scores would have been fine. But in either case, it’s still our fault.

So now what?

Now that the servers are working fine we’re moving away from the “#$R@#@# Demigod sux!” posts and into the regular new game release issues.

So what issues are we seeing and working on? Here are a few at the top of our lists:

1. Players getting disconnected during games.  Demigod’s lag tolerance is fairly low resulting in disconnects if a player lags out a bit. This is fairly easy to fix.  You get a player in Australia playing a user in Europe and there will be times when there’s a hicup in their connection and POW, disconnect and it’s extremely frustrating.  I played all day today and it happened to me.  This is a very high priority.

2. NAT negotiation. For users outside the United States in particular using DSL, this is a problem. This is a case where player A can’t see player B and thus they can’t play together.  This is something we will be aggressively looking at next week. If we hadn’t had the server overload, we likely would have this addressed already.

3. Pantheon games.  Right now, the system is excessively biased for new players – it wants to include AI players as cannon fodder.  While this is fine and dandy for new players, once you know what you’re doing, having AI players in the game is incredibly annoying.  So next week we’re going to change it so that if you have played a few games, it will wean the player off of AI players until you only play humans online. Same for Skirmish.

4. Favor Points and such. There’s some annoying stuff happening with favor points which is related to the ranking system in Demigod as well. First off, we plan to reset the rankings at some point when we enter Epoch 1 (we are in Epoch 0 presently game-wise, I’ll explain Epochs later but in short, every so often we’re going to reset the rankings, save the results as an epoch and go from there).  In Epoch 1, custom games won’t count anymore as there’s too many ways to game it.

5. Misc. Bugs.  There was a crush bug in the game that we’ve fixed internally but it got to be too late in the evening to package it up and get it released tonight.  We have internally fixed the annoying issue of “A ppplayer has left the game” at the start of skirmish and pantheon games. Basically, when you get a new game going, there’s some data that has to be exchanged between players at the start.  But the threshold of time you’re given to do this before it decides you’re too slow is set too low so we’re changing this to be more tolerant. In our tests with players we found on the chat channel who were having problems, it took care of it. 

So that’s what we have in store for next week. After that, we can look at the typical balance requests, cheese stuff, and whatever else crops up.

Now, while the reviews are likely to have a pretty serious impact on sales, it does not affect our plans to continue to release updates and enhancements to the game nor does it affect our plans to release additional free Demigods.

For most of you, have fun this weekend!  For those of you who are having a problem with something, hang in there. We’ve got your back!

Draginol is my home account in case people are confused.

Brad "Frogboy" Wardell

After a marathon coding session with lots of server moving we were able to release a Day 2 update which, from what we can tell from both the logs and from talking to people in chat has made a drastic improvement in the multiplayer experience.  In a nutshell, today’s update let us connect to a different multiplayer NAT facilitator than the original. This one isn’t touched by users who get the game at retail (or those who have a warez version of the retail).

Now, I do want to address something about the update that I’ve seen users talk about.  We aren’t blaming piracy for the fact that the day 0 multiplayer experience absolutely sucked.  The issue boiled down to us having put together a multiplayer infrastructure that was designed to handle around 50,000 or so connected users. If the game took off, we would simply add more servers as the load increased. 

But what happened was that we ended up with 140,000 connected users, of which about 12% were actually legitimate customers.  Now, the roughly 120,000 users that weren’t running legitimate copies of the game weren’t online playing multiplayer or anything.  The issue with those users was as benign as a handful of HTTP calls that did things like check for updates and general server keep alive. Pretty trivial on its own until you have 120,000 of them. Then you have what amounts to a DDOS attack on yourself.

So the day 2 update we released basically made sure legitimate customers were no longer being affected by those users. As a side note, no we can’t just eliminate the infrastructure being used up by warez users because they’re running the unprotected retail version and we can’t make a distinction between retail and pirated since there’s no copy protection. It’s not a huge deal in the long run (except to our metacritic score), it was just an unexpected challenge that made day 1 a very bad multiplayer experience.

What’s next…

So we still have lots of areas to improve the multiplayer experience such as dealing with European DSL stuff (one example) and we need to make the pantheon and skirmish stuff more robust and provide more feedback.  The team is probably exhausted so the next update may not happen until next week since the day 2 update seems to have made a pretty big difference for the majority that I think we can probably take the weekend off.

But next week I expect we’ll have an update that addresses a lot of the above plus starts to look at balance and regular in-game bug reports and such that we’ve been collecting.

Anyway, that’s all for now.  I’m going home to sleep. No really, I mean it this time!

A new update has arrived that should significantly improve the multiplayer experience.

See the change log here:

http://forums.demigodthegame.com/346928

Impulse Version: 1.0.48
Demigod EXE version: 1.00.0093

After a long night of connection server moving around along with a new ingame connection client, we are happy to release the day 2 update.

Change log:

+ Connection dialog now allows the host to eject players trying to connect.

+ New NAT Connection Server now isolated so not affected by general server load resulting in much faster, better connectivity.

+ Eliminated check for update on launching the game

+ CVP calls asynchronous (a server hicup on our side is a lot less likely to cause users to be disconnected)

Notes:

This isn’t a silver bullet. This is basically what Demigod would have been like on day 0 (minus the eject button) if the connection server wasn’t getting pounded.

That said, we’ve noticed some issues that you should be aware of that we plan to iron out this week including:

1. Because connections happen much faster in custom games, you can occasionally end up with players in the lobby that aren’t truly connected to everyone. Their ping will be yellow (or 0). The host will need to eject one of those players and have them rejoin.

2. This update does nothing for users who are behind a firewall or router that has ports locked. See http://forums.demigodthegame.com/345345 for tips in that area.

3. Obviously this is an update that we would normally want to run through a 2 or 3 limited beta since major surgery was done to the servers to reduce their load. But we felt strongly that we needed to get a release out that showed improvement for most people.  We expect to release another update in the next few days once we can see the effect of the connection server in conditions we consider normal.

Good luck!

I broke my 56 hour day this morning and took a nap before coming back.  One of the nice features of Stardock is that it has developers in the US and in Europe so we can, technically, work on things around the clock if necessary which is something we’re taking advantage right now.

So where are we at?

Today

The system works pretty well if you have a few thousand people online at once.  The system works…less well if there are tens of thousands of people online at once. And if there are over 100,000 people, well, you get horrific results such as the game being incredibly unresponsive due to simple web service calls that were considered pretty benign during the beta that suddenly start to bring down firewalls and such due to the sheer massive number of calls that are being made.

Sadly, most of the ~120,000 connections are not customers but via warez.  About 18,000 are legitimate.  So anyway, we spent a lot of time today  trying to isolate out the warez users from the legitimate users (it would require a lot of surgery to actually break them and even if we did, there’d be no friendly “ha ha pirate” message which would result in people just saying the game is buggy).  Mind you, the game makes relatively few server calls, it’s just the sheer number of people.

Our stress tests had counted on having maybe 50,000 people playing at once at peak and that wouldn’t be reached for a few weeks by which time we would have slowly seen things becoming problematic.  With Sins of a Solar Empire, the game was incredibly successful but its user base grew slowly and steadily over time.  Sometimes on a peak time the server would start to get slow and we would adjust so that it would be better next time.

But here, when you’re getting that many connections at once, you’re no longer talking about dealing with the basics like having a good SAN and lots of redundant servers. You instead fall into all kinds of weird secondary issues that start to pop up like yesterday’s case where an older network card couldn’t handle all the packets (not bandwidth limited but # of tiny packets being sent out at once).  

So during the day today, people couldn’t even logon and in some cases, the Demigod forums, which use one of the affected databases for some piddly thing were even down. Even getting the game running was a pain today because a simple HTTP call to see what the latest version would get hung leaving people looking at a black screen. Stuff of nightmares.

Before the game shipped, I wrote a scary email to our team saying how disastrous things would be and predicted doom for us and GPG if there were problems with multiplayer. At the time, my worry was about things like disconnects and CVP. It didn’t occur to me that we’d have near MMO user connections to throw in.

Tonight

Last night and all day today we’ve been working on multiple fronts. 

First, we’ve been trying to shuttle off the warez users so that they’re not slamming our servers. Well, not slamming the same servers that legitimate customers are using.  I’m feeling pretty confident that that is working better because there’s a lot more people in games with lots of people in them.

Second, we’ve developed a new connection system (pictured below).  The connection stuff in Demigod is not part of Demigod but is rather part of Impulse Reactor which in turn is talking to the various connection servers we have set up to get player A to connect to player B and so on.  It’s not done yet but amongst its new features is the ability for the host to eject players who just can’t connect.

This whole new connection service has been written in the past 48 hours.

Third, we’ve greatly expanded our capacity.  I tell you, anyone who wants to know why we delayed Society (MMO RTS we are slowly working on) need look no further than how painful the first 24 hours since the official release of Demigod have been. That was done today and hopefully users have noticed an improvement.

We’re in the process of testing out this new update to put up tonight (less than 24 hours after the street date) that should largely fix this.

Early reviews

The early reviews of Demigod have been very good – in general.

Neoseeker gave us a 10 out of 10.

Crispy Gamer gave us a “Buy It”

But the GameSpot review nailed us because of the connectivity with a 6.5. I’m pretty disappointed about that. [Begin Rant] First, I totally understand that connectivity is central to a game like this. I totally agree. But I think that should be weighted with what the average user who gets Demigod will experience and in reality, as annoying as this issue is, it’s not something that’s going to be an ongoing issue, it’s something that is likely to be taken care of in the next day or two.  So this time next week, players will be happily playing but GameSpot’s review will live on.  I suppose the solution going forward isn’t to send out review copies until the game is released and we make sure there’s no network issues. Afterall, in a week from now, the guy buying it from retail will have a great experience. [End Rant]

The IGN and 1Up reviews are coming up.  So we could be looking at a lot of reviews like that if every reviewer decides to take the same approach Gamespot did which I hope they don’t (luckily for every other on-line centric game that’s had a rough first day or two most reviewers haven’t taken Gamespot’s approach).

Video

We took a video about some of the stuff we’re working on and a little quick tour of Stardock. You can see it here:

IGN Video

Latest status report here:

http://forums.demigodthegame.com/346815/

Hi gang!

What an exciting day! I've spent literally the last 40 or so hours playing Demigod on-line or working with people on the Demigod in game chat to show them how to debug some issue or other with either their PC, their router, or some flaw in Demigod.

For those of you having trouble getting multiplayer games going, I apologize.  

You know, funny thing about running a game developer/publisher.  You look at every game that launches with some multiplayer connectivity problem or some server overload problem or whatever and think "Good grief, didn't they see this coming!?"  Well, it turns out apparently that no, no matter how well prepared you think you are, there are always going to be issues.

While I can say that there are, even as I type, many hundreds of people playing online together, mostly in Skirmish and Pantheon games, there are many hundreds of others having trouble connecting to people in custom games or having skirmish or pantheon doing something odd due to the server overload.

We are working to address it and what I mean by that is that we intend to release another update this week to deal with as much of what we've found as possible.

It's too early to tell whether Demigod is going to match the success Sins of a Solar Empire has enjoyed. But we're pretty happy with early sales even if we're not totally satisfied with the game's network performance (which is not developed by GPG but by Stardock so don't blame GPG).

With the first official day of the game's availability almost over, I'd like to give you a status report on 4 critical issues we're working on addressing:

#1 Custom Games.  The lobby and the NAT connection are seperate which, in hindsight, wasn't an ideal design decision. It's something that can be addressed but it'll take some days to do it.  Basically, the connection system needs to be more like a lobby where the host can eject users who have problematic setups, slow ping times, etc.  In any other game, if you create a game and you see a user who's got a slow ping, you'd eject them probably.  But here, they have to first connect to everyone else.  And that assumes their router isn't set up like fort knox whicha lot of users clearly have.

#2 You're connected but not in the game. This appears to be a genuine bug on our part. We're looking to address this and with any luck we'll be able to provide an updated build today or tomorrow. This is a very frustrating problem for users.

#3 The Pantheon / Skirmish games create uneven teams and such.  This is mostly due to a lack of data for the system to work off of. When it was developed, it had a ton of beta tester win loss data to create balanced teams. Plus, from a "fun" point of view it was designed to try to get a skilled player versus 2 or 3 unski-er less experienced players as we found that pretty fun during the beta but the problem is, with everyone starting out with the same rating, it ends up with some bad results.  This situation will iron itself out in the next couple of days on its own but now you know why it's happening.

#4 Server Load.  We can't complete this list without talking about the overall server load Demigod's Pantheon and community features are using. A lot of this is simply volume but part of this is just finding areas where certain calls were being made too often. This will take a bit of time to address but I expect by the end of the week it'll be pretty responsive if not sooner.

As some of you know, we're all over the forums and on the chat channels hanging out. 

So how can we make any of this up to you? Well, we've decided one way is to expand on something that our partners at Ironclad (developers of Sins of a Solar Empire) really got going when Sins came out and that is put a lot of "Stardockians" online to play multiplayer games with people later this week.

These Stardockians will be on to help show people how to play the game, give them tips and provide just a good online experience. For people who found themselves trying to get a game going today and couldn't due to server overloads or network connectivity, you'll be able to get a good game going with us.

Thanks for your patience. We'll keep you up to date! 

Cheers

Brad "Frogboy" Wardell
President / CEO of Stardock

p.s. I'm on the chat channels if anyone needs me. Though I do plan to get some sleep in the next few hours once we have made some progress on an updated DLL for improving network connectivity.