I broke my 56 hour day this morning and took a nap before coming back.  One of the nice features of Stardock is that it has developers in the US and in Europe so we can, technically, work on things around the clock if necessary which is something we’re taking advantage right now.

So where are we at?

Today

The system works pretty well if you have a few thousand people online at once.  The system works…less well if there are tens of thousands of people online at once. And if there are over 100,000 people, well, you get horrific results such as the game being incredibly unresponsive due to simple web service calls that were considered pretty benign during the beta that suddenly start to bring down firewalls and such due to the sheer massive number of calls that are being made.

Sadly, most of the ~120,000 connections are not customers but via warez.  About 18,000 are legitimate.  So anyway, we spent a lot of time today  trying to isolate out the warez users from the legitimate users (it would require a lot of surgery to actually break them and even if we did, there’d be no friendly “ha ha pirate” message which would result in people just saying the game is buggy).  Mind you, the game makes relatively few server calls, it’s just the sheer number of people.

Our stress tests had counted on having maybe 50,000 people playing at once at peak and that wouldn’t be reached for a few weeks by which time we would have slowly seen things becoming problematic.  With Sins of a Solar Empire, the game was incredibly successful but its user base grew slowly and steadily over time.  Sometimes on a peak time the server would start to get slow and we would adjust so that it would be better next time.

But here, when you’re getting that many connections at once, you’re no longer talking about dealing with the basics like having a good SAN and lots of redundant servers. You instead fall into all kinds of weird secondary issues that start to pop up like yesterday’s case where an older network card couldn’t handle all the packets (not bandwidth limited but # of tiny packets being sent out at once).  

So during the day today, people couldn’t even logon and in some cases, the Demigod forums, which use one of the affected databases for some piddly thing were even down. Even getting the game running was a pain today because a simple HTTP call to see what the latest version would get hung leaving people looking at a black screen. Stuff of nightmares.

Before the game shipped, I wrote a scary email to our team saying how disastrous things would be and predicted doom for us and GPG if there were problems with multiplayer. At the time, my worry was about things like disconnects and CVP. It didn’t occur to me that we’d have near MMO user connections to throw in.

Tonight

Last night and all day today we’ve been working on multiple fronts. 

First, we’ve been trying to shuttle off the warez users so that they’re not slamming our servers. Well, not slamming the same servers that legitimate customers are using.  I’m feeling pretty confident that that is working better because there’s a lot more people in games with lots of people in them.

Second, we’ve developed a new connection system (pictured below).  The connection stuff in Demigod is not part of Demigod but is rather part of Impulse Reactor which in turn is talking to the various connection servers we have set up to get player A to connect to player B and so on.  It’s not done yet but amongst its new features is the ability for the host to eject players who just can’t connect.

This whole new connection service has been written in the past 48 hours.

Third, we’ve greatly expanded our capacity.  I tell you, anyone who wants to know why we delayed Society (MMO RTS we are slowly working on) need look no further than how painful the first 24 hours since the official release of Demigod have been. That was done today and hopefully users have noticed an improvement.

We’re in the process of testing out this new update to put up tonight (less than 24 hours after the street date) that should largely fix this.

Early reviews

The early reviews of Demigod have been very good – in general.

Neoseeker gave us a 10 out of 10.

Crispy Gamer gave us a “Buy It”

But the GameSpot review nailed us because of the connectivity with a 6.5. I’m pretty disappointed about that. [Begin Rant] First, I totally understand that connectivity is central to a game like this. I totally agree. But I think that should be weighted with what the average user who gets Demigod will experience and in reality, as annoying as this issue is, it’s not something that’s going to be an ongoing issue, it’s something that is likely to be taken care of in the next day or two.  So this time next week, players will be happily playing but GameSpot’s review will live on.  I suppose the solution going forward isn’t to send out review copies until the game is released and we make sure there’s no network issues. Afterall, in a week from now, the guy buying it from retail will have a great experience. [End Rant]

The IGN and 1Up reviews are coming up.  So we could be looking at a lot of reviews like that if every reviewer decides to take the same approach Gamespot did which I hope they don’t (luckily for every other on-line centric game that’s had a rough first day or two most reviewers haven’t taken Gamespot’s approach).

Video

We took a video about some of the stuff we’re working on and a little quick tour of Stardock. You can see it here:

IGN Video

Latest status report here:

http://forums.demigodthegame.com/346815/

Hi gang!

What an exciting day! I've spent literally the last 40 or so hours playing Demigod on-line or working with people on the Demigod in game chat to show them how to debug some issue or other with either their PC, their router, or some flaw in Demigod.

For those of you having trouble getting multiplayer games going, I apologize.  

You know, funny thing about running a game developer/publisher.  You look at every game that launches with some multiplayer connectivity problem or some server overload problem or whatever and think "Good grief, didn't they see this coming!?"  Well, it turns out apparently that no, no matter how well prepared you think you are, there are always going to be issues.

While I can say that there are, even as I type, many hundreds of people playing online together, mostly in Skirmish and Pantheon games, there are many hundreds of others having trouble connecting to people in custom games or having skirmish or pantheon doing something odd due to the server overload.

We are working to address it and what I mean by that is that we intend to release another update this week to deal with as much of what we've found as possible.

It's too early to tell whether Demigod is going to match the success Sins of a Solar Empire has enjoyed. But we're pretty happy with early sales even if we're not totally satisfied with the game's network performance (which is not developed by GPG but by Stardock so don't blame GPG).

With the first official day of the game's availability almost over, I'd like to give you a status report on 4 critical issues we're working on addressing:

#1 Custom Games.  The lobby and the NAT connection are seperate which, in hindsight, wasn't an ideal design decision. It's something that can be addressed but it'll take some days to do it.  Basically, the connection system needs to be more like a lobby where the host can eject users who have problematic setups, slow ping times, etc.  In any other game, if you create a game and you see a user who's got a slow ping, you'd eject them probably.  But here, they have to first connect to everyone else.  And that assumes their router isn't set up like fort knox whicha lot of users clearly have.

#2 You're connected but not in the game. This appears to be a genuine bug on our part. We're looking to address this and with any luck we'll be able to provide an updated build today or tomorrow. This is a very frustrating problem for users.

#3 The Pantheon / Skirmish games create uneven teams and such.  This is mostly due to a lack of data for the system to work off of. When it was developed, it had a ton of beta tester win loss data to create balanced teams. Plus, from a "fun" point of view it was designed to try to get a skilled player versus 2 or 3 unski-er less experienced players as we found that pretty fun during the beta but the problem is, with everyone starting out with the same rating, it ends up with some bad results.  This situation will iron itself out in the next couple of days on its own but now you know why it's happening.

#4 Server Load.  We can't complete this list without talking about the overall server load Demigod's Pantheon and community features are using. A lot of this is simply volume but part of this is just finding areas where certain calls were being made too often. This will take a bit of time to address but I expect by the end of the week it'll be pretty responsive if not sooner.

As some of you know, we're all over the forums and on the chat channels hanging out. 

So how can we make any of this up to you? Well, we've decided one way is to expand on something that our partners at Ironclad (developers of Sins of a Solar Empire) really got going when Sins came out and that is put a lot of "Stardockians" online to play multiplayer games with people later this week.

These Stardockians will be on to help show people how to play the game, give them tips and provide just a good online experience. For people who found themselves trying to get a game going today and couldn't due to server overloads or network connectivity, you'll be able to get a good game going with us.

Thanks for your patience. We'll keep you up to date! 

Cheers

Brad "Frogboy" Wardell
President / CEO of Stardock

p.s. I'm on the chat channels if anyone needs me. Though I do plan to get some sleep in the next few hours once we have made some progress on an updated DLL for improving network connectivity.

The official 0-day update will have more connection info as an option:

#1 cause of custom game pain: People being impatient.  The NAT connection system won’t insert people into the lobby until they’ve connected together. But if people leave, you can end up with it taking a lot longer.

Here’s another cause: Too many people trying to join a given game. Users aren’t inserted into the lobby until they’ve all connected.  Now, you might ask, why not just insert people as they connect to the host? The answer is because it would open a new socket for each one due to the way the lobby interacts with the NAT facilitator. The solution will be to have this dialog become richer so that the host can take people in.

Make sure you go to the technical support group to get answers to common Demigod issues:

http://forums.demigodthegame.com/forum/526

Problem: You load up Demigod, click Multiplayer, choose Internet, and type in your Impulse User Name and password and the game does not accept it.

Most common cause: You purchased the game via Impulse with a different email address (i.e. different account) than the one you're using.

Solution: Merge your accounts: https://www.stardock.com/support/registrations/

Second most common cause: You created a second Impulse account some other way. Solution is same as above.

To recap: Demigod's street date is tomorrow (4/14). But one retailer released it early which, in itself was problematic. Then, a couple days later, warez versions from the retail version started showing up on the net.  Until release day, only the test servers are up.

The result is that multiplayer is likely to be unbelievably slow (in terms of interacting with the Demigod UI).  Now, had the street date had been adhered to, it wouldn't have been a problem (even with piracy) because the servers will be up tomorrow.

However, we are taking this opportunity to launch these servers onto a different cluster and update Demigod today (we hope) with a version that will point to this new data cluster.

The good news is that people with legitimate licenses can download a very small update to Demigod and get very fast performance.  The bad news is that the gameplay experience for warez users will be awful as we plan to migrate most of our remaining test servers over to the release server cluster at the earliest opportunity.

As some people know, Gamestop violated the street date of Demigod and released it last week. There have been plenty of discussions about whether Gamestop did this intentionally or not which I won’t go into here.

Obviously there have been some significant consequences as a result:

First, it was Easter weekend. And many of us had just finished a good solid 8 weeks of massive crunch and were looking forward to the weekend to recover. Instead, we found ourselves back at work having to turn on and configure the multiplayer matchmaking servers (we had enough for a beta but not for thousands of people).

Second, since Demigod has zero copy protection on it, it meant that that piracy on this title will, in theory, be maximized. Stardock’s position on piracy is pretty straight forward but to repeat it here: It’s not that we don’t think piracy is massive. We just aren’t convinced that it results in that many lost sales. Or more to the point, we don’t think intrusive, obnoxious copy protection will result in more sales than we lose from people who don’t want to mess with it.

Imagine if Demigod had shipped with some nasty retail copy protection and got released early. Someone “cracks” it but doesn’t do a good job and the game ends up crashing randomly or something. Most people would just assume the game was buggy and that would likely cause more lost sales than the initial piracy did.

The good news, however, is that we’ll get to actually see the effects of piracy. Since we know Demigod is an outstanding game with a wide appeal with equal retail and digital distribution to other titles that came out this year with similar demographics we can see how well it sells compared to them. 

Third, while the game won’t be on the shelves at other retailers until tomorrow morning, we have elected to make the digitally distributed version available today with beta testers and pre-order customers getting it immediately (as in right now).  Those who have legally purchased the game can get the 0-day update that has all the game play changes that the beta group asked for, multiplayer match-making, and LAN support.

I want to thank the Stardockians who came in on their day off (we are supposed to have the Monday after Easter off) to make sure people who pre-ordered got the game today as well as the beta testers.

See you guys online!

As most of you know, Stardock games ship with zero copy protection.

That means someone could take Demigod and post it up somewhere.  But so far, no one has even though Gamestop violated their agreement and released the game early.  So the potential for "0 day warez" with Demigod is more so than most any other game because there's nothing to "crack".

We (Stardock) have always taken the position that people are fundamentally honest. I tend to think that things get warez'd because the copy protection provides a sort of challenge to overcome.  Since Demigod has no such copy protection on the DVD, there's no challenge and hence no point.

Incidentally, we are working on a demo version as well that should be out as soon as we recover from the initial launch.

Both Gas Powered Games and Stardock have poured their life and souls into this game. We're very proud with how it turned out and we thank those of you who have pre-ordered for sticking with us despite it being available. We'll definitely find some way to thank those users in the future.  We look forward to playing with you guys online and hanging out together in the growing Demigod community.

Happy Easter everyone!