If I wrote a post saying that Demigod sales were far below what we had hoped for and I said that the reason was due to piracy and that the answer was that we should have put some nasty copy protection on those DVDs to have prevented early piracy what do you think people would say?

I know what my answer to that would be.  I would say that Stardock couldn’t blame poor sales on piracy but rather the fact that the game’s built-in multiplayer match-making was totally broken for the first day of release due to its underestimation of network resources that a mainstream game would take and even when that got addressed, the multiplayer match-making for two weeks and counting has been incredibly flakey which affected reviews and word of mouth.  That’s what I would say.

And yet…

http://www.gamasutra.com/php-bin/news_index.php?story=23381\

Demigod debuts at #3 for top selling PC games at retail – bearing in mind that that was a partial week and that the majority of units sold were digital sales which weren’t counted.

But…but…what about those hundreds of thousands of pirates? Yep. Demigod is heavily pirated. And make no mistake, piracy pisses me off.  If you’re playing a pirated copy right now, if you’re one of those people on Hamachi or GameRanger playing a pirated copy and have been for more than a few days, then you should either buy it or accept that you’re a thief and quit rationalizing it any other way.

The reality that most PC game publishers ignore is that there are people who buy games and people who don’t buy games. The focus of a business is to increase its sales.  My job, as CEO of Stardock, is not to fight worldwide piracy no matter how much it aggravates me personally. My job is to maximize the sales of my product and service and I do that by focusing on the people who pay my salary – our customers.

As Ars Technica quoted over a year ago:

"The reason why we don't put copy protection on our games isn't because we're nice guys. We do it because the people who actually buy games don't like to mess with it. Our customers make the rules, not the pirates. Pirates don't count," Wardell argues. "When Sins popped up as the #1 best selling game at retail a couple weeks ago, a game that has no copy protect whatsoever, that should tell you that piracy is not the primary issue."

Even Demigod, a game that shipped with no copy protection on the DVD, was massively pirated, and has had, to put it mildly severe launch issues with its multiplayer match-making which has had a negative impact on its Metacritic score has still managed to debut at the top of retail sales charts  (not counting our digital sales).

Why is that?  At that point I can only speculate but the first reason is pretty straight forward: Demigod is an awesome game. Second, while the multiplayer matchmaking that comes with the game currently sucks, our customers know it will get fixed. Part of that is the demographic of Stardock customers. They’re more experienced, they know that some of the issues with the MP matchmaking aren’t due to rushing the game out or negligence but rather the fact that complicated systems sometimes don’t scale well and there is no substitute for time when it comes to fix them.

I think there are many lessons to be learned from Demigod.  For example, if I had to do it over again, I would be inclined to require a valid user account to play LAN even if it only has to be validated one time. That way, we could also make it a lot easier for a legal user to have a LAN party with a single license. Anyone who has played Demigod on Game Ranger probably knows what and why I'm bringing that up.

When the focus of energy is put on customers rather than fighting pirates, you end up with more sales.  It seems common sense to me but then again, I’m just an engineer.

To say it has been a difficult 2 weeks would be an understatement.  I’ve been making PC games for about 15 years and never in that time have I encountered a situation quite like what we’ve had with Demigod.

As most of you know, Demigod is an awesome game. If I may be so bold, Demigod is one of the best games that has been released in the last couple of years.  And yet, here we are, 2 weeks after release and the included multiplayer matchmaking is still flakey.

I would put it like this (based on logs and talking to people)

• 30% of users think the multiplayer matchmaking works flawlessly. You don’t hear much from them (obviously).
• 60% of users are finding the multiplayer matchmaking to be flakey. It works but it’s a pain in the ass to get into a game sometimes.
• 10% of users cannot be connected to other users period.

That last number will go up if the proxy servers aren’t put in place prior to the official European release as that last 10% is almost exclusively made up of people using ADSL in Europe.

So how did this happen?

Losing Messages

The single biggest problem has to do with lost messages.  In the beta, when we had only a couple thousand people using the system, it worked pretty well.  At first, it was pretty flakey but as we added more servers to handle the connections, the success rate went way up and by the end it was pretty perfect.  To be safe, we added extra servers to handle the expected extra players.

Then, one retailer broke the street date over the Easter break. This in itself wasn’t as big of a problem as one might think as the system could handle the legitimate players. But when the game inevitably got into warez channels, the pirates saturated our other servers (not the NAT server). Those servers handled things like validation and checking for updates. The validation kept pirates from being able to play in the Pantheon, skirmish or custom games but it also ate up a lot of our extra capacity.  The result was that for the first 3 days after the official release of the game, the online experience was totally hosed.  The early reviews reflected that unfortunately.

So for the first week, rather than focusing on the issue that would come to bite legitimate players for this past week, we were scrambling to create a mirror series of servers to isolate the pirates from the legitimate users.  That took us into mid last week.

By mid last week, it became apparent that the something was causing players to not get into the game.  We assumed it was simply bugs in Impulse Reactor since it was new and on Friday released a new version of it.  But it turned out there was nothing wrong with Impulse Reactor, the problem was coming from the NAT facilitator itself – it was sending messages but the game was never receiving them – or at least not before the time out period.

Adding more servers over the past few days has improved things bit by bit but not acceptably.

More Specific

When Bill and Rick decide to play Demigod together. The NAT facilitator connects them together, sends a message to Impulse Reactor which in turn hands over the socket to Demigod’s lobby.  We have verified that the NAT facilitator is sending the message but the message is never being received.  We know it has something to do with the load because when we try it out with a relatively small group of testers (~100) it works flawlessly.  It’s when it is put out there for tens of thousands of people that it falls apart.

Solving it

And that is where we are on Tuesday with regards to the connectivity. Figuring out why the message is sent from the NAT but not received by the game. There’s a lot of thoughts on this.  UDP (unlike TCP) isn’t guaranteed. You can learn more on this here.  Another thought is that the NAT queues up its messages and that by the time it sends it back to the game, it has timed out.    The team tells me they think they’ll have this mystery solved today.  If so, I’ll report here on this.  But this will put us up to 90%.

Proxy Servers

If I had a time machine and knew about the pain we would all be experiencing in the first couple of weeks after release, I would have simply had us set up a series of host servers around the world. When someone started a game, they would simply connect to one of those servers and there’s be no connection issues.  That’s what we have been planning to do with Elemental.  But I was of the same mind that many here have said “Hey, lots of games do this, how hard can it be?”

Well, it turns out, it is very hard.  Most vendors have used GameSpy over the years. Blizzard developed Battle.net over many years. Relic migrated to a system it developed with Quazal to develop. 

The only reason we should be able to develop a Proxy system quickly is because we already have a worldwide network infrastructure because of Impulse so the hardware is already there, the agreements with ISPs in various countries already in place.  Work on this system has been going on in parallel with the NAT issue.  What will happen is rather than getting a NAT failed message, the user will connect to one of our servers and have their traffic routed through that.  I don’t have a good ETA other than (very soon – days not weeks we expect).

On a personal note

We read the forums and we hang out on the chat and we even read comments elsewhere when we’re at home.  First, we appreciate all the support and understanding our community has had.  To those who are experiencing problems, we share your frustrations. Many of us at Stardock and Gas Powered Games have spent an immense amount of time making Demigod a great game.  For those of you who are parents, I suspect you can empathize with what those of us with small children at home who have gone weeks without seeing their dad (or mom). As a game designer, it has been quite maddening to see the launch of a great game marred by IT and network stuff.  And I can tell you, getting this network stuff nailed down is easily as complicated as dealing with all the various video cards and sound cards and other typical PC pain. 

The teams at Stardock and GPG do believe those issues will be solved soon. And we thank you for your continued patience.

WEDNESDAY UPDATE

I am currently sitting in the lab with the GPG, Stardock, and Raknet teams interacting.  GPG feels good about having taken care of the in-game disconnect issue. The "big" connectivity issue as described above seems to increasingly appear to be related to having to hand off the socket to the Demigod client. This creates a series of timing issues that gets exponentially worse the more people trying to connect multiplied by the # of users on the NAT server.   

These are just my ideas, feel free to add.

Name of Demigod: TBD

Type of Demigod: General

General description: More of a classic wizard. Carries a staff, wears long robes.

Abilities:

Ability 1.

1. Set totems that give friendly Demigods 20% more experience in a given area.  (level 1)
2.  Totems  now also give friendly Demigods 20% more HP in a given area. Last 30 seconds. (level 3)
3.  totems now also  give friendly Demigods 20% more attack damage in a given area. Lasts 30 seconds. (level 8).
4. totelms now also give friendly demigods 20% more armor in a given area. (level 10)

1. Spell: Immunity. Makes friendly demigods immune from attacks for 2,3,4 seconds (depending on level).

Last week we saw some more reviews of Demigod. G4’s Xplay gave us 4 out of 5. Gamervision gave us a 9.5 out of 10, GameZone gave us an 8.7 ouf of 10 and a number of other positive reviews that I don’t have handy also came out.

Despite that, the custom game experience of Demigod has remained an exercise in frustration.  The skirmish games seem to be working better but largely because it’s picking 2 on 2 games with only occasionally 3 on 3 games.

So why is this so hard to nail down? And what is being done to solve it?

Some people are blaming Peer to Peer as the source of the pain. But most RTS’s use some form of P2P. What makes Demigod different?

The biggest difference is the form in which Demigod takes its P2P. In a game like Warcraft 3, the players route their traffic through the host. Hence, some people can join Warcraft games but can’t host them. Now you know why.

Supreme Commander is the same P2P model as Demigod and it works pretty well. The difference, however, is that most SupCom games are duels (1 vs. 1) and when they’re teams they’re 2 on 2. This tends to hide the complexity of the system.  By contrast, in Demigod, people expect to do 4 on 4 games or 5 on 5 games and it’s a significant challenge to connect them all reliably.

From Friday

On Friday, the team implemented a new system to try to make setting up the connections more orderly. After 3 days of working on it internally with great results, it was released and promptly blew up requiring us to replace it with an alternative update we had “just in case”.

I read a lot of the comments on the forums and I can say we share the frustration others have.  Developer Gas Powered Games is working hard on their end and we are working hard on our end to resolve the issues. 

This week

Monday:

Start the next pantheon tournament and continue to tweak how it works to satisfy the largest % of players.

Tuesday:

Found and fixed the cause of negative favor points. Favor points should now work though we need another client update to store purchased favor items.

Wednesday:

We got a new build of Demigod from Gas Powered Games that aims to address: 

·         Eradicates once and for all the problem where one person disconnecting or lagging would cause the peer-to-peer networking to unravel, collapse, and spawn multiple games. This is cross-functional for all matchmaking setups (LAN, Skirmish, Pantheon, Custom). 

·         Introduces a cumulative tolerance timer of 30 seconds for each player, which gives shaky connections better opportunities to stay connected, and eventually culls serious problem connections. 

·         Reintroduces an informational UI for players that lets them know who is having a connection problem, and what their cumulative timer is up to. It is important to note that this UI is NOT Interactive, merely informative. It is also a bit ugly, but compared to not giving players information, this is the much lesser evil. If need be we can revisit later for UI improvements as that is much lower priority than the information it presents. 

Meanwhile, Stardock worked with the developer of the network library it uses (Raknet) to find a way to make connecting players not be flakey.  The bad news is that it increasingly looks like problems are resulting from timing issues in the complex system of handing off sockets between Raknet to Impulse/Demigod. 

UDP Proxy server code implemented internally with testing expected to begin on Thursday.

In the forums

We are also going to start some pinned topics on things we to expand the game with (for free) as the connectivity mess gets sorted out. 

Topics will be:

• New Demigod #1
• New Demigod #2
• How do you want Clans to work
• Group Joins

The big update we were working on to get up this week went down to the wire which was a new NAT facilitator that made joining games extremely fast and extremely robust. 

Since we’re doing some of this on the fly, we obviously didn’t get to stress test scale.  And so what works great for say 50 people testing things out fell apart when a few thousand people jumped on. 

We had considered that possibility so we had an alternative, more conservative build already wound up and flipped the switch on that.  But the result is that the progress for those having connectivity issues in MP was not as far as we had hoped.

Gas Powered Games and Stardock are taking the weekend off to try to recharge a little bit. There is high confidence on both teams that those people having problems connecting will see their problems resolved shortly. 

Like those of you running into those kinds of issues, I get frustrated at this kind of thing. Tonight was particularly tough since it was working so well.

We appreciate your patience and on behalf of everyone at Stardock and GPG, have a good weekend and you have our dedication.

Hopefully next week we can start talking about additional Demigods.

In the meantime, I hate to have to recommend a third party program but Game Ranger does work pretty well as long as you’re playing other people with the legitimate version: www.gameranger.com.

Demigod: v1.0.105 (EXE version) v1.0.055 on Impulse

(tip, if you put your mouse over the Demigod icon in the bottom left in Impulse it will tell you the revision).

It’s Friday not and we’ve got the Demigod update out.  So what’s new?

• The disconnect threshold has been increased which should hopefully decrease in-game disconnects
• Various timing changes in-game to make connecting to games more effective

What isn’t in this update:

- The disconnect issue when joining a skirmish game. Highly frustrating. It’s some sort of timing issue where it’s just thinking people are too slow. GPG is working on this and we hope to have another update early next week.

- This is not the update with the Proxy Server setup. That’s also for next week. So if you simply cannot even see other players, this update won’t fix that, that’s something we’re going to try to get into next week. But I have a feeling that a lot of people were having timing issues which hopefully, at least in custom games, this version helps.

- We initially had a new NAT fascilitator which addressed quite a few things but quickly discovered that it wouldn't scale well to the # of users trying to use it thus creating a deadlock so we had to pull it. We'll give it a go again on Monday. We apologize for the delay on that.

We’ll be around this weekend to see how things go.

Good luck and thank you again for your patience.

For your amusement…

[A scene from outside]

When the update gets done and the gears start to turn to start uploading it to servers, there’s little to do except wait for it. Unless, of course, you’re a kook with bees.

Not much information in this video so skip it unless you want to see what the outside of the Stardock building is like.

A lot of exciting things are in the works for WindowBlinds 7.  The target date for it hasn’t been set. It’ll be “when it’s done” but it will be, by far, the most significant update to WindowBlinds ever. We’re taking it in a pretty different direction.

The above is just a mockup but give you an idea of some of the things we’re thinking for for the config UI.

We uncovered a bizarre but reproduceable scenario where Google Desktop running while Demigod was running caused Demigod's framerate, particularly in MP, to absolutely tank. It comes across as making the game extremely laggy (i.e. fine for 3 seconds, then a pause, etc.).

We don't know why this is, but I wanted to make sure you guys are aware of it.

Me (Draginol is my other handle) and my friend at GPG working on a problem

So today we got a pretty good new update from Gas Powered Games that we’ve been testing with today.

A lot of the changes have to do with tolerances. How much time should the game wait for others to join before it assumes they’re too slow and kicks them out before the game even starts? How much time should it wait for someone to join a lobby or a game before it says “No, they’re not going to get in?” and blocks them?

So today we played a lot of games with the new build and saw a lot of improvements. Thanks to tricks we’ve picked up from last week when the servers were being crushed, we have gotten a lot better at simulating peek server hours where the connection stuff is most likely to fail.

Right now, the issue we’re trying to fix is where people join a custom game and because they joined at the same time, they are not connected to each other.   That’s what we’re play testing right now and the results look very promising.

Developing games is like a strategy game

Like a strategy game, I’m in the unpleasant position of having to make choices on what people work on and what doesn’t.

If, by now, you have concluded that Demigod is an unusual game in many ways, you’re right.  Typically, the publisher (Stardock) releases the game, collects data and sends them off to the game’s developer (Gas Powered Games) and waits for a “patch” which the publisher then puts through QA and then releases.

Demigod, as you probably have surmised, has been developed differently.  Early last year, I pushed for Demigod to have more single-player elements. But over time, it became clear that everyone involved (at both Stardock and GPG) really enjoyed it more as a multiplayer game.  But in my mind, if you’re going to do that, you need to have lots of cool things like match making, and tournament stuff and long-term expandability.

That’s where Stardock started getting involved in development little by little. At first, it was just a single developer. Then another, and then another.  Pretty soon, we were adding overlays to present more player data, developing a friends list and other community features along with a multiplayer match-making, pantheon stuff, etc.

Stardock and GPG get along great. But you are talking about two veteran development teams working on some pretty sophisticated stuff at great distances.  That’s where I’ve increasingly become involved in pressing both Stardock and GPG development teams on what needs to go in when.  So if you have an issue with Demigod, don’t blame anyone but me because since the game’s release, I’ve essentially taken on the overall development prioritization for good or ill.

Ramifications

So for example, we know that most people can connect to other people. We know this for a fact. The problem is that for a significant percentage of people, it’s #$%# flakey. 

You know what I mean. You set up a custom game and it becomes a fiasco getting people into that custom game because people aren’t connected. Or you try to get a pantheon game going and players get kicked out or you end up with 2 on 2 with a bunch of bots.  Or the same thing with skirmish.

And then there’s the people who just get NAT failed messages or who just can’t connect no matter what.

I’ve gotten a lot of email telling me that I’m “killing Demigod” because I keep making things out to be “worse than they are”.  It’s not my job to paint a happy picture. I’m an engineer, not a marketer. And I hang out enough on the chat channel and on the forums to know that the problems mentioned above are real, must be addressed ASAP!

So anyway, during this week the big architecture change going on is the UDP proxy server setup.  But IT has told me that it will take a few days to do proper load testing on it (i.e. what happens when there’s 20,000 people on there?) 

But the tactical things I’ve been focusing Stardock/GPG onto this week are fixing the connection issues for those who play but find the game flakey, making Pantheon and Skirmish not have bots in them if people have experience, working on the favor point tracking, etc. 

Since I have only N developers at Stardock and M at GPG, that means that if I’m focusing them on those things, other things will go more slowly (including the Proxy stuff).

Tonight’s build is being tested

So like on Tuesday, we do have a new build, this one much more significant because it includes a new Demigod.exe too, but we’re testing it and since it’s 9pm as I write this, it’s not a good idea to put it up this late so it looks like it’ll be tomorrow.

Status Updates long term

I’ve read comments that some people do like these status reports.  Since we’re starting to come to the end of the serious connection problems (it may not seem like to some externally but it really is getting pretty close) I was wondering if people would like to continue to see these things when we start to talk about ideas for new Demigods, new items, new multiplayer modes, how you’d like to see clans and groups and friends handled.  I really don’t mind doing these but obviously don’t want people to get sick of reading these walls of text.

Cheers all and see you tomorrow!