The big update we were working on to get up this week went down to the wire which was a new NAT facilitator that made joining games extremely fast and extremely robust. 

Since we’re doing some of this on the fly, we obviously didn’t get to stress test scale.  And so what works great for say 50 people testing things out fell apart when a few thousand people jumped on. 

We had considered that possibility so we had an alternative, more conservative build already wound up and flipped the switch on that.  But the result is that the progress for those having connectivity issues in MP was not as far as we had hoped.

Gas Powered Games and Stardock are taking the weekend off to try to recharge a little bit. There is high confidence on both teams that those people having problems connecting will see their problems resolved shortly. 

Like those of you running into those kinds of issues, I get frustrated at this kind of thing. Tonight was particularly tough since it was working so well.

We appreciate your patience and on behalf of everyone at Stardock and GPG, have a good weekend and you have our dedication.

Hopefully next week we can start talking about additional Demigods.

In the meantime, I hate to have to recommend a third party program but Game Ranger does work pretty well as long as you’re playing other people with the legitimate version: www.gameranger.com.

Demigod: v1.0.105 (EXE version) v1.0.055 on Impulse

(tip, if you put your mouse over the Demigod icon in the bottom left in Impulse it will tell you the revision).

It’s Friday not and we’ve got the Demigod update out.  So what’s new?

• The disconnect threshold has been increased which should hopefully decrease in-game disconnects
• Various timing changes in-game to make connecting to games more effective

What isn’t in this update:

- The disconnect issue when joining a skirmish game. Highly frustrating. It’s some sort of timing issue where it’s just thinking people are too slow. GPG is working on this and we hope to have another update early next week.

- This is not the update with the Proxy Server setup. That’s also for next week. So if you simply cannot even see other players, this update won’t fix that, that’s something we’re going to try to get into next week. But I have a feeling that a lot of people were having timing issues which hopefully, at least in custom games, this version helps.

- We initially had a new NAT fascilitator which addressed quite a few things but quickly discovered that it wouldn't scale well to the # of users trying to use it thus creating a deadlock so we had to pull it. We'll give it a go again on Monday. We apologize for the delay on that.

We’ll be around this weekend to see how things go.

Good luck and thank you again for your patience.

For your amusement…

[A scene from outside]

When the update gets done and the gears start to turn to start uploading it to servers, there’s little to do except wait for it. Unless, of course, you’re a kook with bees.

Not much information in this video so skip it unless you want to see what the outside of the Stardock building is like.

A lot of exciting things are in the works for WindowBlinds 7.  The target date for it hasn’t been set. It’ll be “when it’s done” but it will be, by far, the most significant update to WindowBlinds ever. We’re taking it in a pretty different direction.

The above is just a mockup but give you an idea of some of the things we’re thinking for for the config UI.

We uncovered a bizarre but reproduceable scenario where Google Desktop running while Demigod was running caused Demigod's framerate, particularly in MP, to absolutely tank. It comes across as making the game extremely laggy (i.e. fine for 3 seconds, then a pause, etc.).

We don't know why this is, but I wanted to make sure you guys are aware of it.

Me (Draginol is my other handle) and my friend at GPG working on a problem

So today we got a pretty good new update from Gas Powered Games that we’ve been testing with today.

A lot of the changes have to do with tolerances. How much time should the game wait for others to join before it assumes they’re too slow and kicks them out before the game even starts? How much time should it wait for someone to join a lobby or a game before it says “No, they’re not going to get in?” and blocks them?

So today we played a lot of games with the new build and saw a lot of improvements. Thanks to tricks we’ve picked up from last week when the servers were being crushed, we have gotten a lot better at simulating peek server hours where the connection stuff is most likely to fail.

Right now, the issue we’re trying to fix is where people join a custom game and because they joined at the same time, they are not connected to each other.   That’s what we’re play testing right now and the results look very promising.

Developing games is like a strategy game

Like a strategy game, I’m in the unpleasant position of having to make choices on what people work on and what doesn’t.

If, by now, you have concluded that Demigod is an unusual game in many ways, you’re right.  Typically, the publisher (Stardock) releases the game, collects data and sends them off to the game’s developer (Gas Powered Games) and waits for a “patch” which the publisher then puts through QA and then releases.

Demigod, as you probably have surmised, has been developed differently.  Early last year, I pushed for Demigod to have more single-player elements. But over time, it became clear that everyone involved (at both Stardock and GPG) really enjoyed it more as a multiplayer game.  But in my mind, if you’re going to do that, you need to have lots of cool things like match making, and tournament stuff and long-term expandability.

That’s where Stardock started getting involved in development little by little. At first, it was just a single developer. Then another, and then another.  Pretty soon, we were adding overlays to present more player data, developing a friends list and other community features along with a multiplayer match-making, pantheon stuff, etc.

Stardock and GPG get along great. But you are talking about two veteran development teams working on some pretty sophisticated stuff at great distances.  That’s where I’ve increasingly become involved in pressing both Stardock and GPG development teams on what needs to go in when.  So if you have an issue with Demigod, don’t blame anyone but me because since the game’s release, I’ve essentially taken on the overall development prioritization for good or ill.

Ramifications

So for example, we know that most people can connect to other people. We know this for a fact. The problem is that for a significant percentage of people, it’s #$%# flakey. 

You know what I mean. You set up a custom game and it becomes a fiasco getting people into that custom game because people aren’t connected. Or you try to get a pantheon game going and players get kicked out or you end up with 2 on 2 with a bunch of bots.  Or the same thing with skirmish.

And then there’s the people who just get NAT failed messages or who just can’t connect no matter what.

I’ve gotten a lot of email telling me that I’m “killing Demigod” because I keep making things out to be “worse than they are”.  It’s not my job to paint a happy picture. I’m an engineer, not a marketer. And I hang out enough on the chat channel and on the forums to know that the problems mentioned above are real, must be addressed ASAP!

So anyway, during this week the big architecture change going on is the UDP proxy server setup.  But IT has told me that it will take a few days to do proper load testing on it (i.e. what happens when there’s 20,000 people on there?) 

But the tactical things I’ve been focusing Stardock/GPG onto this week are fixing the connection issues for those who play but find the game flakey, making Pantheon and Skirmish not have bots in them if people have experience, working on the favor point tracking, etc. 

Since I have only N developers at Stardock and M at GPG, that means that if I’m focusing them on those things, other things will go more slowly (including the Proxy stuff).

Tonight’s build is being tested

So like on Tuesday, we do have a new build, this one much more significant because it includes a new Demigod.exe too, but we’re testing it and since it’s 9pm as I write this, it’s not a good idea to put it up this late so it looks like it’ll be tomorrow.

Status Updates long term

I’ve read comments that some people do like these status reports.  Since we’re starting to come to the end of the serious connection problems (it may not seem like to some externally but it really is getting pretty close) I was wondering if people would like to continue to see these things when we start to talk about ideas for new Demigods, new items, new multiplayer modes, how you’d like to see clans and groups and friends handled.  I really don’t mind doing these but obviously don’t want people to get sick of reading these walls of text.

Cheers all and see you tomorrow!

[Video Status report]

For those of you who are interested in what is happening, what things we’re looking to do, here’s a video where I walk around and ask some of the key people on the Stardock side of things what they’re doing to keep improving the Demigod player experience.

Well I hope you guys aren’t getting sick of these status reports.  As I type this the build from last night is going up. Here’s what’s new:

• Added check in Pantheon and Skirmish games to detect if host has cancelled or all the other human players quit, and pops up a message if this happens so that you can cancel instead of being stuck.  GPG is working on the other side of this fix to make it so that you won't have to cancel and the remaining human players can continue to the game.
• Tweaked NAT punchthrough settings to try and make success more likely
• Put in a proper error message for the case when the player quits rather than generic “Disconnected from NAT”
• Put in a proper error message for the case when someone who started a pantheon game quits before others get in (should largely take care of the staring at the connecting thing forever).
• Added additional check in the connection code to make sure that both players have connection data for each other before handing off the connection data to Demigod, which may prevent unidirectional
• Fixed the connection dialog to refresh the players connecting more aggressively.
• Optimized Friends list code for people with no friends
• Messages for Disconnected from NAT facilitator now contain more useful information (disconnected because game was cancelled, game has already started, etc)
• Reduced delay before allowing people in the lobby, which may cause more people to get in the lobby without being connected to all players.
• Fixed an incompatibility between Direct3D and the Overlay which would cause players to crash, particularly on alt-esc.
• Ground work for UDP Proxy, you should notice connections being faster and better if you’re in Europe.

This update should hit Impulse later tonight.  As you can tell by the change log, this is really an interim update.  We have a much more significant update in testing but we didn’t want to wait until that build was ready to get a new build out there that improves things. 

A preview of what’s next:

I can’t promise that we’ll have another update tomorrow. There’s a lot of coordination that has to take place between GPG and Stardock and we’re 3 hours apart time-zone wise.  But let me give you a head’s up on things we’re pretty close on:

1. Taking care of the timing problem where player says they’re connected to everyone but they don’t get into the game. It’s a timing issue that’s hard to replicate but it doesn’t happen for us under normal circumstances. But it is probably the #1 issue for those who don’t have NAT issues. This really shows up in pantheon and skirmish games in particular (a player has left the game – right at the start).
2. Custom Game player queuing. You set up a game, someone joins, another person joins but they have a yellow ping. That’s because they didn’t know about the other player. This is something that if we had another few hours would have been in tonight’s update.
3. Experienced players not getting AI players in skirmish/pantheon.  This has been a lot trickier to do than it might seem because we really are trying to keep expert players from just reaping havoc on relatively inexperienced players.  The consensus seems that people would rather have the touch players against them than AI so we’re going to broaden things.

So those are the 3 things we are hoping to deliver tomorrow.  You may have noticed there aren’t a lot of “bug fixes” here.  That’s been the most challenging thing here. Like I say in the video below, launching Demigod has been practically like launching an ISP for us in terms of infrastructure.  What once seemed like a simple matter of tossing up a NAT negotiator that puts people into games has really become a lot more than that.  Our fault of course, we couldn’t leave good enough alone and had to have the Pantheon and the smart Skirmish mode and such but I think in the long run it’ll be worth it.

In defense of Peer to Peer

One thing I really want to stress here is that there is nothing inherently wrong with peer-to-peer.  Supreme Commander is peer-to-peer and it works fine.  The difference, of course, is how they were released.  Digital distribution is a godsend in most ways but having people from around the world playing together immediately has created a lot of the challenges that we must overcome.  That’s an over simplification here I just don’t want to see people bashing Gas Powered Games for choosing to make Demigod P2P.  They’ve had great success with it in the past and Demigod will soon be up to speed as well.

My point is, GPG has a lot of experience in this area with a proven game (Supreme Commander) that demonstrates that this works. They know their stuff.

Greetings!  Tons of different things to talk about today so bear with me.

Based on the logs, multiplayer is working pretty decently for most people but for others it is completely broken.  A program called Game Ranger has helped a lot of people but there’s a lot of warez users on there.

Let’s talk about the pirates

I’ve read a lot of misconceptions out there. First, no, we very much don’t like it when people pirate our stuff.  Our position has simply been that people are going to pirate a game whether it has draconian copy protection or not and the people who will buy a PC game will buy it and the people who won’t won’t. 

The first few days of Demigod being miserable on-line was largely the result of us simply not having taken into account the effect pirated copies would have on our server resources. So we had to scrounge servers and get users onto there.

Once we got legitimate users onto the servers that were isolated from the infrastructure that was being pounded on by the pirates, the connection situation improved dramatically as you know.

It’s easy and convenient to blame piracy for the launch debacle of Demigod but that would be inaccurate. Even had Demigod had some nasty copy protection on the DVD, it would still have been cracked and this problem would have come up anyway.

So contrary to what some have said, Stardock’s anti-copy protection stance is not the cause the first few days of Demigod’ connectivity. More like, our inexperience in publishing multiplayer-centric games caused us to underestimate the server infrastructure needed.

We do not intend to change our policy on copy protection. This leads us to the next part.

Understanding how Stardock supports games

Our “copy protection” is to treat our games as a service rather than a product.  Those of you who have our non-games like Object Desktop or Multiplicity know what I speak of.

That is, we keep releasing updates to our games for a long time.  Our publishing agreement with Gas Powered Games includes game updates to the game for at least a year. And after that, we still expect to release updates.

Once the multiplayer connectivity stuff is straightened out, we can start to talk about the kinds of things we can add like team based joining, how clan support should be, what kinds of items people would like to see, what kinds of additional demigods people want, and so on.

But our first priority has to be to those users who find connectivity flakey or just plain non functional. They paid $40 for a game and we have an obligation to help them before we start adding in new features and such.

Stardock’s business model is that our titles generate revenue over a long period of time rather than in the first 90 days. So we look at our journey with you being at the very beginning.

Why we love the community

Another thing that really keeps us going is just how wonderful the community is.  Obviously, it should be clear by now that we weren’t anticipating the kinds of connection challenges some people have run into. Our guys have been working with the guys at Gas Powered Games night and day on resolving them. 

I can assure you that Gas Powered Games is as interested as we are in making sure problems people run into are solve quickly.  Not in some patch a month from now but like updates in a matter of days.

But what really also has made this experience a bit amazing is just how great the community has been to us.  One user (OrleanKnight) sent our team a box of Mrs. Fields cookies.  Other users have sent us cards and pictures their kids have made of various Demigods.  How can you not love such a group of people?

Let’s face it, you guys just rock out there.

Getting the next update out

We’re going to start to try to be a bit more careful about how quickly we put out updates. The last update went out without updating the pantheon security checks to support the new version which resulted in most games not being counted. That’s the kind of thing that haste brings.

That said, as I type this, the team is working on an interim update to Demigod.  It doesn’t “Fix” connectivity issues one way or the other but does provide better error messages (and we still have work there).

For example, one “Fix” that still needs to go in is the “Failed to join game” message in custom games. What it should say is “Game is full”. But a lot of frustration people run into is in the kinds of error messages that are shown (or lack there of). 

I’m pretty sure this update tonight does start to wean experienced users off of AI players in skirmish and pantheon. At the very least, it has the support so it can become a server side change tomorrow.

Eliminating the connection issue

For those who can’t connect (you know who you are) the solution will be the UDP proxy servers. They spent a lot of time today getting them up and starting to test them to see how they work.  Unfortunately, it’s not something that can be done overnight. The team says later in the week they hope to have it ready.

But what that will do is alter Demigod from a peer-to-peer game to a quasi-client-server game that maintains the peer-to-peer performance.  If there are any IT people reading this, feel free to explain to lay people the significance of being able to do this in a couple of days without having to change the game client.  

Long Term

For us, Demigod is a long term commitment. So people who worry about this being like a typical release and move on type thing need only to look at our other games.  We will be continuing to work to ensure Demigod evolves over time.  I look forward to having official tournaments between individuals, clans, etc.  So there is definitely a lot to look forward to.

If any of you have any questions, just comment here and I’ll do my best.

Cheers!

Evening update

Just wanted to give you a head's up that we're not going to put out the update tonight. We have it staged but there is a concern about releasing it and then having the team go home in the event there is some unknown problem. So we're going to put this up tomorrow afternoon. Stay tuned.

Tomorrow will mark the 1st week since Demigod’s official launch.  The newer reviews that have come out since the initial launch have come back to more where we’re expecting:

Firing Squad: 84 of 100

http://www.firingsquad.com/games/demigod_review/page6.asp

Cosmos Gaming: 95 out of 100

http://www.cosmosgaming.com/articles.php?id=1963&articletype=review

GameFocus: 92 out of 100 and an Editor’s Choice

http://www.gamefocus.ca/?nav=reviewCard&fid=7318

Where are we at?

First, one of the side benefits from the day 0 debacle is that I have fallen in love with a new program called GameRanger. It’s hard to remember but back in the Total Annihilation days, playing multiplayer involved knowing someone’s IP address and trying to connect to their game via IRC or something else. But then program like TEN and Kali and such came out and made things better.  But if you’re having problems with getting multiplayer going, I have to say, GameRanger works pretty well and it is doing basically the same thing we’re in the process of setting up.

Now right now, Demigod, as a game, is awesome.  Demigod, as an on-line service, however, sucks ass. It works fine for what I dare say most people but for others, it's just awful. Our job is to make sure it works fine for everyone. 

Talk about being babes in the woods. Sins of a Solar Empire’s network “infrascture” to this day involves 1 server.  Demigod’s involves dozens of servers now because of so many people using it. I guess it turns out most people playing Sins of a Solar Empire were playing it single player.

So what is the cause of our woes?

Let’s talk about peer-to-peer networking.  Here’s how it works:

Everyone has to connect to everyone. And unfortunately, not everyone can.  Worse, the connection servers we’ve had have been just being pounded by the number of requests.

Why wasn’t this found in beta? There’s two reasons:

1) Our beta group was overwhelmingly located in North America and they didn’t have problems connecting to each other. That is still the case now.  But people in Europe tend to have problems connecting to people in the United States.

2. Our synthetic stress tests on the server were through pretty ideal connections where connections were pretty easy.

So what can do do about this?

This is a bit of an over simplification but you get the idea. We have the connections pass through these proxy servers on-route to the other player. The geographic location of these servers is key to keep users from getting lag.  I’m told it shouldn’t take very long to get this part going.

It’s not all bad

I will say that the current system does a pretty decent job.  Once we were able to throw enough servers at the problem we’re seeing a lot of successful MP games.  But that does little good for the people who just can’t connect and no amount of port forwarding is going to solve the case where two players just can’t talk to each other.

Expectations

My expectation is that by the end of this week, the connectivity problem will be largely solved. I hopefully won’t have to eat my words. But the fact that things like GameRanger works so well indicates that there’s little problem with Demigod itself but rather the service infrastructure behind it.

Stay tuned.