Brad Wardell's Blog


Life in the Demigod lane

Published on Saturday, May 2, 2009 By Brad Wardell In Demigod Journals

Last night we got the update out.  The change log can be found on Impulse (right click and view the change log).  Lots of improvements.

We ultimately marked it as a beta because we ran out of time and couldn't integrate the updated connection code which meant that the existing connection DLL had to be included which has a launch bug that causes an occasional crash when combined with this new build.

Anywyay, we had been hoping to surprise people by having the update include the Proxy server update.  But it too wasn't quite ready. We had pushed until Friday to try to surprise everyone but we didn't want to go a second weekend without showing some substantial improvements for those people who have been having difficiulties.

We're going to have an update probably on Monday to move the beta from release. 

The problem with betas, as some of you no doubt have noticed, is that you split the multiplayer community and that can make the skirmish and pantheon games a lot less fun because you end up with a lot fewer 3 on 3 human games.  So we have to do these sparingly.

Now, that said, I want to address a nmber of points in one place that keep coming up online:

Comment 1: "Stardock is overstating Demigod difficulties, to get coverage, the game works fine for me"

First, yes, while the game works fine for most people. There are a significant number of people who are having problems connecting through the built in multiplayer match making.  While I may seem to focus a lot on those issues, in the long run, this is a game that will live and die on its ability to sustain a MP community and it is important that people know that we are taking their issues seriously.

Comment 2: "The multiplayer match making is totally buggy, how could they release it like this???"

So at the opposite end of the spectrum, we have the people who are having trouble getting into games with the built in match making.  I share your frustration.  But Stardock was in no rush to release Demigod.  Demigod was done as far as we were concerned. 

Now, as the publisher, it's our responsibility to make sure problems in anything we release are addressed. But we haven't been fixing "bugs".  We have been getting a crash course in P2P scaleability (or lack there of). 

Should Demigod have had a big huge open MP beta? Yea, in hindsight, yes. If we had a time machine, we would do that.  But we've released games that have MP before (Sins of a Solar Empire, The Political Machine, The Corporate Machine, and OS/2 games before that). The difference here is the difference between MP-centric and a game that's mostly played SP that has MP.

And how does that difference translate? Well, in Sins, you might (and Sins players can attest to this) a game that tops out with 200 people playing online at once versus a game that has 1000 people in games online and another 8000 people who are online in the process of looking for games.  And as many people have said, it works decently overall.

There are programs like Game Ranger that largely take care of the match making for people who want to use that while we migrate towards a system where we host the games for players but it's not something that's easily done overnight. 

PC game expectations have come a long way over the past several years. Leaders in our industry like Blizzard have raised the bar with things like Battle.net and our friends at Relic have created RelicOnline which works very well. And of course there is GameSpy Arcade which has evolved over a period of years.

We appreciate the patience of those who are having difficulties. I can assure you that Gas Powered Games and Stardock are working inhuman hours to solve their problems as quickly as possible (70 to 90 hours last week for instance per person involved).  But these hours aren't about "finishing" a game. The game shipped finished. It's the scale of things that we were (clearly) unprepared for. We can't change the past, we can only take things from where they are now.

A lot of progress has been made internally. In less than 2 weeks, we've taken a game that is a peer-to-peer game and developed an infrastructure from scratch to make it so that the publisher's servers will be handling potentially 100% of the network traffic for all games played throughout the world.  This was not contemplated prior to release but we are commited to making sure that MP is bullet proof even if it takes longer than we all wish.

We're working on it even now.

 

 

 

Late Thursday night status update

Published on Friday, May 1, 2009 By Brad Wardell In Demigod Journals

We got a really good new build from GPG this evening that has a cure for those people who have experienced in-game disconnects along with some in-game exploits and a few other things (favor points, achievements, etc.).  So we spent a lot of time this evening working on that.

Meanwhile, I turned over the Proxy server setup to the enterprise team at Stardock to speed that up (for those of you not familiar with Stardock, we’re 70% an application company, 30% games with very distinct development teams).  Basically I turned over the Proxy server work to the Impulse development team since they already have built the infrastructure around Impulse.

The connection servers, which is one of the areas some people (and we have reproduced this) are having a lot of pain with is something we’re still working on (2:30am as I write this) but we’re nearing the end of productive work on this.  Basically, it’s a problem of scaling. The classic what works for 500 people falls apart when used by 5000 people at once.  But a lot of progress has been made there as well.

Anyway, we’re going to call it a night and come back tomorrow to continue on the integration.

Now, as some people have heard, we are going to release this update as a “beta”.  Impulse has an option for users to be able to try out pre-release software on it.  Warning though, if Demigod MP is already fine for you, don’t mess with this.

Here’s how you make use of this feature:

image

We expect to have this up tomorrow.  But I’m not making any promises as we don’t want to take a chance of putting something up that leaves people stranded (or having to re-download the whole release game again).

Update: Friday afternoon

We've finished integrating GPG's changed with the Raknet changes with the CVP changes.  But we have still not resolved the issue of why it takes so long for 2 people, after being connected, to get into the lobby when there's a heavy load as the database seems to now be able to handle it (we set up a bunch of new servers) but there is still a random several second delay between when it says someone is connected and when it actually even attempts to insert them into the game lobby.

Not having a fun time

Published on Thursday, April 30, 2009 By Brad Wardell In Blogging

(whining begins)

I’ve been a software developer for over 15 years and for the most part, I’ve enjoyed it.  But this past month has been sheer agony. 

Spending immense amounts of time at work away from my little boys and my daughter and my wonderful wife has been taking its toll and I feel so helpless because my development area isn’t in the area that my teams are focused on – networking.

My friends at Gas Powered Games, even right now, are working the midnight oil to do their best with Demigod and similarly, my friends and coworkers here at Stardock are outside my office working diligently on the same kinds of things.

It’s so frustrating because where I come from in normal game development, your code either works or it doesn’t. If it doesn’t, it’s a “bug” and you fix it. You make sure you don’t have any show stopper bugs before you ship a game.

But with Demigod, the complexities of a vast matchmaking system is just a huge pain. Besides sheer compatibility nonsense to deal with (Europe with their VDSL and the US with ISPs sometimes blocking P2P traffic) you also have systems that are brand new that work fine under a reasonable load but fall apart when there’s thousands of people hammering them at once.

I feel like I’ve let down our community and our fans. People know we are diligent about this kind of thing but this has been so heart breaking to watch. While I know that the team will get it fixed, the good will of players is finite.  Sure, it works for most people but that’s pathetic. It should work for basically anyone without a firewall.

It’s also putting a lot of stress on our teams, particularly the games team which I’ve had to put onto this project to help solve things faster. This isn’t game development, this is network administration. It’s just obnoxious.

The teams are working their hearts out both here and in Seattle (GPG) and progress is being made but what’s really depressing is seeing the check-in comments. They’re not fixing “bugs” they’re having to do new design for things that just didn’t scale well. 

There’s a reason why only a few companies have put together these kinds of systems, it’s a huge pain.  I am hoping they can nail this stuff down before the weekend.

Meanwhile, we have one of our distribution partners making threats because we won’t budge from our 1 price for 1 world model (i.e. Demigod costs the same in USA as it does anywhere else in the world).

(whining ends)

I just look forward to getting back to working on actual GAME development.

Favor Points relief...

Published on Wednesday, April 29, 2009 By Brad Wardell In Demigod

We made some server side changes today that work around some of the Favor point issues and gave everyone 150 favor points.

We are still waiting on an update to address it not storing items you purchased on the database but your favor points should now be accumulating again.

So what caused it? In a nutshell, it got broken when we did that day 0 server move. The data was going off to a table on the old database.

Let me know if you're still experiencing problems. You shouldn't get negative favor points (but you will have to keep rebuying the items until we get the GPG fix).

When pure isn’t pure

Published on Wednesday, April 29, 2009 By Brad Wardell In Demigod Journals

There is a website called Reclaim your game that purports to be an anti-DRM site.  I received an email today from a user that includes a link to an article on that site arguing that Stardock can no longer be considered DRM free because Demigod requires a user create an Impulse account (UserID and password) to logon and download updates to Demigod.

I would happily respond there but their site requires me to create a Prism forum account (UserID and password) to logon to make posts and I just can’t abide by such anti-consumer DRM…

Demigod Wednesday update

Published on Wednesday, April 29, 2009 By Brad Wardell In Demigod

We're waiting on some bug fixes from RakNet (they're the company that supplies the networkintg libraries) to incorporate into what we and GPG have been working on.  When we get those in, which we're hoping will be today, we can start putting up a new build.

The new Demigod build we plan to put up though will be marked as a BETA as we are going to begin the transition to the proxy server system we have previously been discussing.

See the Journals for more info.

Update: GPG has made significant progress on the infamous disconnect issue so we're going to hold off on this update. Stay tuned.

Elemental: Modding & Single Player

Published on Wednesday, April 29, 2009 By Brad Wardell In Elemental Dev Journals

image While a lot of Stardock’s development team has been interfacing with Gas Powered Games to continue to improve Demigod, the Elemental team continues forward.

Now a couple of things about Elemental that should be made clear:

  1. Elemental will be a single-player focused game.  It will have multiplayer but most of the development time will be spent on single player.
  2. Elemental will have a story-based campaign. While Sins of a Solar Empire and Demigod do not have story-based campaigns, it’s important to bear in mind that Stardock doesn’t develop those games. Galactic Civilizations II and its expansions all have story based campaigns.
  3. I don’t like traditional story-based campaigns.  I.e. we are not going to do a scripted campaign but instead tell the story through the campaign game that will play a little bit like Defender of the Crown (incidentally, designed by my friend Kellyn Beeck, VP of BizDev at Gas Powered Games).

Now, with those 3 points out of the way, let’s talk about modding.

In Elemental, we are building an engine that we are then “modding” into Elemental. The idea is that if we approach the game in that way, the game will, over time, become flexible enough so that players can mod it into all kinds of different land-based, turn-based games.  If we get our way, a modder could take Elemental and make RISK, or a Lords of Conquest game or all kinds of other things like that.

For those of you not familiar with modding let’s break this down into its elemental (no pun intended) parts:

  1. You have asset creation. That is, the ability to make things you can see in the game.
  2. You have item creation. That would be spells, armor, weapons, etc.
  3. You have resource creation. That would be everything from iron ore to magical gems to wood.
  4. You have gameplay scripting. We are using Python and that is where the rules of the game itself are modded.
  5. You have the AI scripting.  Again, using Python, that is where the AI behavior is modded.

For items 1, 2, and 3 you will have the built in modding tools (early version pictured above). What you see above is where the modder can create a particular tile in the game. Maybe it’s a dungeon. Maybe it’s a going to be a city improvement. Maybe it’s ruins on the map.  Who knows.

The plan is that players will be able to save these creations and post them to share with other players.  When these creations are saved, they can choose what “mod” they go with.  Elemental (for instance) would be the default choice that your creation could be used with.  But if someone made a different game mod, they could save it there as well.  That way, a mod could continue to be improved over time by the community.  Players would continue to use the bestiary (you’ll see that late Summer) to choose which mods they want to make available for themselves. 

For items 4 and 5, how flexible this will be will largely depend on time.  Since I code the AI in Stardock games, I’ll have to be careful to make C++ APIs in the game that are fairly generic that I can then call from python scripts. I don’t know Python right now so there will be a learning curve.  Users will be able to mod the scripts as they see fit for single player.  In multiplayer, the scripts will come from our servers in order to make sure everyone is playing the same game (it avoids all that desync pain you see in games when someone has modded something and forgotten). 

So that’s the direction we’re going.  Now, bear in mind here, what I’m outlining here is what are goal is. These aren’t “promises” because as with anything, economic reality may come into play.  But so far, things are going pretty well.