Brad Wardell's Blog


Favor Points relief...

Published on Wednesday, April 29, 2009 By Brad Wardell In Demigod

We made some server side changes today that work around some of the Favor point issues and gave everyone 150 favor points.

We are still waiting on an update to address it not storing items you purchased on the database but your favor points should now be accumulating again.

So what caused it? In a nutshell, it got broken when we did that day 0 server move. The data was going off to a table on the old database.

Let me know if you're still experiencing problems. You shouldn't get negative favor points (but you will have to keep rebuying the items until we get the GPG fix).

When pure isn’t pure

Published on Wednesday, April 29, 2009 By Brad Wardell In Demigod Journals

There is a website called Reclaim your game that purports to be an anti-DRM site.  I received an email today from a user that includes a link to an article on that site arguing that Stardock can no longer be considered DRM free because Demigod requires a user create an Impulse account (UserID and password) to logon and download updates to Demigod.

I would happily respond there but their site requires me to create a Prism forum account (UserID and password) to logon to make posts and I just can’t abide by such anti-consumer DRM…

Demigod Wednesday update

Published on Wednesday, April 29, 2009 By Brad Wardell In Demigod

We're waiting on some bug fixes from RakNet (they're the company that supplies the networkintg libraries) to incorporate into what we and GPG have been working on.  When we get those in, which we're hoping will be today, we can start putting up a new build.

The new Demigod build we plan to put up though will be marked as a BETA as we are going to begin the transition to the proxy server system we have previously been discussing.

See the Journals for more info.

Update: GPG has made significant progress on the infamous disconnect issue so we're going to hold off on this update. Stay tuned.

Elemental: Modding & Single Player

Published on Wednesday, April 29, 2009 By Brad Wardell In Elemental Dev Journals

image While a lot of Stardock’s development team has been interfacing with Gas Powered Games to continue to improve Demigod, the Elemental team continues forward.

Now a couple of things about Elemental that should be made clear:

  1. Elemental will be a single-player focused game.  It will have multiplayer but most of the development time will be spent on single player.
  2. Elemental will have a story-based campaign. While Sins of a Solar Empire and Demigod do not have story-based campaigns, it’s important to bear in mind that Stardock doesn’t develop those games. Galactic Civilizations II and its expansions all have story based campaigns.
  3. I don’t like traditional story-based campaigns.  I.e. we are not going to do a scripted campaign but instead tell the story through the campaign game that will play a little bit like Defender of the Crown (incidentally, designed by my friend Kellyn Beeck, VP of BizDev at Gas Powered Games).

Now, with those 3 points out of the way, let’s talk about modding.

In Elemental, we are building an engine that we are then “modding” into Elemental. The idea is that if we approach the game in that way, the game will, over time, become flexible enough so that players can mod it into all kinds of different land-based, turn-based games.  If we get our way, a modder could take Elemental and make RISK, or a Lords of Conquest game or all kinds of other things like that.

For those of you not familiar with modding let’s break this down into its elemental (no pun intended) parts:

  1. You have asset creation. That is, the ability to make things you can see in the game.
  2. You have item creation. That would be spells, armor, weapons, etc.
  3. You have resource creation. That would be everything from iron ore to magical gems to wood.
  4. You have gameplay scripting. We are using Python and that is where the rules of the game itself are modded.
  5. You have the AI scripting.  Again, using Python, that is where the AI behavior is modded.

For items 1, 2, and 3 you will have the built in modding tools (early version pictured above). What you see above is where the modder can create a particular tile in the game. Maybe it’s a dungeon. Maybe it’s a going to be a city improvement. Maybe it’s ruins on the map.  Who knows.

The plan is that players will be able to save these creations and post them to share with other players.  When these creations are saved, they can choose what “mod” they go with.  Elemental (for instance) would be the default choice that your creation could be used with.  But if someone made a different game mod, they could save it there as well.  That way, a mod could continue to be improved over time by the community.  Players would continue to use the bestiary (you’ll see that late Summer) to choose which mods they want to make available for themselves. 

For items 4 and 5, how flexible this will be will largely depend on time.  Since I code the AI in Stardock games, I’ll have to be careful to make C++ APIs in the game that are fairly generic that I can then call from python scripts. I don’t know Python right now so there will be a learning curve.  Users will be able to mod the scripts as they see fit for single player.  In multiplayer, the scripts will come from our servers in order to make sure everyone is playing the same game (it avoids all that desync pain you see in games when someone has modded something and forgotten). 

So that’s the direction we’re going.  Now, bear in mind here, what I’m outlining here is what are goal is. These aren’t “promises” because as with anything, economic reality may come into play.  But so far, things are going pretty well.

Demigod: So much for piracy

Published on Wednesday, April 29, 2009 By Brad Wardell In Demigod Journals

If I wrote a post saying that Demigod sales were far below what we had hoped for and I said that the reason was due to piracy and that the answer was that we should have put some nasty copy protection on those DVDs to have prevented early piracy what do you think people would say?

I know what my answer to that would be.  I would say that Stardock couldn’t blame poor sales on piracy but rather the fact that the game’s built-in multiplayer match-making was totally broken for the first day of release due to its underestimation of network resources that a mainstream game would take and even when that got addressed, the multiplayer match-making for two weeks and counting has been incredibly flakey which affected reviews and word of mouth.  That’s what I would say.

And yet…

http://www.gamasutra.com/php-bin/news_index.php?story=23381\

Demigod debuts at #3 for top selling PC games at retail – bearing in mind that that was a partial week and that the majority of units sold were digital sales which weren’t counted.

But…but…what about those hundreds of thousands of pirates? Yep. Demigod is heavily pirated. And make no mistake, piracy pisses me off.  If you’re playing a pirated copy right now, if you’re one of those people on Hamachi or GameRanger playing a pirated copy and have been for more than a few days, then you should either buy it or accept that you’re a thief and quit rationalizing it any other way.

The reality that most PC game publishers ignore is that there are people who buy games and people who don’t buy games. The focus of a business is to increase its sales.  My job, as CEO of Stardock, is not to fight worldwide piracy no matter how much it aggravates me personally. My job is to maximize the sales of my product and service and I do that by focusing on the people who pay my salary – our customers.

As Ars Technica quoted over a year ago:

"The reason why we don't put copy protection on our games isn't because we're nice guys. We do it because the people who actually buy games don't like to mess with it. Our customers make the rules, not the pirates. Pirates don't count," Wardell argues. "When Sins popped up as the #1 best selling game at retail a couple weeks ago, a game that has no copy protect whatsoever, that should tell you that piracy is not the primary issue."

Even Demigod, a game that shipped with no copy protection on the DVD, was massively pirated, and has had, to put it mildly severe launch issues with its multiplayer match-making which has had a negative impact on its Metacritic score has still managed to debut at the top of retail sales charts  (not counting our digital sales).

Why is that?  At that point I can only speculate but the first reason is pretty straight forward: Demigod is an awesome game. Second, while the multiplayer matchmaking that comes with the game currently sucks, our customers know it will get fixed. Part of that is the demographic of Stardock customers. They’re more experienced, they know that some of the issues with the MP matchmaking aren’t due to rushing the game out or negligence but rather the fact that complicated systems sometimes don’t scale well and there is no substitute for time when it comes to fix them.

I think there are many lessons to be learned from Demigod.  For example, if I had to do it over again, I would be inclined to require a valid user account to play LAN even if it only has to be validated one time. That way, we could also make it a lot easier for a legal user to have a LAN party with a single license. Anyone who has played Demigod on Game Ranger probably knows what and why I'm bringing that up.

When the focus of energy is put on customers rather than fighting pirates, you end up with more sales.  It seems common sense to me but then again, I’m just an engineer.

Demigod: Inside the sausage factory

Published on Tuesday, April 28, 2009 By Brad Wardell In Demigod Journals

To say it has been a difficult 2 weeks would be an understatement.  I’ve been making PC games for about 15 years and never in that time have I encountered a situation quite like what we’ve had with Demigod.

As most of you know, Demigod is an awesome game. If I may be so bold, Demigod is one of the best games that has been released in the last couple of years.  And yet, here we are, 2 weeks after release and the included multiplayer matchmaking is still flakey.

I would put it like this (based on logs and talking to people)

  • 30% of users think the multiplayer matchmaking works flawlessly. You don’t hear much from them (obviously).
  • 60% of users are finding the multiplayer matchmaking to be flakey. It works but it’s a pain in the ass to get into a game sometimes.
  • 10% of users cannot be connected to other users period.

That last number will go up if the proxy servers aren’t put in place prior to the official European release as that last 10% is almost exclusively made up of people using ADSL in Europe.

So how did this happen?

image

Losing Messages

The single biggest problem has to do with lost messages.  In the beta, when we had only a couple thousand people using the system, it worked pretty well.  At first, it was pretty flakey but as we added more servers to handle the connections, the success rate went way up and by the end it was pretty perfect.  To be safe, we added extra servers to handle the expected extra players.

Then, one retailer broke the street date over the Easter break. This in itself wasn’t as big of a problem as one might think as the system could handle the legitimate players. But when the game inevitably got into warez channels, the pirates saturated our other servers (not the NAT server). Those servers handled things like validation and checking for updates. The validation kept pirates from being able to play in the Pantheon, skirmish or custom games but it also ate up a lot of our extra capacity.  The result was that for the first 3 days after the official release of the game, the online experience was totally hosed.  The early reviews reflected that unfortunately.

So for the first week, rather than focusing on the issue that would come to bite legitimate players for this past week, we were scrambling to create a mirror series of servers to isolate the pirates from the legitimate users.  That took us into mid last week.

By mid last week, it became apparent that the something was causing players to not get into the game.  We assumed it was simply bugs in Impulse Reactor since it was new and on Friday released a new version of it.  But it turned out there was nothing wrong with Impulse Reactor, the problem was coming from the NAT facilitator itself – it was sending messages but the game was never receiving them – or at least not before the time out period.

Adding more servers over the past few days has improved things bit by bit but not acceptably.

More Specific

When Bill and Rick decide to play Demigod together. The NAT facilitator connects them together, sends a message to Impulse Reactor which in turn hands over the socket to Demigod’s lobby.  We have verified that the NAT facilitator is sending the message but the message is never being received.  We know it has something to do with the load because when we try it out with a relatively small group of testers (~100) it works flawlessly.  It’s when it is put out there for tens of thousands of people that it falls apart.

Solving it

And that is where we are on Tuesday with regards to the connectivity. Figuring out why the message is sent from the NAT but not received by the game. There’s a lot of thoughts on this.  UDP (unlike TCP) isn’t guaranteed. You can learn more on this here.  Another thought is that the NAT queues up its messages and that by the time it sends it back to the game, it has timed out.    The team tells me they think they’ll have this mystery solved today.  If so, I’ll report here on this.  But this will put us up to 90%.

Proxy Servers

If I had a time machine and knew about the pain we would all be experiencing in the first couple of weeks after release, I would have simply had us set up a series of host servers around the world. When someone started a game, they would simply connect to one of those servers and there’s be no connection issues.  That’s what we have been planning to do with Elemental.  But I was of the same mind that many here have said “Hey, lots of games do this, how hard can it be?”

Well, it turns out, it is very hard.  Most vendors have used GameSpy over the years. Blizzard developed Battle.net over many years. Relic migrated to a system it developed with Quazal to develop. 

The only reason we should be able to develop a Proxy system quickly is because we already have a worldwide network infrastructure because of Impulse so the hardware is already there, the agreements with ISPs in various countries already in place.  Work on this system has been going on in parallel with the NAT issue.  What will happen is rather than getting a NAT failed message, the user will connect to one of our servers and have their traffic routed through that.  I don’t have a good ETA other than (very soon – days not weeks we expect).

On a personal note

We read the forums and we hang out on the chat and we even read comments elsewhere when we’re at home.  First, we appreciate all the support and understanding our community has had.  To those who are experiencing problems, we share your frustrations. Many of us at Stardock and Gas Powered Games have spent an immense amount of time making Demigod a great game.  For those of you who are parents, I suspect you can empathize with what those of us with small children at home who have gone weeks without seeing their dad (or mom). As a game designer, it has been quite maddening to see the launch of a great game marred by IT and network stuff.  And I can tell you, getting this network stuff nailed down is easily as complicated as dealing with all the various video cards and sound cards and other typical PC pain. 

The teams at Stardock and GPG do believe those issues will be solved soon. And we thank you for your continued patience.

 

WEDNESDAY UPDATE

I am currently sitting in the lab with the GPG, Stardock, and Raknet teams interacting.  GPG feels good about having taken care of the in-game disconnect issue. The "big" connectivity issue as described above seems to increasingly appear to be related to having to hand off the socket to the Demigod client. This creates a series of timing issues that gets exponentially worse the more people trying to connect multiplied by the # of users on the NAT server.   

New Demigod #1 Concepts

Published on Monday, April 27, 2009 By Brad Wardell In Demigod

These are just my ideas, feel free to add.

Name of Demigod: TBD

Type of Demigod: General

General description: More of a classic wizard. Carries a staff, wears long robes.

Abilities:

 

Ability 1.

 

  1. Set totems that give friendly Demigods 20% more experience in a given area.  (level 1)
  2.  Totems  now also give friendly Demigods 20% more HP in a given area. Last 30 seconds. (level 3)
  3.  totems now also  give friendly Demigods 20% more attack damage in a given area. Lasts 30 seconds. (level 8).
  4. totelms now also give friendly demigods 20% more armor in a given area. (level 10)
Abiilty 2.
  1. Spell: Immunity. Makes friendly demigods immune from attacks for 2,3,4 seconds (depending on level).
Ability 3.
      1. Magic Missile. Sends 1,2,4,8 magic missles (depending on level) at a target area that does area damage.
 
Ability 5.
  1. Charm beast.  Causes 15%, 30%, 45% of enemy reinforcements to switch sides depending on level.
Ability 6.  Magic Armor. Provides 400, 800, 1200, 1600 armor depending on level.
 
Ability 7.  Melee Combat.  Base melee damage with staff adds 5 times level, 10 times level, 20 times level.
 
Ability 8. Money Purse. Provides 2, 4, 8 more gold per second to that player.
Thoughts?

 

Demigod: Battle plan for the week

Published on Monday, April 27, 2009 By Brad Wardell In Demigod Journals

Last week we saw some more reviews of Demigod. G4’s Xplay gave us 4 out of 5. Gamervision gave us a 9.5 out of 10, GameZone gave us an 8.7 ouf of 10 and a number of other positive reviews that I don’t have handy also came out.

Despite that, the custom game experience of Demigod has remained an exercise in frustration.  The skirmish games seem to be working better but largely because it’s picking 2 on 2 games with only occasionally 3 on 3 games.

So why is this so hard to nail down? And what is being done to solve it?

Some people are blaming Peer to Peer as the source of the pain. But most RTS’s use some form of P2P. What makes Demigod different?

The biggest difference is the form in which Demigod takes its P2P. In a game like Warcraft 3, the players route their traffic through the host. Hence, some people can join Warcraft games but can’t host them. Now you know why.

Supreme Commander is the same P2P model as Demigod and it works pretty well. The difference, however, is that most SupCom games are duels (1 vs. 1) and when they’re teams they’re 2 on 2. This tends to hide the complexity of the system.  By contrast, in Demigod, people expect to do 4 on 4 games or 5 on 5 games and it’s a significant challenge to connect them all reliably.

From Friday

On Friday, the team implemented a new system to try to make setting up the connections more orderly. After 3 days of working on it internally with great results, it was released and promptly blew up requiring us to replace it with an alternative update we had “just in case”.

I read a lot of the comments on the forums and I can say we share the frustration others have.  Developer Gas Powered Games is working hard on their end and we are working hard on our end to resolve the issues. 

This week

Monday:

Start the next pantheon tournament and continue to tweak how it works to satisfy the largest % of players.

Tuesday:

Found and fixed the cause of negative favor points. Favor points should now work though we need another client update to store purchased favor items.

Wednesday:

We got a new build of Demigod from Gas Powered Games that aims to address: 

·         Eradicates once and for all the problem where one person disconnecting or lagging would cause the peer-to-peer networking to unravel, collapse, and spawn multiple games. This is cross-functional for all matchmaking setups (LAN, Skirmish, Pantheon, Custom). 

·         Introduces a cumulative tolerance timer of 30 seconds for each player, which gives shaky connections better opportunities to stay connected, and eventually culls serious problem connections. 

·         Reintroduces an informational UI for players that lets them know who is having a connection problem, and what their cumulative timer is up to. It is important to note that this UI is NOT Interactive, merely informative. It is also a bit ugly, but compared to not giving players information, this is the much lesser evil. If need be we can revisit later for UI improvements as that is much lower priority than the information it presents. 

Meanwhile, Stardock worked with the developer of the network library it uses (Raknet) to find a way to make connecting players not be flakey.  The bad news is that it increasingly looks like problems are resulting from timing issues in the complex system of handing off sockets between Raknet to Impulse/Demigod. 

UDP Proxy server code implemented internally with testing expected to begin on Thursday.

 

In the forums

We are also going to start some pinned topics on things we to expand the game with (for free) as the connectivity mess gets sorted out. 

Topics will be:

  • New Demigod #1
  • New Demigod #2
  • How do you want Clans to work
  • Group Joins