Brad Wardell's Blog


Demigod: Scheduled Games = the eVite of death

Published on Wednesday, May 13, 2009 By Brad Wardell In Demigod Journals

One of the things we want to really expand on in Demigod is the notion of multiplayer options.

As many of you know, one of the things that absolutely kills MP games in the long-run is the ability to get matches going with people of equal skill level.

The solution to this is to create a system that works inside and outside the game for setting up scheduled games.  Today, the web team submitted a proposal to provide an iPhone page to support this as well.

image

Here’s how it will work:

A user comes to the Demigod website and in the Pantheon area clicks on a future tab called “Schedule a Game”.  On that tab, the person can either create a game or participate in a scheduled game.

If you choose to create a scheduled game, you would then pick:

  1. Name of your game
  2. What time
  3. Number of players
  4. Map
  5. Game Mode
  6. Minimum XP level
  7. Maximum XP level
  8. Maximum number of No Shows allowed
  9. Maximum number of disconnected allowed

When a user tries to join it, if they aren’t on the host’s “ban list” (which will be implemented this Spring), they are then added to the list.  The game creator can reject people for their game if they want.

At the scheduled time, players load up Demigod, choose “Scheduled Game” and it shows all the games that they’ve committed to.   At the scheduled time, they choose “Fight” and off they go.  There will be a 5 minute window for players to get into the game. After 5 minutes, if a player doesn’t show they get a “no show” stat added to their account.

So when?

I’m afraid to give estimates other than “sooner rather than later”.  It will probably be part of the Demigod: Clan Wars DLC.

Silly after 48 hours

Published on Wednesday, May 13, 2009 By Brad Wardell In Demigod Journals

Last week the team worked 108 hours on the multiplayer match-making.  What most people don’t realize is that on day 0, the v1.0 of Demigod, the connectivity servers were all licensed code.  Then, after a week of mucking around trying to improve it, it was decided to add the Impulse team to come up with a new system.  But since time has been of the essence, there’s been a lot of all nighters lately.

At 4am or so, things can get pretty silly…

Flow control?

Published on Wednesday, May 13, 2009 By Brad Wardell In Demigod Journals

One of the things we’ve been struggling with, even since the beta, when it comes to match making is the seemingly exponential server requirements for match making.  That’s why when we’d release a beta things would be peachy and then they’d go to the public, it would fall apart.

Well, yesterday they finally found the issue deep deep in one of the licensed network libraries was something called flow control. This limits the number of packets that will get processed based on a number of criteria.  There’s good reason for flow control but because this code is a “black box” to us, until we could meet with the developer in person and go through his code, we had no idea that you’d end up with scenarios where a player might only be processing 3 or 4 packets per second.

Believe it or not, 3 or 4 packets per second isn’t necessarily a bad thing in a client/server game. You connect to the server and they send back the connection info. Maybe that connection info is 10k. So it takes a second or two to get that info. It’s not that big of a deal if it’s 1 vs. 1.  It’s not even that big of a deal if it’s 2 vs. 2.

But naturally, in Demigod, people want to to 3 vs. 3 or 4 vs. 4.  And if you do that, well, very bad things start to happen because it starts to back up. You can tell it’s messed up because your connection window will not seem to make sense and it doesn’t – because it’s incredibly outdated.

If you have to get info every few seconds from say 8 people in a lobby, you’re going to quickly get backed up.  Now, until we fix this, you can resolve a lot of this by keeping your games at 2 vs. 2 or maybe 3 vs. 3 if the people in the lobby have decent machines.  It’s fixed on our end but we have to merge the fixed library into our code which, if you’re a developer, you know how carefully that needs to be done, even with version control.

But this problem I just described has been there since the beginning (i.e. February).  The only reason connectivity has gotten better is because we’ve thrown a ridiculous number of servers (I think there’s like 8 servers now dedicated to just handling requests) plus the work we’ve been doing for the past 2 weeks has been improving things.

BTW, the only reason why the day 0 update worked better for some people than the most recent release is because the day 0 update used NAT and that’s a lot slower (so a lot fewer messages). The faster the system, the more it aggravates this.

I’m going to ask our technology architect to do a write up on this once this is fixed.  I am heading back to doing more testing scenarios.

I plan to also write up an article on how games are made so people can get a better idea of how this sort of thing can happen in a major commercial game in the first place.

Tuesday's update notes

Published on Tuesday, May 12, 2009 By Brad Wardell In Demigod

We just published a new update today that should clear up a lot of the problems people had with the Monday update.

HOWEVER... it is only half of the solution.  The other half has to do with the server database which we are increasingly thinking has been a major source of our pain and why things work "great" with a few users but become a nightmware when lots of users get on.  The good news is that the change won't require any updates to the game, it'll just suddenly "work" much better. They're looking at it right now though.

 

Demigod: Tuesday 5/12/2009

Published on Tuesday, May 12, 2009 By Brad Wardell In Demigod Journals

Well last night we released an update to the general public and it was a cluster. After investigating it, it appears that much of it had to do with users with the beta and users with the release getting into each others games (the version numbers of the exe and dll were the same and that’s what the servers check on). We also found and fixed the issue where people were ending up with “Test” names which occurred mostly when people with the beta were joining games hosted by someone with the release. And lastly, we found a case where a debug statement could cause a lobby crash when someone exited.

Anyway, they’re copying up the new update now but due to the time of day, the interwebs, they be slow this time of day.

This update doesn’t have the new EXE we received from GPG.  We wanted to get an update out quickly that took care of as much of the pain from last night as possible.

Meanwhile, the Raknet developer is here working with the Impulse team on the proxy updates that we hope will be up this week also.  I want to double check with GPG to make sure the favor point stuff is really nailed down here.

For those of you having problems, I share your continued frustration. I apologize for the inconvenience.  We have our top people working on this.  I wish I could say “Oh, it’s a bug and here’s how to fix it” but it’s simply a lot more complex than that when it comes to putting together a P2P match making system that has to work with so many different setups.

One of the things we’re doing on our end is to come up with an internal proxy system that will let people connect to other people via people they can connect to (got that?).

Let me draw a picture.

Here’s how P2P works today:

imageAlice must be able to connect to Bill, Sally, and John.

1.00

When Demigod shipped (1.0) Alice would call a NAT facilitator that would give Alice the IP address and port number to connect with Bill, Sally, and John.  The problem with that is that if Alice failed to connect to anyone, she couldn’t get into the lobby and NAT facilitation is slow.

1.00.076

With today’s update, Alice will first try to directly connect to Bill, Sally, and John on port 6112. If those people have that port open and forwarded to their PC, it will connect quickly. If that fails, it will resort back to the slow NAT.  If they connect to at least the host in a custom game, they enter the lobby and then begin trying to connect to everyone else in the lobby. Then, the host can at least pick and choose which people to keep and not keep if there’s someone with a poor net connection.

1.01

With the next update, it will work like this:

imageIn the event Alice can’t connect directly to Sally but could to Bill and John, our system will route the traffic through Bill and John based on who has the best ping.

Now, if Bill and John both disconnected from the game, Alice would get zapped.  But on the other hand, if Alice could only connect to 1 person (some people with Qwest have this problem) she could still play in a 10 person game because the other 8 people would be connected to her via the person she has a direct connection to.  The good news is, we have this working internally right now.  It’s not some theory but something we’re testing.  The bad news is that it probably won’t come up until next week.  But if this system works, it opens the door to all kinds of highly robust multiplayer scenarios.  Now if only we had thought of this a month ago…


 

…for tomorrow…

Published on Monday, May 11, 2009 By Brad Wardell In Demigod Journals

Well I have good news and I have bad news.  The good news is, based on talking to people in game lobbies, the new update works great for a lot of people who couldn’t previously connect and are blissfully unaware that anyone is having any problems.  The bad news is that there are people who had good times previously now having problems. Sigh.

Well, we have on deck another update for tomorrow fresh from GPG:

· Lord Erebus: Bite's Mana Cost increased, Damage decreased, Armor Reduction decreased

· Integrated newest DLLs

· Fix for looping audio track after leaving character select screens

· Fix for audio volumes not being reset on restart

· Adding /skipintro command line to bypass intro movies

· Cleaned up the achievement code so that the CVP Interface no longer maintains a copy of Guest data per peer. Instead you now only have your own achievement data.

· Demigod now checks the return code on GetPlayerData when OnNatComplete is called. If it fails Demigod does not accept the connection.

· OnNatComplete now returns 1 for success.

· OnNatComplete now returns failure if the playerName is empty.

 

This, in theory, combined with some tweaks on our end, should, I hope, take care of the flakiness some have had with lobby. But I make no promises.

Our end all, be all, update is scheduled (note: scheduled not promised) for Thursday. That’s where proxies and such are scheduled to go live. If you still have matchmaking connectivity you then have real reason to worry.

We’ll keep you update to date.

Monday update is up but…

Published on Monday, May 11, 2009 By Brad Wardell In Demigod Journals

Against the team’s better judgment, we went ahead and put up the update tonight since I had said we would have an update today. It has the major improvements promised but there’s still some flakey custom lobby behavior that will need to be looked at (play skirmish and pantheon for best results).

We had the game staged and testing for a long while but we kept trying to reproduce a bug reported from the beta where ImpulsePlayer was being inserted into lobbies.  Anyway, the temp fix for that will result with some flakey custom game lobby behavior (skirmish and pantheon are fine).  So odds are, we’ll release a minor update tomorrow to tweak that.

The symptom, for those of you who run into it, is players called “test” as well as players not being removed from the connection list. 

More to come.

Demigod: Clan Wars concepts

Published on Monday, May 11, 2009 By Brad Wardell In Demigod Journals

In late Summer, we plan to release Demigod: Clan Wars. 

So how will it work?

Besides the obvious – creating and joining of clans, the practical meat of Clan Wars is ability to schedule matches.  Right now, everything is in real-time. The functionality that Clan Wars will bring in is the ability to schedule matches at specific times and then join in with your friends into that battle at that time.

As it starts to come together, we’ll start to put up screenshots.  This will be a free download via Impulse for all registered players.