Ok guys. Let us have it.  We’ve largely rewritten the way users get connected for custom games.

So…

Is it better?

Let us know.

Update:

We are asking about the initial connectivity - getting connected to other players.  Not your experiences with other people once in the game. That is a seperate discussion.

I remember when Impulse first came out, some people didn’t like it.  But you know one fringe benefit of Impulse? Very very easy to release updates to lots of people in short amounts of time.

Anyway…

We released an update today and we’re extremely pleased with how it’s gone. We can actually go into the chat area without getting flamed! 

In early blogs, I’ve mentioned that developer Gas Powered Games and publisher Stardock both have multiple sub-teams working on various things in Demigod. 

Or more specifically, this past Monday, I mentioned that there would be a Monday update and a Thursday update.  Well, the Monday update was something of a disaster.  Yea, it worked better than what was up but we were extremely disappointed. Those of you who know me know I was extremely frustrated online about it in the chat area because I had such high hopes for it.  So today we released the update that fully implemented the changes needed to make Monday’s update be what it was supposed to be and that’s why we’re in such good moods.

Anyway, I also had said there would be a Thursday update that would have a lot of good stuff in it.  Well, it’s Thursday and the Impulse team now has their update in.  I think I referred to it as the “end all, be all” update to some people online.  Well, we’re staging it right now.

After this update, the only thing we’ll really have left to do is to turn on the proxy servers for those people who absolutely, positively cannot connect to people along with making the connectivity more robust for people with multiple network cards in their computer (you can do yourself a favor if you get stuck and if you’re like me with 3 network cards and wireless, turn some of them off when playing online until we come up with a good way to bind it).  But those solutions are much easier than what the team has had to do which is major surgery on the network libraries. 

To use an analogy, imaging if a game was released with a new version of a popular 3D engine and it turned out that 3D engine wouldn’t work for people with a popular model of ATI cards and that the only way to get it to work was to largely rewrite the 3D engine in house.  And even though it normally would take months and months to write a 3D engine, you really only have 3 weeks because people are really ticked off.  That’s essentially what we’ve been doing with the MP match making network stuff. 

UPDATE: 

It's here. People seem happy.  

If you have a low end machine and run into any error messages such as "failed to connect to NAT" try this:

download http://draginol.stardock.net/files/impulsereactor-0514.zip 

Unzip it into your Demigod\bin directory (program files\stardock games\demigod\bin).  If you're not having problems don't bother. But we found a Raknet bug that this works around for now on lower end CPU systems.

It’s been another crazy week of re-architecting the way multiplayer match-making works to handle the number of users. 

Today we are pleased to announce the new update:

Here’s what’s new for this week:

• Additional connectivity optimizations and improvements
• Lord Erebus: Bite's Mana Cost increased, Damage decreased, Armor Reduction decreased
• Fix for looping audio track after leaving character select screens
• Fix for audio volumes not being reset on restart
• Adding /skipintro command line to bypass intro movies
• Players will remain connected to facilitator if they fail to connect to other players, provided they can connect to the host. The host ultimately can kick people with problematic connections to others.
• Favor Items fixes, but still some assorted issues that should be addressed in the next update.
• Major optimizations to audio on low fidelity settings
• Fixed bug that occurred if you used an ability after issuing an attack command. This is the "won't chase the demigod I clicked" bug.
• Sending deltas for partially completed achievements.
• Each player now saves their own achievements and favor items
• Fixed partial Achievement saves bug.
• Fixed a bug that occurred if you accomplished an achievement in game.
• Fixed a bug in disconnect dialog that would try to update it even if we had determined we did not want to show it
• Changed resample rate to 24k in Low settings
• Delayed FMOD initialization until audio LOD is set via options
• Disabled hitbox interference from announcer messages.
• Life's Child no longer removes Inner Grace (Sedna fix)

Now, that said, what’s NOT in this update?

• Robust multiple IP support.  If you are talking to the Internet with more than 1 IP, you may have a harder time connecting. This is because you might connect to 1 player with 1 IP and to another player with a different IP.  We have gotten new code from Raknet to address this but it will take time to integrate and test.
• Better NAT. We have a slew of improvements to the NAT thanks to the Raknet update we got this week but again, it will take time to integrate and test.
• Route through host.  This works and we came “this” close to putting it in.  The problem is, not all players are really in a position to be routed through.  Now, my machine could do it. But we don’t have a way for Billy Bob with his DSL line from trying to host a game with 10 players. 
• Route through dedicated proxy.  This too works but we do not have the code in that lets it intelligently pick which proxy (i.e. someone in Japan is not going to want to use a proxy in Virginia). 

And lastly, let’s talk about 5 on 5.

There have been other peer to peer games.  But the connectivity complexity is C = D * N*(N-1) where N is the number of players and D is the raw data being exchanged.   While Demigod supports 5 on 5 multiplayer in custom, LAN, and single player, I have to tell you, unless you know who you are playing with, you’re asking for trouble if you try to do this with strangers.  Even if you manage to find 10 random people to play with online and they all manage to connect to each other, when you get in game, because the game is synced, you’re going to likely have a non-ideal experience. 

Let’s work through the formula where let’s assume D = 10.

With a 3 on 3 game, which, according to what I’ve seen in Supreme Commander, is the typical largest game people get into, C = 300.  On Battle.net with Starcraft, the max number of players is 8.  C = 560.  But when you go to 10 players, C = 900, nearly double the complexity of trying to do an 8 player game.

What our work has been these last 2 weeks has been to re-architect the system so that we can bring down that D multiplier. That’s the only part we can control.

When trying to connect players together, each player, based on their internet connection and PC performance has their own threshold of what variable C can reach before things fall apart.

In the beta, we were able to bring our variable D down by adding more servers.  And most beta testers had high end connections and PCs so they had relatively high thresholds before variable C had problems.

But clearly, in release, that threshold turned out to be a lot less.  The first inkling that our D variable was way too high was that the pirates who were pounding the server that first week were able to overwhelm our servers.  To put things in perspective, Impulse typically gets around 300,000 users using it each day and it doesn’t even break a sweat.  But a mere 140,000 connections that first week brought Demigod’s online experience to its knees.

We were able to get a reprieve that first week by shuffling users off to a new set of servers to get them away from the warez users. But as the game has continued to sell (and it looks like it’s going to break the 100k milestone shortly).

So the thing we continue to do is try to lower that D variable so that more and more people with marginal systems can get in.  There will likely be lots more updates as we find new ways to squeeze it. In the meantime, your best hope is to try not to go nuts with the # of players unless you know them.

We will probably have another small update tomorrow as we continue to refine this.

To see where we’re at as of 12:26am I recorded part of a beta where we got a 5 on 5 game going. Slowly but surely we’re replacing third-party code with Stardock code.

In this demo, there’s no port-forwarding going on. Port forwarding is a good idea if you can do it but it’s not necessary.

Problems I’m seeing:

1) If I have multiple IP addresses (I have a fiber line and a T1 connection and a Comcast connect – 3 NICs) it has a real hard time connecting me to other people. So that has to be addressed.

2) We have turned off routing through the host. Connections are great but if the host has a low end connection it slows things down. Will have to look to sending people to the dedicated servers but that’s for another night.

But hopefully you can see it’s sped up a lot. On Friday, the international release of Demigod goes out in Europe and Asia so we’re working to make sure Europeans get a good experience by this weekend (along with everyone else).

By the time we’re done, there will have been nearly a 100% transfusion of Stardock Impulse code into this connectivity system (so those of you at Stardock on the Impulse team reading this yes, you’re going to get the credit or blame next week <g>).

Stay tuned.

One of the things we want to really expand on in Demigod is the notion of multiplayer options.

As many of you know, one of the things that absolutely kills MP games in the long-run is the ability to get matches going with people of equal skill level.

The solution to this is to create a system that works inside and outside the game for setting up scheduled games.  Today, the web team submitted a proposal to provide an iPhone page to support this as well.

Here’s how it will work:

A user comes to the Demigod website and in the Pantheon area clicks on a future tab called “Schedule a Game”.  On that tab, the person can either create a game or participate in a scheduled game.

If you choose to create a scheduled game, you would then pick:

1. Name of your game
2. What time
3. Number of players
4. Map
5. Game Mode
6. Minimum XP level
7. Maximum XP level
8. Maximum number of No Shows allowed
9. Maximum number of disconnected allowed

When a user tries to join it, if they aren’t on the host’s “ban list” (which will be implemented this Spring), they are then added to the list.  The game creator can reject people for their game if they want.

At the scheduled time, players load up Demigod, choose “Scheduled Game” and it shows all the games that they’ve committed to.   At the scheduled time, they choose “Fight” and off they go.  There will be a 5 minute window for players to get into the game. After 5 minutes, if a player doesn’t show they get a “no show” stat added to their account.

So when?

I’m afraid to give estimates other than “sooner rather than later”.  It will probably be part of the Demigod: Clan Wars DLC.

Last week the team worked 108 hours on the multiplayer match-making.  What most people don’t realize is that on day 0, the v1.0 of Demigod, the connectivity servers were all licensed code.  Then, after a week of mucking around trying to improve it, it was decided to add the Impulse team to come up with a new system.  But since time has been of the essence, there’s been a lot of all nighters lately.

At 4am or so, things can get pretty silly…

One of the things we’ve been struggling with, even since the beta, when it comes to match making is the seemingly exponential server requirements for match making.  That’s why when we’d release a beta things would be peachy and then they’d go to the public, it would fall apart.

Well, yesterday they finally found the issue deep deep in one of the licensed network libraries was something called flow control. This limits the number of packets that will get processed based on a number of criteria.  There’s good reason for flow control but because this code is a “black box” to us, until we could meet with the developer in person and go through his code, we had no idea that you’d end up with scenarios where a player might only be processing 3 or 4 packets per second.

Believe it or not, 3 or 4 packets per second isn’t necessarily a bad thing in a client/server game. You connect to the server and they send back the connection info. Maybe that connection info is 10k. So it takes a second or two to get that info. It’s not that big of a deal if it’s 1 vs. 1.  It’s not even that big of a deal if it’s 2 vs. 2.

But naturally, in Demigod, people want to to 3 vs. 3 or 4 vs. 4.  And if you do that, well, very bad things start to happen because it starts to back up. You can tell it’s messed up because your connection window will not seem to make sense and it doesn’t – because it’s incredibly outdated.

If you have to get info every few seconds from say 8 people in a lobby, you’re going to quickly get backed up.  Now, until we fix this, you can resolve a lot of this by keeping your games at 2 vs. 2 or maybe 3 vs. 3 if the people in the lobby have decent machines.  It’s fixed on our end but we have to merge the fixed library into our code which, if you’re a developer, you know how carefully that needs to be done, even with version control.

But this problem I just described has been there since the beginning (i.e. February).  The only reason connectivity has gotten better is because we’ve thrown a ridiculous number of servers (I think there’s like 8 servers now dedicated to just handling requests) plus the work we’ve been doing for the past 2 weeks has been improving things.

BTW, the only reason why the day 0 update worked better for some people than the most recent release is because the day 0 update used NAT and that’s a lot slower (so a lot fewer messages). The faster the system, the more it aggravates this.

I’m going to ask our technology architect to do a write up on this once this is fixed.  I am heading back to doing more testing scenarios.

I plan to also write up an article on how games are made so people can get a better idea of how this sort of thing can happen in a major commercial game in the first place.

We just published a new update today that should clear up a lot of the problems people had with the Monday update.

HOWEVER... it is only half of the solution.  The other half has to do with the server database which we are increasingly thinking has been a major source of our pain and why things work "great" with a few users but become a nightmware when lots of users get on.  The good news is that the change won't require any updates to the game, it'll just suddenly "work" much better. They're looking at it right now though.

Well last night we released an update to the general public and it was a cluster. After investigating it, it appears that much of it had to do with users with the beta and users with the release getting into each others games (the version numbers of the exe and dll were the same and that’s what the servers check on). We also found and fixed the issue where people were ending up with “Test” names which occurred mostly when people with the beta were joining games hosted by someone with the release. And lastly, we found a case where a debug statement could cause a lobby crash when someone exited.

Anyway, they’re copying up the new update now but due to the time of day, the interwebs, they be slow this time of day.

This update doesn’t have the new EXE we received from GPG.  We wanted to get an update out quickly that took care of as much of the pain from last night as possible.

Meanwhile, the Raknet developer is here working with the Impulse team on the proxy updates that we hope will be up this week also.  I want to double check with GPG to make sure the favor point stuff is really nailed down here.

For those of you having problems, I share your continued frustration. I apologize for the inconvenience.  We have our top people working on this.  I wish I could say “Oh, it’s a bug and here’s how to fix it” but it’s simply a lot more complex than that when it comes to putting together a P2P match making system that has to work with so many different setups.

One of the things we’re doing on our end is to come up with an internal proxy system that will let people connect to other people via people they can connect to (got that?).

Let me draw a picture.

Here’s how P2P works today:

Alice must be able to connect to Bill, Sally, and John.

1.00

When Demigod shipped (1.0) Alice would call a NAT facilitator that would give Alice the IP address and port number to connect with Bill, Sally, and John.  The problem with that is that if Alice failed to connect to anyone, she couldn’t get into the lobby and NAT facilitation is slow.

1.00.076

With today’s update, Alice will first try to directly connect to Bill, Sally, and John on port 6112. If those people have that port open and forwarded to their PC, it will connect quickly. If that fails, it will resort back to the slow NAT.  If they connect to at least the host in a custom game, they enter the lobby and then begin trying to connect to everyone else in the lobby. Then, the host can at least pick and choose which people to keep and not keep if there’s someone with a poor net connection.

1.01

With the next update, it will work like this:

In the event Alice can’t connect directly to Sally but could to Bill and John, our system will route the traffic through Bill and John based on who has the best ping.

Now, if Bill and John both disconnected from the game, Alice would get zapped.  But on the other hand, if Alice could only connect to 1 person (some people with Qwest have this problem) she could still play in a 10 person game because the other 8 people would be connected to her via the person she has a direct connection to.  The good news is, we have this working internally right now.  It’s not some theory but something we’re testing.  The bad news is that it probably won’t come up until next week.  But if this system works, it opens the door to all kinds of highly robust multiplayer scenarios.  Now if only we had thought of this a month ago…

Well I have good news and I have bad news.  The good news is, based on talking to people in game lobbies, the new update works great for a lot of people who couldn’t previously connect and are blissfully unaware that anyone is having any problems.  The bad news is that there are people who had good times previously now having problems. Sigh.

Well, we have on deck another update for tomorrow fresh from GPG:

· Lord Erebus: Bite's Mana Cost increased, Damage decreased, Armor Reduction decreased

· Integrated newest DLLs

· Fix for looping audio track after leaving character select screens

· Fix for audio volumes not being reset on restart

· Adding /skipintro command line to bypass intro movies

· Cleaned up the achievement code so that the CVP Interface no longer maintains a copy of Guest data per peer. Instead you now only have your own achievement data.

· Demigod now checks the return code on GetPlayerData when OnNatComplete is called. If it fails Demigod does not accept the connection.

· OnNatComplete now returns 1 for success.

· OnNatComplete now returns failure if the playerName is empty.

This, in theory, combined with some tweaks on our end, should, I hope, take care of the flakiness some have had with lobby. But I make no promises.

Our end all, be all, update is scheduled (note: scheduled not promised) for Thursday. That’s where proxies and such are scheduled to go live. If you still have matchmaking connectivity you then have real reason to worry.

We’ll keep you update to date.