Here’s an interesting easter egg that I didn’t know about. You can drop items for others to pick up by just right-clicking on the item in your item bar.
This post here gives a good understanding of where we’ve been.
But here I want to discuss the technical issues that have plagued the built in MP match-making.
I’m going to start this by sounding like a weasel but it is also the truth: The Demigod MP match-making is basically the same as nearly every other RTS out there. It’s no better or worse. The difference – the fatal difference – is the demographic of who is trying to play multiplayer is radically different and the number of people playing multiplayer is radically different. In short, Demigod’s built in MP, as it is today and has been since launch, is no better or worse than other RTSs on the market today.
Now, before some of you hit the flame key, let’s talk about why the experience of people using the built in MP match making has sucked so bad.
First person shooters vs. Real-Time Strategy Games
In games like Call of Duty or Counterstrike or Left4Dead, you are largely dealing with people picking a server from a list, typically based on ping time, and joining it. The only connection one needs to make is to that server and that server has typically been set up to be as easy to connect to as possible.
In a real-time strategy game, like Supreme Commander or Company of Heroes the connections are peer-to-peer. Each player has to connect to each other player.
GPGNet (Supreme Commander)
In Supreme Commander, Gas Powered Games developed GPGNet. The way it worked is that a person would create a game. Subsequent people would connect to the host and then enter the lobby. The game would then try to connect each of the players up and if they failed to connect to one another, they would get a yellow ping and the host would have to eject them.
However, the demographic of Supreme Commander is significantly different from the demographic of Demigod (as SupCom players here can probably tell you). Casual players of SupCom played mostly single player and the people who played SupCom online tended to be technically savvy and knew to port forward 6073 (that’s the port it used). And people who didn’t know what they were doing either quickly got ejected from games or learned to set up their routers.
Demigod
Demigod’s setup is similar to GPGNet with a couple critical differences. First, to get into the lobby you currently have to connect to everyone who is in the lobby before getting into the lobby. However, if you enter the lobby at the same time as someone else, you will get a yellow (or red 0) ping to them. If you can’t connect to them, someone will have to leave and try again or need to be booted out.
Super NAT
To improve connectivity we worked with RakNet to develop a kind of Super NAT Facilitator. The idea is that we would make it so that everyone could connect to each other and avoid the whole "people need to open their ports” thing entirely because we knew Demigod would be a multiplayer heavy game. Unfortunately, the system developed is resource intensive on the servers and when thousands of people showed up, the system started to fall apart.
The result is that people could not connect to people easily inside of the lobby because of the increasing lag (not ping lag but lag from the servers being overloaded) and users would just sit there waiting in frustration to get into a game.
During the first week, we basically lost out because of the well known warez users hitting validation and check for updates so we couldn’t even begin to dig into the issue of the NAT servers being overwhelmed until last week.
Super NAT fails
But we have encountered edge cases that have turned out to be catastrophic. Over the past year it seems, an increasing number of ISPs (we believe, we don’t have proof so I’m not naming names here) are trying to shape their bandwidth use and doing things that make peer-to-peer connections more problematic (bit torrent uses P2P).
The net effect is that some people can only literally connect to 1 or 2 people and are then blocked by their ISP. So what happens? Other people try to join a game and they cannot connect to that guy who is in the lobby because that guy can’t accept any more connections. So then the Super NAT goes into play to try to find a way which in turn takes more and more resources on the NAT. You get even a handful of those types of users and it gets to be a real problem.
Blocking Port 6000
Then we find out that some routers block incoming traffic to port 6000. If you very quickly get a “NAT connection failed” that is why. You will need to tell your router to knock it off – port forward UDP traffic from port 6000 to your IP (or just have it unblock that port). I only know of a few routers that do this (Some Dlinks for instance).
So why is this taking so long to fix for those people?
The reason we have to connect to players in the lobby before putting them into the lobby is because Demigod, not us, handle the in game connections. We hand off the connection (the socket) to Demigod once users are connected. So we’ve had to spend time developing something that will let us connect people who are in the lobby even which is some very specialized coding.
We have also had to work on setting up Proxy servers for people who just can’t connect and that’s been no trivial thing either.
The build we put up tomorrow is expected to let people get into the lobby as long as they can connect to the host and then connect people once they are in the lobby first by trying direct connect (that’s if you port forward 6000 to 6200 to your PC’s IP) and then if that doesn’t work to use NAT facilitation and if that doesn’t work Proxy servers. But it’s complex with a lot of moving parts being developed by a team that has been working non-stop for weeks.
Will everyone be happy?
No. Because there are people out there for whatever reason just can’t connect to other people on the Internet. There aren’t many of them but at least we can work with them and see what we can do without those users impacting everyone else.
So that’s where we’re at.
Personally, I think this kind of stuff should be built into the OS. As a game developer, I can’t even begin to express how frustrated I am that we have to do all this. If this were an Xbox game or a PS3 game, we wouldn’t have to do any of this. We’d simply hand the user off to the manufacturer’s already included super mega system and it would take care of it. I could rant for hours on (as you can imagine, I’m extremely tired) how ridiculous that our reputation, game reviews, and our technical competence are being brought down based on how quickly we can implement something that should already exist by the OS maker (maybe Games for Windows Live will eventually evolve into this I hope).
We expect to put up a new BETA of this tonight and if all goes well release it tomorrow afternoon. That’s the plan. It’s not a promise. Given what I’ve learned in the past 3 weeks, when it comes to Windows networking, router manufacturers, and ISPss never assume anything.
Before a hundred posts drown the forums:
The main build of Demigod (public) has been slightly updated to prepare for the Proxy server move that is scheduled for Thursday.
The beta build of Demigod (for those who turn on "show pre-release" in Impulse) will get a new Impulse Reactor that tests DIRECT CONNECT. This build is only useful for those who have proper port forwarding otherwise it'll cause you pain.
The "big day" is Thursday afternoon, EST.
Our story so far...
The initial release of Demigod was marred due to us not taking into account pirates trying to get validated when we set up our network infrastructure. This resulted in the multiplayer game being basically unplayable which resulted in some pretty negative first day reviews.
48 hours later, we set up an alternative network for legitimate players, released a Demigod update and most people were able to connect - but with a big catch:
Demigod also debuted with a technology internally referred to as "Super NAT". The idea was to let people be able to connect to other people without having to mess with routers or anything. And it worked great in theory. In the beta, we noticed that the connection servers required a lot of resources to do their thing quickly. No problem, we added more servers and added still more again for the release. But that turned out to have been a vast under-estimation still as it wasn't a linear progression but an exponential one.
So over the past 2 weeks, we ditched what we had and started from scratch. Late last week, because we realized that time is of the essence, the Impulse team at Stardock got assigned to the project and we began to use Impulse tech to build something new.
The new system supports Direct Connection (which is pretty traditional - port forwarding). If that fails, it will use NAT negotiation. And if that fails it'll hook up to a series of Proxy servers to handle it.
Tonight
The proxy services will not be in tonight's build because of the lateness and we don't want to put that up and then leave for the evening only to wake up in the morning, go over to Shack News or Voodoo Extreme and see a headline "Stardock's Network becomes sentient, begins world conquest!" or something. So we'll wait until Wednesday during the daylight hours to turn that part on.
We are marking this build as a BETA still since it's so late into the evening we don't want to take any chances.
Expected Results
This update isn't the end all, be all update. This week is the week of putting this pain behind us. Tonight's update is just the first step. Without the proxy, you'll still have people who will fail on the NAT and we are sure, given the sheer quantity of changes, that there will probably be some problem in there since we've had to take out most of the licensed code and replace it with Impulse code. But based on what we're seeing it should be much better for most people.
If you are one of the people who it doesn't work for, hang in there. If you are part of the majority who is having no problem with Demigod, I realize that we keep focusing on connectivity versus game enhancements or improving the Pantheon and such it's just that my first priority is to make sure everyone can play the game online as well as single player.
The team at GPG and at Stardock are already working on v1.01 which will have the kinds of new features and such our users are used to seeing in Stardock games. But we have felt it would be in bad taste to talk about new features while there are people who couldn't play online so I hope you understand why we've not been talking about that yet.
Testing
So now we’re testing it with people on the chat channels and learning lots of things that we can now pass on to you.
Below are some suggestions for connecting faster. You don't have to do them but they will make a huge difference in how fast you connect and how widely you can connect to other people.
First if your router has an SPI “firewall” you may find you have a better experience if you turn it off. SPI firewalls provide good protection against random packets coming in but the downside is that on some consumer routers, it will prevent you from directly connecting to another player forcing the NAT servers (which are far slower) to pick up the slack.
Next: You need to make sure your router is port forwarding to ports 6100 through 6200 to whatever IP address your computer is using. Don’t assume your machine is that IP address because there’s a good chance it’s not.
If you do these things right, you’ll be able to DIRECT-Connect with the new Impulse Reactor we’re releasing tonight. If you don’t, it will fail over to the new Impulse Reactor NAT negotiation system we’ve been working on the past 2 weeks which is a thousand times better than what we already have out but it will slow everyone down.
If THAT fails, then it really gets painful because then you’re going to be sent over to one of the Proxy servers. These sound great in theory and they have the nice advantage of working basically 100% of the time but it will probably take us weeks to set up enough of these to make everyone happy because we have to locate them near you otherwise lag will be high.
If you're afraid of port forwarding go here: http://www.portforward.com/
ONCE IN GAME
The next thing we’ve discovered: There is a lot of software out there now that packet sniffs. Anti-virus programs even Google Desktop. How do you know if something is sniffing packets on your system? The game will stutter – go for a couple seconds, then pause a bit, go for a couple seconds, pause, and so on. There’s not much we can do on our end to stop that. You can try lowering your graphics settings but odds are, someone you’re playing has something that’s sniffing through your packets and slowing things down. And it only takes one person in the game to have this to make it stutter for everyone.
BTW, in the 10 person game we were playing, *I* was the one with the firewall applet I had forgotten to turn off that was monitoring packets. So it can happen to the best of us.
The update status
So they’re putting it together now. It’ll still be a couple hours because of all the uploading that has to do. Everyone’s very excited.
Again: If there are any IT people hanging around reading this, I’m not a network guy but I am being told by people that the fact that the entire backend for this game has been completely re-designed and re-implemented in 2 weeks is super insane.
Once this is up, our team will be talking at length at what was learned. What I keep being told is this: What works fine for 500 people can totally fall apart for 5,000 people and that’s what we’ve run into in a nutshell for these past 2 weeks.
Known bugs
Tonight's build does have an annoying bug in that it says Sim speed is 0 for you so you can't tell how well you're running. We've informed GPG and it'll get fixed in an upcoming update.
Favor points should acrue without a problem with tonight's build as well.
Check it out:
Okay, I now feel confident in publicly saying that I think that the connectivity issues some have had is going to go away next week.
Here's what we've built:
1. If you open ports 6112 to 6200 and port forward them to your IP on your router (this is for people who are familiar with this stuff) you'll get what is called direct connect.
2. If you don't do this, we will try to do a quick NAT connect. This takes longer but works for most people.
3. If NAT fails, you will be connected to our Proxy server system. This is not going to be super desireable at first because the physical location of the Proxy server is going to make the difference between a good and a lagfest. When the proxy server is nearby (east coast US player for instance to the east cost proxy server) it's perfectly good. But, for instance, if you're in Japan and it uses a proxy server in Singapore, you're adding 200ms to your latency which can make the game a total lag nightmare. so this is clearly your last result.
...
Now, right now, the current version of Demigod uses a connectivity server that ONLY does option 2. That's because it was decided early on that users shouldn't have to open ports. This, IMO, should NEVER have been the first option because when you force even technically sophisticated users into this system, you end up having a LOT of unnecessary connections to these NAT servers.
...
So, I feel confident that this next week is going to be the turning point for those people who have had connectivity difficulties PLUS the whole range of stat stuff, pantheon stuff, etc. will all come into line as well because, as I've said elsewhere, because the connectivity thing was our #1 priority, everyone, including the people who would resolve favor items and favor points and accurate win/loss stuff have been pulled onto this.
Heck, this past week, I had the ART team from Elemental working on doing benchmarks for speeing up NAT (That's where we finally concluded that NAT, for P2P is just feasible when there's thousands of players, it would simply require so many NAT servers, located around the world, to make it not totally awful that it wasn't financially logical - we'd be better off just hosting all the traffic which is better anyway).
..
We absolutely do not want to do any more "betas". The betas don't really solve our problems. We're getting rave emails from people in support today saying how this beta "totally fixes connectivity". NO, it doesn't. The only reason it seems that way is because there's so fewer people using the beta servers. If you check impulsereactor.dll, it's still the same one from day 0. So the betas don't really help because it gives us the same false sense of success that the actual open beta had.
The other problem with the beta is that it splits the multiplayer community which is really a problem. Right now, the Demigod MP community is fragmented. You've got a ton of people on Game Ranger. a Bunch of people on Himachi, others on Demigod release and some on the beta. And you get some very poor results hwen you play Skirmish or Pantheon or just find games.
..
Now one of the things that I've always been painfully aware of in online games is that you need a critical mass of users to make it work. With Demigod, we have some plans on how to make sure there's always lots of people to play with once the MP is bullet proof both in terms of connectivity and stat tracking. As someone who has been ejected from a pantheon game during connection, I know we have to do some QUICK changes to some of this stuff in order for the MP experience to have any integrity.
First, one of the things we did is we have everyone's email address who has updated the game. When we start the next-gen pantheon with favor points, achievements, favor items, stat tracking, disconnects dealt with and of course bullet proof MP (all of which I expect to be done this week) we will email everyone letting them know where we are at.
Second, the Stardock marketing push will actually begin. After that day 0 debacle with the MP connectivity being so crummy, I made marketing halt all Demigod marketing efforts. Those will go into high gear probably in the third week of May.
Third, and this is where we need your help: The demo. We will be working with GPG on the demo but I was thinking of proposing that the demo for Demigod be multiplayer ONLY. Moreover, I would like the demo players to be able to play online with legitimate players. Here's where we need your help: What do you think would be a good way to limit this demo then in order to get people to still buy the game?
Fourth, we plan to start formal Demigod tournmants with cash and prizes for the winners.
Fifth, Stardock is working on an update to Impulse Reactor that will allow people to have more control over the types of people people are matched up with. Think of it being like eHarmony but for gaming. 
You'll be able to describe the type of gamer you are and the match making system will pick out people for you. This feature is actually being developed for Elemental but because Demigod uses Impulse Reactor, it'll get that for free.
Sixth, Stardock moderators will soon be introduced into the online community. This will probably happen in mid-May. The idea is, if we have forum moderators, why not multiplayer ones? Their job will be to provide enjoyable multiplayer experiences (i.e. good sports, skilled player, etc.) PLUS they will be able to add or subtract in-game karma from players. People who are just nasty or horrible to other players will lose karma points. People who are good sports and good citizens will get points. Those with higher karma will tend to get matched up. Yes, we know this is controversal. Players will see the karma as part of their account and there will be a way for users who feel they were unfairly negated to appeal.
Seventh, all people who have bought Demigod at retail or direct up May 10th (and didn't return the game obviously) will be getting a coupon that will let them purchase a second copy of Demigod for 50% off for a limited time as a way for us to show appreciation for this community that has put their faith and trust in us.
Thanks everyone!
Last night we got the update out. The change log can be found on Impulse (right click and view the change log). Lots of improvements.
We ultimately marked it as a beta because we ran out of time and couldn't integrate the updated connection code which meant that the existing connection DLL had to be included which has a launch bug that causes an occasional crash when combined with this new build.
Anywyay, we had been hoping to surprise people by having the update include the Proxy server update. But it too wasn't quite ready. We had pushed until Friday to try to surprise everyone but we didn't want to go a second weekend without showing some substantial improvements for those people who have been having difficiulties.
We're going to have an update probably on Monday to move the beta from release.
The problem with betas, as some of you no doubt have noticed, is that you split the multiplayer community and that can make the skirmish and pantheon games a lot less fun because you end up with a lot fewer 3 on 3 human games. So we have to do these sparingly.
Now, that said, I want to address a nmber of points in one place that keep coming up online:
Comment 1: "Stardock is overstating Demigod difficulties, to get coverage, the game works fine for me"
First, yes, while the game works fine for most people. There are a significant number of people who are having problems connecting through the built in multiplayer match making. While I may seem to focus a lot on those issues, in the long run, this is a game that will live and die on its ability to sustain a MP community and it is important that people know that we are taking their issues seriously.
Comment 2: "The multiplayer match making is totally buggy, how could they release it like this???"
So at the opposite end of the spectrum, we have the people who are having trouble getting into games with the built in match making. I share your frustration. But Stardock was in no rush to release Demigod. Demigod was done as far as we were concerned.
Now, as the publisher, it's our responsibility to make sure problems in anything we release are addressed. But we haven't been fixing "bugs". We have been getting a crash course in P2P scaleability (or lack there of).
Should Demigod have had a big huge open MP beta? Yea, in hindsight, yes. If we had a time machine, we would do that. But we've released games that have MP before (Sins of a Solar Empire, The Political Machine, The Corporate Machine, and OS/2 games before that). The difference here is the difference between MP-centric and a game that's mostly played SP that has MP.
And how does that difference translate? Well, in Sins, you might (and Sins players can attest to this) a game that tops out with 200 people playing online at once versus a game that has 1000 people in games online and another 8000 people who are online in the process of looking for games. And as many people have said, it works decently overall.
There are programs like Game Ranger that largely take care of the match making for people who want to use that while we migrate towards a system where we host the games for players but it's not something that's easily done overnight.
PC game expectations have come a long way over the past several years. Leaders in our industry like Blizzard have raised the bar with things like Battle.net and our friends at Relic have created RelicOnline which works very well. And of course there is GameSpy Arcade which has evolved over a period of years.
We appreciate the patience of those who are having difficulties. I can assure you that Gas Powered Games and Stardock are working inhuman hours to solve their problems as quickly as possible (70 to 90 hours last week for instance per person involved). But these hours aren't about "finishing" a game. The game shipped finished. It's the scale of things that we were (clearly) unprepared for. We can't change the past, we can only take things from where they are now.
A lot of progress has been made internally. In less than 2 weeks, we've taken a game that is a peer-to-peer game and developed an infrastructure from scratch to make it so that the publisher's servers will be handling potentially 100% of the network traffic for all games played throughout the world. This was not contemplated prior to release but we are commited to making sure that MP is bullet proof even if it takes longer than we all wish.
We're working on it even now.
For those of you who do not know where the Development Journals area is, go here to see what we're all working on.
We got a really good new build from GPG this evening that has a cure for those people who have experienced in-game disconnects along with some in-game exploits and a few other things (favor points, achievements, etc.). So we spent a lot of time this evening working on that.
Meanwhile, I turned over the Proxy server setup to the enterprise team at Stardock to speed that up (for those of you not familiar with Stardock, we’re 70% an application company, 30% games with very distinct development teams). Basically I turned over the Proxy server work to the Impulse development team since they already have built the infrastructure around Impulse.
The connection servers, which is one of the areas some people (and we have reproduced this) are having a lot of pain with is something we’re still working on (2:30am as I write this) but we’re nearing the end of productive work on this. Basically, it’s a problem of scaling. The classic what works for 500 people falls apart when used by 5000 people at once. But a lot of progress has been made there as well.
Anyway, we’re going to call it a night and come back tomorrow to continue on the integration.
Now, as some people have heard, we are going to release this update as a “beta”. Impulse has an option for users to be able to try out pre-release software on it. Warning though, if Demigod MP is already fine for you, don’t mess with this.
Here’s how you make use of this feature:
We expect to have this up tomorrow. But I’m not making any promises as we don’t want to take a chance of putting something up that leaves people stranded (or having to re-download the whole release game again).
Update: Friday afternoon
We've finished integrating GPG's changed with the Raknet changes with the CVP changes. But we have still not resolved the issue of why it takes so long for 2 people, after being connected, to get into the lobby when there's a heavy load as the database seems to now be able to handle it (we set up a bunch of new servers) but there is still a random several second delay between when it says someone is connected and when it actually even attempts to insert them into the game lobby.
(whining begins)
I’ve been a software developer for over 15 years and for the most part, I’ve enjoyed it. But this past month has been sheer agony.
Spending immense amounts of time at work away from my little boys and my daughter and my wonderful wife has been taking its toll and I feel so helpless because my development area isn’t in the area that my teams are focused on – networking.
My friends at Gas Powered Games, even right now, are working the midnight oil to do their best with Demigod and similarly, my friends and coworkers here at Stardock are outside my office working diligently on the same kinds of things.
It’s so frustrating because where I come from in normal game development, your code either works or it doesn’t. If it doesn’t, it’s a “bug” and you fix it. You make sure you don’t have any show stopper bugs before you ship a game.
But with Demigod, the complexities of a vast matchmaking system is just a huge pain. Besides sheer compatibility nonsense to deal with (Europe with their VDSL and the US with ISPs sometimes blocking P2P traffic) you also have systems that are brand new that work fine under a reasonable load but fall apart when there’s thousands of people hammering them at once.
I feel like I’ve let down our community and our fans. People know we are diligent about this kind of thing but this has been so heart breaking to watch. While I know that the team will get it fixed, the good will of players is finite. Sure, it works for most people but that’s pathetic. It should work for basically anyone without a firewall.
It’s also putting a lot of stress on our teams, particularly the games team which I’ve had to put onto this project to help solve things faster. This isn’t game development, this is network administration. It’s just obnoxious.
The teams are working their hearts out both here and in Seattle (GPG) and progress is being made but what’s really depressing is seeing the check-in comments. They’re not fixing “bugs” they’re having to do new design for things that just didn’t scale well.
There’s a reason why only a few companies have put together these kinds of systems, it’s a huge pain. I am hoping they can nail this stuff down before the weekend.
Meanwhile, we have one of our distribution partners making threats because we won’t budge from our 1 price for 1 world model (i.e. Demigod costs the same in USA as it does anywhere else in the world).
(whining ends)
I just look forward to getting back to working on actual GAME development.