For those of you who are interested in what is happening, what things we’re looking to do, here’s a video where I walk around and ask some of the key people on the Stardock side of things what they’re doing to keep improving the Demigod player experience.
Well I hope you guys aren’t getting sick of these status reports. As I type this the build from last night is going up. Here’s what’s new:
This update should hit Impulse later tonight. As you can tell by the change log, this is really an interim update. We have a much more significant update in testing but we didn’t want to wait until that build was ready to get a new build out there that improves things.
A preview of what’s next:
I can’t promise that we’ll have another update tomorrow. There’s a lot of coordination that has to take place between GPG and Stardock and we’re 3 hours apart time-zone wise. But let me give you a head’s up on things we’re pretty close on:
So those are the 3 things we are hoping to deliver tomorrow. You may have noticed there aren’t a lot of “bug fixes” here. That’s been the most challenging thing here. Like I say in the video below, launching Demigod has been practically like launching an ISP for us in terms of infrastructure. What once seemed like a simple matter of tossing up a NAT negotiator that puts people into games has really become a lot more than that. Our fault of course, we couldn’t leave good enough alone and had to have the Pantheon and the smart Skirmish mode and such but I think in the long run it’ll be worth it.
In defense of Peer to Peer
One thing I really want to stress here is that there is nothing inherently wrong with peer-to-peer. Supreme Commander is peer-to-peer and it works fine. The difference, of course, is how they were released. Digital distribution is a godsend in most ways but having people from around the world playing together immediately has created a lot of the challenges that we must overcome. That’s an over simplification here I just don’t want to see people bashing Gas Powered Games for choosing to make Demigod P2P. They’ve had great success with it in the past and Demigod will soon be up to speed as well.
My point is, GPG has a lot of experience in this area with a proven game (Supreme Commander) that demonstrates that this works. They know their stuff.
Greetings! Tons of different things to talk about today so bear with me.
Based on the logs, multiplayer is working pretty decently for most people but for others it is completely broken. A program called Game Ranger has helped a lot of people but there’s a lot of warez users on there.
Let’s talk about the pirates
I’ve read a lot of misconceptions out there. First, no, we very much don’t like it when people pirate our stuff. Our position has simply been that people are going to pirate a game whether it has draconian copy protection or not and the people who will buy a PC game will buy it and the people who won’t won’t.
The first few days of Demigod being miserable on-line was largely the result of us simply not having taken into account the effect pirated copies would have on our server resources. So we had to scrounge servers and get users onto there.
Once we got legitimate users onto the servers that were isolated from the infrastructure that was being pounded on by the pirates, the connection situation improved dramatically as you know.
It’s easy and convenient to blame piracy for the launch debacle of Demigod but that would be inaccurate. Even had Demigod had some nasty copy protection on the DVD, it would still have been cracked and this problem would have come up anyway.
So contrary to what some have said, Stardock’s anti-copy protection stance is not the cause the first few days of Demigod’ connectivity. More like, our inexperience in publishing multiplayer-centric games caused us to underestimate the server infrastructure needed.
We do not intend to change our policy on copy protection. This leads us to the next part.
Understanding how Stardock supports games
Our “copy protection” is to treat our games as a service rather than a product. Those of you who have our non-games like Object Desktop or Multiplicity know what I speak of.
That is, we keep releasing updates to our games for a long time. Our publishing agreement with Gas Powered Games includes game updates to the game for at least a year. And after that, we still expect to release updates.
Once the multiplayer connectivity stuff is straightened out, we can start to talk about the kinds of things we can add like team based joining, how clan support should be, what kinds of items people would like to see, what kinds of additional demigods people want, and so on.
But our first priority has to be to those users who find connectivity flakey or just plain non functional. They paid $40 for a game and we have an obligation to help them before we start adding in new features and such.
Stardock’s business model is that our titles generate revenue over a long period of time rather than in the first 90 days. So we look at our journey with you being at the very beginning.
Why we love the community
Another thing that really keeps us going is just how wonderful the community is. Obviously, it should be clear by now that we weren’t anticipating the kinds of connection challenges some people have run into. Our guys have been working with the guys at Gas Powered Games night and day on resolving them.
I can assure you that Gas Powered Games is as interested as we are in making sure problems people run into are solve quickly. Not in some patch a month from now but like updates in a matter of days.
But what really also has made this experience a bit amazing is just how great the community has been to us. One user (OrleanKnight) sent our team a box of Mrs. Fields cookies. Other users have sent us cards and pictures their kids have made of various Demigods. How can you not love such a group of people?
Let’s face it, you guys just rock out there.
Getting the next update out
We’re going to start to try to be a bit more careful about how quickly we put out updates. The last update went out without updating the pantheon security checks to support the new version which resulted in most games not being counted. That’s the kind of thing that haste brings.
That said, as I type this, the team is working on an interim update to Demigod. It doesn’t “Fix” connectivity issues one way or the other but does provide better error messages (and we still have work there).
For example, one “Fix” that still needs to go in is the “Failed to join game” message in custom games. What it should say is “Game is full”. But a lot of frustration people run into is in the kinds of error messages that are shown (or lack there of).
I’m pretty sure this update tonight does start to wean experienced users off of AI players in skirmish and pantheon. At the very least, it has the support so it can become a server side change tomorrow.
Eliminating the connection issue
For those who can’t connect (you know who you are) the solution will be the UDP proxy servers. They spent a lot of time today getting them up and starting to test them to see how they work. Unfortunately, it’s not something that can be done overnight. The team says later in the week they hope to have it ready.
But what that will do is alter Demigod from a peer-to-peer game to a quasi-client-server game that maintains the peer-to-peer performance. If there are any IT people reading this, feel free to explain to lay people the significance of being able to do this in a couple of days without having to change the game client. 
Long Term
For us, Demigod is a long term commitment. So people who worry about this being like a typical release and move on type thing need only to look at our other games. We will be continuing to work to ensure Demigod evolves over time. I look forward to having official tournaments between individuals, clans, etc. So there is definitely a lot to look forward to.
If any of you have any questions, just comment here and I’ll do my best.
Cheers!
Evening update
Just wanted to give you a head's up that we're not going to put out the update tonight. We have it staged but there is a concern about releasing it and then having the team go home in the event there is some unknown problem. So we're going to put this up tomorrow afternoon. Stay tuned.
Tomorrow will mark the 1st week since Demigod’s official launch. The newer reviews that have come out since the initial launch have come back to more where we’re expecting:
Firing Squad: 84 of 100
http://www.firingsquad.com/games/demigod_review/page6.asp
Cosmos Gaming: 95 out of 100
http://www.cosmosgaming.com/articles.php?id=1963&articletype=review
GameFocus: 92 out of 100 and an Editor’s Choice
http://www.gamefocus.ca/?nav=reviewCard&fid=7318
Where are we at?
First, one of the side benefits from the day 0 debacle is that I have fallen in love with a new program called GameRanger. It’s hard to remember but back in the Total Annihilation days, playing multiplayer involved knowing someone’s IP address and trying to connect to their game via IRC or something else. But then program like TEN and Kali and such came out and made things better. But if you’re having problems with getting multiplayer going, I have to say, GameRanger works pretty well and it is doing basically the same thing we’re in the process of setting up.
Now right now, Demigod, as a game, is awesome. Demigod, as an on-line service, however, sucks ass. It works fine for what I dare say most people but for others, it's just awful. Our job is to make sure it works fine for everyone.
Talk about being babes in the woods. Sins of a Solar Empire’s network “infrascture” to this day involves 1 server. Demigod’s involves dozens of servers now because of so many people using it. I guess it turns out most people playing Sins of a Solar Empire were playing it single player.
So what is the cause of our woes?
Let’s talk about peer-to-peer networking. Here’s how it works:
Everyone has to connect to everyone. And unfortunately, not everyone can. Worse, the connection servers we’ve had have been just being pounded by the number of requests.
Why wasn’t this found in beta? There’s two reasons:
1) Our beta group was overwhelmingly located in North America and they didn’t have problems connecting to each other. That is still the case now. But people in Europe tend to have problems connecting to people in the United States.
2. Our synthetic stress tests on the server were through pretty ideal connections where connections were pretty easy.
So what can do do about this?
This is a bit of an over simplification but you get the idea. We have the connections pass through these proxy servers on-route to the other player. The geographic location of these servers is key to keep users from getting lag. I’m told it shouldn’t take very long to get this part going.
It’s not all bad
I will say that the current system does a pretty decent job. Once we were able to throw enough servers at the problem we’re seeing a lot of successful MP games. But that does little good for the people who just can’t connect and no amount of port forwarding is going to solve the case where two players just can’t talk to each other.
Expectations
My expectation is that by the end of this week, the connectivity problem will be largely solved. I hopefully won’t have to eat my words. But the fact that things like GameRanger works so well indicates that there’s little problem with Demigod itself but rather the service infrastructure behind it.
Stay tuned.
Most of the current status can be found here.
So I played a lot of games of Demigod online today. And the issues mentioned in the Friday status report were ones I experienced occasionally. I didn’t experience many problems with the exception of people getting kicked out of a Pantheon game. But I’ve gone through the responses here and forwarded them on to both Stardock’s NetDev team and GPG after doing my own analysis.
One of the things I’ve concluded we’re going to need to do to make things better quicker is to set up proxy servers for people who simply can’t directly connect. So the player would connect to us and then to that user. In theory, it should add relatively little lag but it would dramatically ease the pain some users, especially those overseas, are having.
The plan is to bring the team together tomorrow (Sunday) to finish up what we had on Friday which should help a lot with skirmish and pantheon.
One request I do have for the community: While we know statistically that multiplayer is working in most cases, there are a lot of people who cannot connect. Please refrain from telling users that they need to learn how to use their router or whatever. The fact is, people shouldn’t have to configure their routers to play a video game. We do ask those who are having problems to have as much patience as they can muster. Both Stardock and GPG are doing their best to make the multiplayer experience good for everyone.
Anyone familiar with Stardock can also tell you that we support our games – for years. We’re on the job.
I see reports of "bad connectivity". But please understand, when we play the game with people online it works perfectly so we need to try to understand specifically what you're seeing.
So when you say "I can't get into a multiplayer custom game" for instance, what specifically do you mean?
Can you bring up the connectivity dialog? What does it say?
That kind of thing. The more you tell us, the quicker we can resolve it.
See http://forums.demigodthegame.com/347467 for where we're at.
Thanks!
Well, what a dramatic week it’s been. The teams at Stardock and GPG have been burning the midnight oil this week.
As those of you who have the game can already see, the server issues are gone. We’ve recreated a duplicate of the server infrastructure we had but dedicated to users who have the most recent version of the game and a valid CD key (serial #).
Based on the logs, we are seeing lots of games being played on-line now. Yay. Average game has approximately 4.7 humans in it which is a good sign.
Some clarifications
I’ve seen a lot of news articles this week and a lot of confusion about what occurred this week. The issue isn’t terribly complicated.
Ars Technica had a good article that describes what happened. But still, a lot of people seem to think warez users are able to play multiplayer games. No, they can’t. Even the retail box has a serial # in it that users have to use and be validated to play online. What brought down servers was a lot more benign than that. It was the HTTPS requests to inform users if there was a new version along with checking the community features for info (friends lists, chat channels, etc.) and things like that. Things like that are pretty piddly. It’s only when you get a ton of users doing that at the same time that it becomes a problem as we saw.
But here’s the thing: While piracy is annoying, you can’t blame piracy for this problem. Let’s face it, there’s plenty of data out there about how many pirated games are being played. We should have looked at that. We assumed since Sins of a Solar Empire and Galactic Civilizations, both of which sold extremely well and got great reviews, that the # of pirated copies of Demigod in use would probably be in the same ballpark, maybe twice as much. But had we looked at what other publishers have said, we would have known that it’s not unusual for there to be hundreds of thousands of warez copies in use. And if we had, we could have simply had the retail version not have any HTTP calls in it and instead just had an update button on the main menu to check for updates and voila, problem solved.
The second misconception is the argument that because Demigod’s retail version is heavily pirated that it costs massive sales. But that, again, puts the blame on the wrong parties. If you want to talk about the horrible multiplayer experience on launch day, well, that’s our fault because of what I said above. If you want to say that the horrible day 1 multiplayer experience resulted in negative game reviews which will seriously damage the game’s sales then I say again, that’s our fault too because of what I said above OR we could have just sent out the review copies on release day (Tuesday) and reviewers wouldn’t have had it until Thursday by which point the problem had largely been resolved and the review scores would have been fine. But in either case, it’s still our fault.
So now what?
Now that the servers are working fine we’re moving away from the “#$R@#@# Demigod sux!” posts and into the regular new game release issues.
So what issues are we seeing and working on? Here are a few at the top of our lists:
1. Players getting disconnected during games. Demigod’s lag tolerance is fairly low resulting in disconnects if a player lags out a bit. This is fairly easy to fix. You get a player in Australia playing a user in Europe and there will be times when there’s a hicup in their connection and POW, disconnect and it’s extremely frustrating. I played all day today and it happened to me. This is a very high priority.
2. NAT negotiation. For users outside the United States in particular using DSL, this is a problem. This is a case where player A can’t see player B and thus they can’t play together. This is something we will be aggressively looking at next week. If we hadn’t had the server overload, we likely would have this addressed already. 
3. Pantheon games. Right now, the system is excessively biased for new players – it wants to include AI players as cannon fodder. While this is fine and dandy for new players, once you know what you’re doing, having AI players in the game is incredibly annoying. So next week we’re going to change it so that if you have played a few games, it will wean the player off of AI players until you only play humans online. Same for Skirmish.
4. Favor Points and such. There’s some annoying stuff happening with favor points which is related to the ranking system in Demigod as well. First off, we plan to reset the rankings at some point when we enter Epoch 1 (we are in Epoch 0 presently game-wise, I’ll explain Epochs later but in short, every so often we’re going to reset the rankings, save the results as an epoch and go from there). In Epoch 1, custom games won’t count anymore as there’s too many ways to game it.
5. Misc. Bugs. There was a crush bug in the game that we’ve fixed internally but it got to be too late in the evening to package it up and get it released tonight. We have internally fixed the annoying issue of “A ppplayer has left the game” at the start of skirmish and pantheon games. Basically, when you get a new game going, there’s some data that has to be exchanged between players at the start. But the threshold of time you’re given to do this before it decides you’re too slow is set too low so we’re changing this to be more tolerant. In our tests with players we found on the chat channel who were having problems, it took care of it.
So that’s what we have in store for next week. After that, we can look at the typical balance requests, cheese stuff, and whatever else crops up.
Now, while the reviews are likely to have a pretty serious impact on sales, it does not affect our plans to continue to release updates and enhancements to the game nor does it affect our plans to release additional free Demigods.
For most of you, have fun this weekend! For those of you who are having a problem with something, hang in there. We’ve got your back!
Draginol is my home account in case people are confused.
Brad "Frogboy" Wardell
After a marathon coding session with lots of server moving we were able to release a Day 2 update which, from what we can tell from both the logs and from talking to people in chat has made a drastic improvement in the multiplayer experience. In a nutshell, today’s update let us connect to a different multiplayer NAT facilitator than the original. This one isn’t touched by users who get the game at retail (or those who have a warez version of the retail).
Now, I do want to address something about the update that I’ve seen users talk about. We aren’t blaming piracy for the fact that the day 0 multiplayer experience absolutely sucked. The issue boiled down to us having put together a multiplayer infrastructure that was designed to handle around 50,000 or so connected users. If the game took off, we would simply add more servers as the load increased.
But what happened was that we ended up with 140,000 connected users, of which about 12% were actually legitimate customers. Now, the roughly 120,000 users that weren’t running legitimate copies of the game weren’t online playing multiplayer or anything. The issue with those users was as benign as a handful of HTTP calls that did things like check for updates and general server keep alive. Pretty trivial on its own until you have 120,000 of them. Then you have what amounts to a DDOS attack on yourself.
So the day 2 update we released basically made sure legitimate customers were no longer being affected by those users. As a side note, no we can’t just eliminate the infrastructure being used up by warez users because they’re running the unprotected retail version and we can’t make a distinction between retail and pirated since there’s no copy protection. It’s not a huge deal in the long run (except to our metacritic score), it was just an unexpected challenge that made day 1 a very bad multiplayer experience.
What’s next…
So we still have lots of areas to improve the multiplayer experience such as dealing with European DSL stuff (one example) and we need to make the pantheon and skirmish stuff more robust and provide more feedback. The team is probably exhausted so the next update may not happen until next week since the day 2 update seems to have made a pretty big difference for the majority that I think we can probably take the weekend off.
But next week I expect we’ll have an update that addresses a lot of the above plus starts to look at balance and regular in-game bug reports and such that we’ve been collecting.
Anyway, that’s all for now. I’m going home to sleep. No really, I mean it this time!
A new update has arrived that should significantly improve the multiplayer experience.
See the change log here:
http://forums.demigodthegame.com/346928
Impulse Version: 1.0.48
Demigod EXE version: 1.00.0093
After a long night of connection server moving around along with a new ingame connection client, we are happy to release the day 2 update.
Change log:
+ Connection dialog now allows the host to eject players trying to connect.
+ New NAT Connection Server now isolated so not affected by general server load resulting in much faster, better connectivity.
+ Eliminated check for update on launching the game
+ CVP calls asynchronous (a server hicup on our side is a lot less likely to cause users to be disconnected)
Notes:
This isn’t a silver bullet. This is basically what Demigod would have been like on day 0 (minus the eject button) if the connection server wasn’t getting pounded.
That said, we’ve noticed some issues that you should be aware of that we plan to iron out this week including:
1. Because connections happen much faster in custom games, you can occasionally end up with players in the lobby that aren’t truly connected to everyone. Their ping will be yellow (or 0). The host will need to eject one of those players and have them rejoin.
2. This update does nothing for users who are behind a firewall or router that has ports locked. See http://forums.demigodthegame.com/345345 for tips in that area.
3. Obviously this is an update that we would normally want to run through a 2 or 3 limited beta since major surgery was done to the servers to reduce their load. But we felt strongly that we needed to get a release out that showed improvement for most people. We expect to release another update in the next few days once we can see the effect of the connection server in conditions we consider normal.
Good luck!
