The house progresses. 1 year down, 2 more years to go.

I’m a map junkie. I love making maps.  I even bought a program called Campaign Cartographer years ago just to make maps.

One of the features of Elemental is, not surprisingly, a map editor. But the map editor is also being designed so that you can export your creations so you can easily print them out and such.

The map editor will let users build theri own mountain ranges, forests, swamps, you name it.

Of course, it’s also useful that you will be able to then play on these maps with your friends and share them online but just the ability to make your own worlds sounds fun to me anyway.

Gas Powered Games is hard at work on Demigod v1.01.  These are done separately of the recent updates Stardock has done to put in its own connectivity library to help the online multiplayer experience.

Here are some of the things already being tested:

• Bandwidth used reduced by 35% (more people with marginal connections able to play). This should reduce some of the “stutter” some people run into when they play with someone with a marginal Internet connection.
• Updated the way stats are submitted to make stat reporting significantly more robust.
• If a player loses connection with another player (or one player with a group of players but not all) only 1 user is disconnected instead of multiple.
• Demigod is better at detecting the difference between quitting and a disconnect (no connection dialogs when someone leaves).
• Significantly better handling of users who have shaky connections to keep them in game without adversely affecting in game performance.
• Favor Points, Achievements, Items stored on server more reliably (thanks to the updated stats system).

I don’t have an exact ETA as there needs to be a lot of testing on this (next week maybe).  But so far, it’s looking pretty good.

I’m in love.

I’m in love with proxy servers.  They didn’t just solve a lot of problems for people (and if you are out there and have any problem getting connected to people in the lobby you need to tell us specifics because we’re increasingly finding it difficult to find anyone who is having a problem getting into the game lobby (this discussion has nothing to do with people who drop in a game or pantheon or rage quitters, we’re just talking about getting player A to see player .

First, proxy servers are, relatively speaking, cheap. Second, it turns out, they don’t add nearly as much latency as we thought. Maybe 10%. But don’t take my word for it, maybe someone who’s used them already can testify in a comment about it.

We’re increasingly tempted to do away with NAT traversal entirely and go with a system that where you either can directly connect to someone or we just host it for you. NAT traversal is slow and while I’m a network noob, I just don’t trust it given what I’ve learned in the past week about how crazy some network connections have gotten.

In the good old days of Starcraft, people had either phone modems or a cable modem that connected directly to the Internet. In those days, security was so bad that I could go to Network Neigbhorhood and go on to my neighbors computers if I wanted. Not that I did of course…

Anyway, over the years, ISPs have changed. Security is better and bandwidth is higher thanks to things like multicasting.  People are behind various kinds of routers and on top of that people run various firewalls of various degrees of quality.

The nice thing about the proxy servers, however, is that it doesn’t care about any of that.

Do you know why so many people couldn’t connect before the proxy servers went up? The answer is that many ISPs (Cox cable for instance) use multicasting.  Your Internet connection becomes many Internet connections.  This is a good thing for consumers 99% of the time because you can get much better performance if the ISP can pick amongst many different sub-providers in real time.  Blended bandwidth is how I often hear it called. 

But it is, in my opinion, the death knell of pure peer to peer gaming in the long term because more and more ISPs will be using this. Heck, *I* use this on my machine right now.  I have 3 NICs on my computer each with its own ISP. I have a T1, a Comcast Cable and  a US Signal and Windows automatically goes between the 3.  Try doing a NAT punchthru on that.  Now, the ISPs obviously put all this stuff back together on their end to send the packet back to you. 

But consider this:  What if you’re playing your friend Bill and you did a direct connect to him and during the connection phase, the ISP, using multicast, happened to connect to Bill on an IP known as 79.134.34.5:7092.  Forget what your external IP is from your router. That’s irrelevant in this example.  Bill thinks your IP is that.  But if you then send another message and the ISP sends it through a different sub-provider because it happens to be the fastest at that moment Bill is going to see he got a packet from 209.69.142.34:8073.  Well, that’s not you right?  Bill’s computer might assume that that’s a different person. So what happens? Bad things.

Now, some network engineer is likely yelling “Yea but you can fix this with an intelligent NAT server.” Sure, you could keep band-aiding that stuff forever.  But a proxy server doesn’t care about how many IPs you transmit at.  It just has a record of you with your various blended IPs and sends back to the one it received it from and life is good.

Don’t take what I’m writing here as gospel. I’m still a network newbie here and this is just a blog where I’m seeing just how crazy connections have gotten since even a couple years ago. 

If any network engineers reading this want to comment, by all means, do so.  But from what I’m seeing, in the long-term, I think it makes a lot more sense for even synced RTS’s to just use dedicated servers because you can set them up anywhere in the world and the ping times today are nothing like the old days and bandwidth is cheap.

Incidentally, we are hoping to put this to the test today by having Pantheon games only use the proxy servers.  So we’ll be able to find out pretty quickly if this works. 

I don’t know about you but I find this extremely cool stuff.

The dev team says that only people who have serious connection issues should use this (and we’ll have a smarter parameter shortly) but we now have:

/serverproxyonly which will force Demigod to have our servers host your connection. You put this as a parameter for your Demigod shortcut.

We spent the evening playing with people online who have previously been unable to play with fantastic results.

Everyone up there who has the word “(proxied)” in the above screenshot are likely people who would not have been able to connect previously.

Mr. Glass Half Empty

But not all is perfect in Demigod land. 

First, this update is only for connecting people together. Developer Gas Powered Games is cooking up v1.01 that has a lot of in-game networking changes. No official ETA on that yet.

Second, there’s still some logic changes needed IMO for Pantheon games because it’s still hit or miss. We got a lot of good data tonight but I’d say Pantheon (and skirmish too) are still pretty crummy in terms of reliably getting a 6 or 8 person game going with qualified people. I’m tempted to just have all pantheon/skirmish games hosted by us but that’s a bit of a last resort.

Possibly tomorrow, we may put up a Memorial weekend update with some new parameters. And yes, we do plan to have these parameters make their way into the game itself once we finalize them and set things up with Gas Powered Games (we did some voodoo to let us get those command line parameters in the first place without having to change the game EXE).

Hope you guys like what you see.  We are seeing some strange things from Version DSL and Cox Cable ISPs in terms of how they do their connections so we’re looking at that. 

Goodnight/good morning. Cheers!

I recall someone saying that it would take many months for someone to develop, from scratch, their own client/server proxy system and even then, the publisher couldn’t do it because it would require changes to the game.

For those of you in software development what I'm going to write is known to you so you can skip the next paragraph.

So as some of you have heard, we have a lot of developers now on Demigod at Stardock. Since it became clear shortly after launch that there were tremendous scaling issues with the network connectivity for setting up games (i.e. when thousansd of people are online, messages between the NAT server and the clients got backed up) Stardock has taken it upon itself to write its own system in order to try to solve things faster.

You saw the initial results of those efforts late last week.

Tonight, we're launching the update that tells Demigod to have Stardock servers host the game if players have trouble connecting.  There's a lot of different modules involved for this and so different modules pass QA and are approved for integration.  

Normally, these modules would get put together into a "big" update but with Memorial weekend coming up which is a prime time for people playing games like this, we're taking those updates as they come out of QA, putting them in and taking advantage of Impulse's rapid deployment capabilities to put out these updates. They're tiny (less than a megabyte) each.

So tonight it looks like we're going to have 3 in one day.  Each one progressively better.

Now, will this make everyone happy? Probably not because there's some logic behind deciding when it should give up trying to connect someone and just use the proxy server.

If you're someone who has serious problems connecting to other people, you can force it to just have our servers host you by adding the command line parameter /serverproxyonly to the Demigod short-cut.  Though, note, this will add some latency but probably not much (we had test games all week between USA and Europe with total ping times >100ms).

So it's going to be a long night for us but we're extremely excited to see how this series of updates works out.  If anyone has any problems AFTER the third update (please don't post until 1.00.088 is being run -- that's the official version off of Impulse - mouse over the version number in Impulse to in the bottom left to see what sub-revision you have).  

Stay tuned!

Recently I posted a blog asking what people’s speed test rating:

http://frogboy.impulsedriven.net/article/351550/Whats_your_speedtest_rating

Lots of interesting data there.

Most people tend to only think about their download speeds and little about their upload speeds. But if you look, at that thread, a lot of people have marginal upload speeds as in well less than a single megabit.

Since I know I’m not a typical user, I talked to a friend of mine whose connection is about 2 megabits down but only .16 megabits up.  Looking at that thread, there are quite a few people who are getting less than a half megabit up.

Then I asked him to open up task manager in Vista and open up networking:

I had him add more columns so I could see the bytes sent per interval.  Turn out, he was already sending around 10k a second (bit torrent client in the background) plus a bunch of little tray items that occasionally tossed another 5k to 20k now and then.  That’s kilobytes btw.

Now, when you see that you have say a 0.2 megabit upload speed, that means your system tops out at 25k per second. 

Demigod, being a game that is synchronous (what you see on the screen is the same as what other people see) uses 1k per second up to over 20k per second as you add players.  And it’s not linear. I.e. it takes a lot more to handle 8 people than 6 people and a lot lot more to do 10 people than 8.

So let me ask you a question: What do you think the in-game behavior would look like if you’re playing someone who has an internet connection that can’t do more than 0.4 megabit upload in an 8 person game? Their sim speed would be fine. Depending on the way their system handles ping requests their ping might be fine or it might quickly jump to several seconds to respond.  Toss in some “security” programs that sniff every packet coming in to that too.

Now luckily, there are some things that can be done about this. 

First, there’s just plain awareness. Since I’m always in debug mode, whenever I get into a game that’s “stuttering” I ask if anyone is running any network programs. I usually end up finding out that someone has bit torrent running in the background but they have a “5 megabit” connection.

You know, 5 megabit like this:

Second, it’s pretty clear (to me anyway) that we’re going to have to make some tools that are accessible in game that will help identify people who are (need a new term) band-lagging out. They  might have a great ping (26ms for instance) but they can’t play with 8 people unless they turn off the other junk they have going on.

Third, we can look for ways to reduce the amount of data being sent across. I’m not privy to the network traffic but I do know that things like the friends list and chat windows in game and such do add to the mix.

Fourth, those who have seen people disconnect during the middle of the game can open up task manager (on Vista) and do what I asked my friend to do. It’s the type of thing that might help people with other games too. I suspect my friend’s Word of Warcraft experience will improve after I told him to turn off some of the junk he had loaded.

Two-Way education

One of the things I’ve learned during Demigod’s release is just how unaware some people are on how this stuff works.  You’ll get someone with a connection like what I showed above saying “Well, Left 4 Dead works fine!”.  Well, of course, you’re connecting to a dedicated server in a non-synced game.  A first person shooter is a totally different system than a strategy game.

I had one user I was talking to who was telling me how “broken” networking is in Demigod because every time he plays he lags out.  I asked him a few questions and eventually discovered that he has an Xbox 360 and is almost constantly downloading movies from Xbox Live.  When I brought that up, he said “But that’s on a different network cable.” 

I highly recommend looking at that thread though:

http://frogboy.impulsedriven.net/article/351550/Whats_your_speedtest_rating

It’s very educational.

If you find someone in game behaving utterly horrible, feel free to take a screenshot and post it here.

One of the most infuriating things about the Demigod pantheon is that it assumes players will be able to connect. The problem is, you get one person who can’t connect and bad things happen:

Now, in theory, with the proxy code (that’s staged but still being tested – there are a lot of fail scenarios to be gone through) the two players who couldn’t connect would get routed through the proxy. Will it work for sure? There’s no way to know until it’s out there and we see. 

Look closely at the screenshot. Those people were able to connect to other people fine. But not to each other.  Why? Some router thing? Some bug? Something else? I don’t know but like you, we are aware of it and it drives us nuts too.

Now ideally, what the system should do is pick one of those guys and eject them and find someone else. But that’s a level of sophistication that isn’t in there yet because, again, the system was designed on the assumption that we’d find a way to get everyone to connect (seems quaintly naive in hindsight). 

But the overall experience is maddening and frustrating beyond belief.  The proxy servers better fix this though, I fear, that the other issue (a person having more than 1 IP address) could be the cause as well (a person with multiple network cards / wireless cards that are connected to the Internet – that’s coming up next).