Stickiness and Networking

(TL;DR: We got the first version of multiplayer working. Skip to the end of the post if you want to read more about that.)

Our debug model placer interface would like to tell you about the fine qualities of the monarchy.

Our debug model placer interface would like to tell you about the fine qualities of The Monarchy.

My time right now on CE is evenly divided between three things: the completion of new features (finishing the dynamics system, at present); revival of old features which stopped working due to bit rot at various points in development (animals, steps leading into buildings, etc.), or moving the pre-alpha from “let’s get it running on computers” to “let’s get it playable on computers where everybody actually moves, and does their jobs under real world conditions.” Hooray! Bug busting.

I like to think of this as making the game less sticky. Basically, what the last part means is this: sometimes, in the current build, people may never pick up a job, or may stop working altogether. Sometimes, under rare circumstances, the game will crash. (this has now been fixed.) Less damning problems include things like misbehaving UI widgets – things that don’t work correctly, things that accept mouse clicks that they shouldn’t (buttons in the UI accepting clicks from scroll bars, etc.) to latency. It is very much like somebody has dumped an enormous jar of honey into your video game, and you need to remove it all – ergo, stickiness. As you may imagine, chasing these through a game with the complexity of Clockwork Empires is quite a mess. Right now, for instance, I am working on the cheerful problem of ensuring people do not get trapped in modules when they are built, which means that before building a building characters must wait for all characters to leave the build zone before they actually go ahead and change the obstruction layer. The problem, of course, is that the builder is also a character – and he’s in the build zone, waiting for everyone else to leave – which means I need to calculate a list of all the squares next to a building or module that are next to it (so the character can build), but not in it (so that the character gets trapped).

"It's got me!  ... Go! Save yourself! There is no life for us, together, any longer. My place is here now, embedded in the floor of the carpentry shop."

“Go! My place is here now, embedded for all my days in the floor of the carpentry shop I myself built. Alas, but for what cruel irony was wrought by these, mine hands.”

These are all easy things, but annoying, and a lot of them have been fixed. Highlights include: not using a destroyed item as a building material for another building (crashes the game); ensuring that if your job item is destroyed, your job cancels correctly (people would get stuck); ensuring that, if your job got cancelled, you started at the start of the next job and not half-way through it (random things, breaking, forever); etc., etc. This is all standard bug hunting, but it’s the difference between a sort of wobbly tech demo and something people can actually play. I’m happy to report we’re grinding our way through it at a good speed, so it’s feeling more and more like a game every day. I know I keep saying that, but it’s true!

What is more interesting, though, is networking.


The half-eaten tub of chili in the middle of this photo really sells it. In other, related, news Nicholas is never allowed to take pictures of anything for Gaslamp Games ever again.

So – way back when we announced this game in 2012, we mentioned support for two types of multiplayer gameplay. The first, round-robin gameplay, provides a way to share worlds automatically between players for Boatmurdered-style hijinks. (This is totally not done yet.) The second mode we talked up was the ability to play online, with up to four of your friends.

This is now working: we can have two to four players connect to each other in an exciting game of networked Clockwork Empires. Well, it’s not very exciting yet – you basically run the same colony, “cooperatively”, and there’s no real notion of “your characters” versus “your friends’ characters” – this is all fairly easy stuff to put in. The hard part – taking an extremely complicated, multi-character simulation and getting it to work over a network? That’s done.

So how do you take a game where you may have two hundred characters, and get it working over a network? Answer: we cheat. Instead of sending either a) the game state, or b) “the parts of the game state that have changed” over the network (usually called deltas, after the fraternity), we run a deterministic copy of Clockwork Empires on each machine in the network game. By determinism, this means that “each person, playing the game with the same set of commands, will get exactly the same output.” In practice, this is very fussy – for instance, we can’t use floating point math, because floating point math may, in very rare cases, do totally different and wacky things on different machines that will throw off the determinism. Therefore we go through Great Pains to ensure everything we do is deterministic; our worlds are generated using old-fashioned fixed point arithmetic, just like we used to do Back In The Day before the original Pentium chip came out, and we had to do 3D on a 486DX/33. (Okay, I’m showing how long I’ve been doing this here.) We have to make sure that everything is sorted by consistent game IDs, and referenced by consistent game IDs; one of the last bugs causing this to blow up was caused by a linked list class sorting things by their location in memory as opposed to their game ID, which is a no-no. Anyhow, it all works, and two copies of Clockwork Empires running with the same random seeds for terrain and simulation generation will run in perfect sync.

Above, Micah furiously heliographs gamestate updates.

Pictured: Micah furiously sending gamestate update packets via the heliograph.

In fact, there’s even more fakery going on. Despite the fact that Clockwork Empires looks like everybody’s smoothly moving from place to place, and dodging around each other, internally Clockwork Empires is discretized and everybody is on a fixed grid – to make the obvious comparison, it’s *exactly* like Dwarf Fortress. We just use colossal amounts of fakery to disguise this important fact from you. A cool bonus feature coming soon from all this work is debug logging. Since the simulation is deterministic, our testers can send very small logs back to us so that we can figure out why their game crashed or did Something Weird – which will, in turn, help remove stickiness.

Most of the work on networking was done by Micah and Ryan. (I just sat in the back of the car and gave incomprehensible directions.)

Imagine an image of UI annotated with a numbered list of everything you have to do stamping on a human face forever. This is Nicholas' life.

Imagine an image of UI annotated with a numbered list of everything you have to do stamping on a human face — forever. This is Nicholas’ life.

There’s a lot of work to be done here, of course. Daniel and David have both written an enormous list of tickets related to UI concerns and gameplay concerns, which I had to promise to do before I complete this bug post. More problematic for me is making the networking feel solid, and to give users the ability to automatically set up games despite being behind firewalls and routers. This is called NAT punching; the Steam API provides one solution, but we need to come up with something for users who do not have, or do not want to use, Steam for this purpose. Indie games traditionally do a poor job of this, probably because NAT punching is a pain in the neck.

There is also the longer, more interesting problem of game play. Right now, everybody plays in the same colony, in an anarchic free-for-all. At some point, we will need to address the issue of team-based gameplay: multiple players, on multiple teams, competing for the Glory of the Empire. With this, there are questions about balance and the Regulation of Gameplay. How sandboxy will the game be? How do multiple players colonize and fight on the same map? Is siccing the fishpeople on your opponent a viable strategy? (Yes. -ed)

What does a multiplayer simulation of a hundred or more discrete, interesting characters look like?

What is the gameplay here?

I don’t know, but I’m very excited to find out.

Posted in Clockwork Empires | Tagged , , , , , , , , , , , , , , , , , , ,

18 Responses to “Stickiness and Networking”

  1. Matt says:

    I’m all for multiplayer if it’s going to have a voice chat feature in which 13-year olds call each other cads and bounders.

    As for the determinism solution to keeping everything synchronized, it reaffirms my opinion of, and my prognosis for, your mental health.

    { reply }
    • Cam says:

      I’d like to see a system where you can’t communicate directly, but rather have the option to send a procedurally generated Sternly Worded Missive to your opponents.

      { reply }
      • Tyler says:


        Also I feel a group needs to be started to encourage all gamers to use Victorian slang when gaming online in any circumstance.

        { reply }
  2. Aegis says:

    Is that Diggle in the photo giving you strategy advice? (Or is he wishing you didn’t order Diggle Eggs alongside the chili…)

    This is lookin’ good, I like the comparison to honey. Guess this makes programming bugs “bees” and you have to squish them without getting stung…

    { reply }
  3. J23 says:

    So, if I understand this multi-player contraption of yours, clients just send commands to the server, and the server outputs the game based on a certain few possible ways the commands can work out?

    Does this mean multiplayer games will essentially play out very formulaically?

    { reply }
    • No. Basically, what it means is this: everybody’s game executes in the same way. You’re still pulling random numbers. For instance, imagine that we each have a deck of cards. I instruct you to deal four cards to yourself, and four cards to “me”. I then deal four cards to you, and four cards to me. If we deal from a pack shuffled the same way, and in the same fashion, we will end up with identical hands. We can then play poker against each other as long as we agree that what we are doing is the same at each step of the game.

      That’s all we’re doing, just on a larger scale.

      { reply }
  4. Micah J Best says:

    I would be happy to share my Paleo Chilli recipe, full of meaty protein and spice that a programming brain needs to thwart the demons of parallelism and networking.

    Also, I must report that my attempt to port CE to run on Diggle has so far met with failure. So far he just curses at me, though I believe that he is cursing at me in a more Steampunk way than before.

    { reply }
  5. JC says:

    I’m curious how you make sure the commands from players get synced in the correct order. If everything is fully deterministic, then you’d have to make sure if the host pressed X and client Y took 300ms to receive the button, that event was retroactively applied and the whole model was updated correctly.

    I’m sure there is a clever way around this, but its baffling for the moment to imagine what that is.

    { reply }
  6. Micah J Best says:

    The cleverness comes in the general architecture of the system. There is a disconnect between the visual representation and the actual simulation values. Once an action is performed it is sent to an arbitrary ‘server’ game instance with the information on when it was performed. All actions for a very very short period of time (yet to be determined) are resolved and the complete set of actions are sent to each game instance. At which point the actual simulation effects of the action are computed. These actions are very lightweight so this can be done in a fraction of second and because the the presentation started the moment the player clicks the button that ever so slight delay is actual effect is hidden. This approach might not work amazingly well for deathmatch style twitch-fest, but for a colony builder it should be ideal. This would be next to impossible if the system wasn’t designed this way from the ground up to have this kind of separation.

    I mean… I do it all with pixie dust and good intentions.

    { reply }
  7. Electro says:

    How well is fixed-point working out for you? I used to be concerned about the performance hit versus floating point, but never got around to quantifying my concerns. (Also, good job on going the determinism route. Woo! [insert RTS engine nostalgia here])

    { reply }
    • Back from Steam Dev Days, working through comments in random order:

      Fixed point performance versus floating point is basically a non-issue. (How’s that for a turnup for the books?) All you’re really talking about is sticking some shifts on front of integer arithmetic, and a good compiler will work ten kinds of sorcerous magic on it before you have a chance to say “Michael Abrash.”

      If you’re worrying about that on modern hardware, I envy you of your problems.

      { reply }
  8. Eric says:

    That chili is only half-finished if you’re accustomed to eating the tub as well. 😛

    { reply }
  9. Mike says:

    What would be the main focus – singleplayer or multiplayer?

    { reply }
  10. DelicateTask says:

    Will Capture the Diggle be a game mode? Also, will round-robin mode automatically move on after a certain about of in-game time, gameplay time, whenever the player decides, whenever the other players decide? How will that be defined?

    { reply }
    • I have no idea about other game modes, although Daniel and I did discuss a few things on the train. One idea that seemed like an interesting one would be domination – “who holds onto the island for the longest amount of time over ten years”, that sort of thing. I’m not sure if this would actually be fun, though.

      Round-robin: yes, it would decide after a certain amount of in-game time, probably decided by whoever starts the robin. We’ll figure this one out once I get here.

      { reply }
  11. Farmerbob says:

    That was a big bowl of chili. Gaslamp games, you say? I think I understand now.

    On the whole “making sure not to have the last construction event trap NPC’s in buildings” Why not have a state change event separate from the last build event, and require the state change event take place at a location outside the construction. Call it “Inspection” or whatever.

    { reply }
  12. Kai says:

    I’m super excited for this game! Keep up the great work, I can’t wait to play it! 😀

    { reply }

Leave a Reply

Your email address will not be published. Required fields are marked *