Technical Debt Payment Plans

Technical Debt is a handy metaphor that has been used in the programming community for awhile now. I sort of think it’s one of those things that probably got coined by somebody in the Lean Startup community, or the Agile community, or something like that. The idea of the metaphor is simple: you incur technical debt at the start of a project for a variety of reasons – getting your software started or out the door, for instance; or meeting a release target. At some point, however, the debt collector comes calling. This week, I seem to be paying my debts off: in the case of Clockwork Empires, we incurred a lot of technical debt getting the game into Early Access, with the knowledge that at some point during the process we would have to pay it off in order to get out of Early Access. Well, the point of paying it off seems to be this week.


Let’s examine some of our debts:

  • The speed of loading save games. Save games save very quickly, but they load very slowly. In a previous version of the game, I switched how what the save system calls “nodes” were stored, from an STL container (bloated!) to a simple list in order to save memory and ensure that the game could load without crashing due to running out of memory. We fixed that, but the cost of the linked list approach is, of course, that load times are slower. A new game, saved, and then loaded takes two minutes on my machine. Pretty rough. What motivated me to fix this was a user commenting that typically when the game was loading a save game was when he’d go get stoned.

    Well, that’s one form of user feedback…For 49C my first thought was to implement a balanced binary tree to speed up the search for nodes with specific names (where the bottleneck lies.) I spent Friday doing that, to not much success; in fact, it made things slower. I threw that out, and tried something else: just getting rid of a bunch of the general case stuff that went through the save system and should, probably, have been saved as binary files. Much faster. Now saves load in 30 seconds or so on my machine. Still not fast enough, though. The next step is to eliminate the amount of time we spend allocating and deallocating memory in the loader – doing one trip to and from the allocator every time we load an individual object is slow, especially when we know at the start of the load time how many objects we saved. Now we allocate all of that at the start of the program. After much running about figuring out where the heck all the allocations and deallocations in the load/save code actually were, save game performance is now hopefully much better with none of the memory hiccups of previous versions.

    “The first living thing to go through the device was a small white rat. I still have him, in fact. As you can see, the damage was not so great as they say.”

  • Memory usage in general. For some reason, our new biome code blows the roof off of our memory footprint. In the process of auditing this, I made two unpleasant re-discoveries: every object in the world allocates its own tooltip, and those tooltips took up a lot of memory. I mean a lot. 300 megabytes worth. Well, that’s getting killed off; we now have one tooltip per type of interactive object, and can share them between them. (You figure out which tooltip you’re using for your string using a hash table. I don’t use hash tables enough and I should!)

    “What actually transpires beneath the veil of an event horizon? Decent people shouldn’t think too much about that.”

  • I also discovered that we were allocating 150 megabytes for a largely empty data structure that is mainly used to find the top of the world. Well, that can be sparsified, and that will save some memory as well.We also allocate memory for some data structures on rendered models that is never used anywhere (edge adjacency information.) Bye-bye.

    “No matter how beautiful the theory, one irritating fact can dismiss the entire formulism, so it has to be proven.”

  • Glow is broken. This is important for a thing David is doing. It got broken when the FBO format was changed around for the multipass rendering for the… uh, umpteenth time or so. Less than ideal. Lights cannot be attached to static props. This is important for standalone things that you might want to cast light, like lamps. This also needs to be fixed with some expediency.

    “Humans are natural-born scientists. When we’re born, we want to know why the stars shine. We want to know why the sun rises.”


And the list goes on and on. This is all stuff that has worked well enough up until Revision 49, and frankly could probably keep working for a bit longer; however, it’s stuff that needs to be fixed so we can get out of Early Access and actually ship the full version of the game. It’s not stuff that is necessarily *directly* obvious to the user, but it does exhibit itself as various symptoms. A lot of the time, this sort of technical work is invisible even to members of the same team as a systems programmer, but it does pile up and it does need to be done and done well.

“What we usually consider as impossible are simply engineering problems… there’s no law of physics preventing them.”

At the end of the day, nearing release, development becomes a question of “what things are still good enough to ship, even though they aren’t ideal programming practice?” That’s a fine and interesting line!

“The popular stereotype of the researcher is that of a skeptic and a pessimist. Nothing could be further from the truth! Scientists must be optimists at heart, in order to block out the incessant chorus of those who say ‘It cannot be done.'”

Posted in Clockwork Empires | Tagged ,

7 Responses to “Technical Debt Payment Plans”

  1. Alavaria says:

    I’m now wondering about the “large colony” slowdown, which some people (including me) seem to get around 50+ colonists.

    Now, the initial release of Happiness Workshifts really nuked performance (thankfully not anymore) but the colony runs appreciably slower at say 60 colonists than at 20.

    { reply }
    • The happiness parameter was intentionally overtuned for the current version of the game while we add some other things for players to optimize against. It’ll very likely have a less serious impact on progress when other in-development things get dropped in.

      { reply }
      • Alavaria says:

        Yes, but this is about the game running slow and lagging because…. not sure, is the game constantly checking every colonist or something?

        Before, it was because it was calculating the workshifts every 3 seconds (now 60 seconds), but whatever else it does is noticeable at 50+

        { reply }
        • I dug into this this morning. Basically, this is due to characters having a lot more friends and enemies to blame for their woes, hug, punch, yell at, etc. So when we go through the AI behaviour loop for “mandatory job evaluations”, we end up doing a *lot* of pathfinding checks. I’ll fix this, but I don’t know if it will make into Alpha 50.

          { reply }
  2. Oxi says:

    Thanks so much for working on this. The Devil is in the details. Keep up the hard work!

    { reply }
  3. Oxi says:

    This is a game I Hope to be playing over many, many years. Heck, I still play Settlers IV sometimes.

    So all the work you put into making it great in every way is very appreciated.

    { reply }
  4. Daniele says:

    What language are you using?

    { reply }

Leave a Reply

Your email address will not be published. Required fields are marked *