Game Development Snake Oil

I am actually sick with lung flu, which means I have some time to write angry rants, inspired by things on my Twitter feed, and then post them to the company blog.
(Actually, this is just shameless bait for sites like Y Combinator, who love to hear Angry Young People railing about the world at large. I… I should give up now.)

That said, we have been falling short on technical commentary here, and I did get linked to two Twitter posts this morning that are worth discussing in some detail. So let’s have at ’em.

The first item is from id Software’s John Carmack, who does things like writing an entire photon mapper in a day and then tells people that he did it – and, it’s not a big deal, you know? His contribution to the discussion:

“Floating point trick: If ( a != a ) a is a NaN”

I took a few minutes to puzzle out how this could possibly work. It turns out that in C++ – and in fact, according to IEEE floating point standards – NaNs (or not-a-numbers) will cause ANY expression to return true if they are used in an inequality comparison. Clever!

The second item that caught my attention was an advertisement for a course with a “Certified ScrumMaster for Agile Game Development”, to be held two days before GDC. This course promises that we will, with the ScrumMaster’s help and guidance, learn such things as:

“The essentials of getting a project off on the right foot”,
“How to successfully scale Scrum methods to hundreds of participants”,
“How to help both new, and experienced teams, be more successful,”

and so on and so forth. In just two days, you too can sip at the mystical elixir of Scrum, which is guaranteed to make your game ship on time, your Metacritic scores improve, and as an added bonus it’ll make all your hair grow back and your girlfriend will stop complaining about all the overtime you put in at the office. The cost of this affair? $1500 for a two day seminar, although you get $250 off if you register early. As a bonus, after you take this course (and fill in some kind of online quiz), you too can call yourself a Certified Scrum Master!

*sigh*

Needless to say, this little advertisement sent me into hysterics; quite frankly, this is nothing but snake oil and cargo cult science of the worst kind. Today, you get to enjoy a slightly discombobulated rant about it, fuelled by the unholy mixture of caffeine and cold medicine. I have a feeling this is going to bite me in the posterior, so I may as well get on with it.

First, some background: for those of you not in the know, the Scrum process is one of the subprocesses and methodologies espoused by the “Agile Software Development” movement slash umbrella. The Agile movement, itself, evolved from the “eXtreme Programming” movement, and many of the ideas of eXtreme Programming (XP) have been incorporated into Agile. (That said, as a general rule: never
trust anything that chooses to capitalize the “X” in extreme, while ignoring the “e”.)

The Agile manifesto states, in a few pithy lines:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

There are also twelve principles attached to the manifesto, most of which we won’t get into here. There is one principle that I think stands out:

“The best architectures, requirements, and designs emerge from self-organizing teams.”

I am going to come back to this idea, because it’s important. So if this is Agile, then what is Scrum?

Scrum, itself, is sort of… well, it is sold, at least, as one way of implementing the Agile manifesto, even though the actual Scrum methodology predates the Agile manifesto by some time. The Scrum people would also like you to believe that Scrum is *the only* way to implement Agile… which is interesting, because Agile is a manifesto and a set of core values. Participants in the development process are divided into categories: there are “pigs”, or people who are actively committed to the project, and “chickens” who are people who are invested in the result, but whose bacon is not on the line. Through a 24-hour style daily feedback process, Scrum teams work towards meeting goals based around 30-day sprints, with the end goal of each 30-day sprint being the production of, quote, “working code” – I’ve put quotes around this because I think that the actual definition of working code varies from team to team, or from situation to situation.

Before we dig into what is wrong with Scrum, and more specifically what is wrong with a two-day ScrumMaster course, let’s start by revealing a dirty secret about software programming.

The reason why we have software methodologies, “best practices” and rituals like Scrum and XP is simple: if your company hired consistently better programmers, you wouldn’t actually need a methodology beyond a shared set of coding conventions. Everything that you could want would emerge from the team itself, because at the top levels of software development great programmers have a communal, shared, and internalized understanding of what to do to write the best code possible, and to produce the best code possible. Joel Spolsky writes brilliantly and passionately about this; in fact, one can argue that the entire reason behind his weblog is his desire to get the best programmers possible for his startup. Spolsky notes – and I am paraphrasing here – that the great programmers, the very best programmers, are never available for hire on the job market. Even if you wanted to hire John Carmack to work on your game, you couldn’t. The best thing you could do, at this point, would be to try to buy his entire company and then hope for the best. (Zenimax Media did this.)

Sometimes, but not always, the smartest and brightest programmers join the ranks of academia or pure research. Other times, they wind up starting their own companies. More likely, they end up getting embedded in a company like Google. Google understands this principle, and internalizes it in their company culture. They hire the best – the very, VERY best – of the programming breed, and do their utmost to keep their developers fat and sassy. In the game development community, Valve follows the same model. (So, actually, does RAD Game Tools – you know, the guys who made that Bink video codec that every single game uses, ever.)

Unfortunately, your company isn’t Google, or Valve, or even RAD Game Tools. So what do you do? Well, you hire developers that are less than fantastic. It happens. You may, if you’re lucky, get a brilliant developer who has slipped through the cracks, but most likely your developer talent is going to be drawn from some sort of normal distribution of programming talent. As such, you’re going to get some better than average developers, and some worse than average developers, but mostly you will get average developers. Here, then, is where software development methodologies enter the picture: they exist to ensure that the worse than average developers, and the average developers, vaguely sort of do some of what the brilliant developers would do instinctively, without questioning it in the first place. This is the entire purpose of test driven development: an automated system to deal with the fact that average programmers can, and will, cause regressions. The best software methodologies – and by best, I mean most popular, seek to encode and enshrine in a methodology what good, smart programmers already do – and, if you asked them why they did the things that they do, they would probably roll your eyes at you and say, “Well, it’s obvious, really.”

Agile takes this one step further: Agile seeks to reproduce or imitate the flexibility and freedom that small companies, with universally brilliant programmers, have. Unfortunately, you lose that flexibility as your team size expands. Right now, I do all the programming for Gaslamp Games; this works, although we have accumulated a little technical debt in a few places and some of that debt would probably be reduced if I had a peer offering some feedback. To compensate, I try to learn from other people who I respect, and the next project will offer more opportunities to try and do things… just a little bit better. I would like to see the company grow to the point where we have more than one programmer, if only so I don’t go insane; when this happens, there will have to be a partition of our products and codebases into sections that are “owned” by certain people, and everybody will probably have to obey a set of coding conventions that I concoct (one of the pleasures of being a lead programmer) so that the code looks, and feels, as close to to the work of one person as it can be. This decision – to bring on additional programming talent – comes with a cost; I will have to decide that this cost is worth eating, and I will have to ensure that we bring on only those programmers that make that cost worth it.

Let me state, for the record, that a lot of the ideas behind Agile are really, really good ideas. I wish all game development teams were empowered with the ability to control their end product, free from publisher interaction. The fiscal reality of this, of course, is that publishers working with third-party developers are very, very good at sticking their nose in other people’s business. I think that game designs do change and evolve, and that the development process must incorporate that. (Look at Dredmor!) Yes, there’s a lot of good stuff in that manifesto.

So now that we’ve talked about Agile, let’s talk about Scrum. First off, Agile isn’t Scrum. This is something that Scrum people would like you to believe, but it’s not true. Gaslamp has some internal processes that are, themselves, Agile in nature; however, they are not processes that have been handed down to us by some consultant. Rather, we do what we currently do because we’ve spent two years figuring out how the heck to make the damn company work, and we have paid for this in sweat, blood, tears, and missed internal goals. If you are willing to embrace what Agile offers, and to let a methodology evolve AT your company or WITH your team, then you have a reasonable chance of achieving success and making life better.

If, on the other hand, you drop ANY methodology on a project, well… let’s just say that there is no such thing as a silver bullet.

Blindly obeying a set of Scrum principles isn’t going to make your life any better. For one thing, I strongly feel that you want a process in place that evolves with your company, hopefully developed by brilliant people to make their lives better. I like to talk about IMVU’s Agile implementation a lot for several reasons. First, I have a lot of friends there. Second, I interviewed there once, and it was an interesting (albeit sleep-deprived) experience. Third, IMVU’s processes – continuous deployment, test-driven deployment, the whole “rapid startup” mentality, et cetera. – have all evolved naturally as part of the company’s ongoing development, with a certain amount of trial and error involved. As the company advances, the processes change and mutate. As a result, they have an Agile process, and they picked up the parts of methodologies – including Scrum – that worked for them. It happens over time. You work with it. It’s a natural process. We did it at Gaslamp Games, despite the fact that we’re not really an Agile company per se; everything we’re doing on Dredmor now is the result of David, Daniel, Derek and myself spending two years trying various things, figuring out what works and what doesn’t, and hitting our heads against everything and the kitchen sink. (Dredmor itself extends back six years, including some time when it was the product of another company; Gaslamp Games itself was working on a game prior to Dredmor, but we canned that, as well as two previous employees, one of whom basically went nuts; et cetera, et cetera. I’m saving all of the juicy details for the inevitable post-mortem article.)

Real agile programming, with a lowercase a, involves spending a lot of time in the sink with your hands dirty, wondering why the drain isn’t unclogging.

With that in mind, in the other corner we have… well, folks like the gentleman behind this Certified Scrum Master business. The CSM folks would like to sell you a process – and what a process it is! Without having to spend two years, or more, in the gutters figuring out how to make things work, you too could go off and learn how to be a Certified Scrum Manager, and all your problems are solved. The simple truth: CSM courses are big business for CSM people, because if you have fifteen attendees at fifteen hundred smackers for a two day course, you’ve just pulled down $22,500 for two days worth of work. The game development industry, which has massive problems of its own right now, is desperate for a solution to the problem of making games ship on time, with the increasingly large amounts of people and money involved. The CSM folks can smell a sucker from a mile away, and we’re off to the races. This is happening, ironically, just as the regular software development community is starting to twig to the fact that they’re being played for fools.

Incidentally, the same fifteen hundred dollars will buy you the all-access, no-holds barred pass to the Game Developer’s Conference itself, where you can enjoy excellent lectures by some of the best talent that the game development industry and the graphics card manufacturers have to offer: Dan Baker, Christina Coffin, Squirrel Eiserloh (who shipped Anachronox through sheer force of will), the always delightful Richard Huddy, Marco Salvi, etc, etc. (And this is a quiet year.)

Anyhow, there you have it. My advice, if you want to build a good game in this day and age, and to ship well and on time? Hire the best damn people you can get ahold of, through any means necessary, but don’t hire too many of them. Give them the freedom to create. Light the fuse and stand back. This has been the formula for success throughout the video game industry since its inception, from ID Software to Media Molecule… and, hopefully, Gaslamp Games.

Also? When John Carmack says something about floating point, it’s usually worth listening to. Now there’s a man who has earned the right to have the word “Master” attached to his name… and I bet the Scrum Masters don’t know about that cute little floating point trick.

Footnotes:

1. Yes, I know that “Master” in Scrum Master refers to “master of ceremonies”; it is deliberately deceptive. At the very least, it is a master course in marketing.
2. Yes, I read the Y Combinator news page. Sometimes, it has some interesting things.
3. The ultimate demonstration in the difference between a great programmer and a merely average programmer is probably the famous Sudoku debacle. In short, Ron Jeffries (Agile/Scrum/TDD “Guru” and Founding Light) spent an inordinate amount of time trying to write a program to solve Sudoku puzzles using test-driven development, and ended up abandoning the project when he just couldn’t do it. By contrast, Peter Norvig, director of R&D at Google, decided to write a Sudoku problem solver and ended up doing it in one, rather nice, little blog post using a method called constraint propagation. His approach occupies about one page of code, plus a few embellishments. Jeffries’ approach is about five blog posts full of waffle that doesn’t actually DO anything. Read more about that here.
4. The history of Media Molecule: here.
5. Next week, we will get into Gaslamp Games’s practices, and figure out which ones are Agile and which ones are not. Actually, I might just write about constraint propagation.
6. IMVU company blog: here.
7. Alternate, but sadly rejected post title: “John Carmack versus the Evil ScrumMasters of Foulhaxx 9”

Posted in Games, Programming | Tagged , , , , , ,
8 Comments

8 Responses to “Game Development Snake Oil”

  1. Jeff Craig says:

    Great caffeine-and-codeine fueled rant.

    I remember first hearing about SCRUM about six or seven years ago while still in college, and feeling that it sounded like a scam. I’ve had a low opinion of certifications for some time, not that the impetus to learn more is negative, but the idea that passing a test (which is often WRONG) and getting a piece of paper makes you intrinsically more valuable is ridiculous, but then I test well, so I don’t put much value in them.

    Where I’m working now, I’ve implemented continuous integration, code quality tools, and some level of Unit Testing, though I tend not to test as much as I could, and possibly should. However, the way that most agile consultants and SCRUM masters seem to be just trying to pile guilt on average and below-average developers to do the ‘right thing’. Best practice is just very difficult to teach from a pure consulting role.

    The benefit to struggling through to create your own set of practices is that it gives you battle-tested tools to improve your operation next time around.

    { reply }
  2. Brian says:

    Rather informative, even if a tad over my head given that I’ve not yet gained any chops as far as programming goes.

    { reply }
  3. J Brian says:

    Everybody (including me) hates process. But it’s not very interesting to talk about the successful processes of small teams who generate their own requirements, because that’s a trivial problem to solve and these teams self-organize. Hire “only the best” and you can maybe create slightly larger teams that self-organize.
    As you mentioned, on big teams who have requirements at least partially handed down from on high (like a big budget game at the mercy of a publisher), just hiring great people doesn’t solve the problem.
    The core nugget of wisdom in scrum is to try and break work into pieces that are small enough in scope that they can be accomplished by one of these nice small self-organizing teams. Sadly in practice that is much easier to say than to do. Not all functionality breaks down into those small pieces, and even if it does figuring out what the bite sized pieces are and assigning the right people to them is itself a huge amount of management effort that scrum claims to make unnecessary.
    So indeed, there is no silver bullet for making large projects go. Including “Hire the best damn people you can get ahold of, through any means necessary, but don’t hire too many of them.”

    { reply }
    • What you’re actually talking about, though, is a Mythical Man-Month problem. (I hold that Brooks’s work is one of the best tomes on software engineering ever written, mainly because it is the product of a significant number of battle scars.) Brooks’ main argument against large programming teams is that the communication flowlines for a large team are equivalent to the number of edges of the complete graph K_n – that is, if you have n developers, there are (n)(n-1)/2 ways for them to interact. If you have 15 developers, you end up with 105 communication paths. If you have 50 developers, you get 1225 communication paths. That’s not the only reason why the Man-Month is mythical, but it is his primary argument. So let’s take it as granted.

      Does Scrum address this? Not really, no – not unless you’ve taken your developers and already partitioned them into small groups. In fact, I believe one of the strains of Scrum that maintains that it’s the One True Way of Scrum has the condition that everybody can take any task that they want from the board, and everybody who is designated a pig shows up for the daily Scrum. So Scrum doesn’t address this at all. In fact, a lot of the Scrum devotees claim that Scrum works best precisely when you have a small team – 6 people or less – a state which certainly doesn’t describe AAA game development.

      (There is an additional factor here, which is that I believe that small teams make better – specifically more innovative – games. That is neither here, nor there. It is also a reflection of my personal biases.)

      What I am specifically railing against, though, isn’t Scrum. If Scrum, or Kanban, or LEAN, or Agile, or XP, or the Kent Beck Dead-Chicken-Innards Oracular Scheduling System works for you, great, do it. I have two key objections and one key thesis point – and these will be a little more clear now that I’m not on cold medication:

      1. The Scrum organizations and individuals, specifically the CSM program, promote the top-down “let me drop this process on you from a great height” method, rather than letting processes evolve organically. I feel that this is a capital mistake. What is right for you is not right for another company. (Scrum inflexibility amongst its disciples is also a part of this objection; saying “you must do Scrum our way” seems to me to be rather antithetical to being agile, which is theoretically why you chose to adopt Scrum in the first place.) Additionally, evolution of processes (and organisms) is how we naturally weed out the bad ones and move on with the good ones.

      More importantly:

      2. CSM training programs, such as the one that offended me, offer you a two-day course in “Scrum Mastery” that then certifies you as a “Scrum Master.” Mastery, in my mind, is an earned title, and two days is not sufficient time to achieve this in anything, let alone implementing and maintaining an entire software development process. The fact that the Scrum organizations claim that “Master” in this case refers to “Master of Ceremonies” is a pathetic cop-out at best. CSM course are very clearly sold under the false pretenses that, by completing them, you will have Mastered the Scrum Process. (What you actually get is somebody who doesn’t know what’s going on running into the studio waving a stack of multi-colored pens, and next thing you know all hell has broken loose.)

      2b. Furthermore, a lot of these people selling themselves as Scrum Masters or teaching Scrum Mastery or whatever the heck it is they’re offering fail to convince me that they are, in fact, Masters of anything. The post in Footnote 1 is a good example of this. Read Jeffries’ five posts where he attempts to build a TDD-based Sudoku solver before losing interest. Read Norvig’s post where he builds a Sudoku solver. (He throws a few tests in there too, but mainly out of boredom.)

      2.b.i.: Okay, this is going to be a bit of an aside, and will also contain some character assassination. I find it very interesting that the same people pushing Agile and Scrum, and who are busy serving as consultants on Agile, Scrum, and so on and so forth, are also invariably the same people who were pushing Extreme Programming back in the day. Let us not forget the following fact that has been conveniently swept under the rug: the big shining star of Extreme Programming was the Chrysler payroll system. The work on C3, and the overhaul that was started by Kent Beck (who later brought on Martin Fowler and Ron Jeffries) was cancelled, after four years, when the system ended up only being able to service 10% of the company. Chrysler determined, in fact, that the entire C3 project was a total failure, and anybody who wishes to be taken seriously by management at Chrysler dare not utter the words “Extreme Programming” for fear of being laughed at.

      C3 got canned in 2000. The original Agile Manifesto was published by Beck, Fowler, Jeffries and a few others in 2001. I think that a lot of good has come out of that manifesto, and I respect some of the original seventeen signatories, but I wonder if some of the people involved have started shopping Agile and Scrum “coaching” simply because they couldn’t sell Extreme Programming coaching any more. There are people who are involved with Agile that I have a great deal of respect for – I have a fair amount of props for Eric Reis at IMVU, who is involved with this Lean Startup thing that all the cool kids do (and which will be the source of another blog post) – but there is a lot of pushing of Scrum, and coaching of Scrum, and selling of courses and DVDs and magical whiteboards by people who… well, haven’t actually shown me that they can ship incredible, or even good, software!

      (I did, however, like Martin Fowler’s book on refactoring.)

      My general thesis: there is a clear difference between the upper echelon of programming talent, and the average programmer; software methodologies exist to minimize the amount of damage that poor programmers can do and to attempt to point them in the right direction, and do not themselves make the software product better.

      Does this method – hire only the damned best – scale to large projects? I think it does. Again, look at Valve. They have Yahn Bernier, who wrote BSP. They have Jason Mitchell, who used to run ATI’s Demo Squad, and Alex Vlachos, who also used to run ATI’s Demo Squad. These are the guys who had to figure out what the heck you’re going to do with pixel shaders back when nobody had any clue. They have a lot – a LOT – of Looking Glass Studios alumni (the people who made Thief: The Dark Project and System Shock.) They have Gary McTaggart, who did a lot of the original demos for the 3DFX Voodoo Graphics accelerator. Hell, they even have Zoid there these days.

      And guess what? Valve dominates the entire PC electronic game distribution market, and has a heck of a lot of critically acclaimed titles under its collective belt. Portal 2 is going to sell a heck of a lot of copies and will probably go on my list of “games I wish I’d made.” In the mean time, they also publish a lot of frequently-cited papers.

      Google is another prime example of it. Nobody can argue that Google isn’t a large company that has scaled well and has a stupidly large market cap. I think the successful bits of Apple (i.e. NOT whoever is responsible for the OpenGL stack) have operated like this.

      … hmm, that was a bit longer than intended. Sorry about that!

      { reply }
      • J Brian says:

        I don’t disagree with any of your points about process consulting fads current or past. I’m not convinced, though, that elite talent makes software projects work at large scale. At 10 devs? I could buy that although I’ve never seen a collection of 10 truly elite devs all working on the same project. At 20 devs? 50? No, I simply can’t imagine it just working without any process (and you likely don’t have time to organically grow a process at that scale).
        Valve is an interesting example and I must admit I can’t say that you’re wrong there. I don’t have any contacts there so I don’t really know how it works, but I suspect they have a relatively small number of people devoted to any 1 project, and bet they have a really clear individual leader for each one.
        My understanding from some folks at Google is that they are really starting to feel their weight. They are huge at this point, and there is a lot of software development going nowhere in those halls, as you would find at any large software company I expect.

        So I guess my take-away is just “don’t do huge projects if you can at all avoid it.” You will probably get a higher quality result if you scale down your requirements to a level that can be accomplished by a manageable sized team.
        Of course, that’s probably why you do what you do now, rather than whatever it is you did before. 🙂

        { reply }
  4. J Brian says:

    Hmm, on second reading that came off more snotty than intended. I don’t think I significantly disagree with the thrust of your rant, heh.

    { reply }
  5. There is some information on Valve’s development processes on their website. They employ an in-house approach known as the \Cabal\ method – or at least they did for both Half-Life games – and there’s some information on their webpage here:

    http://developer.valvesoftware.com/wiki/Valve_in_the_press

    { reply }
  6. Devlin Vining says:

    I’m going to go ahead and save everyone 1500 bucks and do a full scrum training course from this blog comment.

    1: You work in small teams focused specific areas of the product
    2: Every morning your team has a meeting where you tell people what you did yesterday.

    { reply }

Leave a Reply to J Brian { cancel }

Your email address will not be published. Required fields are marked *