I am actually sick with lung flu, which means I have some time to write angry rants, inspired by things on my Twitter feed, and then post them to the company blog.
(Actually, this is just shameless bait for sites like Y Combinator, who love to hear Angry Young People railing about the world at large. I… I should give up now.)
That said, we have been falling short on technical commentary here, and I did get linked to two Twitter posts this morning that are worth discussing in some detail. So let’s have at ’em.
The first item is from id Software’s John Carmack, who does things like writing an entire photon mapper in a day and then tells people that he did it – and, it’s not a big deal, you know? His contribution to the discussion:
“Floating point trick: If ( a != a ) a is a NaN”
I took a few minutes to puzzle out how this could possibly work. It turns out that in C++ – and in fact, according to IEEE floating point standards – NaNs (or not-a-numbers) will cause ANY expression to return true if they are used in an inequality comparison. Clever!
The second item that caught my attention was an advertisement for a course with a “Certified ScrumMaster for Agile Game Development”, to be held two days before GDC. This course promises that we will, with the ScrumMaster’s help and guidance, learn such things as:
“The essentials of getting a project off on the right foot”,
“How to successfully scale Scrum methods to hundreds of participants”,
“How to help both new, and experienced teams, be more successful,”
and so on and so forth. In just two days, you too can sip at the mystical elixir of Scrum, which is guaranteed to make your game ship on time, your Metacritic scores improve, and as an added bonus it’ll make all your hair grow back and your girlfriend will stop complaining about all the overtime you put in at the office. The cost of this affair? $1500 for a two day seminar, although you get $250 off if you register early. As a bonus, after you take this course (and fill in some kind of online quiz), you too can call yourself a Certified Scrum Master!
Needless to say, this little advertisement sent me into hysterics; quite frankly, this is nothing but snake oil and cargo cult science of the worst kind. Today, you get to enjoy a slightly discombobulated rant about it, fuelled by the unholy mixture of caffeine and cold medicine. I have a feeling this is going to bite me in the posterior, so I may as well get on with it.
First, some background: for those of you not in the know, the Scrum process is one of the subprocesses and methodologies espoused by the “Agile Software Development” movement slash umbrella. The Agile movement, itself, evolved from the “eXtreme Programming” movement, and many of the ideas of eXtreme Programming (XP) have been incorporated into Agile. (That said, as a general rule: never
trust anything that chooses to capitalize the “X” in extreme, while ignoring the “e”.)
The Agile manifesto states, in a few pithy lines:
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
There are also twelve principles attached to the manifesto, most of which we won’t get into here. There is one principle that I think stands out:
“The best architectures, requirements, and designs emerge from self-organizing teams.”
I am going to come back to this idea, because it’s important. So if this is Agile, then what is Scrum?
Scrum, itself, is sort of… well, it is sold, at least, as one way of implementing the Agile manifesto, even though the actual Scrum methodology predates the Agile manifesto by some time. The Scrum people would also like you to believe that Scrum is *the only* way to implement Agile… which is interesting, because Agile is a manifesto and a set of core values. Participants in the development process are divided into categories: there are “pigs”, or people who are actively committed to the project, and “chickens” who are people who are invested in the result, but whose bacon is not on the line. Through a 24-hour style daily feedback process, Scrum teams work towards meeting goals based around 30-day sprints, with the end goal of each 30-day sprint being the production of, quote, “working code” – I’ve put quotes around this because I think that the actual definition of working code varies from team to team, or from situation to situation.
Before we dig into what is wrong with Scrum, and more specifically what is wrong with a two-day ScrumMaster course, let’s start by revealing a dirty secret about software programming.
The reason why we have software methodologies, “best practices” and rituals like Scrum and XP is simple: if your company hired consistently better programmers, you wouldn’t actually need a methodology beyond a shared set of coding conventions. Everything that you could want would emerge from the team itself, because at the top levels of software development great programmers have a communal, shared, and internalized understanding of what to do to write the best code possible, and to produce the best code possible. Joel Spolsky writes brilliantly and passionately about this; in fact, one can argue that the entire reason behind his weblog is his desire to get the best programmers possible for his startup. Spolsky notes – and I am paraphrasing here – that the great programmers, the very best programmers, are never available for hire on the job market. Even if you wanted to hire John Carmack to work on your game, you couldn’t. The best thing you could do, at this point, would be to try to buy his entire company and then hope for the best. (Zenimax Media did this.)
Sometimes, but not always, the smartest and brightest programmers join the ranks of academia or pure research. Other times, they wind up starting their own companies. More likely, they end up getting embedded in a company like Google. Google understands this principle, and internalizes it in their company culture. They hire the best – the very, VERY best – of the programming breed, and do their utmost to keep their developers fat and sassy. In the game development community, Valve follows the same model. (So, actually, does RAD Game Tools – you know, the guys who made that Bink video codec that every single game uses, ever.)
Unfortunately, your company isn’t Google, or Valve, or even RAD Game Tools. So what do you do? Well, you hire developers that are less than fantastic. It happens. You may, if you’re lucky, get a brilliant developer who has slipped through the cracks, but most likely your developer talent is going to be drawn from some sort of normal distribution of programming talent. As such, you’re going to get some better than average developers, and some worse than average developers, but mostly you will get average developers. Here, then, is where software development methodologies enter the picture: they exist to ensure that the worse than average developers, and the average developers, vaguely sort of do some of what the brilliant developers would do instinctively, without questioning it in the first place. This is the entire purpose of test driven development: an automated system to deal with the fact that average programmers can, and will, cause regressions. The best software methodologies – and by best, I mean most popular, seek to encode and enshrine in a methodology what good, smart programmers already do – and, if you asked them why they did the things that they do, they would probably roll your eyes at you and say, “Well, it’s obvious, really.”
Agile takes this one step further: Agile seeks to reproduce or imitate the flexibility and freedom that small companies, with universally brilliant programmers, have. Unfortunately, you lose that flexibility as your team size expands. Right now, I do all the programming for Gaslamp Games; this works, although we have accumulated a little technical debt in a few places and some of that debt would probably be reduced if I had a peer offering some feedback. To compensate, I try to learn from other people who I respect, and the next project will offer more opportunities to try and do things… just a little bit better. I would like to see the company grow to the point where we have more than one programmer, if only so I don’t go insane; when this happens, there will have to be a partition of our products and codebases into sections that are “owned” by certain people, and everybody will probably have to obey a set of coding conventions that I concoct (one of the pleasures of being a lead programmer) so that the code looks, and feels, as close to to the work of one person as it can be. This decision – to bring on additional programming talent – comes with a cost; I will have to decide that this cost is worth eating, and I will have to ensure that we bring on only those programmers that make that cost worth it.
Let me state, for the record, that a lot of the ideas behind Agile are really, really good ideas. I wish all game development teams were empowered with the ability to control their end product, free from publisher interaction. The fiscal reality of this, of course, is that publishers working with third-party developers are very, very good at sticking their nose in other people’s business. I think that game designs do change and evolve, and that the development process must incorporate that. (Look at Dredmor!) Yes, there’s a lot of good stuff in that manifesto.
So now that we’ve talked about Agile, let’s talk about Scrum. First off, Agile isn’t Scrum. This is something that Scrum people would like you to believe, but it’s not true. Gaslamp has some internal processes that are, themselves, Agile in nature; however, they are not processes that have been handed down to us by some consultant. Rather, we do what we currently do because we’ve spent two years figuring out how the heck to make the damn company work, and we have paid for this in sweat, blood, tears, and missed internal goals. If you are willing to embrace what Agile offers, and to let a methodology evolve AT your company or WITH your team, then you have a reasonable chance of achieving success and making life better.
If, on the other hand, you drop ANY methodology on a project, well… let’s just say that there is no such thing as a silver bullet.
Blindly obeying a set of Scrum principles isn’t going to make your life any better. For one thing, I strongly feel that you want a process in place that evolves with your company, hopefully developed by brilliant people to make their lives better. I like to talk about IMVU’s Agile implementation a lot for several reasons. First, I have a lot of friends there. Second, I interviewed there once, and it was an interesting (albeit sleep-deprived) experience. Third, IMVU’s processes – continuous deployment, test-driven deployment, the whole “rapid startup” mentality, et cetera. – have all evolved naturally as part of the company’s ongoing development, with a certain amount of trial and error involved. As the company advances, the processes change and mutate. As a result, they have an Agile process, and they picked up the parts of methodologies – including Scrum – that worked for them. It happens over time. You work with it. It’s a natural process. We did it at Gaslamp Games, despite the fact that we’re not really an Agile company per se; everything we’re doing on Dredmor now is the result of David, Daniel, Derek and myself spending two years trying various things, figuring out what works and what doesn’t, and hitting our heads against everything and the kitchen sink. (Dredmor itself extends back six years, including some time when it was the product of another company; Gaslamp Games itself was working on a game prior to Dredmor, but we canned that, as well as two previous employees, one of whom basically went nuts; et cetera, et cetera. I’m saving all of the juicy details for the inevitable post-mortem article.)
Real agile programming, with a lowercase a, involves spending a lot of time in the sink with your hands dirty, wondering why the drain isn’t unclogging.
With that in mind, in the other corner we have… well, folks like the gentleman behind this Certified Scrum Master business. The CSM folks would like to sell you a process – and what a process it is! Without having to spend two years, or more, in the gutters figuring out how to make things work, you too could go off and learn how to be a Certified Scrum Manager, and all your problems are solved. The simple truth: CSM courses are big business for CSM people, because if you have fifteen attendees at fifteen hundred smackers for a two day course, you’ve just pulled down $22,500 for two days worth of work. The game development industry, which has massive problems of its own right now, is desperate for a solution to the problem of making games ship on time, with the increasingly large amounts of people and money involved. The CSM folks can smell a sucker from a mile away, and we’re off to the races. This is happening, ironically, just as the regular software development community is starting to twig to the fact that they’re being played for fools.
Incidentally, the same fifteen hundred dollars will buy you the all-access, no-holds barred pass to the Game Developer’s Conference itself, where you can enjoy excellent lectures by some of the best talent that the game development industry and the graphics card manufacturers have to offer: Dan Baker, Christina Coffin, Squirrel Eiserloh (who shipped Anachronox through sheer force of will), the always delightful Richard Huddy, Marco Salvi, etc, etc. (And this is a quiet year.)
Anyhow, there you have it. My advice, if you want to build a good game in this day and age, and to ship well and on time? Hire the best damn people you can get ahold of, through any means necessary, but don’t hire too many of them. Give them the freedom to create. Light the fuse and stand back. This has been the formula for success throughout the video game industry since its inception, from ID Software to Media Molecule… and, hopefully, Gaslamp Games.
Also? When John Carmack says something about floating point, it’s usually worth listening to. Now there’s a man who has earned the right to have the word “Master” attached to his name… and I bet the Scrum Masters don’t know about that cute little floating point trick.
1. Yes, I know that “Master” in Scrum Master refers to “master of ceremonies”; it is deliberately deceptive. At the very least, it is a master course in marketing.
2. Yes, I read the Y Combinator news page. Sometimes, it has some interesting things.
3. The ultimate demonstration in the difference between a great programmer and a merely average programmer is probably the famous Sudoku debacle. In short, Ron Jeffries (Agile/Scrum/TDD “Guru” and Founding Light) spent an inordinate amount of time trying to write a program to solve Sudoku puzzles using test-driven development, and ended up abandoning the project when he just couldn’t do it. By contrast, Peter Norvig, director of R&D at Google, decided to write a Sudoku problem solver and ended up doing it in one, rather nice, little blog post using a method called constraint propagation. His approach occupies about one page of code, plus a few embellishments. Jeffries’ approach is about five blog posts full of waffle that doesn’t actually DO anything. Read more about that here.
4. The history of Media Molecule: here.
5. Next week, we will get into Gaslamp Games’s practices, and figure out which ones are Agile and which ones are not. Actually, I might just write about constraint propagation.
6. IMVU company blog: here.
7. Alternate, but sadly rejected post title: “John Carmack versus the Evil ScrumMasters of Foulhaxx 9”