Software Carpentry

Helping scientists make better software since 1997

Das Kapital, Computational Thinking, and Productivity

Somewhere in The Age of Uncertainty, Galbraith wrote that what made Das Kapital and the Bible great books was that they were so large, and so full of contradictions, that everyone could find support in them for anything they wanted. I have felt the same way about the phrase “computational thinking” ever since I attended a workshop at Microsoft Research in September 2007. In one of the breakout sessions, six of us tried to operationalize our understanding of the term by coming up with questions for a quiz that could be given to someone to determine if he or she was thinking computationally. It quickly became clear that we meant very different things when we used those two words. It was also clear (to me at least) that this ambiguity was socially very useful, since (to switch metaphors) it allowed people to attend the same church while disagreeing on the nature of salvation. It’s not a polite fiction per se, but rather a—um, damn, I don’t know the word—a thing that no one looks at closely because doing so would cause discomfort or friction.

Eventually, though, things do have to be looked at closely.  In this case, it’s the productivity of scientific programmers.  Based on feedback from people who’ve taken it, I believe that Software Carpentry significantly increases how much scientists can do with computers, but I don’t have anything that would pass muster as “proof”.  I’m actually not even sure what form such proof would take, since I don’t know how to measure the productivity of  programmers of any other kind either—not in any reasonable amount of time, either.  (Waiting to see if alumni produce more papers would take at least a couple of years, maybe more.)  If someone could figure out how to measure computational thinking ability, on the other hand, before-and-after testing might be good enough.  Any thoughts?


Written by Greg Wilson

2009/02/23 at 20:57

Posted in Opinion

4 Responses

Subscribe to comments with RSS.

  1. Sounds like you won’t find this “proof” unless you have decent, well grounded constructs of what “productivity of scientific programmers” means, and the best way to develop these constructs is through case studies.

    On the other hand, I recognize that more and more I see case studies as the sensible choice for our field, and that your tolerance for them is lower than mine.


    2009/02/23 at 21:57

  2. You can do an education-style pre-test / post-test experiment. Give the subjects some kind of test before they take the course, and then give them the course, and give them an equivalent test, and check their improvement. Ideally, you’d have a control group that doesn’t take the course, but does do the pre-test and post-test. You then compare the improvement in the treatment group to the control group.

    The operationalizing bit where you define the test is tricky, it’s the ever-present construct validity problem that we always do battle with in empirical software engineering research. I am a fan of the Tom Gilb quote: “Anything you need to quantify can be measured in some way that is superior to not measuring it at all.”

    For productivity, I like to measure task completion time (e.g., time to solve a programming problem) along with some measure of quality (e.g., tests passed, expert rating of code quality with blinding). If you are concerned about, say, the subject writing really unmaintainable code, you can structure the task so that they have to make changes they weren’t informed about in advance. If you think of computational thinking as being more conceptual, you can have them solve a problem on paper.

    Measuring the productivity/quality of any knowledge worker (e.g., scientist, programmer, teacher, manager, doctor) is a very difficult task. And yet, somehow, I’d be willing to wager that we can all name scientists, programmers, teachers, managers, or doctors we’ve encountered who we recognize as being very productive of very high quality. You’ll never measure it perfectly in a study, but I think you always learn something by trying.

    Lorin Hochstein

    2009/02/23 at 22:53

  3. operationalize?

    Titus Brown

    2009/02/24 at 20:59

  4. […] and argumentation Jump to Comments Just a reflection spawned from a recent post by Greg Wilson and a later discussion at the lab (I had to leave early, so I don’t know how it […]

Comments are closed.

%d bloggers like this: