Software Carpentry

Helping scientists make better software since 1997

Quick Quiz to Measure What Scientists Know

Suppose you have a room full of scientists—hundreds of ’em—and want to find out how they actually use computers in their work. There isn’t time to interview them individually, or to record their desktops during a typical working week, so you’ve decided to ask them to self-assess their understanding of some key terms on a scale of:

  1. No idea what it is.
  2. Use it/have used it infrequently.
  3. Use it regularly.
  4. Couldn’t get through the day without it.

My list is below; what have I forgotten, and (more importantly) how would you criticize this assessment method?

  1. A command-line shell
  2. Shell scripts
  3. Version control system (e.g., CVS, Subversion)
  4. Bug tracker
  5. Build system (e.g., Make, Ant)
  6. Debugger (e.g., GDB)
  7. Integrated Development Environment (e.g., Eclipse, Visual Studio)
  8. Numerical Computing Environment (e.g., MATLAB, Mathematica)
  9. Inverse analyzer (e.g., Inane)
  10. Spreadsheet (e.g., Excel)
  11. Relational database (e.g., SQLite, MySQL, Oracle)
  12. Layout-based document formatting (e.g., LaTeX, HTML)
  13. WYSIWYG document formatting (e.g., Word, PowerPoint, OpenOffice)

Now, you have the same room full of scientists, and you want to find out how much they know about software development. There still isn’t time to interview them or have them solve some programming problems, so again you’re falling back on self-assessment. This time, the scale is:

  1. No idea what it means.
  2. Have heard the term but couldn’t explain it.
  3. Could explain it correctly to a junior colleague.
  4. Expert-level understanding.

and the terms themselves are:

  • Nested loop
  • Switch statement
  • Stable sort
  • Depth-first traversal
  • Polymorphism
  • Singleton
  • Regular expression
  • Inner join
  • Version control
  • Branch and merge
  • Unit test
  • Variant digression
  • Build and smoke test
  • Code coverage
  • Breakpoint
  • Defensive programming
  • Test-driven development
  • Release manifest
  • Agile development
  • UML
  • Traceability matrix
  • User story

Once again, my questions are (a) what have I forgotten, and (b) how “fair” is this as an assessment method?


Written by Greg Wilson

2008/07/23 at 07:41

Posted in Content, Research

12 Responses

Subscribe to comments with RSS.

  1. I think your first scale is missing something between “Haven’t heard of it” and “use it infrequently”. Perhaps something like “I know what it is, but I don’t use it”


    2008/07/23 at 08:16

  2. Howdy from Texas.

    Is Q9 a trick question? Not only have I never heard of an inverse analyzer, Google hasn’t either. The first hit on “inane” “inverse analyzer” is this blog entry, and the pickin’s are mighty slim after that.

    Michael Tobis

    2008/07/23 at 10:49

  3. The upper scale doesn’t have a good response if they know perfectly well what X is, but have never used it.

    Matthew Woodcraft

    2008/07/23 at 11:52

  4. I’d suggest that you add a number into your first scale:

    0. No idea what it is.
    1. Know what it is, but never used it.
    2. Use it/have used it infrequently.
    3. Use it regularly.
    4. Couldn’t get through the day without it.


    2008/07/23 at 12:04

  5. What I don’t like about your scale is that it’s loaded and doesn’t cover all the options. There’s pretty much a built-in assumption that if you know about this stuff you will use it. What about people who have good knowledge of something but either have chosen not to use it or have used it a lot and stopped because it was bad (or bad for them)? For example, I’ve not only lived in debuggers, I’ve written them; but I haven’t used one for years.

    Personally, I think the most obvious missing thing on the (first) list is a text editor…but I know you think emacs users would still be using punched card if only there decent readers available 🙂

    (And on a different note, I can’t resist asking, if Word is WYSIWYG system, why does it have a preview button?)

    Nick Radcliffe

    2008/07/23 at 13:18

  6. I might add something like “automated builds” to the first list. And perhaps release management or “how do you deploy the software?”

    Matt Doar

    2008/07/23 at 13:45

  7. > Perhaps something like “I know what it is, but I don’t use it”

    Or even, “I don’t know what it is, but Greg told me I need to use it” 😉

    Matt Doar

    2008/07/23 at 13:47

  8. Some trick questions could be useful for filtering bozo responses. “And all of you who said that they used back-scattering inheritance in their code are big ol’ liars!”


    Matt Doar

    2008/07/23 at 13:49

  9. For the first list you’ve missed out:

    14. Computerised Simulation e.g. Monte Carlo methods, discrete event simulations.
    15. Computerised visualization software e.g. Thermal analysis.

    On the second list, you could add:

    * Recursion
    * OO
    * functional programming style.
    * declarative programming style.
    * Profiling.
    * Refactoring.

    The last list doesn’t address the team management side of software development though.

    – Paddy.

    * Style guides


    2008/07/23 at 14:48

  10. Your first order question is define ‘scientist’. I would hazard that there are many disciplines that not pass any of these tests. eg. Medical researcher. But that does not mean that they aren’t a scientist.

    Fact much of this I would pose to the team IT guy not the Principal Investigator.

    Just an observation.


    2008/07/23 at 15:18

  11. “I know all about it and you couldn’t pay me enough to use it again”.

    Michael Tobis

    2008/07/23 at 15:36

  12. This post is also a high Google hit for “Variant digression”.

    define:”variant digression” and define:”variant regression” return nothing.

    Yahoo returns little of interest. So … what is it?

    Sam Penrose

    2008/07/23 at 16:09

Comments are closed.

%d bloggers like this: