But now there’s this:

BioTorrents: A File Sharing Service for Scientific Data

The transfer of scientific data has emerged as a significant challenge, as datasets continue to grow in size and demand for open access sharing increases. Current methods for file transfer do not scale well for large files and can cause long transfer times. In this study we present BioTorrents, a website that allows open access sharing of scientific data and uses the popular BitTorrent peer-to-peer file sharing technology. BioTorrents allows files to be transferred rapidly due to the sharing of bandwidth across multiple institutions and provides more reliable file transfers due to the built-in error checking of the file sharing technology. BioTorrents contains multiple features, including keyword searching, category browsing, RSS feeds, torrent comments, and a discussion forum. BioTorrents is available at http://www.biotorrents.net.

It’s a neat idea, and will become neater once scientists routinely put DOIs on data as well as papers. I’d be very interested in a usability study to see how easy or hard it is for the average grad student in botany to get this plugged in and turned on.

We’re very pleased to announce that Scimatic Software, a Toronto based company that specializes in the development of software for the scientific community, has come on board as a sponsor of this project. Many thanks to Jamie McQuay and Jim Graham!

From the abstract:

Although unguided or minimally guided instructional approaches are very popular and intuitively appealing…these approaches ignore both the structures that constitute human cognitive architecture and evidence from empirical studies over the past half-century that consistently indicate that minimally guided instruction is less effective and less efficient than…approaches that place a strong emphasis on guidance of the student learning process. The advantage of guidance begins to recede only when learners have sufficiently high prior knowledge to provide “internal” guidance.

A few selections from the main body:

Minimally guided instruction appears to proceed with no reference to the characteristics of working memory, long-term memory, or the intricate relations between them. The result is a series of recommendations that most educators find almost impossible to implement…because they require learners to engage in cognitive activities that are highly unlikely to result in effective learning. As a consequence, the most efefctive teachers may either ignore the recommendations or, at best, pay lip service to them. (pg. 76)

Inquiry-based instruction requires the learner to search a problem space for problem-relevant information. All problem-based searching makes heavy demands on working memory. Furthermore, that working memory load does not contribute to the accumulation of knowledge in long-term memory because while working memory is being used to search for problem solutions, it is not available and cannot be used to learn… The consequences of requiring novice learners to search for problem solutions using a limited working memory or the mechanisms by which unguided or minimally guided instruction might facilitate change in long-term memory appear to be routinely ignored. The result is a set of differently named but similar instructional approaches requiring minimal guidance that are disconnected from much that we know of human cognition. (pg. 77)

None of [this] would be important if there was a clear body of research…indicating that unguided or minimally guided instruction was more effective than guided instruction. In act…the reverse is true. Controlled experiments almost uniformly indicate that when dealing with novel information, learners should be explicitly shown what to do and how to do it. (pg. 79)

After a half-century of advocacy associated with instruction using minimal guidance, it appears that there is no body of research supporting the technique. In so far as there is any evidence from controlled studies, it almost uniformly supports direct, strong instructional guidance rather than constructivist-based minimal guidance during the instruction of novice to intermediate learners. Even for students with considerable prior knowledge, strong guidance while learning is most often found to be equally effective as unguided approaches. Not only is unguided instruction normally less effective; there is also evidence that it may have negative results when students acquire misconceptions or incomplete or disorganized knowledge. (pg. 83)

There are well over a hundred references into the literature. If they’re right (and I’m now convinced), then the material for this course should be presented in smaller chunks than I’ve used in the past, and each should be accompanied by several worked examples.

[1] This is, by the way, why I believe that attempts to teach “computational thinking” without first teaching programming are doomed to fail, but that’s a rant for another time.

• Existing metrics have known flaws
• A reliable, open, joined-up data infrastructure is needed
• Data should be collected on the full range of scientists’ work
• Social scientists and economists should be involved

The same points could be made about evaluating software developers (or any other kind of knowledge worker). The devil, as always, is in the details, and unfortunately I have to start doing evaluations before those details are worked out. Several of the sponsors for this course need me to demonstrate its impact on the productivity of the scientists who take it (so that they can in turn justify their contribution to their funders). It isn’t enough to ask students who have completed the course whether they think they know more about programming than they used to: ignoring the obvious problems of survivor bias and self-assessment, I would still have to demonstrate that making people better programmers also makes them better scientists. I believe it does, but belief is not evidence, and doesn’t convey scale.

The best plan I’ve been able to come up with so far is to look at how scientists spend their time before and after taking the course, but that would require resources I don’t have.  If you’re interested in studying scientists or software developers empirically, and would like some raw material, I’d like to hear from you.

There are two kinds of infrastructure: technical and social. Each shapes the other, so decisions about them have to be made together. On the technical side, Software Carpentry must have:

1. a way to deliver content to students, including text and images, audio/video, exercises (with solutions), sample data sets, and useful software;
2. a way for students to feed questions back to the course organizers (asynchronously through email and bulletin boards and/or synchronously through VoIP and desktop sharing);
3. a way for instructors (who may or may not be contributors) to respond to students;
4. a way for lay contributors (who may also be students) to offer new content, from pointing out typos to providing exercises or whole new lectures; and
5. a way for core contributors to manage and edit contributions, create some of their own, et cetera.

This description implies some social infrastructure, including:

1. some core contributors who are creating lots of course content, and probably teaching the course as well;
2. a second tier of instructors who are creating less content but also interacting with students; and
3. students, who may be registered in a course of some kind or working through the material on their own (and who may eventually move up to answering others’ questions and eventually to creating content).

This sounds like what you’d find in an open source project (regular users, occasional testers or bug reporters, and contributors) at least as much as it sounds like a traditional college course (students, teaching assistants, and professors). The most important difference is that the divisions between the latter three roles are sharper and deeper than the divisions in open source projects: some undergrad students eventually go to grad school and become TAs, and some of those eventually become faculty, but it’s almost unheard of for someone to be in two of those categories at once, or to “bubble up” from one to the next based solely on ability and enthusiasm. That must happen if Software Carpentry is to become self-sustaining: while over 140,000 people have looked at the existing material in the past five years, only three dozen have ever sent bug reports, and only four of those have contributed any substantial content. Whatever we use to accomplish tasks 1-5 above must therefore draw people in and make it easy for them to use, ask, answer, and contribute.

With that out of the way, here are a few options for discussion:

• Project: use SourceForge or Google Code as a host.
• Retro: a classic turn-of-the-century web site with static HTML for content, bulletin boards and/or mailing lists for discussion, Trac with Subversion for project management.
• Social: a WordPress blog with lectures and other content as posts (updated several times, with threaded comments for feedback).
• Turnkey: a fully-fledged learning management system such as Moodle.
• Wiki: like Retro, but with the content in a wiki.

So how do they stack up?

A bit of explanation:

• Project hosting services require little or no setup, but would force us to manage this as a software development project, rather than as a writing project: self-tests and “try this at home” examples aren’t built in, and neither of the big open source project hosting sites would be happy if we used them as a video server.
• The “retro” option would require us to “roll our own” on a lot of things, which would be fun (I like to program) but wouldn’t directly deliver value to scientists.
• WordPress is easy to set up, but doesn’t have very many development-oriented or education-oriented plugins, and isn’t designed to host video snippets or live examples. I think that building the course as a blog is a neat idea, but the novelty would soon wear off…
• Learning management systems like Moodle have a lot we don’t need (recording grades, for example), but a lot that we do (managing course bundles). I gave this category -1 for ease of setup because I’m unfamiliar with it, and 0 for “easy for people to contribute” because the LMSes I’ve looked at (ATutor, OLAT, Moodle, and Sakai)  all seem to have significant learning overheads for creating content—they’re a bigger hammer than I (think I) need. Of course, that could just be unfamiliarity again…
• A wiki would be easy to set up and maintain, but in my experience, editing large volumes of material in a browser is unpleasant, and there’s little support for managing updates, particularly by concurrent authors.

As always, I’d welcome your thoughts…

Three of the students and five of the TAs spent an hour with me on Thursday discussing what went right and what went wrong. The chance to work through their own problems with some expert assistance was at the top of the former list for all of the students: they all felt that they learned most by bringing their own “homework” to class and having the TAs show them how to tackle it. I’m pleased that this was so useful, but if the Version 4 lectures are recorded for online viewing, this will aspect of the course be lost. I asked whether being able to work with a TA remotely (via Skype and desktop sharing) would be an acceptable substitute for in-class hacking, but no one in the room had ever tried it. There’s a lot of research out there into virtual office hours and remote tutoring; I’d like to try a few experiments in May or June to see how well it might work for Software Carpentry.

The two things students like least were the general disorganization of the course and the fact that a lot of the material felt like what we computer scientists though they ought to know, rather than what they could see as being immediately useful. The disorganization reflects the grassroots nature of this round of the course, and the fact that it was our first time teaching in MATLAB. Next time around, we’ll use a more natural order for material in MATLAB, rather than sticking to the order that makes sense for Python, but forces students to grapple with some of the more obscure features of MATLAB early on.

The “eat your vegetables” tone of the material is going to be much harder to deal with. Software Carpentry is meant to be a second course in computing, not an introduction to programming in general: as the last of the user profiles says:

This course is probably too advanced for [a novice], as it assumes familiarity with basic programming concepts like loops, conditionals, arrays, and functions. [They] should probably audit a first-year introduction to programming or find an intensive two-week summer school course before tackling this one.

The problem is that if we’d actually applied that rule last November, we would have turned away more than half of the students, most of whom would never have acquired those basic concepts. So, do we:

1. ignore the problem and hope that these people will somehow pick up the basics on their own (despite the fact that most scientists never do), or
2. broaden the course’s mission to include basic programming as well.

My phrasing makes my preference for the second option clear, but feature creep is the biggest risk this project faces. Teaching the basics of Python to people who already know a bit about programming takes 4-5 lectures out of the 25 budgeted; if they don’t know how to program, that figure probably triples, leaving only 10 lecture hours to cover a much-reduced subset of the planned material. On the other hand, sticking to the plan means condemning the majority of potential students to wander lost and frustrated through a bewildering maze of seemingly inconsistent behavior, and to hour upon wasted hour of heartbreaking frustration. (That was a bit melodramatic, but not necessarily inaccurate.)

Another argument against option #2 is pacing. Software Carpentry has been run four times at the University of Toronto (twice as non-credit tutorials and twice as a regular for-credit course). Each time, wide variation in students’ prior experience levels meant that no matter how material was paced, one third of the class would be bored or another third bewildered. On balance, therefore, I think Software Carpentry has to continue to assume a more advanced starting point than most of its potential audience currently has. If things go well, I hope we’ll be able to backfill with more accessible introductory material in a year’s time.