I’m working on a very different sort of website for a big publisher. The plan is that in a few years, tens or hundreds of thousands of college students will be using our book / website combo to learn physics. There will be many differences between our site and our major competitors (Mastering Physics and WebAssign). I really admire this publisher for taking a risk and trying something different, but sometimes it’s tough to not play the feature list game.
A while ago I was given a list of features that homework questions should have. This list included, among other things, having a spell-checker check spelling of free response answers and awarding partial credit for misspelled answers , specifying a regular expression to use on student’s answers to determine correctness, and about 10 different ways to specify the required precision of a numerical answer.
I won’t claim that there is no possible reason for someone to want these features, but what I will claim is that adding these features would do more harm than good. I won’t get into details of each feature, but take the spell check – partial credit feature as an example. Suppose the correct answer to a question is “Newton’s 2nd Law”. I want to give partial credit for misspelled answers. Ok, how much credit? I’ll pick 70%. Should capitalization matter? Hmm, no. What about the apostrophe? Take off points if it’s missing? Yeah, because the student might actually think “Newton” is plural rather than possessive. That was a few extra things to do, but now I’ve got a great partial credit for misspelled words free response question.
But then the students start doing the assignment. I get an e-mail from a student. “Dr. Feil, I only got 70% on the question because I just forgot to put in the apostrophe. I didn’t think it mattered.” And another one, “Dr. Feil, I typed ‘Newton’s second law” and it didn’t give me credit. Yet another, “Dr. Feil, I typed ‘Netwon’s 2nd Law’ and I got 0%, but my friend typed “Newton’s 2nd Lwa” and he got 70%. That’s not fair!”
It’s not uncommon for large physics classes to have 500 – 1,000 students. Does the professor really want to deal with these kinds of questions and complaints?
Changing the question to a multiple-choice format removes all these issues. It’s a better experience for the instructor. It’s a better experience for the students. It’s one less feature!
So why the obsession with long lists of features? Well, in the world of education websites, so I’m learning, it’s all about comparisons with the competition. It’s something like, “Our Product X has features A, B, C, D… and Products Y and Z don’t have them.” So that’s marketing, and I think there is some validity to the obsession with features in this respect. It’s tough when your product doesn’t have that check-mark on the feature list.
But there’s another reason, I think, and having a lot of experience with social science research, I’m pretty sure I know what the problem is. Every good company does market research to find out what their customers want. In this case, the website companies survey professors. I think the problem is the surveys ask the wrong questions. Given a list of features and a rating scale of 1 (I wouldn’t use this feature) to 5 (I have to have this feature), you can easily fool yourself into thinking that many features are more important than they really are. If you ask somebody a “would you like…” question, they’re probably going to be somewhat favorable to it unless you’re offering them a kick in the shin. So you end up with a bunch of surveys and 80% of the professors rated “partial credit for misspelled answers” a 3 or better.
The lesson here is to be mindful of people’s knowledge areas. Most professors know a lot – about their field. No matter what they think about themselves, professors don’t have any kind of superior wisdom or knowledge of all things. Yes, you should find out from your users what they want, but don’t ask them questions that they aren’t qualified to intelligently answer. I know that partial credit for misspelled answers is just going to lead to trouble, so I don’t care that some professors think that it would be a neat thing to have.