Wrapping Up Round 5.2, Starting Round 5.3

2013-06-20
Meeting of the Software Carpentry Instructors Study Group
Round 5.2/5.3

Every conference call ever: http://teaching.software-carpentry.org/2013/06/09/conference-calls/

Agenda

• Did people understand your questions correctly?
• Did they separate novice/competent/expert accurately?
• Is it even possible to come up with a succinct question that separates competent from expert?
• How long did it take to come up with good questions? (Or based on feedback, how long would it take?)

Entries

• Overall: http://teaching.software-carpentry.org/category/proficiency/round-5-2/
• Shell assessment (Preston): http://teaching.software-carpentry.org/2013/06/12/shell-assessment/ (in the end I feel better about my novice question — being able to identify the main basic use of a common tool or concept — I found coming up with a master question very tricky, as to be advanced it had to be detailed, but with then at the same time becomes very narrow — and so subject to being hit-or-miss with regard to an expert’s specific mosaic — this took me maybe 15-20 minutes)
• Python data structures (Philipp): http://teaching.software-carpentry.org/2013/06/12/basic-python-data-structures-assessment/ (My intermediate question was probably too easy — I tried to check whether people knew about the behaviour (and ideally big O-attributes) of these data structures, especially when it comes to large amounts of data)
• List comprehension (Karmel): http://teaching.software-carpentry.org/2013/06/12/multiple-choice-assessment-list-comprehension/
• Pipe and redirect (Yuxi): http://teaching.software-carpentry.org/2013/06/13/multiple-choice-assessment-piperedirect-comprehension/ about the expert question, my point is, if you are a master, you should have at least 50% of right answers. And in fact, I’m not expert about this, so, maybe some questions are not suitable.
• File permissions (David): http://teaching.software-carpentry.org/2013/06/14/multiple-choice-assessment-file-permissions/
• Java (Jordan): http://teaching.software-carpentry.org/2013/06/14/java-proficiency-questions/ All 4 answers were correct. The novice question didn’t take long to come up with, the expert question was another story. I found it difficult to come up with a succient question for intermediate/expert separation.
• Regular expressions (Martin): http://teaching.software-carpentry.org/2013/06/15/py-regex/
• (took quite some time to design the question, particularly the expert vs competent- question)
• Regular expressions (Julia): http://teaching.software-carpentry.org/2013/06/16/regular-expressions/ (I’m not satisfied with the novice question — people got it wrong despite being competent. I came up with the question before the topic.)
• MPI (Neal): http://teaching.software-carpentry.org/2013/06/17/multiple-choice-assessment-basic-mpi-routines/ (I don’t know MPI — but I was tempted to look up the API documentation when reading these questions !)—only one answer which (correctly) surmised the answer to the novice/competent question, but no tries or discussion on the expert/competent question. MPI’s probably a little esoteric for most folks. I found it challenging to frame a multiple choice question which distinguished expertise from competency, particularly as context starts to matter a lot more there. So the question style isn’t generally adequate for that distinction, I think.
• Python — yes, all of it (Luke): http://teaching.software-carpentry.org/2013/06/17/python-noviceintermediateexpert-assessment/ (Novice question took only a few minutes, but expert question took lots of on and off thinking. I think the questions seemed to do a decent job at separating the groups. The most difficult part was coming up with plausible chioces for the expert question. I think questions to determine experts are more suited for discussion, short answer, etc.)
• SQL — yes, all of it (David): http://teaching.software-carpentry.org/2013/06/18/sql-noviceintermediateexpert-assessment/ Found that designing the expert question was much more subtle than I had expected
• Bears (Kirsten): http://teaching.software-carpentry.org/2013/06/18/beginner-and-intermediate-questions-about-bears/ (and may I say, I didn’t expect that one)
• Genetics (Billy): http://teaching.software-carpentry.org/2013/06/18/assessing-proficiency-at-classical-genetics/ I think the questions were understood. Question 1 separated novices from the unwashed masses, and question 2 seemed difficult enough, but I’m not sure if it was “competent” or “expert” level. It was difficult to write a succinct expert question, which was why I ended up writing a glorified vocabulary question.
• Pass by ref/value (Ted): http://teaching.software-carpentry.org/2013/06/18/passing-by-reference-and-by-value-in-python/ Yes, respondents understood. I think my questions separated levels of knowledge, but not necessarily “mastery” as we defined it last week. I think this first pair of questions, with improved distractors, would have taken a half hour.
• Functions vs. methods (Jessica): http://teaching.software-carpentry.org/2013/06/19/python-functions-vs-methods/ I think people understood the first question, the second question maybe wasn’t clear enough. My first question was actually meant to separate two levels of intermediate — there were two correct answers and I think if you get one but not the other, you’re competent, and if you get both, you’re at a higher level of competency (but not necessarily at expert, which the second question was meant to address more). Coming up with the questions took me a good portion of yesterday, and it was maybe even a bit easier because I based the ideas I was testing off of a blog post I’d written in the past.
• SQL grouping (Shoaib): http://teaching.software-carpentry.org/2013/06/20/multiple-choice-sql-group-by-again/ (coming up with the questions took longer than I thought — as I had to think of realistic choices or mis-understandings)
• Basic bash (Katie): http://teaching.software-carpentry.org/2013/06/20/3264/ I was late in getting the question up, so no one answered, but it was a good exercise to think about the questions. (Anybody want to answer now? :] ) It was difficult to come up with them. I think having specific learning goals would make it easier. I wanted to make expert questions that involved creating something, but that might make a lot of time to answer.
• Python operators (Itamar) -http://teaching.software-carpentry.org/2013/06/17/python-operators-quiz/ people did understand my question, and I’m pretty sure it separated decently, with the assumption at least that expertise is fairly rare. I definitely would have preferred a follow up “and *why* did you choose that answer” for people who got expert answer correctly, to verify it’s not just someone who is confused.
• Python difference between module & package (Promita)
• I thought it was difficult to only rely on a few questions to separate the 3 groups

Thoughts

http://www.amazon.com/Understanding-Design-Expanded-2nd-Edition/dp/0131950843/:

• questions, then lessons
• the first thing to do is to decide on assessment
• then backtrack to identify key practice skills for exam preparation
• then identify concepts and lessons for preparatory tasks.
• Then construct coherent lessons on this basis (feels backwards).

Work backwards: what I want to assess, what they need to know/do to pass that assessment