Fourth Meeting for Round 12

Mar 20, 2015 • Greg Wilson

Assignment:

  1. Write a short blog post describing something that happened in your schooling that demotivated you, and discuss what could have been done afterward to fix it.
    • Add the file to _submissions/round-12/3/yyyy-mm-dd-firstname-lastname.md
    • Use /2015/03/firstname-lastname-motivation/ as the permalink
    • Use Motivation as one of the tags
  2. Fill in http://doodle.com/7w9vyq7mhncqzc7t#table to let Greg know what time zone you’re in so that he can put you in a group of three for teaching practice.
  3. Do at least one online teaching session with your group (preferably two).
    • Each person picks something from the existing lessons that they can teach in 5 minutes.
      • Note: most of Software Carpentry’s lessons take longer than this, so pick a subsection that you think will fit into this time.
      • Note also: 5 minutes really means “5 minutes”…
    • Use Skype or Google Hangout to screenshare while talking.
    • Each person teaches while live coding for 5 minutes, then their partners spend 5 minutes giving feedback.
    • Then the next person, then the next.
  4. Read:
    • Chapters 3 and 4 of How Learning Works
    • the teaching tips section of this website.
  5. Greg will send a note separately about Git/GitHub/Markdown tutorials to those who signed up.

Groups for Practice Lessons

  • East Asia/Australia 1
    • Will Cornwell
    • Victor KC Lee
    • Belinda Weaver
  • East Asia/Australia 2
    • Karl Broman
    • Darya Vanichkina
  • Europe 1
    • Donna Henderson
    • Johan Hjelm
    • Laurie Baker
  • Europe 2
    • Matt Probert
    • Laura Graham
    • Sarah Mount
  • Europe 3
    • Horacio Vargas
    • Javier Garcia-Algarra
    • Fran Navarro
  • Europe 4
    • Konrad Forstner
    • Hugues Fontenelle
    • Richard Tomsett
  • Europe 5
    • Owen Stephens
    • Simon Fraser
    • Marios Isaakidis
  • Europe 6
    • Arvind Sundaram
    • Martin Bentley
    • Evgenij Belikov
  • Europe 7
    • Malvika Sharan
    • John Constable
  • South America / Eastern North America 1
    • Jon Borrelli
    • Ariel Bowers
    • Kris Keipert
  • South America / Eastern North America 2
    • Dirk Eddelbuettel
    • Sarah W.
    • Violet Zhang
  • South America / Eastern North America 3
    • Meredith Durbin
    • Arliss Collins
  • South America / Eastern North America 4
    • Daiva Nielsen
    • April Clyburne-Sherin
    • Remi Daigle
  • South America / Eastern North America 5
    • Vladimir Sudilovsky
    • Jeremy Gray
    • Brian Magill
  • South America / Eastern North America 6
    • Auriel Fournier
    • Derek Howard
    • Andrew MacDonald
  • South America / Eastern North America 7
    • Elizabeth Wickes
    • Sue McClatchy
    • Narayanan Raghupathy
  • South America / Eastern North America 8
    • Heather Gibling
    • Kathy Chung
    • Kim Moir
  • South America / Eastern North America 9
    • Filipe Pires Alvarenga Fernandes
    • Matthew Collins
    • Morgan Taschuk
  • South America / Eastern North America 10
    • Kyler Brown
    • Luke Johnston
    • Matthew Bourque
  • South America / Eastern North America 11
    • Pawel Pomorski
    • Joseph Long
    • Catherine Devlin
  • Western North America 1
    • Kai Yang
    • Tessa Pierce
    • Amy Boyle
  • Western North America 2
    • Michael Sarahan
    • Phil Rosenfield
    • Matt Dickenson
  • Western North America 3
    • Sarah Stevens
    • Dorota Jarecka
    • Ben Weinstein
  • Western North America 4
    • Adam Richie-Halford
    • Jessica Gallinger
    • Daniel Wheeler
    • John Moreau

Questions

  • What did you learn about your topic (and how to teach it) from thinking through your MCQ?
  • What did you learn about your topic and/or about writing MCQs from the comments you got?
  • What did you learn about writing MCQs and/or giving feedback from looking at other people’s?

10:00 Eastern

  • Greg Wilson (Software Carpentry Foundation)
  • Filipe Fernandes: No post
    • My topic was python conditionals and I learned that the topic is trickier than I first though. (I actually found a bug in Shapely’s logic after creating the MCQ.)
    • No feedback (No post).
    • I learned that teaching how to reverse a list can be more useful than I imagine. And how they :: can be confusing.
  • Daniel Wheeler: http://swcarpentry.github.io/training-course/2015/03/daniel-wheeler-mcq/
    • Differentiate between pipes and passing result of commands with $(…)
    • The level of abstraction when phrasing the question. Difficult issue. Make it specific or abstract.
    • Make it possible for the student to implement the question in a program at the command line quickly so they can experiment if they don’t understand it. Useful for me when reading other’s questions.
  • Marios Isaakidis: https://swcarpentry.github.io/training-course/2015/03/marios-isaakidis-mcq/
    • Finding misconceptions helped me think of what could confuse students and what I should clarify during the workshop
    • Giving options at the MCQ that refer to concepts out of the scope of the course might be misleading
    • Giving datasets or results of commands can help “simulate” a real world scenario
  • Stephanie Mark (University of Toronto)http://swcarpentry.github.io/training-course/2015/03/stephanie-mark-mcq/
    • helped me think about the concepts behind what I was doing: unix commands has become automatic for me so I had to think hard about what a person new to unix might be thinking
    • questions were better received than I thought they would be! and that there are better ways to ask the same question
    • I’m more comfortable receiving feedback than giving it
  • Pawel Pomorski (University of Waterloo) http://swcarpentry.github.io/training-course/2015/03/pawel-pomorski-mcq/
    • Just general understanding that variable assignment is not trivial, and might be confusing to someone who has never done programming before
    • Writing good MCQs is difficult, coming up with more than a few options which are actually of use to the student and instructor is hard
    • Review by a second person always improves the questions
  • Pandurang Kolekar (University of Pune, India)
  • John Constable (Welcome Trust Sanger Institute) http://swcarpentry.github.io/training-course/2015/03/john-constable-mcq/
  • Kai Yang (U. of Washington) http://swcarpentry.github.io/training-course/2015/03/kai-yang-mcq/
    • I learned how to evaluate prior/posterior knowledge.
    • Unfortunately, I did not receive any feedback.
    • For the topic itself, I learned nothing from it although I spent much time in iterating versions of my MCQ before I pick up an appropriate one.
    • I particularly paid attention to the distraction in the MCQ, whether intended or not. Those can give us hints about potential misconceptions.
  • Laura Graham (University of Nottingham) http://swcarpentry.github.io/training-course/2015/03/laura-graham-mcq/
    • That I probably don’t have a good idea of common misconceptions (it was difficult to think of distractors) and that by understanding these it might make me a better coder.
    • One comment suggested that I could be biasing the response against one of the wrong answers because it was so obviously different from the others - it’s difficult to come up with distractors which are diagnosable, but also don’t bias the answers.
    • I was more likely to offer feedback on shorter, well formatted questions, which suggests that these types of questions are most likely easier to answer (and therefore more useful as a diagnostic tool?)
  • Heather Gibling http://swcarpentry.github.io/training-course/2015/03/heather-gibling-mcq/
    • I realized a good way to learn about how to teach a topic is to think about what I struggled with when learning it myself. (only had to think back ~1.5 years)
    • Things that I think are clear might not be to others. Helpful to have feedback
    • Simpler is better–I found I got tripped up in details in other people’s questions that distracted a bit from what I should be focusing on. Also, I was much more inclined to read through and attempt the simpler questions than the long/detailed/complicated ones… less intimidating, especially on topics that are not my strengths.
  • Amy Boyle (Washington State University Vancouver) http://swcarpentry.github.io/training-course/2015/03/amy-boyle-mcq/
    • I learned different ways people might think about python indexing that makes sense, but would be wrong, that I hadn’t thought of before
    • I had only positive feedback, so I suppose I learned writing MCQ is not as difficult as I thought
    • The quality and approach of the questions varied greatly, some I found kind of confusing, some were good; I could see how the MCQ as a tool could be an aid if it’s good, or probably demotivating if its bad.
  • Adam Richie-Halford (University of Washington) https://swcarpentry.github.io/training-course/2015/03/adam-richie-halford-mcq/
    • Writing the MCQs gave me a better understanding of the improper conceptual connections that novices can make. e.g. the syntax of piping commands can get in the way of the concept of piping commands. We’re trying to teach new concepts in a new language.
    • I didn’t receive any comments.
    • I learned that writing a good MCQ can be very challenging. I spent about 45 minutes writing two questions. When I told my friends about this they were very surprised at the low question output. I’m guessing that it gets easier with time. But I’m also guessing that good, productive questions might just take a lot longer than everyone thinks.
  • Michael Sarahan (Nion Co) https://swcarpentry.github.io/training-course/2015/03/michael-sarahan-mcq/
    • Coming up with MCQs that were single-topic was very hard. Much easier to talk about procedures, but then those felt like they expected too much.
    • The comments pointed out a good error I had made - the question I wrote was invalid. I think formatting issues in my post (i didn’t do the numbered list properly) prevented people from giving more feedback, because the question was hard to look at.
    • Other people often had very long, involved questions. That reinforced my belief that questions need to be short and single-topic.
  • Karl Broman (UW-Madison) http://swcarpentry.github.io/training-course/2015/03/karl-broman-mcq/
    • I really had to choose the topic to match what I could write an MCQ about rather than being able to write an MCQ for a random topic of importance. The MCQ exercise really hammered home the importance of my understanding what students’ misconceptions and difficulties might be. I have surprisingly little understanding of what are the important mistakes that students might be making for many topics.
    • It was nice to get some positive feedback, and a sense that others have the same difficulty with this exercise that I did, but I don’t think I learned much particularly from the feedback.
    • Some others were really good at this. Looking at a lot of nice MCQs was really helpful for thinking about what I might have done, or what I should do next.
  • Owen Stephens (Owen Stephens Consulting) GMT until 29th March, then GMT+1 (No post) (sorry :()
    • You have to know your topic very well, but at same time you need to understand the assumptions about existing knowledge
    • Writing MCQs that target specific knowledge and enable you to understand the students understanding through incorrect answers is very challenging - ruling out what the incorrect assumptions might be is impossible so need to have some baseline
    • I had to understand something of the area to give feedback, but knowing too much could be a disadvantage - I felt I was able to give the best feedback where I was a ‘good student’ level rather than an expert level - back to points about baseline assumptions above. Also much easier to give feedback on the MCQs than to write good ones of my own!
    • Time: Haven’t posted but 1+ hour even so
  • Daiva Nielsen (University of Toronto) - No post on SWC Training Course Repo, but this is a link to the MCQ on my own github – I was very late getting these up and didn’t submit a PR since late submissions weren’t merged. https://github.com/daivanielsen/training-course/blob/gh-pages/_submissions/round-12/2/2015-03-18-daiva-nielsen-mcq.md
    • I understand commands in Unix Shell a lot better now and how the Shell is very useful for easily navigating your way through your files and directories.
    • N/A
    • It was good to think about the Distractors and the purposes they serve. Now I have a greater appreciation for the way that understanding why an option is incorrect helps with learning.
  • Meredith Durbin (Space Telescope Science Institute): https://swcarpentry.github.io/training-course/2015/03/meredith-durbin-mcq/
    • MCQs are a lot harder to write than they look
    • It was easiest to approach it for me by thinking about some of the misconceptions I had when I started working with my topic, and how I unlearned those; however, other people may have had very different initial misconceptions than my own
    • I saw several good MCQs that were fairly complex in what they addressed, and several good ones that were very simple; there are multiple ways to create good MCQs
  • Jessica Gallinger (University of Saskatchewan): No post
  • Narayanan Raghupathy (http://swcarpentry.github.io/training-course/2015/03/narayanan-raghupathyr-mcq/)
    • Writing good MCQs are difficult :)
    • Had to choose the familiar topic to write good MCQs.
    • Even after covering for all bases (supposedly), there is always better way to write MCQs
  • Javier Garcia Algarra (UPM) (http://swcarpentry.github.io/training-course/2015/03/javier-garcia-algarra-mcq/)
  • R. Burke Squires (NIAID) No post
    • The topic can be difficult to understand - python programming
    • Difficult to think of how someone would misunderstand the concepts
    • Good MCQ take a long time to write and think about
  • Matthew Bourque (Space Telescope Science Institute) http://swcarpentry.github.io/training-course/2015/03/matthew-bourque-mcq/
    • I learned that python strings are immutable :) Also, trying to come up with distractors made me view the topic in a whole different way; it made me consider all aspects of the problem.
    • That I probably could have made my questions a bit more complex and that would have been OK.
    • Questions can be improved by simple criticism. Also, some MCQs test the student’s memorization while others test the student’s ability to reason/deduce.
  • Victor Kwangchun Lee (xwMOOC, Korea) http://swcarpentry.github.io/training-course/2015/03/victor-lee-mcq/

15:00 Eastern

  • Greg Wilson (Software Carpentry Foundation)
  • Phil Rosenfield: http://swcarpentry.github.io/training-course/2015/03/phil-rosenfield-mcq/
    • I learned that I don’t have the correct vocabulary when it comes to programming.
    • That I made the mistake of covering in the MCQ what I wanted to have seen covered in the lesson (oops!)
    • I am a better editor than writer!
  • John Pormann http://swcarpentry.github.io/training-course/2015/03/john-pormann-mcq/
    • To state the obvious: I learned that writing MCQs is hard; and it is easy to get tripped up by natural language (words/phrases) that sound clear (and unique) but then get interpreted many different ways
    • I was expecting a bit more “high level” responses, but got lower-level ones .. which is probably good since we’re targeting novice programmers
    • Comp-sci allows for many different ways of doing anything .. which complicates how to teach it (but also makes it fun!)
  • Sarah W: http://swcarpentry.github.io/training-course/2015/03/sarah-w-mcq
    • I learned that it is very hard to create meaningful MCQs and that made me view my topic differently.
    • That it’s a difficult topic, and it’s hard to write good analytical MCQs when you yourself are still learning your topic.
    • It was great to see other examples and see the different takes on MCQs.
  • Matt Dickenson: http://swcarpentry.github.io/training-course/2015/03/matt-dickenson-mcq/
  • Arliss Collins: https://swcarpentry.github.io/training-course/2015/03/arliss-collins-mcq/
    • Although my topic was something I was familiar with I did have try to read through the lesson material as if I was a new learner to the topic in order to figure out what might be challenging to a new learner.
    • The assignment was very interesting. There were portions of the lesson that I needed to read a couple of times to make sure I understood them enough to teach them. I hope we can still make comments on the MCQ’s for a bit longer so that we have time to look through all of the material
    • Coming up with good, relevant distractors takes a bit of thought - and then to know if they were thought-provoking for the learner.
  • Catherine Devlin: http://swcarpentry.github.io/training-course/2015/03/catherine-devlin-mcq/
    • I’d never before thought of a cartesian join as the fundamental join and everything else as a variant on that (b/c after all cartesian joins are generally an error), but I realized that’s a clear way to think about it
    • Got a good reminder to be super-careful with the vocabulary used in questions. It’s all too easy to assume I’m writing clearly because I already know what I mean.
    • There’s such a sharp divide between MCQs for remembering terminology and MCQs for understanding logic that they almost deserve different names
  • Tessa Pierce: http://swcarpentry.github.io/training-course/2015/03/tessa-pierce-mcq/
    • It’s fairly difficult to write good, clear MCQ’s.
    • People came up with additional distractors that would help diagnose issues with python lists and errors.
    • Giving feedback: It’s easy to see how something was clear in somebody’s mind, but very difficult to interpret for someone else. It is hard to create simple examples that don’t require a ton of reading or code before getting to the actual learning/assessment component.
  • Morgan Taschuk No post
    • I actually didn’t find it too hard to make the questions… but then I have no feedback yet so my questions are possibly complete crap. :)
    • No feedback
  • Ariel Bowers http://swcarpentry.github.io/training-course/2015/03/ariel-bowers-mcq/
    • I learned that it is difficult to create multiple choice questions for novice programmers. Simplicity is key and you can loss sight of that sometimes.
    • I learned that everyone doesn’t see the same things you see.
    • Giving feedback can be is difficult.
  • Sarah Mount (University of Wolverhampton) http://swcarpentry.github.io/training-course/2015/03/sarah-mount-mcq/
    • Some corner cases in git - e.g. if two users make identical changes to the same branch on the same repo and push to the same remote the second push will be rejected.
    • How difficult it is to write a clear and unambiguous question / answer about really detailed bits of syntax
    • It’s difficult to think of really good “wrong” answers and think through all the relevant options. In some cases it was interesting to see what possibilities people had left out of their MCQs.
  • Donna Henderson http://swcarpentry.github.io/training-course/2015/03/donna-henderson-mcq/
    • I learned that you need a specific idea in mind for each question. You can’t test too much knowledge with a single question
    • I learned the importance of thinking through your phrasing. Something that you’ve written can easily be interpretted differently by someone else
    • Others thought of good distractors that I never would have come up with. Highlights the reason to collaborate on lessons
  • Sue McClatchy (The Jackson Laboratory) http://swcarpentry.github.io/training-course/2015/03/sue-mcclatchy-mcq/
    • I would second the comment about lacking proper vocabulary. I thought I knew the language of computer science, but now know that the terminology doesn’t roll of the tongue so easily.
    • I submitted my MCQs late and didn’t receive comments. My bad.
    • I discovered how difficult it actually is to write a good question. It seems very simple when you’ve taken so many multiple choice exams and quizzes, however, writing a good one is not easy or straightforward.
  • Sarah Stevens (University of Wisconsin - Madison) : http://swcarpentry.github.io/training-course/2015/03/sarah-stevens-mcq/
    • I learned that some of the basics behind my topic could easily be confusing, as it was tough to design a question which would cover many of the possible misconceptions
    • I actually learned that if writing a question that purposely uses broken code, you should state it in the question. My comments were more about correcting the code than the question.
    • I found it was much more difficult to create a good MCQ than I expected. I found it really interesting to see the logic behind other people’s plausible distractors.
  • Malvika Sharan (University of Wuerzburg, Germany): http://swcarpentry.github.io/training-course/2015/03/malvika-sharan-mcq/
    • Keep the general confusions in mind a student might have about the topic so that you can really find out if they really learnt the correct concept
    • Getting second opinion helps alot in thinking more logically
    • It was interesting to learn that we all have different points of view about the same/similar concept and even writing a wrong answer needs to have a logical explanantion
  • Jon Borrelli (Stony Brook University): http://swcarpentry.github.io/training-course/2015/03/jon-borrelli-mcq/
    • Coming up with answers with useful distractors was hard. Especially when picking something from an earlier lesson it was difficult to remember the misconceptions I had when I learned it.
    • It was reassuring when people thought the question and answers were useful.
    • The structure of the MCQ does not have to follow a specific format. I saw some interesting implementation of questions and distractors. Especially with asking how to do something and providing different code that may be able to do the task.
  • Laurie Baker (University of Glasgow): http://swcarpentry.github.io/training-course/2015/03/laurie-baker-mcq/
    • I realise from the comments now that I made mine rather long.
    • it was hard to think about what size chunk to break off for an mcq
  • Francisco Navarro (University of Alicante, Spain): http://swcarpentry.github.io/training-course/2015/03/franz-navarro-mcq/
    • My topic was specially hard for me. Thinking the right questions and looking through the doc helped me to really understand and therefore, be able to explain the topic.
    • That is a hard topic for beginners (so it was for me!)
    • Very nice ideas, and respectful comments that really improved most of them (if not all).
  • Joseph Long (Space Telescope Science Institute): http://swcarpentry.github.io/training-course/2015/03/joseph-long-mcq/ (I may be a little late calling in, as I have another meeting ending right at 15:00 EDT.)
    • A1 & 2: I learned that the examples I would use to illustrate certain concepts rely (perhaps too much) on background knowledge of the topic.
    • A3: I was happy with my MCQs, but it was not easy to come up with them. I think I would improve with practice, and with experience seeing the ways in which students misunderstand.
  • Arvind sundaram (Norwegian Sequecing Centre): No post
    • MCQs would help in getting feedback from the students
    • None
    • Extremely difficult to create MCQs unless you had interacted with students in a given topic. This would have given an overview of mistakes made by students
  • Kim Moir (Mozilla): https://swcarpentry.github.io/training-course/2015/03/kim-moir-mcq/
    • To think of scenarios where the students could get off track and identify the misunderstanding
    • No comments received
    • The examples I looked at - most people had a really good grasp of their topic.
  • Ben Weinstein (Stony Brook University)
  • Johan Hjelm (Technical University of Denmark) https://swcarpentry.github.io/training-course/2015/03/johan-hjelm-mcq/
    • MCQs are harder than I expected. Regarding the topic, I think the process of trying to think of possible misconceptions made it a bit clearer to me how dataframes work,
    • Not so many comments but the one I got helped me see how easy it is to cause confusion if it is not written well.
    • There was a lot of constructive feedback given, and it is difficult to construct good distractors of great diagnostic value
  • John Constable (Welcome Trust Sanger Institute) http://swcarpentry.github.io/training-course/2015/03/john-constable-mcq/
    • I learned how hard it is to write a question that doesn’t lead on the reader, and quite how well you need to understand your subject..
    • About the topic I started to get an idea of how people might think about the subject, hard breaking out of expertise.
    • I only managed to give one set of feedback because I found doing so incredibly hard - providing useful, constructive feedback when you’re not a subject expert yourself. Hard not to default to only critical mode, which isn’t helpful.
  • Konrad Förstner (University of Würzburg, Germany) Konrad Förstner (University of Würzburg, Germany)
    • The topic (Shell Pipes and Filters) was rather clear to me but it took me very long to think about potential misconceptions (going back to “incompetence”) and drafting questions that can asses those.
    • I used “tr” in one of my examples and that seem to be not as widely known as I thought
    • I learned that there are very different perspective of the chosen topics. I would have done most of the question with a complete different focus but can understand why people did it the way they did it. Even those rather simple lessons contain numerous traps that can be addressed by those MCQ.
  • Remi Daigle (University of Toronto) http://swcarpentry.github.io/training-course/2015/03/remi-daigle-mcq/
    • I didn’t know you could write a one-liner custom function without using curly brackets in R
    • I think the comments really highlighted the importance of maintaining consistent formatting between the MCQs and the lessons. It really minimizes potential confusion
    • It’s much easier to improve someone else’s questions than your own, it’s often hard to see your own mistakes
  • Dirk Eddelbuettel http://swcarpentry.github.io/training-course/2015/03/dirk-eddelbuettel-mcq/
    • MCQs are hard, and reasonably difficult to do well. But they also help to delineate subtle distinctions, so they are useful. Unclear, though, how they help us in the SWC classroom.
    • The comments were somewhat incisive, and I agreed with at least a good part of them – but they did not really shed too much new light on the issue.
    • A variety of different approaches and angles, and it helps to see them.
  • Derek Howard: http://swcarpentry.github.io/training-course/2015/03/derek-howard-mcq/
    • I learned how easy it is to confuse a potential student
    • Comments were helpful to minimize confusion for new learners who would be answering the question.
    • Coming up with relevant distractor answers which can help diagnose misconceptions is difficult
  • Andrew MacDonald
    • I learned that R functions do partial argument matching, which I didn’t know. However, after a bit of discussion in the comments I’m now of the opinion that we should not even be teaching that particular fact :P
    • I learned that writing multiple choice questions is REALLY RATHER HARD. The only person who attempted my question answered it incorrectly, despite her own understanding of the topic, because I presented it poorly. Also, one commenter pointed out that some of my answers aren’t really likely points of confusion for students, and are thus not too helpful.
    • From looking at other people’s questions, I think this is rather common. It’s really hard for an expert to imagine what a novice might think, and inevitably at least one of the distractors is an answer that would only be given by somebody who either wasn’t paying attention or who didn’t understand either math or English.
  • John Moreau (University of Missouri, Kansas-City) http://swcarpentry.github.io/training-course/2015/03/john-moreau-mcq/
    • I choose to write questions about different object types in R. As I developed my questions, I learned how many different concepts a student would need to really understand this idea.
    • I learned how my understanding of my topic relies on my implicit connections among different concepts. Several commenters pointed out that one of my questions was a bit longer and would be more difficult to do without expert knowledge or running the code in R. I could interpret the question and think through the right answer, however novice learners might not be able to do so.
    • It was interesting to see how people came at similar topics based on their experience. One person specifically included a distractor based on the differences between R and other languages. I thought that was a good consideration.
  • Martin Bentley
    • I need to actually consider what is the most important piece of understanding to take away from a section/course/question. This will affect what and how to ask things.
    • That sometimes more questions are not always better. A choice of two good questions might be better than 4 ok ones.
    • This is much harder than it seems. Also, to make the question clear is important, it is very easy to make something overly complicated, and just confuse people, rather than make them think about the question.

19:00 Eastern

  • Greg Wilson (Software Carpentry Foundation)
  • Kristopher Keipert (IAState)
    • The logic of the grep tags can be easily obsucated and produce unclear results
    • Most of the SWC curriculum on grep can be covered by evaluating useage of grep tags
    • It can be difficult to think of 3/4 alternate but feasible trains of thought which lead to incorrect answers.
  • Elizabeth Wickes (UIUC)
    • I remembered going through the same mistakes as a beginner and same anger of “why!?”
    • I learned that I need to put in more errors
    • It is hard to go back to the beginner mindset because we’ve all worked so hard to reframe our logical constructs to live and breath how we know that programming world to work.
  • Will Cornwell: http://swcarpentry.github.io/training-course/2015/03/will-cornwell-mcq/
    • Sorry I missed out. Turns out daylight savings time switches at different times in different countries, very confusing
    • I learned that scoping is best taught in multiple steps, especially the weird way that R does it
    • Learned that writing multiple choice question is an art, especially coming up with many plausible wrong answers
    • Good to think about the materials (esp help files) available for students when you give them MCQ’s
  • Richard Tomsett (OIST): http://swcarpentry.github.io/training-course/2015/03/richard-tomsett-mcq/
    • Apologies for missing this - forgot about daylight savings! Answer to Q: I learned that a simple SQL statement can be really quite powerful, and also very speicific - it was hard coming up with distractors I thought weren’t too obvious
    • I learned it can be difficult not to make MCQs leading. And about SQL, that a beginner’s problem is often understanding what you’ll actually get back from a SELECT query
    • People have very varied styles! I Also saw several different ways to go about asking (give a code snippet and ask what it does, or describe an outcome then give choices of code snippets that do it).
  • Brian Magill (SSAI): http://swcarpentry.github.io/training-course/2015/03/brian-magill-mcq/
    • From preparing my own MCQ’s I learned how difficult it is to come up with responses that discern different misconceptions students might have. The other issue was to avoid writing tricky questions that simply measure cleverness instead of assessing comprehension.
    • As of Tuesday no one had left any comments on my MCQ’s. (I did not check yesterday evening.)
    • One thing I found by looking at other people’s MCQ’s is that there is only so much content one can put in a single question and its corresponding responses. Sometimes you need more than one MCQ (to avoid student’s narrowing responses due to process of elimination.)
  • Jeremy Gray (U of Toronto): http://swcarpentry.github.io/training-course/2015/03/jeremy-gray/
    • already had a good understanding of scoping, but thinking of how people might get it wrong helped out a lot
    • People will not always know the surrounding functions you use - testing one thing might require extremely simplified versions of other things - specifically, c() and rm. “https everywhere” browser extension breaks disqus, and makes you think you got no comments.
    • Writing sensible mcqs is hard- you need to have a good understanding of exactly how your students will think.
  • Luke Johnston (U of Toronto) http://swcarpentry.github.io/training-course/2015/03/luke-johnston-mcq/
    • I learned more about how SQL deals with missingness… which is very different from how R or SAS deals with them. I also learned how useful a well designed MCQ is for getting information from the students!
    • I learned from the comment that it is hard to write a clear question
    • Writing a good MCQ is hard. Takes quite a bit of time and mental thought! Based on others MCQ, I learned that formatting (for mutli-line responses) for MCQ are very important. Plus, from reading others’ MCQs I really saw how useful they can be to diagnose problems, including my own misconceptions about a question!
  • Vladimir Sudilovsky (NASA/ADS) http://swcarpentry.github.io/training-course/2015/03/vladimir-sudilovsky-mcq/
    • How to properly construct a MCQ that tests “common” or expected misconceptions
    • That my assumption that the two questions should be read together (as one) was not necessarily valid
    • MCQs are not very easy to construct
  • Matthew Collins (U of Florida) http://swcarpentry.github.io/training-course/2015/03/matthew-collins-mcq/
    • Learned about topic: handling of “$@” in shell is intuitive (just works)
    • Learned from comments: Not sure
    • Learned about MCQs: That last option is hard to come up with. 1-2 distractors are easy. Writing a distractor that tests exactly one misconception is extra hard. 2 or more is much simpler.
  • Violet Zhang (UPenn)
    • I leant more about the correct usage of the the square brackets
    • I learnt that it is very hard to cover all the misconceptions at once, it is tricky to design in all possible mistakes
  • Kathy Chung (U of Toronto) https://swcarpentry.github.io/training-course/2015/03/kathy-chung-mcq/
    • learned the “structural” similarities and differences between different shell commands and functions; also how hard it is to write good diagnostic distractors. You need to really know your subject and think about how others might perceive it
    • alas, no comments
    • learned some ?erroneous? assumptions (about the subject and about the learning?) people made in composing their answers & distractors; also how “easy” it is to select the right question through elimination rather than actually understanding concepts being tested. The elimination process was based on knowledge but not the knowledge being tested. Also degree of complexity suitable or less suitable for a MCQ?
  • Darya Vanichkina (U of Queensland Australia) http://swcarpentry.github.io/training-course/2015/03/darya-vanichkina-mcq/
    • Whether people see learning concepts or only by doing as the optimal strategy for learning SWC material
    • Have never thought so much about two questions
  • Dorota Jarecka (U of Warsaw / NCAR) http://swcarpentry.github.io/training-course/2015/03/dorota-jarecka-mcq/
    • I’ve learned about possible misconceptions people might have and how easy is to not see them
    • I’m still not sure, how easy/hard they should be (mine look a bit too easy)
  • Horacio Vargas (Max Planck Institut)
  • Evgenij Belikov
    • Compiler/interpreter error messages are still relatively poor.
    • Too many options may be misleading or question could lose focus.(Not sure about value of allowing multiple correct answers. What is best number of options?)
    • I learned to focus on meaningful distractors and not choose random wrong answers. Feedback may help discover additional distractors.
  • Simon Fraser (Sanger Institute) Apologies for absence, will be at the next lesson.
  • Hugues Fontenelle (Oslo University Hospital, Norway) https://swcarpentry.github.io/training-course/2015/03/hugues-fontenelle-mcq/
    • I picked up a topic clear to me.. so clear it was difficult to create a good MCQ
    • n/a
    • Actually very hard to write a good one. What are the misconceptions? I couldn’t think of any at first, but actual teaching in the classroom helped me identify some.
  • Kyler Brown: https://swcarpentry.github.io/training-course/2015/03/kyler-brown-mcq/
  • Belinda Weaver The University of Queensland/QCIF http://swcarpentry.github.io/training-course/2015/03/belinda-weaver-mcq/
    • I learned that there is more than one right answer so you have to be careful how you phrase things or you can get unexpected results. Also it is hard to think about what misconceptions others might have when it seems so clear to you.
    • That I hadn’t thought it through enough - the comments were very helpful in highlighting that - I was focusing too much on formulating a question that could have four plausible answers instead of wondering whether the question was any good and would assess knowledge meaningfully!
    • That you have to be really clear on the question as any ambiguity will prevent people from answering correctly. And simple and short questions work better than superlong complicated ones - I found some MCQs too intimidating because they were so long
  • April Clyburne-Sherin (SickKids)
    • I learned that you need to prioritize what information you want before designing the questions, because your questions will determine what information you have to build upon.
    • Feedback (from others, not comments) helped me realize I could have received better information with different questions.
    • Looking at other people’s questions was really useful in showing the variety of approaches and types of information you could retrieve through MCQs. This will really helpful for the next time I need to create a test.
  • Victor Kwangchun Lee (xwMOOC, Korea) http://swcarpentry.github.io/training-course/2015/03/victor-lee-mcq/
    • I learned the real value of MCQs
    • Besides, carefully designing MCQs is also helpful to instructors. In particular, various MOOC courses from Udacity, edX and Coursera seems to have not enough distractors…
    • From comments, I found one mistake, I assumed learners already know terminology and jargons.
  • Ben Weinstein (Stony Brook University) - mine needed to be repulled its here: https://github.com/bw4sz/training-course/blob/gh-pages/_submissions/round-12/2/2015-03-10-Ben-Weinstein.md
    • my apologies, my .md had a bunch of html code that greg (sensibly) rejected.
    • still need to figure out how to just push .md that still keeps the figures, without having html index.If anyone is willing to let me know how they do this with R + knitr, let me know. (Ben, you can’t do that in Rmd. You need to push the figure folder as well.)
      • Yeah, that was what greg rejected, i’ll figure a different approach.Thanks! Np :)
    • Important to clarify the difference between a misconception and a mistake.
    • Trying to empathize with students and ‘get into their shoes’. Ask, what would i select here?

Notes:

  • Ways to administer MCQs
    • Hold up fingers
    • Use four coloured sticky notes.
    • Can use combinations.
    • Important that everyone vote at once.
  • Other Q&A stuff: (If not enough plausible wrong answers for an MCQ)
    • Can also flip, as them to supply why an answer is wrong.
    • (I’m not quite sold on this one. I feel like if you ask the room “why is this wrong?” then strong students will answer with confidence while struggling ones will just feel extra “behind”)
  • Pedagogical content knowledge (PCK)
  • Can start to pool our efforts, by working on what other people have done.
  • Single biggest factor in learning: intrinsic motivation
    • If they have a genuine desire to understand the topic, they will learn it
    • If only have extrinsic motivation (forced to do it by some external process/stimulus) then learning doesn’t happen or is shallow)
  • In SWC almost all attendees are there because they have chosen to be there - so vast majority intrinsically motivated. So SWC job is to avoid demotivating them
  • The major causes of demotivitation:
    • indifference - instructor doesn’t care that students are learning
      • Since most of us are volunteers, we usually do care
    • unfairness - students left behind
      • This is very possible, given that people come in with different levels of skill in what is being covered.
      • Set-up issues are also very likely to happen, which will demoralise people before they start.
  • If an instructor is pacing the class with the fastest/ most “interesting” students then this makes students who are at an earlier stage feel unwelcome or left behind.
  • How do you avoid alienating the people who are beginners while not boring people who know a little?
    • This is almost unavoidable. On a good day, we will have 1/3 bored and 1/3 lost.
    • Can try and turn advanced people into ad hoc helpers.
    • Pair people up, preferably with advanced/beginner to help them.
  • Beware demotivating intimidators:
    • dismissive words like “just” or “trivial”
    • too much jargon
    • using big array of tools
    • Saying that a concept ‘is’ easy - “this is easy, you should have no problem with this”
    • Saying that someone isn’t a computer person, or not a math person
  • If students see others being left behind they also become demotivated - so impacts on more than just the student directly affected
  • Four half-day sessions are better then 2 whole days and better then semester class for non-credit class.
  • Catching ambiguous language is one of the big benefits of working together on writing lessons
  • Show them that your questions are there to guide your teaching, not to evaluate them - disarm defensiveness
  • We re-use code, re-use science, so we should certainly re-use lesson plans and bits of lessons.
  • generating those MCQ is not really economical (they need a lot of time) - sharing them will help to save time
  • Not enough plausible distractors? Can switch to other types of questions, or ask them to explain why an answer is wrong Be aware of the topics/knowledge you test in parallel and that you might not be thinking of. (For example, may test addition and such in addition to assignment.)

Why do programming languages count from zero? http://exple.tive.org/blarg/2013/10/22/citation-needed/