Group 9 had its second online meeting yesterday, during which we discussed what people learned from creating concept maps, the differences between formative and summative assessment, the mis-use of testing in large organizations, and reverse instructional design. Notes are below; comments and corrections would be welcome.

Our next meeting will be on Wednesday, May 21. To prepare for it, each participant should do the following:

1. Choose a concept map created by someone else that describes something you think you could teach in five minutes or less.
2. Create two formative assessment exercises you could use after that 5-minute lesson to determine whether or not you had successfully managed to get the key ideas across. One of these must be a multiple-choice question (MCQ); the other can be a second MCQ, a short programming exercise, or anything else that:
   1. learners could do in 5 minutes or less, and
   2. you could assess quickly in a class of 40 people or more.

   Please post both of your assessments to the blog no later than Wednesday, May 14. Please do not post answers or solutions.</li> * Answer the questions or exercises posted by two other people no later than Tuesday, May 20. Post your answers as comments on their blog posts, and include any comments you have on the questions themselves: were they ambiguous? Are there multiple correct answers? Did they take too long? Etc. * Be sure to have read at least the first two chapters of How Learning Works by the 21st as well.</ol> We'll see you in two weeks…

   ---

   Questions:

   • What (if anything) did you learn about your topic, or how to teach it, from creating a concept map?
   • What did you see in other people's concept maps that was different from what you did or what you would have done (i.e., what did you learn about their thinking, as opposed to their topic, from their concept map)?
   • What (if anything) did you learn from the feedback people gave you?
   • Do you think concept mapping is more helpful for organizing your thoughts than point-form notes, about the same, or not as useful?

   And a request: please do get work up in the first week, so that people have a full week for comments. Much of the value of this class comes from peer feedback, and if you don't give your peers a chance to give feedback before our meeting, everyone gets less out of the exercise than they could.

   19:00 Eastern

   • Greg Wilson (Mozilla, Toronto): no concept map URL
   • Jacob Levernier (U. Oregon, USA): http://teaching.software-carpentry.org/2014/05/01/concept-map-sql-select-statements/
     • 1) Creating the concept map took longer than I thought it would at first — the difficulty was in deciding where (at which node) to start. Once I started, though, the mapping process flowed quickly. Also, it was helpful to make 2 iterations of the map.
     • 2) One of the maps (on Virtual Machines) used nodes-within-nodes in a really effective way to show subconcepts. I'd never seen that used before, but I like it — it adds an extra dimension to the map, allowing a more (literally) layered thought process.
     • 3) (Both from reading feedback for me and reading feedback on others' maps) There are cultural differences (even across different parts of the US and Canada) in how people give feedback. Also, consistency is important — lack of consistency can be distracting to viewers. I also learned a new SQL shortcut!
     • 4) I think that concept mapping is more useful for certain types of domains. I think that it's slower than using point-form notes, but much better for clarifying ideas are.
   • John Mosher: No map posted  (done late and having trouble getting it off my phone)
     • The concept map was harder than I thought and needed several revisions.
   • Simon Michnowicz. (Monash University, Australia) http://teaching.software-carpentry.org/?p=6942
     • I learnt that there were questions I did not consider as I thought it was obvious..
     • Others put a more information in..How cluttered/full should they be?
     • I learnt my map could be improved by small changes
     • Not sure on this one. But it did get me to focus on what I was trying to teach (concept vs facts)
   • Christian Jacobs (Imperial College London, UK): http://teaching.software-carpentry.org/2014/04/23/concept-map-lists-in-python/
     • 1) It made me appreciate the links between concepts, both within my concept map, and between my map and those of others. Concept maps can also give an "overview"/"big picture" view of a rather complex topic.
     • 2) Other people included concrete examples, and (visually) were less hierarchical than mine. The amount of abstraction also varied between the different concept maps.
     • 3) Describing concepts just by using 1 or 2 words is not always enough — sometimes concepts should be split up further (otherwise complexity can become hidden).
     • 4) Concept mapping is definitely helpful when organising thoughts because it makes one think about the relationships between concepts that might not be as easily expressible in point-based form.
   • Jeremiah Lant (Louisville, Kentucky, US): http://teaching.software-carpentry.org/2014/04/30/concept-map-for-python-dictionaries/
     • 1) I learned how to be clearer in explaining the main concepts and include the relationships between each concept.  I learned that focusing on the major concepts and there respective relationships is important. Learned that concept maps are helpful in viewing my knowledge organization.
     • 2) Some other concept maps were more organized in structure; more flow like where mine was more sprawling.  Others were more hierarchical.
     • 3) From the feedback, I learned to try to minimize the number of concepts covered.
     • 4) Concept mapping is more helpful for organizing my thoughts than point-form notes.
   • Devasena Inupakutika (University of Southampton, SSI, UK): http://teaching.software-carpentry.org/2014/05/06/concept-map-shell-paths/
     • 1) Presenting a computing topic in a clear and concise way correlating it with real-world (lay man terms) examples.
     • 2)  I used symbols at certain places which are more specific to Unix environment and windows users might not be able to make out and understand the concept properly. Certain concept maps were detailed and organised, covered all the possible scenarios.
     • 3) Feedback made me consider all the possibilities (e.g all the platforms windows/linux/MAC OSX) and procuring concept map accordingly.
     • 4) CMs are useful and helps in linking and gathering the facts we know about certain computing concept.
   • Dana Bauer (Rackspace, Philadelphia): I didn't finish it, but I started to create a concept map for the geocoding process
     • 1) I learned just how iterative my geocoding process is. I realized how many rounds of geocoding I do— at different geo levels— before I'm satisfied with the result. This exercise helped me really "see" my process. It also made me realize that most people who hate geocoding (even more) if they followed my process.
     • 2) I learned that not everyone considers color to be an important part of a concept map.
     • 3)
     • 4)
   • Matthew Dimmock (Monash University, Australia) http://teaching.software-carpentry.org/2014/04/28/concept-map-image-processing/
     • 1) Which aspects of knowledge I took for granted.
     • 2) Mine appears to be more of a flow chart.
     • 3) Not to imply stages of visualisation and interactivity.
     • How differently everyone approaches the same problem.
     • 4) More useful for organising my thoughts than a linear list.
   • Mark Wilber (UC Santa Barbara) http://teaching.software-carpentry.org/2014/04/30/for-loop-concept-map/#comments
     • I realized that I wanted to draw my concept map as a flow diagram and found it difficult not to do it this way
     • Other people's concept maps, at least the ones I was looking at, did a great job incorporating the "parts" of the concept they were describing without describing a step by step process. Again, I found it difficult to think this way but liked it when others could.
     • I learned that what I thought was confusing on my concept map was confusing to others as well! Tried to include too much contingency
     • About the same
   • Catalina (repeat from this morning — on a train between Toronto and Montreal .http://teaching.software-carpentry.org/2014/04/30/concept-map-function-in-r/
     • 1) Well, don't think I did it right. 2) Learned  from other maps: seeing more the big-picture concepts (whereas I tend  to teach by examples); how different it was from my own from the same topic. (Hi Tim!)
     • 3) From feedback: my thought process was not clear (why my arrows where the way they where)
     • 4) Both point form and concept maps are useful.   (Tend to go from point form to organizing them in concepts for myself.)
   • Yu-Ching Shih (at a loud env. National Taiwan University, Taipei, Taiwan): http://teaching.software-carpentry.org/2014/05/07/concept-map-how-awk-works/
     • 1) I learn how to organize my thought of the command awk.
     • 4)Yes. It's very useful.
   • Tim McNamara (New Zealand eScience Infrastructure, New Zealand)  http://teaching.software-carpentry.org/2014/05/06/concept-map-functions/
     • Functions are complex! There's lots of hidden detail in there. Lots of layers to go into — not sure where to stop. Very difficult to express "how"
     • I tended to spend more time on concepts, whereas some people were more literal. I tried to link concepts together, even if this made for a messier diagram.
     • Missing points
     • I really enjoyed the practice of exploring the whole idea.
   • Wesley GOI (National University of Singapore) http://teaching.software-carpentry.org/2014/05/07/concept-map-ssh-in-a-nutshell/
     • Too much to add into the map, started with a the simple wish to teach SSH but couldn't resist to add the rest of what I knew.
     • Best layout constantly changes and was difficult to determine as I drew the the concept map
     • Concept mapping can be difficult when things link out to something which is not relevant, have to be careful from veering off.
     • Difficult to memorise, feels like a very raw summary.
   • Dan Warren
     • 1) I think I saw some value in the nonlinearity of it.  I tend to think of things in a very sequential manner, so it was interesting to think of things in a less flowchart-y manner.
     • 2) Actually a lot of them did read as flowcharts rather than concept maps.  That was my initial approach as well, and it took me a bit to get my head around how a concept map differs from a flowchart.
     • 3) Concept maps can be useful for organizing topics and figuring out what you may have omitted from a lesson plan.
     • 4) About the same, I think I would find it harder to work out a specific plan of action from a concept map, but it could be better at illustrating the connections between ideas and coming up with new connections.

   14:00 Eastern

   • Greg Wilson (Mozilla, Toronto): no concept map URL
   • Dav Clark (UC Berkeley): http://teaching.software-carpentry.org/2014/05/02/benefits-and-challenges-of-vms/
     • All of the technical features of VMs were easy to organize under benefits vs. difficulties
     • I saw some hybrids of link meaning (e.g., temporal order vs. semantic categorization). Few folks used labels on links, but I think this would allow more clarity (labels were in the example in the reading)
     • Folks are attentive to the way they have addressed a particular challenge
     • I already tend to do this kind of thing a lot — whiteboarding, OmniGraffle. I think it's useful when I'm at an intermediate level of expertise. I'm curious to try it as a novice.
   • Chandler Wilkerson (Rice University): http://teaching.software-carpentry.org/2014/04/23/editing-with-vi/
     • I did learn the formal names for some of the concepts I was mapping
     • I found it different from the way my mind works to see others' maps with bubbles forming sentences, rather than displaying connections
     • I learned that as a self-styled expert in my subject that I can assume too much, and supply too little information in my attempt to be succinct.
     • I think the concept map is a useful drill-down technique for a bullet point in a point form list.
   • Michael Schliephake (KTH): http://teaching.software-carpentry.org/2014/04/30/concept-map-for-make/
     • Efficient way to get overview about a topic
     • Comparison of maps shows the many possible different approaches to a topic
     • Visualisation eases communication about a topic
     • Combination of concept map and bullet lists is most productive
   • Florian Rathgeber (Imperial College London, UK): http://teaching.software-carpentry.org/2014/05/01/concept-map-git-merge-and-rebase/
     • 1) Creating the map forced me to think about tools I used every day in a more conceptual manner
     • 2) Not labelling connections makes the map hard to understand. The way people organised information was rather different: some created a tree-like structure, others more of a circle
     • 3) The topic was considered too advanced by some, but the presentation was considered quite clear by others
     • 4) I found it mapped quite well to the way I think and I applied it again in another context to explain a different concept to a different group
   • Padraic Stack (Dublin, Ireland) http://teaching.software-carpentry.org/2014/05/07/concept-map-setting-up-git/
     • 1. I learned that I had quie a disorganised view of what went on, and I had to tease it out a little.
     • 1. It was clear that peoples' abstractions depended very much on their level of expertise — some took things for granted that I would have thought required explaining.
     • 1. The feedback I received was very useful — the commenter immediately identified a simple way to more clearly illustrate the concept.
     • 1. More useful — can display non linear concepts.
   • Bror Jonsson (Princeton University / Cape Town) http://teaching.software-carpentry.org/2014/05/01/concept-map-read-text-file
     • 1. Concepts and being complete is important.
     • 1. The level of abstraction.
     • 1. Nothing specifically, more that it's really useful to have a second pair of eyes looking at your stuff.
     • 1. Very useful. Especially if used interactively when teaching.
   • Stefan Pfenninger (Imperial College London, UK): http://teaching.software-carpentry.org/2014/05/01/concept-map-while-loop-2/
     • Terminology can hide complexity, concept map can help break out complex concepts into simpler constituent parts
     • Combination of abstract concepts with practical examples
     • Specific ways to improve my concept map
     • Yes, it seems so, maybe because it's an unfamiliar tool and forces clear thinking about the concept and how things link together
   • Isabel Fenton (Imperial College London, UK): http://teaching.software-carpentry.org/2014/05/07/concept-map-loops-2/
     • 1. it clarified my ideas giving me a chance to try to work out the relationships between different loops
     • 1. looking at some of the maps for topics I properly understand, there seem to be multiple ways of linking the same things — different people link things differently
     • 1. for certain topics (i.e. ones that have a series of connections) I think this is a useful way of sorting out thinking. For ideas that have a more linear structure, I don't think I would find concept maps as useful
   • Jon Duncan (UNC Chapel Hill, NC) http://teaching.software-carpentry.org/2014/05/01/concept-map-for-github/
     • Learned more about context vs. mechanics of workflows/programming
     • Learned from other maps- Adding brief sentences to arrows really helps convey goals
     • Learned from feedback- always good to get a new set of eyes- can catch assumptions that are easily made
     • Personally, with an n=1, I think point form notes are more useful, but I will certainly try again, perhaps with an issue that has more feedbacks, linkages among concepts
   • Genevieve Smith (UTexas, Austin) http://teaching.software-carpentry.org/2014/04/29/concept-map-drawing-figures-in-r/
     • From other people's maps I realized how helpful it would have been to label the arrows with informative or descriptive text
     • Making the map helped me realize how overly ambitious I was in the number of sub-topics I thought I could cover in 5 minutes. If I were to use these to develop lectures, I think I'd end up doing a lot of 'pruning' of my maps
     • I think the main advantage is the free-form structure, not getting trapped in a single order of topics.
   • Dureid El-Moghraby (University of Leeds, UK)
     • Excellent way to organise thoughts and refine topic concepts. Have started using it in one-to-one sessions.
     • Other people emphasized different aspects  from what I would have emphasized.
     • seemed better than taking point notes

   10:00 Eastern

   • Greg Wilson (Mozilla, Toronto): no concept map URL
   • Aur Saraf (Tel Aviv, Israel): http://teaching.software-carpentry.org/2014/04/23/concept-map-condition/
     • I didn't learn much about conditions. It must be said that I did choose an easy subject.
     • Many people did a concept map that was more of a flowchart. I got to think a lot about mixing multiple representations graphically. Also, one concept map I saw made it really clear that the UX of the tool he was teaching was (in my opinion) needlessly convoluted — it exposed a flaw in the material's learnability
     • I didn't get enough feedback
     • Concept maps require more work and give more value, so they're a tool to use in situations where you need the notes more.
   • Russell Alleen-Willems (Diachronic Design, Seattle, WA): http://teaching.software-carpentry.org/2014/04/30/concept-map-for-differences-between-relational-and-object-oriented-databases/
     • The map helped me notice how many different subtopics were under my general topic. The nodes also suggested places where I could split up the topic into separate lessons.
     • Some people focused on describing the relationship between ideas or objects while others made more of a flow chart of steps for a function or process.
     • The comments helped me identify that my topic was too large for the time limit of 5 minutes, and also where my concept map needed more fleshing out.
     • I think concept mapping would be a good first step for outlining a lesson plan. Afterwards, I think it will be much easier to fill out point-form notes or a script for each section of the concept map.
   • Dan MacLean (The Sainsbury Lab, Norwich, UK) http://teaching.software-carpentry.org/2014/04/30/concept-map-for-reading-from-a-file-with-ruby/
     • Not much, maybe that the thing isn't as linear as I imagined
     • Putting different things, e.g. verbs and nouns as different entities, using more 'flow' than I did. Its possible to learn more from 5 minutes than I thought.
     • That not all I think is commonly understood is…
     • I think it certainly helps to stop me from forcing a false narrative onto what I know
   • Graham Etherington (The Sainsbury Lab, Norwich, UK) http://teaching.software-carpentry.org/2014/04/29/concept-map-while-loop/
     • Made me think about the individual steps and how they were connected.
     • I though some were a bit complex for a 5 minute introduction to a novice.
     • Some got bogged down in the detail
     • More helpful. Provides a flow of ideas and how they're linked.
   • Mark Stillwell (Cranfield University, UK): http://teaching.software-carpentry.org/2014/04/30/concept-map-for-accessing-resources-through-ssh-gateways/
     • I don't think I learned any more about the topic, but it did help me to organize some of the related concepts in a way that would help me create a lessor or blog post later.
     • Other people had somewhat more detailed concept maps, or mixed in ideas from other types of network graphs, which had mixed success: sometimes the concept maps became more successful, other times they started looking kind of cluttered/confusing/ill-formed
     • The point about labeling the connections with relationship is a good one.
     • I think concept maps are interesting, and probably more flexible than point-notes because they free us from the linear narrative organizational style imposed by text. I'd like to work on this some more.
   • Alex Simperler (Imperial College, UK):http://teaching.software-carpentry.org/2014/04/27/my-concept-map-for-teaching-batch-submission-scripts/
     • I learned that I never learned concepts in Software
     • They think like software experts/trained software people/engineers
     • I permanently think in terms of how do I teach this?
     • I think with a concept map map I will be better able to understand the topic I am teaching and find better solutions.
   • Timothy Warren (University of Washington, Seattle, WA) http://teaching.software-carpentry.org/2014/05/01/list-comprehension-concept-map/
     • I learned about value of emphasizing incremental increases in knowledge/abstraction with terms like 'needs', 'is used for', 'does'. It forces instructor to move in small steps.
     • In general, I was impressed with the way many concept maps lived comfortably in a relatively abstract space, devoid of syntax/examples — which I think is quite useful to orient students before worrying about details.
     • Learned to focus more on clarity
     • At this point I think I would tend to employ concept maps if I wanted to communicate a 'concept map' (i.e. be very explicit about the way information is connected). Less certain that I would use a "covert" one for my own notes for communicating information without referring to the concept map.
   • Hsingtzu Wu (JAEA, Japan) http://teaching.software-carpentry.org/2014/04/30/concept-map-for-an-if-loop/
     • concept mapping may not fit in all materials. i think a flow diagram may fit my topic better, which means that I should have picked a better topic.
     • others' concept maps teach the idea, but mine teaches the steps.
     • yes, i did not think that much. it is nice to have feedback.
     • concept mapping helps organize ideas.
   • Shyam Rallapalli (The Sainsbury Lab, Norwich, UK) http://teaching.software-carpentry.org/2014/04/30/concept-map-for-regular-expressions/
     • I have the content in mind but with my map as pointed out from feedback, was good as a teaching aid
     • Most maps i saw described the flow, while mine tried to show the sub-contents
     • Feeback help me to stream line my teaching points and emphasise
     • Agreed Concept maps does help to organize teaching content
   • Scott Burns (Vanderbilt University, Nashville, TN) http://teaching.software-carpentry.org/2014/04/30/concept-map-for-unix-rsync
     • Short form text explaining edges between concepts is invaluable. That relationship is obvious to the author, but much less so to the reader.
     • More connected maps seem more valuable. Many nodes can overwhelm, while many edges just serve to reinforce the nodes place within the structure.
     • Names of nodes mean a lot because they carry technical denotations & connotations the readers bring. If the denotations != connotations, there can be an "impedance mismatch".
     • Point-form notes are implicilty like Fig2.2.b from HLW. They imply simple relationships between concepts.
     • A potential issue (related to Chp 2) is that diagramming "flows" (ie multiple organizations) through the concept map is difficult with just the map alone. There's a difference between the map itself and how one should move through it.
   • Jeff Hollister (US EPA, Narragansett, RI) http://teaching.software-carpentry.org/2014/04/23/concept-map-basic-syntax-of-an-r-function/
     • R syntax is something I have been working with daily, for years.  It made me actually go back through it and learn real names, formal definitions, etc.
     • I have always had the default stance to connect all the concepts.  Others didn't do this.  I think both are useful.
     • The value of examples!  One of the suggestions was to add one (even though it may not be an "official" part of a concept map) and I think it helped orient the concept map much better.
     • I don't think it is more or less useful.  I have since tried it (for an ignite talk on git and github) and found that it helped me get the bulk of the content out quickly.  The point form part of this helped with the organization of the basic concepts.
   • Huayan Gao (CUHK, Hong Kong)
     • will do! It actually takes me a while to think about an appropriate concept map.

   Greg: How do you tell which concepts to put into a map?

   • If you feel that the concepts are at the same level in your head, put them on the same level
   • If you feel that the concepts are on a "different" or sublevel, they probably belong in a different map

   Do you think it did a better job of making you think about individual steps than point-form notes?

   • Graham
     • Helped me focus on general subjects rather than getting bogged down into too much detail
   • Greg
     • Do you normally sketch or write when you're taking notes?
       • Graham — I'm mostly a writer
     • I have noticed that people who think of themselves as writers, not drawers, seem to get more out of the concept mapping exercise.
     • Could be confirmation bias, but it seems like writers notice and value the new experience moreso than those already used to thinking through drawing

   Sometimes people mix flow charts and decision trees into their concept maps

   • Greg — Did you find it useful to get the idea down on paper in this way?
   • Greg — Exercise idea. Suggest doing concept maps together with other colleagues before a lesson or presentation
     • Then together, you can individually turn them into a linear list. Rarely have I seen individuals come up with the same list.

   Exercise idea — for the next presentation you do — sit down with colleagues and first produce concept map. Then individually come up with order for narrative of presentation. How do each colleagues differ?!

   Differences in the maps you come up with unlikely to arise because of disagreements about content, but different opinions on order

   Greg asked Scott Burns about "denotations vs connotations": name of concept/node has technical definition but also has added connotations as to which the reader understands the word to mean.

   Jeff Hollister — Found concept mapping to be a great way to outline slides for an "Ignite talk" (= 20 slides in 5 minutes, on a timer so that the slides advance automatically every 15 seconds)

   Greg: If anyone has other ways of organizing their ideas that they find useful, please write a blog post for the SW Teaching Blog

   Formative assessment is online feedback of what you're doing well & poorly so as to focus effort. Helps learning, implements the feedback loop.

   In large institutions morphs into summative assessment. Is this group of people ready to move on? X teacher better than Y teacher?

   Push toward national testing in educational systems is an example of how this goes wrong. As soon as performance on these tests becomes linked to reward, then the system gets "gamed" and people "teach to the test".  Example: If publication counts are tied to a reward, there is incentive to "salami" cut your research into small pieces and send out to journals with lower standards in order to get as many publications as possible.

   Summative assessment is necessary, individual things need to be compared and sorted but misapplications of the metrics lead to failure.

   Summative assessment is an assessment at a particular time: Has the student passed a particular requirement?

   Teaching to the test arises when what is meant to be formative assessment becomes taken as summative

   If you write the tests first, then the code, you usually get better code faster (described in Greg & Andy Oram's book "Making Software": http://shop.oreilly.com/product/9780596808303.do)

   Equivalent of test-driven development in education is Reverse Instructional Design.

   Reverse Instructional Design — Normally when people design a course they begin by saying what do we need to teach? A list. Then come up with exercises, then a test based on what was taught. This is backward to what you should do:

   1. Come up with assessment first, (if you don't know how to assess them you aren't ready to start teaching them)
   2. Figure out what they need to practice before that assessment so they can pass it: what will you then have them do as warm up
   3. What will you now teach them so they can do those exercises?

   exam= summative assessment, exercises are formative assessment.

   4-5 laps of assessment and talk per hour

   Compare with coding practice Test Driven Development , write a test, make the code pass — this process is TDD for teaching!!

   recommended book: Seeing Like a State

   Productivity X Uniformity (local variation eg. agriculture)

   Ref: Owning property in russia.

   Pick a topic you think you can teach in 5 minutes.

   Come up with 2 multiple choice questions or 1 longer exercise question that students can do in 5 minutes that will tell you whether you managed to teach them.

   Try someone else's concept map. How will you know whether your learners grasped the concepts you were trying to teach.

   This process can seem to slow your progress, but it also means that you find it hard to leave people behind as checking at very regular intervals means that you ensure more people are brought up to speed. Feedback is important to make sure that you are getting the ideas across and students will learn more.

   Additional benefit: students will feel like you are taking an interest in their progress

   A good Multiple Choice Question has about 4 answers — 1 good, 3 plausible bad, none obviously wrong.

   Avoid MCQs that rely entirely on memorization/recall

   If the question is "open ended", checking it should be scalable

   a programming exercise could have one correct output even if it has many correct solutions ("What is the sum of the numbers 1, 2, 3, …, 1000″)

   Should be able to solve in 5 minutes, don't want lots of wrong answers because of lack of time

   due wed next week

   Influential paper: A Rational Design Process: Why and How to Fake It.

   http://software-carpentry.org/blog/2013/05/rational-computing-process.html

   • Regardless of whether you followed a rational process, it must appear as though you followed a process so that others can understand what you did.

   Interesting tool for real-time assesment during a class: https://github.com/cjlee112/socraticqs