Final Writeup

Summary:
The question my report set out to answer was "can some of the core concepts of parallelism be taught to people with no programming experience?"  I tested two populations, one consisting of CMU students and one consisting of people from the internet via Mechanical Turk, and found moderately positive results.

Approach:

Overall, I tried to teach the following topics, using a very analogy-heavy lesson to make it easy to understand from a non-programming perspective:

Speedup
Challenges of Parallelism:
   Communication Overhead
   Workload Imbalance (*)
   Inherently Sequential Portions of a Task
Pipelining
Deadlock/Livelock(*)/Starvation

(* = added in Phase Two)

Phase One:

In Phase One, I experimented with CMU students.  I had two populations, one who had programming experience but no systems background, and one who had little or no programming experience.  My methodology was to give them a 5 minute quiz, then teach a 5 minute lesson on parallelism, and then give the same quiz again.

Slides for my lesson: https://docs.google.com/presentation/d/1s54jTbYw8fX2KDjsAzvpwQ2ukfGhxhMLBpuGXC0a0mA/edit#slide=id.p  (these were not the focus of the lesson, just a visual aide, so they were pretty hastily done and mostly taken from 418 slides when those demonstrated a helpful analogy)

My quiz (this stayed the same for Phase Two): https://drive.google.com/file/d/0B_D9EQm0S0FPRjJKMUN2eFVhWEU/edit?usp=sharing

My script (this was mostly a guideline, I played it by ear so if people were getting a topic, I wouldn't go through every single detail of my script): https://drive.google.com/file/d/0B_D9EQm0S0FPanZZMTFuZDFhZ0k/edit?usp=sharing

Phase One was pretty helpful in developing my lesson, but it was inherently limited in the number of people I could test the lesson on, since they had to know me well enough that they'd be willing to take 15 minutes of their time on a busy week in exchange for just candy to help me.

I also observed that even among the non-programmers, my lesson was feeling a little light on content (which is why I added additional topics going to Phase 2 and fleshed out the existing ones a little more).  Another note was that all the people with programming experience successfully got the pipelining question right on their first try, compared to half of the non-programmers, and generally did much better.  My quiz tests parallel thinking more so than explicit memorization of concepts or anything programming-intensive, so it makes sense that even without systems background, people more used to problem solving would do better.

The sample was also a little un-diverse in the sense that even non-programmers at CMU pick up a lot of that way of thought from the prevalent culture and often the programming gen-ed that even humanities majors take.  To really test non-programmer performance, I would need to seek (*ominous drumroll*)...the internet.

Phase Two:

In Phase Two, I used Mechanical Turk to experiment.  I had participants take the quiz, then watch a Youtube video, then take the quiz again (they got paid 25 cents each).  There were multiple safeguards to make sure they at least attempted to learn - I said three letters at different points in the Youtube video that they needed to copy down to get paid for the task, and the place to enter that was before the second take of the quiz, so they at least had to listen to the video, if not understand it, to get credit for the task.

Youtube link: https://www.youtube.com/watch?v=_xqHr7ZxOuk

Results:

Overall, both groups seemed to improve.  There were a lot of 'eccentric' answers to the Mechanical Turk quiz, although only accepting correct confirmation codes filtered the most insane gibberish out.  I had 8 answers to the in-person version, and 32 at least somewhat reasonable answers to the Mechanical Turk version.

Some simple measurements:

6/8 before lesson, 8/8 after lesson of CMU sample got a basic pipelining question right (part c of the Cake question), versus 9/32 before lesson vs. 11/32 after of the MTurk sample.

7/8 both before and after lesson of CMU sample got a basic question testing how completely sequential portions of the work affect its parallelism right (part a of the Cake question).  MTurk results for that: 16/32 correct before lesson, 21/32 correct after lesson.

For the most thorough measurement, we need to compare the quiz averages (out of 10) before and after:

CMU pre-average (programmer): 8
CMU post-average (programmer): 8.8
CMU pre-average (non-programmer): 3.67
CMU post-average (non-programmer): 7.67
MTurk pre-average: 3.13
MTurk post-average: 3.26

Statistics aside, it definitely seemed like both populations learned at least a little from the lesson.  Many of the MTurk second-attempt quizzes referenced things like the cost of coordination and terms like deadlock, though not always correctly, and on the whole there was an improvement.

There are many, many future directions such a project could be taken.  Expanding the project to try to test the lesson on specific groups like younger kids, or the elderly, could be interesting.  Making a much more animated video could also be interesting since my current one is basically all spoken with a few visual aids.  The challenge in getting results is finding people who don't have much experience with these topics/ways of thinking (problem with the CMU sample) but are motivated to try their hardest to learn (problem with the MTurk sample).

Presentation: https://docs.google.com/presentation/d/1Or9wFOS6b4QOBKRL381jhkeFpaZpfGWa26eaY89idUE/edit?usp=sharing

Fun class, fun project!

Initial Results

In the end, I ended up running my in-person lesson on about 10 people, with the quiz before and after to test their progress.

I also got a video version of my lesson up and running, and then got people on Mechanical Turk to do the process of "take quiz, watch lesson, take quiz again".  (Using a confirmation code to make sure they actually watched the video)  Currently I have about 50 results from this.

I haven't fully analyzed the results from the quizzes yet, but it seems like overall, people improved in their understanding of the concepts and scored better the second time.

Progress Update

I've run my study on 6 people so far.  So far, most of my sample has been people who know programming but not parallelism concepts, so I'm going to look at the results and think about how to improve my lesson plan for that audience.  I'm also going to look for more people from the non-programmer camp, since I'm currently having trouble finding them compared to the first camp.

The other major direction I'm going to experiment with is trying to make a video with my lecture, post it online, and use Mechanical Turk to get a much larger audience than the convenience sample my real-life results are.