Learning from Research

On Class Debates

2010-11-27T13:07:00.000-08:00

I first came across using debates in the class room to stimulate reflection over technical content in the class Robotics: Science and Systems taught at MIT by Daniela Rus, Seth Teller, Nick Roy and others. In these debates, student teams prepare pro and contra arguments for a statement of current technical or societal concern.

I adopted the debate concept in my Introduction to Robotics class, which is about to conclude its second iteration, and proposed the following debates to teams of 3-4 students:

D1: Robots putting humans out of work is a risk that needs to be mitigated.
D2: Robots should not have the capability to autonomously discharge weapons / drive around in cities (autonomous cars).
D3: Robots do not need to be as cognitive as humans in order to be useful as making the environment intelligent is sufficient.
D4: Robots need to be made differently than from links, joints, and gears in order to reach the agility of people

As the debate statements have lead to ambiguous interpretations in Spring (leading to student groups both defending a "Pro" position), I provided explicit formulations of each statement, starting the sentence with a "Yes" or "No" and including "does"/"does not" where appropriate.

The students are instructed to make as much use as possible of technical arguments that are grounded in the course materials and additional literature researched by the students. For example, students can use the inherent uncertainty of sensors to argue for or against enabling robots to use deadly weapons. Similarly, students get the ability to relate the importance and impact of current developments in robotics to earlier inventions that led to industrialization when considering the risk of robots putting humans out of labor. For instance, students can argue by relating robotic innovation to previous innovations in factory automation and their impact on society, stimulating far reaching and deep discussions. In almost all of the cases, rebuttals and discussions following the Pro and Contra statements led to the emergence of a consensus among the students and a differentiated position.

After a positive first experience in Spring 2010, I surveyed the students of the Fall 2010 iteration of the class. Unlike the MIT class, which consists of multiple weeks of debates, I propose four debate topics during 2 weeks at the end of the class, serving as a capstone experience that complements the presentation of the course project and a written exam.

Overall, I consider the resulting debates a success. Some of the student teams did a great job in pulling example robot systems from the literature to support their argument, and presenting their work also gave those students that are weaker on theory an opportunity to shine. Despite extensive research by some teams, very few of the students used explicit technical arguments. For instance, a strong arguments against the statement D4 is that the ever increasing sampling rate of sensors and computational power has led to manipulators that can simulate any desired dynamics, for example to behave like a spring-mass, using high-speed sampling and force control. Although the students tended to show systems that take advantage of these developments, e.g. "Rollin' Justin" or ''Big Dog'' showed in lecture, they did not identify the underlying technology that enables them. This requirement should be made more explicit in further iterations of the class.

The quantitative perception of the students to the debate format have been generally positive. I asked the students to respond to a series of statements using the terms "Strongly disagree", "Disagree", "Neutral", "Agree" and "Strongly Agree" using Surveymonkey.com. Two solicitations via email have led to a response rate of 18/26 or around 70%.

This is the data in the order the questions where asked.

	strongly disagree	disagree	neutral	agree	strongly agree	Response Count
help me to improve my presentation skills	0.0% (0)	5.9% (1)	41.2% (7)	47.1% (8)	5.9% (1)	17
prepare me for questions that engineers face from society	0.0% (0)	16.7% (3)	11.1% (2)	55.6% (10)	16.7% (3)	18
fundamentally changed my opinion on a topic	11.8% (2)	23.5% (4)	52.9% (9)	11.8% (2)	0.0% (0)	17
are relaxing	0.0% (0)	27.8% (5)	27.8% (5)	38.9% (7)	5.6% (1)	18
let me better understand the technical content of the class	11.1% (2)	16.7% (3)	44.4% (8)	27.8% (5)	0.0% (0)	18
should be part of every class / introduced early in the curriculum	0.0% (0)	29.4% (5)	41.2% (7)	23.5% (4)	5.9% (1)	17
should be replaced by a more in-depth treatment of the technical content	5.9% (1)	35.3% (6)	35.3% (6)	11.8% (2)	11.8% (2)	17
should take less time	11.1% (2)	27.8% (5)	44.4% (8)	5.6% (1)	11.1% (2)	18
should allow for more discussion	0.0% (0)	11.1% (2)	27.8% (5)	55.6% (10)	5.6% (1)	18
reflect up-to-date issues in research and society	0.0% (0)	11.8% (2)	17.6% (3)	29.4% (5)	41.2% (7)	17

The questions were targeted at shedding light on three specific aspects: relevance of debates for the engineering profession, learning experience, and overal format.

While there is agreement that debates help to prepare for the engineering profession by improving presentation skills, prepare engineers to think about questions posed by society, and reflecting up-to-date topics, the debates seem to have little effect on changing the student's actual opinions on a topic (only 2 students say so). Students are also kind of indecisive about whether the debates actually helped them to better understand the technical content of the class.

Yet, students find the debate concept reasonably important to keep it over a more in-depth treatment of the technical content of the class, and disagree that debates should be devoted less time in class. However, the students are indecisive, whether debates are important enough to merit early consideration in the curriculum or should be part of every class.

Concering the overall format, students find that discussion came short in the 75 minute lectures with 2 debates each. With 10 minutes per position, this left around 15 minutes discussion time per debate, although including rebuttals. Also, students tend to agree that debates are an opportunity to decompress ("relaxing"), which is desired as this period of class coincides with wrapping up the course project.

I feel that debates are an important learning tool that provides complimentary skills to theory and exercise work. It is unclear, whether debates make sense in every class, but I feel the format to be particularly adapt to classes with broader scope such as robotics or AI. Having to present in class, hearing and arguing about a position completely opposite to one's own, and thinking about the broader impact of science and technology are important skills that should be encouraged early on. These benefits are understand as such by a majority of the students, who had little expectations to the format at the beginning of the class, which is exemplified in the following comment:

"I was not excited about the idea of a debate when I first heard about it. However, once we got going with the debates, I found it to be fun and interesting. It's great to hear compelling arguments on these issues that could go either way. I really enjoyed the discussion within the class as well. This was also a great opportunity for me to practice my presentation skills."

A drawback of the debates in their current form is the lack of technical depth, which might make them prohibitive in a course with already compressed content, where lack of time and the rather limited contribution of the debates to understanding of the course work, make the debates prohibitive. I believe this can be partly remedied by adding more structure to the assignment by providing 1 or 2 key papers for each topic that provide technical content to support one or the other argument as well as pointing out relevant lecture slides that debaters should recall to the class to support a specific argument. In order to reserve more time for discussion, I plan to forgo the rebuttal phase in the next iteration of the class in favor of a cross-examination of the presenters by the class.

Videos from the debates are available via CAETE:

D1 and D2, November 8, 2010
D3 and D4, November 15, 2010

Symmetry breaking in the class room

2010-11-27T11:12:00.000-08:00

In a dialog between Profs. Patrick Henry Winston and Gerry Jay Sussman on Winston's blog entry on Slice of MIT, Sussman proposes that the first day of classes is determining the mood of the class in the lectures to come. He argues that all students like to be like everyone else and collectively tune in (or not) into the lecture.

"The students don’t realize it, but they all want to be like everyone else, so on the first day of class, they are all sensing the overall mood. Within a few minutes, the symmetry breaks spontaneously, and the class falls into a fixed state."

He therefore suggests to bring graduate students and UROP students to the first lecture to nod approvingly and laugh about jokes to increase the likelihood that the student collective falls into a positive mood. I know from my own experience that the first few minutes of a speech (both at conferences and private venues) will determine the reception of the audience; the observation that this phenomena can extend over an entire class period is intriguing.

Although the article does not discuss the benefits of a positive atmosphere in class - at first sight there should be no correlation between reaching the learning goals and whether the students think the instructor's jokes are indeed funny - I believe that this positive atmosphere will contribute to lower the threshold of students actually asking questions, being attentive, and consequently improve their learning experience.

On assessment of students in mixed graduate/undergraduate classes

2010-10-03T14:33:00.000-07:00

Ten percent of the grade in my class "Introduction to Robotics" is based on reading exercises and filling out a 3-4 question online questionnaire before each lecture. My goal with posing questions before the lecture is to a) stimulate students to actively participate in the class based on questions that arose during reading, b) provide a positive experience for those students whose understanding is enabled by the lecture and c) provide an incentive to actively study for the exam during the entire period of the class. Designing questionnaires that meet these goals is not easy, however. If questions are too hard, students are frustrated. If the questions are too easy, students are lured not to pay attention during lectures. Finally, it is hard to make the questions the right level for everybody. In fact, the questions are perceived as being "ambiguous", "taking too much time", and interestingly, as "poorly correlated" to the textbook content by some of the students. I believe this criticism to stem from the fact that the questions are explicitly designed to 1) differentiate among the students and 2) to be difficult to be answered just from skimming through the text. Yet, what is the right level of difficulty to reach these goals and yet providing positive feedback also to those students that struggle with the questions?

The questions usually follow one of the following schemes:

1) In order to do X, you have to
a) Do A, B, C and then D
b) Do A, C, D and then B
c) Do A, B, and then D

2) When it rains outside, you need to
a) take an umbrella
b) the probability of rain is correlated to the probability of thunder
c) wear a bathing suit

While the first scheme is geared towards understanding of algorithms and systems and requires careful reading of the options and matching them to language in the textbook, the second scheme tests understanding of concepts and is probably mostly responsible for confusion. Indeed, answer (2c) is not really wrong, it just makes a lot less (common) sense than answer (2a), however. Also, option (2b) is a favorite choice of some students as it's content is correct and it looks like the "smartest" answer in the pool. I find the second scheme to be particularly attractive as it requires the students not only to think about the question itself but also about its broader context. Yet, the risk is that the question differentiates too strongly for language skills and attentive reading rather than technical knowledge.

In order to understand whether the questions are indeed too hard, I decided to look at the distribution of the performance after the first 4 weeks of class. The following rules of thumb come to mind:

If the distribution is skewed toward the lower end, i.e. everybody does bad, I probably did something wrong.
If the distribution is slightly skewed toward the upper end, the questions are just right.
If the distribution is strongly skewed toward the upper end, the questions are too easy, do not allow to differentiate among the students, and frustrate the top students.

Indeed, if (1) is the case, questions are just too hard for everyone and will not help in differentiating the students. This is analogous for (3) with too easy questions.

Notice that the students have to answer the questions solely based on reading before the material is actually taught in the lecture. In fact, I consider it desirable if the distribution is skewed toward the upper end after lecture and accompanying exercise as this suggests that a broad student population actually has been reached. The data for all 27 students for the first four week of exercise is shown below. Each exercise is worth 12 points (48 points max).

The data is organized into 9 bins (from 43-48 points, 37-42 etc.). Not submitted assignment are counted as 0. Two students have not submitted any assignments and are therefore not considered in the analysis that follows.

At first glance, the data has a mean of 26.3 and a standard deviation of 13.3. Thus the mean is slightly above half of the possible score (24) and the data strongly differentiates the students. If the data were normal-distributed, however, 50% of the students would score less than half of the points, which would be highly undesirable. Indeed, the distribution is not actually Gaussian, but seems to be bi-modal. Around half of the students in the class are graduate students (MS or PhD program). Their previous training and the fact that MS and PhD students are usually strongly selected upon admission might make them score systematically better than undergraduates. Looking at the overall performance of students in the course (here from the previous iteration of the class) classified into graduate (N=9) and undergraduate students (N=13) leads to the following plot:

Both distributions are clearly "long-tail" and skewed slightly toward full score. They suggest that graduate students perform in average better in class. If this is the case, however, different assessments for both groups should be chosen to reach the same satisfaction in the class for undergraduates and graduate students.

In conclusion, the preliminary data from the first four weeks of class confirms that the testing methodology currently used in class is sufficient to differentiate among a pool of students with strongly varying backgrounds, such as it is the case in a class that is taken by both graduate and undergraduate students. However, as graduate students systematically perform better than undergraduates in this particular course, the assessment methodology currently in use might lead to frustration of the undergraduate population that has to think about questions that are geared to differentiate among graduate students. This frustration is counter-productive as it generates unnecessary pressure and anxiety.

There are three possible solutions: (1) making the questions easier, (2) better communicating the actual goals of this particular assessment, i.e. preparation for class participation as opposed to evaluation of learning goals, and (3) splitting the class into different offerings for graduate and undergraduate students.

Making the questions easier while maintaining the ability to use the test for student differentiation could be easily achieved by adding a set of questions that can be answered by everybody. The drawback of this approach is that also the better students skip on the hard questions as they think they already scored sufficient questions and the lecture might be perceived as pointless as it is unclear what remains to be learned. Also, communicating the particular goals of a testing methodology is hard as students might already be sufficiently frustrated. Thus, the option to split the class into two separate offerings for graduates and undergraduates emerges as the best solution. This insight is corroborated by the fact that some of the undergraduates that dropped the class articulated the concern that "there are too many graduate students in this class". Splitting the classes would compromise possible benefits from student-to-student learning, e.g. during mixed undergraduate-graduate student project work. For these reasons, I will organize future iterations of this class as separate offerings for the 3rd and the 5th year (the latter being "grad-level" classes), including different assessments but with a set of common lectures, laboratory exercises and projects.

Peer-to-Peer Learning

2010-10-03T12:22:00.000-07:00

Results on "peer-to-peer learning" that I posted below have been published in the following paper

N. Correll and D. Rus. Peer-to-Peer Learning in Robotics Education: Lessons from a Challenge Project Class. ASEE Computers in Education Journal. Special Issue on Novel Approaches in Robotics Education. 1(3):60-66, 2010. [preprint]

Robotic Gardening - Project-based engineering education

2009-11-24T11:48:00.000-08:00

Project-based learning gives students the opportunity to immediately put the course materials into action. Also, projects provide a tremendous opportunity for the students to learn from each other. The class "Building a Robot Garden" was taught by Prof. Daniela Rus and myself in Fall 2008 at MIT as a follow-up to "RSS: Robotics Science and Systems", an introductory class in robotics. Our goal was to actively engage students into research by letting them design a distributed robotic system that autonomously waters and harvests tomato plants in CSAIL. We chose this task as it combines navigation, coordination, image recognition, manipulation and networking while imposing "real-world" challenges due to the interaction with real plants instead of objects modified for their interaction with robots. After the class was over and a common paper was published, I polled the class using surveymonkey.com [1] to learn more about the individual learning experience and the efficiency of the different learning modules.

Class Organization
After an introductory lecture and a brief poll questioning their individual preferences, the 12 students were divided into 6 groups addressing the following technical aspects of the problem:

- System architecture: how to connect individual software modules written in different languages? What code organization do we need?
- Navigation: how can the robot move from A to B on the garden platform hosting the tomato plants? How can we avoid collisions between robots?
- Image recognition: how can the robot recognize red and green tomatoes to inventory a plant and harvest tomatoes?
- Visual servoing: how do the joint positions of our four degree of freedom arm relate to the position of a tomatoe that we would like to grasp in an image captured from a camera on the arm?
- Inverse kinematics: how do we control the joint positions of our arm in order to reach arbitrary position in six degrees of freedom (x,y,z,pitch,yaw and roll)? Which positions can we not reach?
- Networking: how can we exchange information wirelessly between robots and embedded devices monitoring the humidity of each pot?

The class contained the following learning modules: ex-cathedra lectures to teach concepts and theory (1h a week), design reviews where the students presented and discussed their progress and next steps with teachers and peers (1h a week), as well as weekly meetings in the garden (1h a week) to teach individual concepts and address problems with hardware, software and algorithm design.

Student Background
Of the 9 male and 3 female students, 10 where in a BS program and 2 in a MS program. 7 of the students studied computer science, 2 were enrolled in both computer science and electrical engineering, 2 in Mechanical Engineering, and 1 student in Aero- and Astronautical Engineering. 9 of the students stated that they planned on attending graduate school, one female student indicated that the course has motivated her to do so, and two students did not change their plans to not attend graduate school after the class.

Technical Content

We asked the students on which topic they have been working, and how well they understand the other technical aspects.

Figure 1: Technical understanding of course modules

It turned out that almost all students perceived their involvement to go beyond their assigned task as they regularly answered "I was working on component X myself" for more than one component. More than half of the class had a solid understanding of the overall system architecture, navigation and visual servoing, but only 20% of the students claimed this for networking.
We believe the high confidence for some technical aspects to be due to the fact that students had significant previous experience with robotics, in particular in the course of the class "Robotics: Science and Systems" that covered system architecture, navigation and visual servoing. (4 students had both course work and practical experience, 4 students did only take courses, and 2 students had only practical experience due to competitions or internships. Only two students didn´t have any previous experience with their project component.) 60% of the students claim to have had only a vague or basic understanding of inverse kinematics and manipulation. We believe this to be an artifact of the fact that one of the students implemented inverse kinematics using a robotics software suite that he had previous experience with, but which has not been introduced during the class.
We observe the lowest ratio of peer-to-peer learning for networking and coordination (around 75% of the class indicate a "vague" understanding). Although networking and coordination were interacting with almost everybody else's modules, interaction between modules was abstracted by an inter-process communication framework and an understanding of the underlying aspects of ad-hoc networking were not necessary for most of the students. Also, while some of the modules required strong mutual understanding of their inner workings, such as visual servoing and inverse kinematics, the actual coordination algorithms were of little importance for students implementing the individual tasks. Finally, as the students needed to overcome various challenges in navigation, perception and manipulation, actual coordination could only be implemented during the very last days of the class, limiting their exposure to the rest of the class.

Efficieny of individual course modules
Learning from the team partner and peers working on other projects of the class, was valued high ("I learned a lot") by more than 50% of the students (Figure 2).

Figure 2: Perceived efficiency of learning modules.

Due to the different backgrounds of students, interaction with the team partner has also seen the highest variance in individual perception (from "I learned nothing" to "I learned a lot"). While all of the students agree that they learned a good deal ("learned something" and "learned a lot") from independently working on their project, only 60% of the students have this opinion on the lecture and 40% of the students reported that they learned "little" during this time. This is also the case for the design reviews - students presenting their progress and ideas in front of the class - received a "I learned a lot" from only two students.
We also asked students, whether they relied on literature not distributed during class in order to research their aspect of the course project, which 50% of the students did.

Summary and Discussion

It turns out that peer-to-pear learning, i.e. learning from other students that work on the same or related project, is perceived as providing a substantial learning experience, and the amount of knowledge acquired seems to be at the same level as that learned during individual study. From this perspective, project-based classes seem to be superior to classical models that rely exclusively on ex-cathedra lecturing and individual study. Designing a project-course that covers the same breadth as a lecture-based course is difficult, however, and an implementation challenging and time intensive for the students. Although lectures and design reviews were not perceived as the most efficient learning vehicles, we believe both of these offers were necessary for the success of the project. The lecture provides the theoretical basis for the students' own exploration and provides a common ground by defining the scope of the project. Also, the fact that the design reviews require deliverables on a weekly basis keeps the progress of individual teams within sync, and it is unclear wether - particularly bigger classes - could efficiently self-motivate themselves. One way to increase the value of the lecture could be to design the lecture in response to problems encountered during the project, in order to maximize the opportunity for the students to put theory into practice.

References

[1] Survey questions