LNCS 3220 - Evaluating the Effectiveness of a Tutorial Dialogue System for Self-Explanation

Evaluating the Effectiveness of a Tutorial DialogueSystem for Self-ExplanationVincent Aleven, Amy Ogan, Octav Popescu, Cristen Torrey, Kenneth KoedingerHuman Computer Interaction Institute, Carnegie Mellon University5000 Forbes Ave, Pittsburgh, PA 15213, USA+1 412 268 5475aleven@cs.cmu.edu, {octav,koedinger}@cmu.edu{aeo,ctorrey}@andrew.cmu.edu,Abstract. Previous research has shown that self-explanation can be supportedeffectively in an intelligent tutoring system by simple means such as menus.We now focus on the hypothesis that natural language dialogue is an even moreeffective way to support self-explanation. We have developed the GeometryExplanation Tutor, which helps students to state explanations of their problem-solving steps in their own words. In a classroom study involving 71 advancedstudents, we found that students who explained problem-solving steps in adialogue with the tutor did not learn better overall than students who explainedby means of a menu, but did learn better to state explanations. Second, exam-ining a subset of 700 student explanations, students who received higher-quality feedback from the system made greater progress in their dialogues andlearned more, providing some measure of confidence that progress is a usefulintermediate variable to guide further system development. Finally, studentswho tended to reference specific problem elements in their explanations, ratherthan state a general problem-solving principle, had lower ...