Understanding multimodal deixis with gaze and gesture in conversational interfaces [Elektronische Ressource] / Thies Pfeiffer. Technische Fakultät

Understanding Multimodal Deixis with Gaze andGesture in Conversational InterfacesThies Pfei erA.I. GroupFaculty of TechnologyBielefeld UniversityP.O. Box 10 01 31D-33501 BielefeldGermanyemail: tpfei e@techfak.uni-bielefeld.deThis dissertation has been approved by the Faculty of Technology at Biele-feld University to obtain the academic degree of a Doctor rerum naturalium(Informatics).Dean of the faculty: Prof. Dr. Jens StoyeFirst reviewer: Prof. Dr. Ipke WachsmuthSecond reviewer: Prof. Dr. Hannes RieserSubmission of the thesis: April 28, 2010Day of the disputation: October 8, 2010The background of the model on the frontpage shows the scene \Venice" byStefan John, copyright 2009.The o cial print version has been printed on age-resistant paper accordingto DIN-ISO 9706.ISummaryWhen humans communicate, we use deictic expressions to refer to objects inour surrounding and put them in the context of our actions. In face to faceinteraction, we can complement verbal expressions with gestures and, hence,we do not need to be too precise in our verbal protocols. Our interlocutors hearour speaking; see our gestures and they even read our eyes. They interpretour deictic expressions, try to identify the referents and { normally { theywill understand. If only machines could do alike.The driving vision behind the research in this thesis are multimodal con-versational interfaces where humans are engaged in natural dialogues withcomputer systems.