3rd quarter, 2016/2017, timeslot A, 7.5 ECTS [CS site | OSIRIS]
INFOMMMI: Multimodal Interaction

Lecturers: Wolfgang Hürst (WWW, Email), Peter Werkhoven (WWW, Email)

News | Description | Schedule | Lectures | Essay | Project


Part 1 (given by Peter Werkhoven) covers mostly multimodal (multisensory) perception and interaction. It is about the fascinating world of human visual, auditory and tactile perception and the use of its potential in designing novel interfaces for interacting with virtual worlds. It is accompanied by a practical part in which students have to read papers, write an essay (on the analysis of a training game) and give a presentation - as specified under Essay.

In the 2nd part of the course (given by Wolfgang Hürst), we will mostly address Augmented Reality plus potential applications. It is accompanied by a practical part in which students have to implement some multimodal interaction system in a related project - as specified under Project.


The lectures will address the following topics:
  • some basics of visual, auditory and tactile perception and the effects of combining them (multimodal presentation and related interface design guidelines)
  • visual communication interfaces in virtual worlds (effects of non-verbal facial communication)
  • navigation interfaces in virtual worlds (head tracked visualization and 'cyber sickness')
  • manipulation in virtual worlds (traditional mouse-cursor interfaces versus virtual hand control)
  • virtual worlds through mobile displays (scrolling interfaces versus virtual windows)
  • emerging interface technology (synaesthetic media and brain machine interfaces)
  • augmented reality (technologies, human factors, interaction)
For related lecture dates, please refer to the course schedule.


Multimodal perception:
  • Obligatory paper 1:
    Ernst, M.O. & Bulthoff, H.H. (2004). Merging the senses into a robust percept. Trends in Cognitive Sciences, 8(4), 162-169.
    PDF (password protected)
  • Obligatory paper 2:
    Petkova, V.I., & Ehrsson, H.H. (2008). If I Were You: Perceptual Illusion of Body Swapping. PLoS ONE 3(12): e3832. doi:10.1371/journal.pone.0003832
    PDF (password protected)
  • Additional reading:
    Shimojo, S. & Shams, L. (2001). Sensory modalities are not separate modalities: plasticity and interactions. Current Opinion In Neurobiology, Vol. 11(4), p. 505-509.
    PDF (password protected)
Control and navigation in (mobile) virtual environments:
  • Obligatory paper 3:
    Emmerik, M.L. de Vries, S.C., & Bos, J.E. (2011). Internal and external fields of view affect cybersickness. Displays, 32(4), 169-174.
    PDF (password protected)
  • Obligatory paper 4:
    Wolbers, T., Hegarty, M. (2010). What determines our navigational abilities? Trends in Cognitive Sciences, 14(3), 138-146
    PDF (password protected)
  • Additional reading:
    Mehra, S., Werkhoven P. & Worring M. (2006) Navigating on hand held displays: Dynamic versus Static Peephole Navigation. ACM Transactions on Computer Human Interaction, 13 (4), 448-457.
    PDF (password protected)
  • Additional reading:
    Jeffery, K.J., et al. (2013). Navigating in a 3D world. Behavioral and Brain Sciences, in press.
    PDF (password protected)
Emotions and stress in virtual environments:
  • Obligatory paper 5:
    Toet, A., Schaik, M.G., van (2012). Effects of signals of disorder on fear of crime in real and virtual environments. Journal of Experimental Psychology, 32(3), 206-276. (http://dx.doi.org/10.1016/j.jenvp.2012.04.001)
    PDF (password protected)
  • Obligatory paper 6:
    Visschedijk G.C. et al. (2013). Modelling human emotions for tactical decision-making games. British journal of educational technology, 44 (2). 197 - 207. ISSN 0007-1013
    PDF (password protected)
  • Additional reading:
    Nacke, L.E., Stellmach, S., Lindley, C. (2010). Electroencephalographic Assessment of Player Experience: A Pilot Study in Affective Ludology. In Simulation & Gaming, vol. 41, 1-24. DOI=10.1177/1046878110378140.
    PDF (password protected)
  • Additional reading:
    Toet, A., Henselmans, M., Lucassen, M.P., & gevers, T. (2011). Emotional effects of dynamic textures. I-Perception, 2, 969-991. (http://i-perception.perceptionweb.com/journal/I/volume/current/issue/current)
    PDF (password protected)
Emerging technologies:
  • Obligatory paper 7:
    Bach-y-Rita, P. & Kercel, S.W. (2003). Sensory substitution and the human-machine interface. TRENDS in Cognitive Sciences, 7(12), 541-546.
    PDF (password protected)
  • Obligatory paper 8:
    Lebedev, M.A. & Nicolelis, M.A. (2006). Brain-machine interfaces: past, present and future. Trends Neuroscience, 29, 536-546.
    PDF (password protected)
Augmented reality:
  • Obligatory paper 9:
    Feiner, S. (2002). Augmented reality: A new way of seeing. Scientific American, 286.4, 48-55.
    PDF (password protected)
  • Obligatory paper 10:
    Azuma, R.T. (1997). A survey of augmented reality. Presence: Teleoperators and virtual environments 6.4, 355-385.
    PDF (external link)