PERIODE 3, 2013/2014, timeslot A, 7.5 ECTS [CS site | OSIRIS]
INFOMMMI: Multimodal Interaction

Lecturers: Wolfgang Hürst (WWW, Email), Peter Werkhoven (WWW, Email)

News | Description | Schedule | Lectures part 1 & part 2 | Practicals part 1 & part 2 

LECTURES (PART 1) given by Peter Werkhoven

This part covers mostly multimodal (multisensory) perception and interaction. It is about the fascinating world of human visual, auditory and tactile perception and the use of its potential in designing novel interfaces for interacting with virtual worlds. It is accompanied by a practical part in which students have to read papers, write an essay (on the analysis of a training game) and give a presentation - as specified under Practicals (part 1).


The lectures will address the following topics:
  • some basics of visual, auditory and tactile perception and the effects of combining them (multimodal presentation and related interface design guidelines)
  • visual communication interfaces in virtual worlds (effects of non-verbal facial communication)
  • navigation interfaces in virtual worlds (head tracked visualization and 'cyber sickness')
  • manipulation in virtual worlds (traditional mouse-cursor interfaces versus virtual hand control)
  • virtual worlds through mobile displays (scrolling interfaces versus virtual windows)
  • emerging interface technology (synaesthetic media and brain machine interfaces)
For related lecture dates, please refer to the course schedule.

PAPERS TO READ (obligatory)

Multimodal perception:
  • Paper 1: Shimojo, S. & Shams, L. (2001). Sensory modalities are not separate modalities: plasticity and interactions. Current Opinion In Neurobiology, Vol. 11(4), p. 505-509.
    PDF (password protected)
  • Paper 2: Ernst, M.O. & Bulthoff, H.H. (2004). Merging the senses into a robust percept. Trends in Cognitive Sciences, 8(4), 162-169.
    PDF (password protected)
  • Paper 3: Petkova, V.I., & Ehrsson, H.H. (2008). If I Were You: Perceptual Illusion of Body Swapping. PLoS ONE 3(12): e3832. doi:10.1371/journal.pone.0003832
    PDF (password protected)
Control and navigation in (mobile) virtual environments:
  • Paper 4: Mehra, S., Werkhoven P. & Worring M. (2006) Navigating on hand held displays: Dynamic versus Static Peephole Navigation. ACM Transactions on Computer Human Interaction, 13 (4), 448 457.
    PDF (password protected)
  • Paper 5: Emmerik, M.L. de Vries, S.C., & Bos, J.E. (2011). Internal and external fields of view affect cybersickness. Displays, 32(4), 169-174.
    PDF (password protected)
  • Paper 6: Wolbers, T., Hegarty, M. (2010). What determines our navigational abilities? Trends in Cognitive Sciences, 14(3), 138-146
    PDF (password protected)
  • Paper 7: Jeffery, K.J., et al. (2013). Navigating in a 3D world. Behavioral and Brain Sciences, in press.
    PDF (password protected)
Emotions and stress in virtual environments:
  • Paper 8: Nacke, L.E., Stellmach, S., Lindley, C. (2010). Electroencephalographic Assessment of Player Experience: A Pilot Study in Affective Ludology. In Simulation & Gaming, vol. 41, 1-24. DOI=10.1177/1046878110378140.
    PDF (password protected)
  • Paper 9: Toet, A., Henselmans, M., Lucassen, M.P., & gevers, T. (2011). Emotional effects of dynamic textures. I-Perception, 2, 969-991. (
    PDF (password protected)
  • Paper 10: Toet, A., Schaik, M.G., van (2012). Effects of signals of disorder on fear of crime in real and virtual environments. Journal of Experimental Psychology, 32(3), 206-276. (
    PDF (password protected)
  • Paper 11: Visschedijk G.C. et al. (2013). Modelling human emotions for tactical decision-making games. British journal of educational technology, 44 (2). 197 - 207. ISSN 0007-1013
    PDF (password protected)
Emerging technologies:
  • Paper 12: Bach-y-Rita, P. & Kercel, S.W. (2003). Sensory substitution and the human-machine interface. TRENDS in Cognitive Sciences, 7(12), 541-546.
    PDF (password protected)
  • Paper 13: Lebedev, M.A. & Nicolelis, M.A. (2006). Brain-machine interfaces: past, present and future. Trends Neuroscience, 29, 536-546.
    PDF (password protected)