Hans Rutger Bosker receives ERC grant to study audiovisual communication

Hans Rutger Bosker and his TEMPOS research group investigate human speech production and perception. His ERC project ‘HearingHands’ focuses on audiovisual communication. In face-to-face conversation, we do not only produce sound. We also produce all sorts of visual communicative cues, including hand gestures. These hand gestures are carefully timed in close synchrony with the speech. However, it remains unclear what this precise timing contributes to speech perception.

In the HearingHands project, Bosker and his team aim to demonstrate that the timing of even the simplest up-and-down hand movements serves as a form of multimodal prosody, with the potential to even guide what words you hear. The researchers will record gesture-speech synchrony in a range of different languages. In addition, they will assess how the human brain integrates visual (gestures) and auditory (speech) input. They will also test the usefulness of enriching virtual avatars with carefully timed gestures to enhance speech intelligibility.

The ERC Starting Grant awards up to €1.5 million to talented early-career researchers for a period of five years.