The keynote talks will be held in Lecture hall L1

Children communicate using gestures before they speak, and continue to use gesture along with speech even after they begin to produce their first words. Does gesturing merely precede talking, or is it itself relevant to the language-learning process? If gesturing not only precedes language, but also reflects knowledge relevant to the developmental process responsible for language, then the differences and/or delays in speech should become first evident in gesture across different learners. I approach this question by examining early gesture and speech production of children with different developmental profiles—including children with autism, Down syndrome and typical development, who show unique strengths or weaknesses in their early gesture production. I ask whether early gesture predicts later speech across different learners, and if so, what underlies the link between early gesture and later spoken language development.

The strong link between prosodic expression in speech and gesture has been established through ample empirical evidence that prosodic prominence, prosodic phrasing as well as floor management is expressed and processed in a multi-modal fashion.
However, the exact functional and formal relationship between the different modalities is still not well understood: 
First, we lack knowledge about which aspects of prosodic expression, namely signal features such as pitch movements or rather structural features such as information structure, are actually reproduced across modalities. To shed light on this issue, I will present a recent series of analyses (Wagner et al., in press), where this question was tackled with a newly developed methodological approach: Listeners were asked to "reproduce" acoustically presented utterances in a drumming task. The results revealed that the patterns of drumming intensities closely resemble patterns of perceptual prominence gathered with established annotation schemes, and involving both prosodic experts and naive listeners. However, as unveiled by a Random Forest Analysis, the gestural reproductions of prosody were driven comparatively stronger by signal cues than by linguistic structure. Also, we found different strategies for the gestural interpretations of prosodic patterns: a largely signal-driven, and a more integrative strategy.
Second, we do not know much about the flexibility or stability of the cross-modal link in prosodic expression. While speech economy models predict a flexible relationship depending on communicative demands and cross-modal compensation, a strong view of cross-modal congruence predicts a stable cross-modal co-ordination. To explore this issue, I will present a series of recent studies of semi-spontaneous, task-oriented interaction (Wagner and Bryhadyr, 2017; Wagner et al., 2019a; 2019b; in prep.) aiming at a better understanding of the impact of (1) information structure and (2) visibility between interlocutors on the cross-modal link in prosodic expression. Our results once again confirm a strong cross-modal temporal co-ordination. Furthermore, we detected a systematic modulation of this co-ordination as a function of communicative demands: In important or unpredictable contexts, co-speech movements occur later and align tightly with corresponding pitch peaks if interlocutors can see each others' hands. Also, a lack of facial visibility between interlocutors leads to a significantly earlier production of corresponding co-speech movements.
In summary, our results show that co-speech movements in general can express a rich set of signal and structural cues inherent in speech prosody, and that the degree of temporal co-ordination between speech and co-speech movements is a function of communicative needs.
As a side result, we found that cross-modal prosodic link also extends to co-speech movements such as drumming or manual moves on a game board, which are not gestures in the traditional understanding of the term.

References

Wagner, P. and N. Bryhadyr (2017). Mutual Visibility and Information Structure Enhance Synchrony between Speech and Co-Speech Movements. Journal of Multimodal Communication Studies 4(1-2): 69-74.
Wagner, P., Cwiek, A., and B. Samlowski (in press). Exploiting the speech-gesture link to capture fine-grained prominence impressions and listening strategies. Journal of Phonetics.
Wagner P., Bryhadyr, N., Schröer, M., and B. Ludusan (2019a). Does information-structural acoustic prosody change under different visibility conditions? In: Proceedings of the International Congress of Phonetic Sciences 2019, Melbourne, Australia.
Wagner P., Bryhadyr N., and Schröer M. (2019). Pitch Accent Trajectories across Different Conditions of Visibility and Information Structure - Evidence from Spontaneous Dyadic Interaction. Proceedings of Interspeech 2019, Graz, Austria.
Wagner, P. et al. (in prep.). The temporal co-ordination between speech prosody and co-speech movements as a function of communicative needs.

Animated Virtual Characters exhibit many desirable aspects for who wants to understand human behavior and language. Like their human counterpart, they can display a broad palette of multimodal stimuli, these stimuli can, however, be very precisely altered, fired at exact timestamps, or triggered by specific reactions of a subject taking part in an experiment. 

Unfortunately, the integration of virtual characters into a full-fledged experiment setup requires a concentration of many diverse and specific skills which are often out of the reach of the team crafting the experiment. A lightweight, modular, well documented, and easy to deploy agent toolkit is still needed.

This talk starts depicting a series of trans-disciplinary collaborations which lead to the design and implementation of interactive virtual humans in experimental setups involving human counterparts. Each experiment was able to shed an original light on specific aspects of human language or behavior.

This talk later focuses on the lessons learned during these trans-disciplinary collaborations and how we could infer from them the leading design principles of a new agent creation and control framework called YALLAH. 

YALLAH stands for Yet Another Low-Level Avatar Handler. It is a framework supporting the creation of real-time interactive virtual humans by non-experts. After a quick overview of YALLAH's features, documentation, and ongoing projects using YALLAH, the talk will conclude by a discussion on how YALLAH could help the community understanding the coordination of gesture with speech.

When we speak, we use rhythmic hand gestures which are coordinated with prominent parts of speech (e.g., beat gestures). In this talk I will discuss several experiments carried out in our research group that deal with how beat gestures and other embodied rhythmic movements facilitate the learning of second language pronunciation. Even though most of the research on the benefits of gesture in the second language classroom has focused on the effects of representational gestures (e.g., for the acquisition of vocabulary), little is known about the potential beneficial effects of embodied rhythmic movements on the learning of L2 pronunciation. A set of experiments will be presented. Experiments 1 and 2 will assess the potential benefits of observing and performing beat gestures on L2 pronunciation learning with intermediate Catalan learners of English. Experiments 3 and 4 will assess the benefits of hand-clapping on L2 pronunciation learning at initial stages of L2 acquisition of French by Catalan and Chinese native speakers. Widening the scope of this investigation, Experiments 5 and 6 will focus on the positive effects of using melodic and singing trainings for pronunciation learning. Based on the positive findings from these experiments, I will conclude that a multimodal approach is essential to understanding L2 speech learning. I will suggest that not only rhythmic trainings with beat gestures or hand-clapping procedures can act as scaffolding mechanisms for L2 speech production but also melodic trainings based on pitch mimicry and singing. Importantly, both types of embodied prosodic trainings could be successfully applied to language teaching and language treatment contexts.