Cohn, Neil: "A multimodal parallel architecture. A cognitive framework for multimodal interactions." In: Cognition 146 (2016), S. 304–323.
Added by: joachim (2017-10-17 00:49)
|Resource type: Journal Article
BibTeX citation key: Cohn2016a
Email resource to friend
View all bibliographic details
Keywords: Cognition, Communications, Intermediality, Language
|Attachments||URLs http://visuallangu ... _multimodality.pdf|
Human communication is naturally multimodal, and substantial focus has examined the semantic correspondences in speech–gesture and text–image relationships. However, visual narratives, like those in comics, provide an interesting challenge to multimodal communication because the words and/or images can guide the overall meaning, and both modalities can appear in complicated “grammatical” sequences: sentences use a syntactic structure and sequential images use a narrative structure. These dual structures create complexity beyond those typically addressed by theories of multimodality where only a single form uses combinatorial structure, and also poses challenges for models of the linguistic system that focus on single modalities. This paper outlines a broad theoretical framework for multimodal interactions by expanding on Jackendoff’s (2002) parallel architecture for language. Multimodal interactions are characterized in terms of their component cognitive structures: whether a particular modality (verbal, bodily, visual) is present, whether it uses a grammatical structure (syntax, narrative), and whether it “dominates” the semantics of the overall expression. Altogether, this approach integrates multimodal interactions into an existing framework of language and cognition, and characterizes interactions between varying complexity in the verbal, bodily, and graphic domains. The resulting theoretical model presents an expanded consideration of the boundaries of the “linguistic” system and its involvement in multimodal interactions, with a framework that can benefit research on corpus analyses, experimentation, and the educational benefits of multimodality.
PHP execution time: 0.03834 s
SQL execution time: 0.10744 s
TPL rendering time: 0.00152 s
Total elapsed time: 0.14730 s
Peak memory usage: 1.3070 MB
Memory at close: 1.2560 MB
Database queries: 64