BOBC

WIKINDX Resources  

Cohn, Neil. "A multimodal parallel architecture: A cognitive framework for multimodal interactions." Cognition 146. (2016): 304–23. 
Added by: joachim (10/17/17, 12:49 AM)   
Resource type: Journal Article
Language: en: English
Peer reviewed
DOI: 10.1016/j.cognition.2015.10.007
BibTeX citation key: Cohn2016a
Email resource to friend
View all bibliographic details
Categories: General
Keywords: Cognition, Communications, Intermediality, Language
Creators: Cohn
Collection: Cognition
Views: 29/1212
Attachments   URLs   http://visuallangu ... _multimodality.pdf
Abstract
Human communication is naturally multimodal, and substantial focus has examined the semantic correspondences in speech–gesture and text–image relationships. However, visual narratives, like those in comics, provide an interesting challenge to multimodal communication because the words and/or images can guide the overall meaning, and both modalities can appear in complicated “grammatical” sequences: sentences use a syntactic structure and sequential images use a narrative structure. These dual structures create complexity beyond those typically addressed by theories of multimodality where only a single form uses combinatorial structure, and also poses challenges for models of the linguistic system that focus on single modalities. This paper outlines a broad theoretical framework for multimodal interactions by expanding on Jackendoff’s (2002) parallel architecture for language. Multimodal interactions are characterized in terms of their component cognitive structures: whether a particular modality (verbal, bodily, visual) is present, whether it uses a grammatical structure (syntax, narrative), and whether it “dominates” the semantics of the overall expression. Altogether, this approach integrates multimodal interactions into an existing framework of language and cognition, and characterizes interactions between varying complexity in the verbal, bodily, and graphic domains. The resulting theoretical model presents an expanded consideration of the boundaries of the “linguistic” system and its involvement in multimodal interactions, with a framework that can benefit research on corpus analyses, experimentation, and the educational benefits of multimodality.
  
WIKINDX 6.10.2 | Total resources: 14672 | Username: -- | Bibliography: WIKINDX Master Bibliography | Style: Modern Language Association (MLA)