Much research has demonstrated that inference processing is critical to understanding verbal narratives. In recent years, empirical studies have offered evidence that inference generation is also crucial to visual narrative comprehension. Given the similarities between verbal and visual narratives, it is reasonable to argue that readers might be able to integrate information across different codalities during inference processing. Meanwhile, given the differences in both verbal and visual narratives across languages, people from different language backgrounds might perform differently in cross-codal inference processing. Other individual differences like working memory capacities and visual language fluency might also play significant roles in this process.

The PhD project focuses on two specific types of inference, namely the anaphoric inference and the bridging inference. Three online behavioural experiments and one event-related potential (ERP) experiment were conducted. The first online experiment focused on the anaphoric inference processing by L1 and L2 English and Chinese speakers, which serves as a background for further study on cross-codal anaphoric inferential processing. Interesting results were revealed. First, concerning the controversy on Chinese anaphora resolution, our results support the theory that the null elements and overt pronouns are pragmatically constrained. Regarding L2 anaphora resolution, the results are consistent with the claim that even the advanced L2ers are susceptible to transfer errors as they “misanalyse” the L2 data to make them compatible with the transferred L1 data.

The second behavioural experiment then further investigated the cross-codal processing of the anaphoric inference in English and Chinese, by both L1 and L2 speakers. Results show that all groups of speakers achieved higher accuracy when the information was compatible. However, the English L1 speakers struggled to keep pictorial information active when processing textual information, probably because they failed to regard the picture as the context for anaphoric inference processing.

The third behavioural experiment investigated the integration of cross-codal information when processing bridging inferences. 4 groups of participants (Chinese monolingual speakers, English monolingual speakers, Chinese learners of English, and English learners of Chinese) read pictorial or textual stories where the bridging event was 1) in the same codality as the context, 2) conveyed in the other codality, or 3) replaced by a blank panel. The results indicated that cross-codal bridging inferential processing could be achieved, but alphabetic texts and L2 texts might be more difficult to process than pictures.

A possible explanation is that the alphabetic texts require phonological processing, whereas orthography is first activated in Chinese visual word recognition, followed by semantics, and phonology may be activated at a later stage. To test this hypothesis, a meta-analysis was conducted with a total of 27 studies. The results confirmed this hypothesis, suggesting that it is indeed possible to retrieve meaning from orthography of the Chinese characters, leading to less challenges in Chinese textual processing.

To further scrutinise the topic, an ERP experiment with the same design was conducted on the Chinese L1 speakers, to see whether there are any nuances in Chinese cross-codal processing that the reaction time failed to reveal. During early processing, the text-to-picture switch requires a higher attention level, indicated by the larger P1 and posterior P200 components. Textual processing involves orthography-phonology mapping, suggested by larger N170 and frontal P200 amplitudes. These components, however, were not observed in the picture-to-text switch, probably due to the phonological activation being suppressed. Larger P300 effects were also found in the text-to-picture switch, demonstrating automatic semantic processing of the pictures. Picture-to-text switch, on the other hand, elicited higher N400 amplitudes, signalling deeper semantic processing of the texts. Text-to-picture switch also led to a larger P600 component, which could be due to the larger surprisal effect, or may further support the shared neurological mechanism between P300 and P600.

All experiments demonstrated significant effects of individual differences such as language background, verbal and visuo-spatial WM capacity, exposure to visual languages such as comics, and socioeconomic status, indicating that these variables collectively play a crucial role in text and picture integration during inferential processing.