Engineering cognitive alignment in interactive systems through sensorimotor regularities

Abstract

The central hypothesis of this thesis is that grounding interactive systems in universal sensorimotor regularities (SRs) provides an effective methodology for creating interactive systems that are intuitive, explainable, and aligned with human cognition. The work is situated at a point where Mixed Reality (MR) and Generative AI (GenAI) systems are becoming increasingly important to human-computer interaction, both of which currently suffer from a fundamental cognitive disconnect with their human users. MR systems often rely on arbitrary system representations and action-outcome mappings; while GenAI systems, such as LLMs, lack embodied grounding, leading to concept representations misaligned with human reasoning. This thesis argues that grounding interactive systems in Sensorimotor Regularities (SRs)—universal image schematic patterns—offers a systematic principle to address these cognitive disconnects. Research Question 1 examines whether users from different demographics converge on the same image-schematic patterns. A comparison between younger and older adults revealed a substantial overlap in their use of image schema patterns, despite systematically different technological experiences between the two groups. Research Question 2 investigates how sensorimotor regularities can be operationalised at the interface level in mixed reality, and how this integration affects learnability, performance, and user experience. An SR-enhanced MR authoring system led to significantly faster task performance, improved learnability, and higher mental efficiency than a conventional baseline. Research Question 3 explores how sensorimotor regularities can be operationalised at the interaction level in mixed reality, and how this integration influences learnability, performance, and subjective perception. Results showed higher success rates, lower workload, faster mastery, and greater agency and presence compared to arbitrary or conventional mappings. Research Question 4 examines how sensorimotor regularities can be used to evaluate and align concept encoding in Large Language Models (LLMs), and the extent to which LLMs reflect human sensorimotor regularities. While LLMs approximate global schema distributions, they diverge in schema–concept associations and co-occurrences, revealing key gaps in embodied alignment. Research Question 5 evaluates how sensorimotor regularities can be operationalised within generative pipelines, and how this integration influences the interpretability and embodied alignment of generated outputs. The resulting tool, CognGen, achieved human-expert-level discoverability, a 75% overlap with human embodied reasoning (versus 17% for a baseline LLM), and produced more diverse, less fixated proposals. Grounding interactive systems in sensorimotor regularities is shown to be a powerful and transferable methodology. This thesis contributes empirical evidence, design frameworks, and generative tools that support the development of next-generation interactive systems that are cognitively aligned, creative, and accessible across diverse user groups.