Multimodal Interaction for Text Entry and Control in Mixed Reality

Abstract

This thesis explores the integration of multimodal interaction methods in Mixed Reality (MR) environments for text entry control. The central hypothesis is that these multimodal approaches can provide an expressive and efficient solution for (1) text input; (2) text editing; and (3) command retrieval, complementing the limitations of traditional input methods in immersive environments. The research addresses four key areas: text caret control, dwell-free eye typing, command search systems through dwell-free eye typing, and eye-hand coordination in mid-air gesture typing. Through iterative prototyping and controlled empirical studies, the research demonstrates that a multimodal interaction method, combining gaze for coarse positioning and hand gestures for fine control, provides significant advantages in terms of performance and reducing physical strain. Moreover, the introduction of dwell-free eye typing methods greatly enhances typing speed without sacrificing accuracy, offering an efficient solution for hands-free input in MR environments. Additionally, this dwell-free method was later utilized for a command search system which outperform traditional menu-based approaches, improving both usability and task efficiency. The thesis also investigates the challenges of eye-hand coordination in mid-air gesture typing, revealing that the absence of tactile feedback necessitates greater visual attention and introduces a tighter synchronization between eye and hand movements. Despite these challenges, participants adapted to achieve stable typing performance after practice. These findings contribute to the design of more user-friendly and effective MR interfaces, offering practical insights for improving the user experience in multimodal MR applications. The thesis concludes by discussing limitations and proposing avenues for future research.