Towards Body-Centric Gestural Interaction in Virtual Reality

Abstract

Gestural interaction is central to virtual reality (VR), yet many existing techniques remain fundamentally disembodied. They rely on unsupported mid-air movement, visually intensive search, and sequential one-to-one actions, making it difficult to sustain precise, efficient, and expressive interaction in productivity-oriented immersive environments. This thesis investigates whether the user’s body can be treated not merely as a source of tracked motion, but as a stable reference frame for interaction design. It investigates how body-centric gestural interaction can improve precision, accessibility, expressivity, and scalability by organising input, interface structure, and control around bodily references. The thesis develops this argument through four strands of work. First, it investigates on-body menus as persistent bodily anchors for interface organisation. Two user studies show that users form meaningful spatial mappings between icon categories and bodily landmarks, and that first-person, third-person, and mirror-based creation perspectives shape how those mappings are created and recalled. Second, it presents microGEXT, a microgesture-based text editing approach that uses fine-grained finger motion to support high-frequency editing commands with reduced physical demand. Across controlled studies and an open-ended note-taking scenario, the findings show that microgestures can provide a low-fatigue alternative for recurrent editing tasks, while also revealing trade-offs involving memorability and recogniser robustness. Third, the thesis introduces a curved selection technique driven by finger curvature and combines it with an on-body disambiguation mechanism that projects candidate targets onto the forearm. A factorial study shows that on-body disambiguation reduces errors and physical demand, while curved trajectories expand reach in dense and occluded scenes. Fourth, the thesis explores swarm manipulation, in which the morphology of the hand is remapped to control multiple virtual entities simultaneously. The results show that one-to-many bodily mappings can improve performance in several manipulation scenarios while also raising broader questions about agency, ownership, and coordination. Taken together, these contributions show that body-centric interaction is not a single technique but a general design stance. The body can function as a spatial scaffold, a kinematic stabiliser, a disambiguation surface, and a morphological template for scalable control. By synthesising these roles, the thesis derives a framework for body-centric gestural interaction in VR and demonstrates how bodily grounding can support more precise, sustainable, and expressive immersive interaction.