Repository logo
 

Simulating realistic human motion trajectories of mid-air gesture typing

Accepted version
Peer-reviewed

Type

Conference Object

Change log

Authors

Dudley, J 
Kristensson, PO 

Abstract

The eventual success of many AR and VR intelligent interactive systems relies on the ability to collect user motion data at large scale. Realistic simulation of human motion trajectories is a potential solution to this problem. Simulated user motion data can facilitate prototyping and speed up the design process. There are also potential benefits in augmenting training data for deep learning-based AR/VR applications to improve performance. However, the generation of realistic motion data is nontrivial. In this paper, we examine the specific challenge of simulating index finger movement data to inform mid-air gesture keyboard design. The mid-air gesture keyboard is deployed on an optical see-through display that allows the user to enter text by articulating word gesture patterns with their physical index finger in the vicinity of a visualized keyboard layout. We propose and compare four different approaches to simulating this type of motion data, including a Jerk-Minimization model, a Recurrent Neural Network (RNN)-based generative model, and a Generative Adversarial Network (GAN)-based model with two modes: style transfer and data alteration. We also introduce a procedure for validating the quality of the generated trajectories in terms of realism and diversity. The GAN-based model shows significant potential for generating synthetic motion trajectories to facilitate design and deep learning for advanced gesture keyboards deployed in AR and VR.

Description

Keywords

Human-centered computing, Human computer interaction (HCI), Interaction techniques, Text input

Journal Title

Proceedings - 2021 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2021

Conference Name

2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)

Journal ISSN

1554-7868

Volume Title

00

Publisher

IEEE
Sponsorship
Engineering and Physical Sciences Research Council (EP/S027432/1)