Supporting Decision Making in Engineering Design Using Parallel Coordinates in Virtual Reality

Computational engineering design methods and tools are common practices in the modern industry. Such approaches are integral in enabling designers to efficiently explore large and complex design spaces. However, they also tend to dramatically increase the number of candidate solutions that decision makers must correctly interpret.Becauseallcandidatesolutionscanberepresentedinadigitalformtogetherwiththeirassessmentcriteria,a natural way to explore and understand the complexities of the design problem is to visualize their multidimensional nature. The task now involves the discovery of patterns and trends within a multidimensional design space. This work aims to enhance the design decision-making process with immersive Parallel Coordinates Plot (IPCP) in virtual reality. This paper presents the design of this system, which allows representation and exploration of multidimensional scientific datasets. A qualitative validation with two surrogate expert users demonstrated that the system can be used successfully to detect both known and previously unknown patterns and support learning the decision-making process in a shorter time. The results serve as a promising indication of how immersive parallel coordinate plots can enhance decision support in complex engineering design processes.

returned by a Google Scholar search on 9 December 2020, reaches over 2.030 million items.The theory and applications of PCP can be found in the textbook written by Inselberg [10].
The wide adoption of two-dimensional (2D) PCP has led to multiple attempts of designing a useful and compelling 3D PCP.For example, Wegenkittl et al. [11] and Gröller et al. [12] describe extruded PCP, which uses a PCP to visualize a complex surface.A different example of using the surface-type design is a density isosurface [13], which was used to visualize cytomics data.Falkman [14] describes the Cube package applied for swift analysis of medical data.
Parallel Glyphs combine Star Glyphs [15] with 3D PCP [16].The usability of 3D Multi-Relational PCP in conjunction with High-Precision Textures has been investigated by Johansson et al. [17].Dang et al. [18] describe a few examples of 3D parallel dot displays and stacked PCP, which use various stacking techniques to augment the standard visualization of dot plots and PCP, respectively.In their related work, Chang et al. [19] compare joined side-by-side 2D visualizations to PCP with scatter plot matrices, concluding that the PCP-based approach offered clear advantages over individual visualizations for most of their participants for a number of their tasks.Furthermore, Johansson et al. [20] present a comparison between 3D PCP visualizations with conventional 2D plots and concluded in favor of the latter.Holten and Wijk [21] discuss different variants and improvements of 2D PCP used to find clusters in the multivariate datasets and remark that only the use of additional scatter plots has a substantial positive impact on the visualization.
ImAxes [22] allows the user to rearrange the axes interactively and supports both 2D and 3D scatter plots.Another example of an immersive PCP is the ART collaborative data analysis tool [23] that distributes the user interface across an augmented reality headset and a touch surface.Rosenbaum et al. [24] took another approach in which the user was treated as an element of the immersive PCP visualization itself.
Our approach, first described by Tadeja et al. [5] and discussed by them later on [6], differs from other attempts in several key ways.First, we denote values in each dimension with interactive volumetric objects (i.e., unit-size cubes) that automatically highlight the item under the user's gaze in order to signal that they can be interactive.The same is true for all the other interactive elements, such as selectors (i.e., various size spherical markers in the interface) used to move or rotate the scatter plots.This in turn opens up new design possibilities [25] that can use characteristics of a 3D shape, such as its orientation in the 3D space [25].In addition, users can freely move within and maneuver in the infinite 3D space to explore all parts of the visualization [5].Furthermore, our system augments the pattern identification task with 3D scatter plots [5][6][7][8], that can substantially aid users in isolating groups of data items that can potentially form patterns.In addition, we also explore bimanual manipulation as a new way of interacting with visualization elements.Here, we are investigating the feasibility of using a gesture-based interface facilitated with an additional sensor § § attached to the VR headset that tracks the user's hands in real time.

III. Problem Context
We emphasize that the purpose of the two case studies is not only to demonstrate how to identify patterns in high-dimensional engineering datasets but also to demonstrate the effectiveness and usefulness of the use of VR technology in the discovery of meaningful information in complex datasets with ease and with common intuition.Both cases represent real-world design problems in the field of turbomachinery.
In a typical process, when an automated shape aerodynamic design optimization process is executed successfully, the designer discovers a set of solutions representing the answer to the multicriteria design problem.This set of solutions, the Pareto set, is an approximation of the Pareto front.To achieve this, the automated optimization process should synthesize three modules: one to modify the geometry, which is associated with the design variables; the second to simulate the flow characteristics, which is associated with the quantification of the objective functions; and the third to navigate through and explore the design space, which is the optimization algorithm.Prior work [1] introduced a stage of postprocessing of optimization results where the user simultaneously considers the design parameters and objective functions.Depending on the complexity of the optimization problem, the number of solutions within the Pareto set can easily exceed a few hundred.The objective at this stage of the design process is to identify a few families, or groups, of solutions to aid the decisionmaking process.The different groups of solutions are identified as clusters within the multidimensional data space.It is this final stage that we enhance in this work with the use of VR-based IPCP.
The first sample dataset is the result of an optimization study for the 3D geometrical design of compressor blades subject to many equality and inequality physical and geometrical constraints.In this study, we use the data produced for a three-objective function formulation of the optimization problem as described in detail by Kipouros et al. [2,3].This dataset (hereafter referred to as DS1) expresses the Pareto front, which contains a set of 54 equally optimum design configurations, and the 29-dimensional data items categorized as design parameters (26 dimensions) and objective functions (3 dimensions).The 2D view of the complete dataset and the two patterns which were identified using the conventional 2D PCP are presented in Fig. 2. The two patterns of interest A (in red) and B (in blue) are composed of 11 and 19 data items, respectively.These two patterns were identified by domain experts, and we consider them as the reference in this work.The user's view of the same dataset through a VR headset in three dimensions is shown in Fig. 3.
The second sample dataset (hereafter referred to as DS2) describes the results of the optimization study of S-duct design (see Fig. 4), as presented by D'Ambros et al. [4], and is composed of 39-dimensional data items: design parameters (36 dimensions) and objective functions (3 dimensions).The user's view through a VR headset in three dimensions is presented in Fig. 5.

A. Overview
The overall objective is to design a visual analytics IPCP system for engineering design processes, supporting efficient identification and comparisons of patterns of interest in a multidimensional dataset.This is a high-dimensional design space with many controllable and uncontrollable parameters.Controllable parameters include rendering parameters, such as colors and shapes, visual/auditory feedback design, interaction techniques, and choice of headset and hardware controllers.Uncontrollable parameters include properties about the datasets, such as size, density, and correlation structure and the degree of domain knowledge and expertise of the user.
As a result of the many controllable and uncontrollable parameters, we have adopted an iterative process of first building an early system that provided direct transplantation of PCP to VR.This early design was then used to carry out qualitative user studies (partially described in a previous poster publication by Tadeja et al. [5]), briefly outlined in Sec.IV.B.We use the information distilled from the user study along with knowledge of the tasks, expert advice, and technical constraints to arrive at a set of design principles (see Sec. IV.D), which guide our task analysis in Sec.IV.E.In the task analysis, we break down each task into functions and translate each function into a solution based on technical constraints, expert advice, and information gathered from the qualitative user study [5].Finally, we provide   verification and validation of the IPCP system in Sec.V, whereas Sec.VII provides a description of the implemented extensions to the system.These are twofold.First, we couple the system with CAD 3D models of the actual geometries of the individual designs associated with individual data items on the IPCP.Second, we develop and implement a different manipulation technique that uses the hand tracking and gesture recognition done with the help of Leap Motion Controller (see footnote § §) coupled with gaze tracking to aid the user in more effective interaction with our system.Finally, in Sec.VIII, we provide a detailed discussion about the entire process of design, development, and validation of the IPCP system.Section IX offers a brief summary of our work.

B. Qualitative Study
To help inform the design, we carried out a qualitative study with seven participants (hereafter referred to as P1-P7).More details about this study can be found in the work by Tadeja et al. [5].The IPCP visualization used in the study can be seen in Figs. 3 and 5.The users could interact with the system using a mixture of gaze tracking and an Xbox gamepad controller, as shown in Figs.6a and 6b.The color-coded cones ¶ were used to denote both the selected dimensions over the IPCP and axes directions at the scatter plots [7,8]; see Figs. 6a  and 6c.
Although the visualized dataset comes directly from the specific domain (the aerodynamic design of turbomachinery components), we wanted to pinpoint the general underlying design principles that can be used in future more advanced versions of the IPCP.Therefore, we broadened our sample of participants to include both fields of expertise and prior familiarity with PCP.The volunteers self-assessed their level of expertise with regards to VR and PCP using a Likert-like scale.As the color coding plays a crucial role in the visualization, participants were prescreened with Ishihara's tests [26], which indicated no color deficiency.
On average, participants needed 25 min during the training phase and 15 min for the study itself.To offer insight into any adverse physiological effects, volunteers were instructed to take the simulation sickness questionnaire [27] before and after each phase (i.e., training or study) to see to what extent and if at all the visualization negatively impacted their perception of comfort.Participants were reminded that they should stop the experiment if they experienced any of the symptoms described in the survey.However, none of the participants decided to stop and discontinue the study.
Before the experiment commenced, each participant sat through a demonstration of the IPCP technique, the studied tool, and all the possible ways in which the user could interact with the system.Moreover, while listening to a verbal description, participants could track in real time the instructor's actions and their corresponding results on a computer screen.
Our participants reported having varying levels of prior familiarity with the PCP and VR.To level the playing field, we provided each volunteer with training data visualization.This process took between 18 and 36 min, depending on the individual participant's needs as judged by the researcher.The primary task of the user was to find patterns within the given dataset using IPCP.In this context, the pattern is understood as a group of data items sharing similar values across all the dimensions visualized as group of polygonal curves [see Fig. 2 (left)].However, such curves can have certain variability between one or more of the line segments.The level of this variance is left to be judged by the user.This primary task was split into subtasks, including 1) selection of both data items and dimensions on the PCP, 2) selection of data points and clusters on the scatter plots, 3) visual analysis of the relation between the visualization elements, and 4) rotation and movement of a scatter plot or generation of more than a single such 3D projection at once.Familiarization with the system was especially important because the training may be very beneficial for the participant's grasp  of the PCP technique, as remarked by Shneiderman and Plaisant [28].
For this purpose, we used a portion of a real-world dataset of 80 multidimensional data points, each consisting of 22 dimensions (20 inputs and 2 outputs).The training was followed up with a semistructured interview.If deemed necessary by the researcher, the participant was asked to repeat previously completed tasks.Each instruction was marked as pass by the study supervisor, who constantly monitored the participant's actions on the accompanying screen.During this process, the researcher provided oral instructions upon completion of a task and whenever the participants requested assistance.
To minimize fatigue, participants were given a break of approximately 15 min after training and before the main experiment commenced.After training, the participants were asked to perform a task within the VR environment without a time constraint.The main task had an identical form compared to the training: identify patterns in a multidimensional dataset of 26 input parameters and 3 output criteria, together with the suggested approach to start developing the scatter plot using the criteria dimensions.We adopted a think-aloud protocol and requested participants to discuss what, why, and how they are trying to achieve as the task progressed.The supervisor, who continuously monitored the experiment, could at any moment inquire for further clarifications to gain more insight into the participant's actions and thought processes.In particular, for the first-time gamepad users, an inquiry on whether the participant is stuck allowed the supervisor to provide guidance, for example, information about which button to click to invoke a specific action.This is important as the controller is not visible through the VR headset.Moreover, after running two participants (P1 and P2), we decided to instruct the remaining participants (P3-P11) that a pattern is constituted by a minimum of a four-points group.Audio and video were captured during the study for later analysis.

Participants' Background and Expertise
All the participants reported having a background and extensive experience in the science, technology, engineering, and mathematics fields in either industrial or academic settings or both.Nine participants were men, and two were women with their ages ranging between 22 and 44.
Participant 1 (P1) reported approximately six years of experience in engineering at undergraduate and graduate levels and two years spent in the automotive industry.This person was not previously exposed to the PCP and had only very limited experience with VR.
Participant 2 (P2) reported less than three months of industrial experience.However, this volunteer has a background in aerospace engineering and engineering design spanning over nine years in total.This person self-rated as a novice in using VR and with even less familiarity with PCP.
Participant 3 (P3) spent seven years in an academic setting and almost a year in the industry.This person reported to have not previously used any VR headset.However, this participant is an expert on the PCP technique and its applications in design.
Participant 4 (P4) reported 16 years of academic experience and three years of prior work in the industry.This participant is also an expert on PCP methods applied in modeling of the design processes.
Participant 5 (P5) has 21 years of academic (15 years) and industrial (6 years) experience in computer science and software engineering.This person also reported basic awareness with respect to VR and extensive exposure to PCP.
Participant 6 (P6) has a background in artificial intelligence and computer science (nine years).During that time, this volunteer also worked for three years with the PCP and multidimensional data visualization methods.
Participant 7 (P7) reported that he has an academic background in mathematics and computational fluid dynamics (eight years).However, this participant was a novice in terms of prior familiarity with VR and PCP.
Participant 8 (P8) was a doctoral student working on the application of analytical methods such us subspace-based dimension reduction to manufacturing deviations in turbomachinery components.
This participant also self-reported having some limited experience with the VR and 2D PCP.
Participant 9 (P9) self-reported an intermediate level of familiarity concerning both 2D PCP and VR.P9 was also an expert in the field of mechanical and manufacturing engineering with vast experience of multi-objective optimization in academia and industry.
Participant 10 (P10) reported having some limited prior exposure to VR and advanced knowledge about the 2D PCP.This person used the 2D PCP for conducting research for a master's thesis in aerospace engineering.P10 also holds an undergraduate degree in the same area.In addition, this participant also recognized the dataset in which he looked for the patterns, as P10 used the same data for prior research involving 2D PCP.However, this person reported not spending a substantial amount of time on the analysis of this data, and P10 did not previously try to specifically identify the contained patterns.
Participant 11 (P11) self-reported to never before using the VR headset and to have some limited prior exposure to 2D PCP.This participant also reported to have undergraduate and postgraduate degrees in mechanical and aeronautical engineering and is currently in his first year of doctoral studies in aerospace engineering.

C. Identifying Patterns
Kipouros et al. [1] provided an analysis that discovered two patterns of importance, shown in Fig. 2b.As the study involved a real-world dataset (DS1), participants were instructed to note that some variability was to be expected in the data.However, they had to decide independently what level of variability would be acceptable.Only two participants (P2 and P5) failed to recognize most of the data points belonging to the two groups as shown in Table 1.In three cases, the participants changed their initial pattern candidates by selecting or deselecting elements originally grouped, thus creating subsets or partial repetitions of the previous selection (P1, P4, and P6).The majority of the participants identified almost the complete pattern A (with completion ratios for P3: 9/11, P4: 8/11, P6: 9/11, and P7: 6/11) or B (with completion ratios for P1: 16/19, P3 :16/19, P4: 15/19, P6 :19/ 19, and P7 :15/19).These differences most likely come from limited knowledge of the area from which the dataset was sourced and, as such, the allowed levels of noise between the respective values.

D. Observed Behavior
The results of the study involving the first seven participants (P1-P7) suggest that the time needed for the analysis was correlated with prior experience with PCP; that is, the more expertise the participant had, the longer it took to finish the training session.Simultaneously, experts took slightly less time to complete the requirements of the main study.This effect may be caused by the bias developed while exposed to 2D PCP.However, these data should be interpreted with caution.The participants did not have a clear cutoff point.For example, the study only finished once all the potential patterns were found as judged by the participant.There was also an age difference between the two groups; novice participants were in their 20s (P1, P2, and P7), whereas experts were in their 30s (P3, P4, and P6) or 40s (P5).Furthermore, the experts had little or no familiarity with VR or the controller, which put them at a disadvantage compared with younger participants, such as P7, who reported extensive experience with this type of gamepad.This may have impacted their approach used during the study.
All the participants quickly became fluent in their chosen movement and manipulation strategies and did not report any comments or suggestions directly linked to these interaction techniques.
None of the participants generated more than a single scatter plot comprising different dimensions, even though they were informed about this possibility existed.Such behavior, however, may have been observed because only three criteria were present in the dataset.Hence, all of them could be mapped onto a single scatter plot.Moreover, the training phase also exposed participants to the methodology of exploring the data mainly through the outputs' scatter plot, which was also used by Kipouros et al. [1] in their exploration of this dataset.However, some of the participants (P1, P2, and P6) reflected on the fact that having more than a single scatter plot may be confusing.
We observed that participants P4 and P6 chose to reposition the scatter plot closer to themselves, or the particular part of the main plot, instead of moving toward the scatter plot floating above the center of the IPCP plot.This behavior would most likely change if more data items were visualized at once.Therefore, the movement would take considerably more time, and the perspective would render the distant selection of the elements at the scatter plot almost impossible.
Only participants P2 and P6 chose to rotate the scatter plot.However, we observed that for all participants the exact mapping of the dimensions to the 3D projection was not critical, and this was also confirmed later in validation.The participants were selecting clusters on the scatter plot irrespective of the current rotation of the projection.One participant (P3) was constantly selecting the same, largest cluster, instead of exploring other groups or individual points.

E. Design Principles
From the technical constraints of what is achievable in VR, prior work [5], and literature review, as well as from expert feedback [1], and observations from our study, we distilled six design principles (D1-D6).
D1 is immersive PCP visualization.The IPCP design should leverage the benefits bestowed by the immersive, spatial environment and remain both useful and compelling to the user.Prior work [20] has shown that standard 2D PCP may not benefit from translation into three dimensions.In fact, it can yield worse results due to, for instance, occlusion effects.Hence, the immersion bestowed by the VR environment should be considered as a key factor when designing a 3D PCP.
D2 is pattern candidate identification.The user must be able to identify candidates, in other words, a group of data items, as possible patterns within the dataset.This is the key functionality, as the main purpose of the PCP visualization is aiding pattern identification within the given multidimensional dataset [10].
The next four design principles D3-D6 are derived from the first two principles D1 and D2.Effective interaction, especially in the case of complex, multidimensional visualization in 3D space, requires the possibility of swift dataset inspection (D3) and exploration (D6).In turn, these two factors influence how the patterns are compared (D4) and stored (D5) for future reference.Without such functionality, the user may not be able to gain an overview of the whole, or parts, of the visualization and effectively analyze the IPCP.This is due to the sheer size in terms of the number of data items and how much space such a visualization can occupy in a spatial VR environment.Further, as the patterns may overlap, the system has to provide the user with the means of comparing previously found patterns with new candidates.
D3 is pattern candidate inspection.The user has to be able to inspect candidates for potential patterns within the dataset to decide if the particular group of data items forms a pattern.
D4 is pattern candidate comparison.To avoid repeated selections of the same pattern, the user has to be able to effectively compare a group of data items that could potentially form a pattern but also be able to compare such a group with previously discovered patterns.
D5 is pattern storage.The possibility of saving and retrieving ondemand information about discovered patterns is necessary for the users to be able to differentiate between potential new candidates and previously discovered patterns.
D6 is visualization exploration.The system should support exploratory visualization by providing mechanisms for zooming and exploring individual visualization elements.

F. Task Analysis
We performed a task analysis by first identifying a suitable user interface workflow guided by the previously described user study.Each task is broken down into functions carried out either in series or in parallel.Thereafter, each function was translated into a function carrier (solution) by considering the requirements, technical constraints, and data from the user study.This analysis was carried out by first modeling the function structures with the help of the Function Analysis Systems Technique (FAST) [29] (see Fig. 7).Next, by using information gained through the formative user study in the form of observation and user comments and discussion, we have arrived with a list of six key tasks.

D1: Visualize Data Using IPCP
This task can be implemented in many variants in both 2D and 3D settings.For instance, one can use parallel planes [20,23] to visualize the parallel axes.Instead, in our case, each data item is essentially a series of connected, interactive cubes spaced in equal intervals over the X axis.Identically to 2D PCP, the Y component denotes the value for each dimension.However, as we are spreading the points also in the third dimension, the cube's position in Z axis signals the data item indexing with the first element starting with the Z component equal to zero and progressing toward the positive direction of the Z axis (see Fig. 3 or 5).As opposed to other 3D implementations, dropping the parallel planes limits the potential effects of occlusion and allows users to gain a birds-eye overview of the dataset by positioning themselves above the IPCP.
None of the participants reported confusion concerning the way the system visualized PCP.Moreover, all of them quickly understood that the interactive cubes indicate the numerical values in each Table 1 The groups of data items identified as potential patterns by the participants (P1-P7); color coding of these data indicates the overlap with the two patterns (two bottom rows) identified by Kipouros et al. [1] as can be seen in Fig. 1  dimension and that a group of line-connected cubes constitutes a single data item.

D2: Pattern Identification
This task can be carried out in two ways: via exploration of the IPCP and visual inspection of a subset of selected data items or with the scatter plot, as it was strongly suggested to the participants during the user study.However, we observed that the participants preferred a mixture of both approaches; that is, in many cases, a participant augmented a subset of data items selected with the help of the scatter plot by adding or removing selected data items individually from the PCP visualization.

D2*: Scatter Plot Generation
This task provides a feasible approach to pattern identification, especially when handling a large volume of data visualized with IPCP.The IPCP is augmented with the help of scatter plots [1,21] generated from values in one, two, or three dimensions across all the data items.We gave the participants the ability to create up to five different 3D scatter plots.We also advised our participants to start by selecting the criteria to develop the scatter plot and see if they can use this aid to identify clusters of data points on the scatter plot to help them efficiently select a group of data items that can potentially form a pattern.
The scatter plots were augmented with additional functionality, such as the ability to apply rotation in 90 deg about a chosen axis [6][7][8].In addition, they could be moved into any position in 3D space [6][7][8].
Based on the participants' comments, we decided to contextualize the data points and scatter plot axes by providing textual labels, as seen in Figs.1a and 1b.The labels contained information of which dimensions, that is, which Y values, were mapped to which coordinates on the scatter plots.However, the two expert participants suggested a reevaluation of this design, as they found the labels not to be very helpful.
If two or more values in a particular dimension are identical within the floating-point precision of their coordinates, they will be presented as two entirely overlapping cubes on the scatter plot.This effect would interfere with the ray tracing used for gaze tracking as the system has no means to decide with which overlapping object the user wants to interact.The analogous situation may also happen in the 2D PCP.In such a case, only a single point will be visible on the scatter plot, the selection of which will highlight multiple corresponding items on the IPCP as those points would be automatically clustered with each other.The simple clustering algorith (see Fig. 8) was nonetheless quite effective.The clustering method only considers the Euclidean distances between the cubes' (data points) centers, from which we estimate the level of overlap between the cubes.Certain restrictions, such as a difficulty of differentiating with which object users want to interact if they are entirely overlapping, cannot be easily solved.We conjecture that different clustering algorithms for multidimensional data, such as one proposed by Wegman and Luo [30] might be applied here.One way to minimize occlusion and cluttering effects in PCP is to use the density information of the data.This is shown by Heinrich and Weiskopf [31] and also by Artero et al. [32], who proposed algorithms that use frequency and density information to decrease the overall number of data points that are shown to the user in PCP.Moreover, giving the user the possibility of zooming in and out of the cluster internal structure may prove to be very convenient in such situations.

D3: Inspection of Selected Data Items
This task is made possible through visual inspection of the selected items by either rendering not-selected points temporally invisible or by making a copy of the subset and placing it in a chosen part of 3D space.In both cases, participants usually first tried to gain an overview of the entire selection by positioning themselves (i.e., zooming out) in front of the subset and then compare and decide if particular data items constitute a pattern.

D4: Comparison of New Candidate to Other Patterns
This task is achievable by placing candidates near each other in 3D space and then visually inspecting possible overlaps.However, one of Fig. 7 The FAST diagram [29] used to reason about the requirements of IPCP.The horizontal axis shows abstraction level: higher-order functions to the left are gradually decomposed into lower-order functions to the right.The vertical axis shows order of execution.
the participants mentioned that having some analytical aid could be helpful, especially if the data items are confounded by many dimensions.This participant also suggested using color coding to provide additional information on how particular subsets potentially overlap with each other.

D5: Storage of Identified Patterns
The system automatically logs a list of data items selected by the user into a file.Second, the subselection of data items can be placed and stored in any part of the 3D space throughout the entire duration of the visualization.However, as the IPCP extends along the Z axis, it may be difficult to compare points that are placed on the opposite sides from each other.
In addition, participants also remarked that a combination of both of these methods could potentially be even more beneficial.The logged data can be made recallable, visualized on demand to assist the user in understanding, for example, if, and to what extent, patterns overlap each other by additional color coding or temporal shape change.

D6: Exploration of Visualization
The movement and manipulation methods allow the user to zoom in or zoom out on any visualization element, which, as reported by the participants, was easy to use with either the Xbox controller or Oculus Touch (see Fig. 6).
Based on the gathered qualitative data, all the participants swiftly understood the proposed movement and manipulation techniques.However, participants unfamiliar with the controller had problems remembering which button activated which action.This problem can be rectified by several strategies.First, the training can be extended.Second, the feature set can be narrowed.Third, the feature set can be initially limited and gradually expanded as the user gains proficiency in the interface.This third strategy is an example of a training wheel interface [33].The ideal solution is probably a combination.In the short term, extending training is preferable as PCP, even in its 2D version, demands intensive practice to gain fluency in their usage [28].Another possible option would be to use a different VR headset (e.g., HTC Vive) that supports extended spatial tracking of the wearer.In such a case, instead of using a controller for movement and maneuvering, the user can move around in the real world, which would be instantly reflected in the VR environment.This would decrease the number of required interaction methods, although it would raise other issues, such as the increased complexity in tracking setup and space demands for the system.

A. Interface Verification
We verified the usability of the system using the widely adopted Nielsen heuristics [34,35].

Visibility of System Status
The system constantly informs the user which visualization elements are in a selected state on both the IPCP (see Figs. 3 and 5) and on the scatter plots (see Figs. 1 and 8).The selected objects remain highlighted (in light green or orange) up to their deselection or when the user invoked visualization reset.This provides the user with continuous feedback about the state of all individual elements.Moreover, the system supports linking and brushing interaction techniques by automatically mapping the current state of each of the visualization elements, in other words, scatter plots and the IPCP onto each other.Furthermore, if the user reaches the limit of possible duplicates, the system will disallow this operation and signal this to the user by playing a short sound.

Match between system and Real World
Both the 3D scatter plots and the cross-hair-based gaze tracking [5][6][7][8] should be familiar to the majority of users.The use of the cross-hair to represent the user's gaze point and the focus the user's attention on an object directly in front of where the user is looking at the moment was implemented based on a well-known realworld concept (see footnote **).This, in turn, allows the system to optimize its resources to provided immediate feedback and response to the user's actions.

User Control and Freedom
The user can reset [5][6][7][8] (see button R in Fig. 6) the visualization to its original state, which will preserve the copied data items (see Fig. 5).However, due to efficiency reasons, these duplicates will no longer support any kind of interaction after the system reboots and remain in a freeze state based on where they were originally copied.To further foster user control and freedom (before validation with the larger dataset), the system was extended to support the undo and redo operations; in other words, the last selection of a data item on the IPCP or data point on the scatter plot can be undone or redone if necessary by the simultaneous press of two thumb-rest buttons (one on each of the controllers; see button H in Fig. 6).

Consistency and Standards
Despite the recent rapid development and advances in the research, VR technology is still in its early stage.As a consequence, the standards regarding the interaction techniques, visualization guidelines, or the overall design principles are not yet fully understood.We were trying to be consistent with well-known concepts; thus, for instance, the selection of the object requires a double-tap of the gamepad button instead of a single click, similar to what is necessary when selecting an object using a computer mouse.

Flexibility and Efficiency of Use
The accelerator provided for the benefit of advanced users was the possibility to generate multiple scatter plots.Each of them could map up to three dimensions from the IPCP.Moreover, some users possessed previous experience with the Xbox controllers (see Fig. 6) and therefore required substantially less time to acquaint themselves with the interaction techniques.

Error Prevention
The most common slip type of an error related to the user failing to focus his or her gaze on a chosen point is easily fixable with maneuvering techniques which allow the user to zoom in the object by just moving toward it.The scatter plots automatically deselect the rotation selector when another one is being selected by the user [5][6][7][8].This prevents the system from confusion about which axis the scatter plot should be rotated.Moreover, the system will omit visualization of the data items on the PCP and the scatter plots if there is a gap in values at the input dataset, hence preventing the user from working with broken or incomplete data.

Help Users Recognize, Diagnose, and Recover from Errors
The system uses the audio feedback to inform the user if no more duplicates of the selected data items can be generated immediately after such an action was performed.

Recognition Rather than Recall
The linking and brushing interaction techniques help to minimize the user's memory load by providing constant feedback about which operations have been previously executed by the user.Moreover, the user can easily duplicate a group of data items for further inspection and comparison with other such groups.This feature can be used, for instance, to keep track of which group of points were previously recognized by the user as a pattern.

Aesthetic and Minimalist Design
The systems keep the information presented to the user to a minimum.Only selected elements are highlighted in green or orange (see selected: movement selector in Fig. 1a, data items in Fig. 3, and cluster in Fig. 8c).Moreover, the gazed-locked textbox is only shown once the user's gaze is hovering over a data point on the scatter plot and automatically disappears otherwise.Thus, minimizing the possible collateral cluttering and occlusion effects.The textbox also contains additional information of which dimensions were used to map into this data points' rearrangement on the particular scatter plot.

Help and Documentation
As it is hard to find users experienced in both the PCP and VR, the participants were trained in how to read the PCP and how to interact with our system through the gaze tracking and the Xbox controller (P1-P7) or the Oculus Touch (P8 and P9).

B. Validation
It is very challenging to meaningfully validate visual analytics tools in short-term laboratory-based studies.Traditional humancomputer interaction (HCI) evaluation methods, such as A/B testing, rely on the experimenter's ability to explicitly control all controllable and uncontrollable parameters of the system and the task as much as possible.However, in visual analytics, this parameter space is very large.For example, the data density, the nature of the task, characteristics of the dataset used in the test, and the experience and motivation of the participants are all variables that are likely to affect the outcome of an A/B test.In addition, easy-to-measure dependent variables, such as task time, are only partially relevant as one of the primary objectives of a visual analytics tool is to provide additional insight rather than enabling users to make decisions faster.
As a consequence, we validated the system design qualitatively in two independent user studies.We recruited two pairs of different surrogate expert users to participate in each study.These participants are referred to as P8 and P9 in the case of the study involving the same dataset as in the previous study (DS1) and P10 and P11 with the new, larger, more complex dataset (DS2).All four volunteers were also prescreened with the complete version of the Ishihara's color deficiency test [26].To validate the system design, we used the same three-part procedure and dataset DS1 (see Fig. 3) as in the first study reported earlier in this paper.However, based on the previous participants' feedback, we skipped parts of the presentations related to having multiple 3D scatter plots, although this possibility was not removed from the IPCP.As such, the training phase was truncated.Overall, the results revealed that participant P8 managed to find the majority of the elements in both patterns, Awith a 9/11 completion ratio and B with an 18/19 completion ratio, whereas the second participant P9 correctly identified only the entire pattern B with a 19/19 completion ratio.

Part II: 166 Data Items with 39 Dimensions per Element (DS2)
After extending the system with additional capabilities, such as the support of basic undo and redo operations, we reran our validation study with another two domain experts (hereafter referred to as P10 and P11).We also used the Oculus Touch instead of the Xbox gamepad controller to facilitate the interaction (see Fig. 6).
We further investigated the feasibility of using IPCP for pattern identification in a substantially larger and more complex dataset DS2.The user study was carried out over a new dataset containing 166 data items with 39 dimensions per element containing four patterns of interest identified by domain experts (see Fig. 5).This dataset describes the results of the optimization study of S-duct design (see Fig. 4), as presented by D'Ambros et al. [4].
In addition, the near misses in terms of selection were the main observable issues with our selection technique.To support the user's selection task, one can use a form of target acquisition aid, that in our case has been loosely inspired by the Bubble Cursor proposed by Grossman and Balakrishnan [36].We leveraged the features natively supported by the Unity game engine by on-the-fly manipulation of the size of the sphere collider ¶ ¶ component attached to the interactive objects (cubes) in the IPCP.The size of the collider, which acts as a target for the ray tracer (see footnote **), is dynamically adjusted based on the size of cross-hair (see the orange circle in Fig. 5), which in turn is dynamically scaled with respect to the user's distance from the target objects (see footnote **).Initially, we set the collider's radius as 3 p ∕2 ≈ 0.866 to encompass the entire cube in the sphere.The further away the user is from the object, the larger the collider becomes.
Table 2 Group of indices of data points identified as patterns by the participants P8 and P9; the two bottom rows contain the two patterns that were identified by Kipouros et al. [1] as can be seen in Fig. 1  P10 found 5/8 of pattern C, 9/13 of pattern E, and almost entire pattern F: 11/12.P11 found 4/8 of pattern C, 4/13 of pattern E, and 9/12 of pattern F in his best attempt.However, none of the two participants found the pattern D. They did, however, identify other potential patterns that were not initially found by the original researchers, which is a concrete demonstration of the utility of the system in enabling users to explore and identify previously unknown patterns in the dataset.Interestingly, some overlapping results were reported by both participants, as can be seen in Table 3, namely, pattern candidate 6 for P10 and pattern candidate 2 for P11.Moreover, both participants identified additional groups of points that in their opinion constituted patterns not originally identified by the experts in the work by D'Ambros et al. [4].As mentioned before, in the case of visual analytics, we aim to provide additional insight into the dataset; hence, we were interested in the new candidates for potential patterns found by the two participants (P10 and P11).To do this, we requested a domain expert to reanalyze and comment on the new pattern candidates identified.After inspection and preliminary analysis, the expert concluded that indeed these are different patterns sharing some common characteristics with those identified previously and as such they are complementary.However, the domain expert commented that they are not as rich as the dominant four patterns previously identified, in other words, C, D, E, and F. Nonetheless, this tentatively indicates the VR analytics system can empower users to detect new patterns.

VI. Design Decisions
We summarize the final design decisions refined after we ran additional experimental studies with domain experts over two separate datasets (see Figs. 3 and 5) varying in both the number of data items and dimensions per data item as well.Moreover, based on the participants' observed behavior and their comments, we also identified a number of potential improvements.

A. D1: Immersive PCP Visualization
We have developed IPCP as a software system running within the Unity (see footnote ¶ ¶) game engine.Our approach represents a 3D version of regular 2D PCP but with values of each dimension denoted as volumetric, interactive objects, that is, cubes spread in the Z axis instead of using parallel planes as regular 3D PCP usually does.However, this way of presenting data has its own challenges.For instance, to clearly see the distant objects with respect to the user's current position, it was necessary to provide adequate means of close inspection of such regions.Here, we give the user the possibility to freely move and maneuver in the 3D space using the controllers (Xbox gamepad in first and Oculus Touches in the following two observational studies).Such an approach has additional advantages, especially when we coupled the IPCP visualization with the CAD models of the respective designs (see Fig. 4), which now can be inspected not only from outside but also from within (see Fig. 4c).Another possibility is to give the user means of manipulating the entire PCP visualization with the user's hands as shown in Figs.9-12.

B. D2: Pattern Candidate Identification
In the context of PCP, the pattern is understood as a group of data items sharing similar To identify and highlight a group of data items that can potentially form a pattern, the user either selects data items one by one or by generating a 3D scatter plot from up to three dimensions.These data are automatically clustered based on the distance between the data points (see Fig. 8) and the selection of such data points belonging to a cluster simultaneously highlights points belonging to this cluster on the scatter plot and the IPCP alike.The larger the dataset and subsequent IPCP visualization, the more the scatter plots seem to aid the user in the task of finding patterns.Here, we noted that the actual orientation of the scatter plot's local axes is not as important as providing the user with a clear way of distinguishing which points on the scatter plot potentially form a cluster.We observed that even a simple clustering algorithm, however, relying on the user's intuitive feel of the grouping of volumetric points placed in close vicinity, can be surprisingly effective.Hence, potential improvements could be threefold.First, other clustering algorithms can be studied here such us Ordering Points to Identify Cluster Structure (OPTICS) [37] or Density-Based Spatial Clustering of Applications with Noise (DBSCAN) [38]  Table 3 Group of indices of data points identified as patterns by the participants; the four bottom rows contain the four patterns that were identified by domain experts [4] (in the case of P11, we excluded repeated pattern selections) manipulation can be used.For instance, instead of rotating the scatter plot along with one of the three main axes, it could be rotated about any vector selected by the user (see Fig. 10).These would potentially help the user swiftly obtain an overview of all the data present in the scatter plot.Third, to avoid repeated selection of the same cluster, and subsequent repeated selection of the same group of data items on the IPCP, we could additionally color-code or otherwise mark previously selected and inspected clusters.Furthermore, additional information could be given in the form of textual labels attached to the data points instead of current axes mapping, such as the scatter plot's coordinates or a number of points within the selected cluster, which could be more informative to the users.

C. D3: Pattern Candidate Inspection
To inspect the pattern candidates, the user can temporarily disable the unselected data items of the IPCP and use the manipulation and movement capabilities in the 3D space to verify if this particular group of data items forms a pattern.Moreover, if a particular data item is upon inspection deemed by the user to not be part of this pattern, it can be easily deselected as well.This is especially useful the more data items are contained within the dataset.As the data items will be spread in the Z axis (see Figs. 3 and 5), the potential occlusion effects are more persistent with the growing number of data items.Here, the ability to toggle between the view with all or only currently selected data items has proven to be very useful.

D. D4: Pattern Candidate Comparison
To compare the currently selected group of data items with previously recognized patterns, the user can make a copy of these data items, in other words, the pattern candidate, and place them in a part of 3D space where the other patterns have been stored.The user may then visually compare them to decide if a new pattern has been found.Here, the system should provide the user with the snapping mechanism that would automatically align the subsets of the data items that the user wants to compare with each other.To some extent, the participants were trying intuitively to obtain a similar effect by placing the duplicates of previously found patterns next to each other in the 3D space.

E. D5: Pattern Storage
Previously discovered patterns of data items can be stored freely in any location within 3D space.Here, even though we could keep the frozen data items as noninteractive for optimization reasons, we could color code the data items overlapping across the previously found patterns and the new group of points to inform the user to what extent they share the same subset of data.

F. D6: Visualization Exploration
Exploration of the entire visualization with all of its elements is made possible through the use of gaze tracking, head tracking, and movement assisted by the controllers.Here, we observed that all our participants swiftly gain fluency in how to move around the virtual space.We can also leverage the tracking of the user in a real space, especially if we combine this with the bimanual manipulation of the PCP elements (see Figs. [9][10][11][12].However, in our case, the IPCP (depending on the input data size) can be potentially vastly extended in at least the XZ plane parallel to the user actual ground plane.As such, it could be unfeasible for the user to walk and see the entire visualization; hence, new interaction and transportation methods would have to be included.

VII. Visualization Extensions
A. Extensions of IPCP After analyzing both the qualitative and quantitative feedback together with the participants' comments and our own observations, we have decided to extend our system with a number of additional features.We have also decided to explore a completely new method of interacting with the system using hand-tracking-based gestural input coupled with gaze tracking (see Fig. 13).The changes were twofold.
First, we have equipped the system with the CAD 3D models of the individual designs to allow the user to inspect and asses the associated 30 geometries on the fly while selecting data items on the IPCP (see Fig. 4).
Second, instead of using hand-held controllers (see Fig. 6), this system iteration explores gaze tracking coupled with hand tracking supported by an additional sensor (see Fig. 13), namely, the Leap Motion Sensor (see footnote § §) attached to the VR head-mounted display to track and recognize the user's hand gestures.
Finally, we report the results of an interface verification of the system guided by Nielsen's heuristics [34,35].This allows us to identify the main usability bottlenecks and issues with respect to the new gesture-based interaction method.

Coupling IPCP with Computer-Aided Design Geometries
The system contains the visualizations of associated 3D models of the full 3D S-duct geometries and 2D flow solutions, an example of which can be seen in Fig. 4. In the future, we are also planning to give the users the possibility of overlaying the various S-duct geometries on top of each other to closely inspect and identify potential nuances in the difference between the shapes as demonstrated in the AeroVR system presented by Tadeja et al. [7,8] and upon which this visualization is built.This allows the users to promptly verify their pattern selections and compare between the S-duct shapes associated with selected data items.This extension also fosters the connectivity with commercial software and simulation packages that are widely used in the industry.

Hand Tracking and Bimanual IPCP Manipulation
The user can manipulate all the visualization's elements with the help of a new, bimanual interaction technique and a range of gestures (see Fig. 13) without the need of using hand-held controllers (see Fig. 6).This was achieved through a mixture of gazetracking and hand-gesture recognition.The left-hand menu (see Fig. 13a) recalls the hand-attached menu with all the available options: 1) [RESTART] resets the visualization to default settings, 2) [DUPLICATE] duplicates the selected data items at the location in 3D space the user is currently gazing while pressing the button, 3) [UNDO] reverts the results of any manipulation of either the IPCP or scatter plot depending on which of the two is selected while pressing the button, and 4) TOGGLE toggles between the full view and the view with only selected data items in the IPCP.The thumbsup gesture (see Fig. 13b) is used to release selection of an interactive object.The pinch gesture (see Figs. 13c and 13d) executed with the index finger and thumb allows the user to select and control interactive objects.After gazing on an interactive object, the user can activate or deactivate control over this object by performing the pinch gesture.
In case of the IPCP main plot, the users can select (see Fig. 9) either a single data item using the single-pinch gesture (see Fig. 9b) or the entire dimension across all data items using the double-pinch gesture (see Fig. 9c).In addition, there are two interactive movement selectors situated above and below the main plot.These selectors have a form of spherical markers whose, similarly to scatter plots, color coding switches to orange after selection as seen in Fig. 11 or in Fig. 12.After activating one of these selectors, users can manipulate the whole IPCP by doing the following: 1) Rotate the 3D scatter plot.See Fig. 10; part a shows the scatter plot generated out of three criteria dimensions of the DS2 dataset (selected in red), b illustrates the user enlarging and simultaneously rotating the scatter plots using both hands, and c depicts the user gazing over a selected cluster of data points in the enlarged scatter plot, which results in a text box with additional information.
2) Move the IPCP main plot.See Fig. 11; part a is gazing at the IPCP's selector (red sphere) and selecting it with the double-pinch gesture, after which the user can in part b move the IPCP with either hand.
3) Scale the IPCP main plot.See Fig. 12; after selecting the IPCP's selector (orange sphere) and articulating a pinch gesture with both hands, the users can scale the IPCP by moving their hands further apart or closer together (a-b).
The 3D scatter plot can be generated out of a maximum of three dimensions selected on the IPCP, and it can be manipulated similarly to the main IPCP (see Fig. 10).Moreover, the system could allow the user to select a group of points that constitute a cluster based on one of the selected clustering algorithms such as the earlier mentioned OPTICS [37] or DBSCAN [38].A selection of the algorithms can be attached to the left-hand pop-up menu in place of the empty buttons shown in Fig. 13a.Any other actions can be attached to those buttons, as well.For instance, we could give the user the possibility of attaching actions facilitated with the execution of the user-indicated external script or another program, thus, vastly enhancing the IPCP's capabilities and simultaneously rendering IPCP more suited to the user's individual needs and wants.

B. Verification of Extended Interface
The usefulness of basic IPCP and 3D scatter plot designs were verified and evaluated previously with the user studies in which the domain experts were able to not only rediscover previously found patterns in the data but also, to a limited extent, to discover new ones.Here, the updated version of IPCP was equipped with the new gesture-based interface.As such, the main goal here is to verify the impact this new, bimanual interaction technique has on the interface and to reason about the potential benefits brought with it to the user.To achieve these, we apply the formative evaluation method; in other words, we assess the interface's usability with the help of Nielsen's heuristics [34,35].

Visibility of System Status
The user is continuously informed about the current selection of data items at the IPCP, data points on the 3D scatter plots (see Fig. 9), activation of objects' selectors (see Fig. 10), and buttons selection (see Fig. 13a).This is achieved through automatic color coding and highlighting.

Match Between System and Real World
The interface provides the user with the possibility to use gestural input coupled with the gaze tracking to select, manipulate, and interact with the main visualization elements the same way as the user would in the nonvirtual world.These include the possibility to 1) rotate (see Fig. 10), 2) move (see Fig. 11), or 3) rescale the size (see Fig. 12) of either the main IPCP plot or the 3D scatter plot alike.

User Control and Freedom
The system features both [UNDO] and [RESTART] options (see Fig. 13a).The former allows the user to undo the last executed manipulation of either the 3D scatter plot or the IPCP.These operations will not result in the loss of the previously made selections of data items nor data points.The restart (see Fig. 13a) will cause the entire visualization to be restored to its defaults.

Consistency and Standards
In the case of the interactive objects'selectors, the same convention is followed; in other words, both the scatter plot and IPCP main plot have the selectors in a form of large, spherical markers that have to be selected with the double-pinch gesture while the user is gazing over them.

Error Prevention
The execution of the two operations that may cause irreversible effects on the current state of the visualization, in other words, the [UNDO] and [RESTART], require the double press of a button to be executed instead of a single press required for the execution of the remaining options.

Recognition Tather than Recall
The user can execute the palms-up gesture which instantaneously recalls the left-hand pop-up menu with all the available options (see Fig. 13a).

Aesthetic and Minimalist Design
Only the main IPCP is initially visible to the user unless the user decides to make copies of the groups of selected data items or generate the scatter plot.However, if no dimensions on the IPCP are selected, the scatter plot will not be visible.

VIII. Discussion
The IPCP tool automatically provides a rudimentary form of collaborative design environment.The first-person view of the headset wearer can also be simultaneously streamed on the PC display as well.As a consequence, multiple designers can look at the steps undertaken by the user and easily collaborate having a verbal dialogue between themselves and the person using the headset.This capability was observed by the authors while working on the design and verification of the IPCP tool presented in this paper.The advantages of such a setup will be further studied in a series of more controlled user studies.

A. Limitations
The evaluation with a group of surrogate expert users, P8 and P9 for the smaller (DS1) and P10 and P11 for the larger dataset (DS2), revealed that users were indeed capable of identifying patterns in both datasets.However, while this evaluation demonstrates the efficacy of the system and serves to provide some evidence to consider the potential merits of 3D PCP, we advise caution against overinterpreting these results given the qualitative aspect of the findings.The sample size is very small, and as a consequence, the findings are highly sensitive to the individual participants' general ability to use novel immersive technology to analyze a demanding real-world multidimensional dataset.
A previous study conducted by Johansson et al. [20] indicated that 3D PCP was, in fact, worse than a conventional 2D version in both the metrics (i.e., response times and error rates).However, the study was conducted using a desktop computer connected to an external display instead of an immersive, VR environment, and thus it is unclear whether the findings generalize to a truly immersive environment.However, such a study is intrinsically limited in terms of the many study parameters that have to be arbitrarily set in order to be able to achieve an A/B comparison.Examples of such study parameters include data density and volume, saliency of data items, field of view, the user's sense of presence, situational awareness of a 3D world, and so on.The objective in this paper is to explore whether VR is feasible for PCP.Our results demonstrate that PCP is indeed viable for VR analytics and thus encourage and motivate further design exploration.However, as PCP in VR is still in its infancy, it is important to avoid a premature 2D vs VR comparison until an effective PCP in VR design has fully emerged.

B. Future Work
Future work includes conducting statistically robust quantitative studies that are able to tease out the merits and flaws of immersive PCP visualization compared with status quo 2D PCP visualizations.However, such studies are difficult to carry out due to the many controllable and uncontrollable parameters of the system and the experimental task.This makes the design of a robust and meaningful experimental design highly challenging.Prior work has reflected about this problem, both in general for evaluating visualization tools [39,40] and specifically for the problem of comparing 2D and 3D visualizations [41].
For the near future, we believe a more fruitful approach is to iteratively evolve a highly promising PCP in VR approach by following the model of this paper of combining design with qualitative insights gathered from domain experts.Once such a design has fully matured, it will become more meaningful to carry out large-scale A/B comparisons capable of teasing out the likely nuances differences between PCP in 2D or VR.
We believe a promising area of future work is to further refine the system design by evolving the task model and the underpinning functions and revisit the process of translating functions to solutions.This can be achieved by identifying controllable and uncontrollable parameters for each overall function necessary to carry out a task and study the effects of modifying these parameters in isolation.
For example, the 3D immersive environment opens up a plethora of methods for selecting data points using input techniques such as head tracking, gaze tracking, touch controllers, game controllers, or direct manipulation mechanics.At the same time, selection feedback can be provided using a variety of output techniques, such as color coding, shape modification, sound, haptic feedback, or a combination.
Further, the data analytics toolkit can be extended with a range of new features, including explainable AI methods such as LIME [42], SHAP [43], or Anchors [44] to allow the user to query more complex questions about the data, such as the following: "Why were these points classified as noise and not included among any of the identified clusters?"Another interesting research direction is to investigate coupling the system with a voice interface.
Our system implementation can aid such future explorations of how to make informed decisions on, for instance, which combination of inputs and outputs to choose to arrive at an effective operating point for immersive data analysis and thereby assist designers in gaining a more complete understanding of the tradeoffs and their effects that are inherent in such design decisions.We believe such technological capability should be harmonized with state-of-the-art interactive computational engineering design methodologies [45] and thereby open up a new generation of engineering design software environments and systems.

IX. Conclusions
The qualitative results in this paper reveal that immersing the user within PCP in a VR environment is, at the very least, complementary and may provide ways to enhance the user's ability to identify patterns.Thus immersive 3D PCP can potentially be considered as a promising way of visualizing high-dimensional data.Most of the participants throughout these qualitative studies managed to identify the same patterns as recognized as interesting by Kipouros et al. [1], which can be regarded as the reference standard.The two participants working with a substantially larger dataset in terms of its complexity and size (the number of dimensions per data item and the total number of data items) discovered similar patterns as those given by D'Ambros et al. [4] and identified additional candidates for potential patterns.This demonstrates the potential for immersive VR in engineering design processes.
The main contribution of this paper is the six design principles that were distilled based on the technical constraints of what is achievable in VR, prior work [5], literature review, expert feedback [1], and from observations from this study.
The second strand of contribution is the task analysis carried out at the functional level and the translation of these functions into solutions that allowed the system to be realized.This scaffolding structure is useful in framing the findings from the preliminary qualitative user study in such a way that the outcomes of the user study could directly shape the refinement of the initial design.
Further, the authors gradually updated the hardware interface for the interaction between the user and the IPCP.They switched from an Xbox gamepad to Oculus Touch controllers specifically designed for VR (see Fig. 6) and replaced the controller-based interface with gestural input coupled with gaze tracking (see Fig. 13).This enabled the user to manipulate the IPCP plot and 3D scatter plot in a manner similar to real-life objects.
To better inform the user's decision making and analysis of the dataset, the authors coupled the individual data items with additional information.These supplementary data can, among other things, include the 3D geometries, such as the S-duct designs for DS2 or the 2D plots of the flow going through the duct as shown in Fig. 4.
In summary, the results in this paper serve as a promising indication that immersive parallel coordinate plots may enhance decision making in complex design engineering processes, which are intrinsically reliant on the user being able to analyze and reason about multidimensional data.

Fig. 1
Fig. 1 3D scatter plot: a) the 3D scatter plot [7,8] generated from selected IPCP dimensions, b) zoomed-in volumetric data points, and c) the cones marking mapped dimensions on the IPCP.

Fig. 3
Fig. 3 The user's field of view of the 29-dimensional dataset of 54 equally optimum design configurations describing the Pareto front (26 design parameters and 3 objective functions).

Fig. 4
Fig. 4 After the selection of an item on IPCP, the user is shown additional information, for example, an S-duct CAD geometry and a 2D plot of the S-duct flow.

Fig. 2
Fig. 2 The left plot shows the complete dataset in the standard 2D PCP.The two patterns are shown in red and blue on the right plot.Adapted from Kipouros et al. [1].Copyright 2013, T. Kipouros, A. Inselberg, G. T. Parks, and M Savill.

Fig. 6
Fig. 6 VR controllers: a,b) an Xbox gamepad and c,d) Oculus Touch.The green markers are showing the mapping between the action buttons on both types of controllers.

Fig. 8
Fig.83D scatter plot: a) the 3D scatter plot[7,8] and b,c) the results of a simple clustering algorithm applied to overlapping data points.

Fig. 9
Fig. 9 Data item selection: a) gazing with a cross-hair, b) selecting data items with single-pinch gesture, c) or selecting the entire dimension with doublepinch gesture.

Fig. 11 TheFig. 12 Fig. 10 3D
Fig. 11 The IPCP movement procedure: a) gazing over and selecting the IPCP's selector with a double-pinch gesture and b) moving the IPCP with either hand.

Fig. 13
Fig. 13 The recognized gestures are a) the left-hand menu, b) the thumbs-up gesture, and the pinch gesture c) before and d) after execution.Icons by Icons8 (https://icons8.com).
in red and blue, respectively in blue and red, respectively (here, we report only the participants' best attempts to identify the two patterns) Unity Game Engine, "Unity," https://unity.com/[retrieved July 2018].