UTOvKrsOty UL CGSbrOdMK :rGIkOTM GTd MUdKRROTM MUtOUT LUr BOUSKIhGTOIGR ATGRysOs Vnyrevs Vristiyou Higheg HUll CUabridge Ubivergihm Ebgibeeribg DedUrhaebh GigbUl Drcceggibg Ubd CcaaibicUhicb Grcid Cchcber 5, 2010 ihis yisszrtvtion is suwmittzy for thz yzgrzz of Doxtor of ehilosophy Copyrighfi c⃝ JHIH Yndreys Yrisfiidofl Typeset by the author with the LATEX2ε documentation system Submitted Version, July, 2010. Resubmitted with minor corrections, October, 2010. SOMTGR PrUIKssOTM GTd CUSSuTOIGtOUT GrUup Department of Engineering University of Cambridge Trumpington Street, CB2 1PZ Dexlvrvtion I, Andreas Aristidou of Hughes Hall hereby declare that, except where specifically indicated in the text, the work submitted herein is my own original work and includes nothing which is the outcome of work done in collaboration. This dissertation is not substantially the same as any that I have submitted for a degree or diploma or other qualification at any other university. This dissertation is the result of my own work. I also declare that the length of this dissertation is less than 65,000 words and that the number of figures is less than 150. Signature: ............................................ Andreas Aristidou Cambridge October 5, 2010 i Vwstrvxt This thesis focuses on the problem of determining appropriate skeletal configurations for which a virtual animated character moves to desired positions as smoothly, rapidly, and as accurately as possible. During the last decades, several methods and techniques, sophisticated or heuris- tic, have been presented to produce smooth and natural solutions to the Inverse Kinematics (IK) problem. However, many of the currently available methods suffer from high computa- tional cost and production of unrealistic poses. In this study, a novel heuristic method, called Forward And Backward Reaching Inverse Kinematics (FABRIK), is proposed, which returns visually natural poses in real-time, equally comparable with highly sophisticated approaches. It is capable of supporting constraints for most of the known joint types and it can be extended to solve problems with multiple end effectors, multiple targets and closed loops. FABRIK was compared against the most popular IK approaches and evaluated in terms of its robustness and performance limitations. This thesis also includes a robust methodology for marker prediction under multiple marker occlusion for extended time periods, in order to drive real-time cen- tre of rotation (CoR) estimations. Inferred information from neighbouring markers has been utilised, assuming that the inter-marker distances remain constant over time. This is the first time where the useful information about the missing markers positions which are partially vis- ible to a single camera is deployed. Experiments demonstrate that the proposed methodology can effectively track the occluded markers with high accuracy, even if the occlusion persists for extended periods of time, recovering in real-time good estimates of the true joint positions. In addition, the predicted positions of the joints were further improved by employing FAB- RIK to relocate their positions and ensure a fixed bone length over time. Our methodology is tested against some of the most popular methods for marker prediction and the results confirm that our approach outperforms these methods in estimating both marker and CoR positions. Finally, an efficient model for real-time hand tracking and reconstruction that requires a mini- mum number of available markers, one on each finger, is presented. The proposed hand model is highly constrained with joint rotational and orientational constraints, restricting the fingers and palm movements to an appropriate feasible set. FABRIK is then incorporated to estimate the remaining joint positions and to fit them to the hand model. Physiological constraints, such as inertia, abduction, flexion etc, are also incorporated to correct the final hand posture. A mesh deformation algorithm is then applied to visualise the movements of the underlying hand iii i– skeleton for comparison with the true hand poses. The mathematical framework used for de- scribing and implementing the techniques discussed within this thesis is Conformal Geometric Algebra (CGA). ceywords The following keywords may be used for indexing purposes: Czntrz of gotvtion ZstimvtionA Computzr kisionA Conformvl Gzomztrix VlgzwrvA FiltzringA Forfivry Vny Wvxkfivry gzvxhing Invzrsz Kinzmvtixs (FVWgIK)A Hvny gzxonstruxtionA Hvny irvxkingA Humvn VnimvtionA Invzrsz KinzmvtixsA Joint ConfigurvtionA jnsxzntzy Kvlmvn FiltzrA bvrkzr przyixtionA botion CvpturzA ehysiologixvl xonstrvintsA kvrivwlz iurn boyzlC Vxknowleygements This is a great opportunity to thank those who supported me through my doctorate experience in Cambridge. Foremost, I am heartily thankful to my supervisor, Joan Lasenby, whose en- couragement, guidance and support from the initial to the final level enabled me to develop an understanding of the subject. Her perpetual energy and enthusiasm in research had motivated me as well as her regular supervision which made this study possible. It was an honor for me to have Prof William Fitzgerald and Prof Adrian Hilton as the examiners for my thesis. Their comments were constructive, enable a considerable improvement of both the content and the presentation of my thesis. I would like to thank them for agreeing to be my examiners and for making my viva a unique experience. I am indebted to many of my colleagues in Signal Processing Group for welcoming and supporting me over this period. More specifically, I owe an enormous dept to Jonathan for his continued help and for introducing me to Geometric Algebra and quaternions and to Paris for his useful comments and feedback. Also, I would like to thank Richard for introducing me to Blender, for his fruitful discussions and help with capturing data and producing video examples. I am also very grateful to Charles for his help with video processing and to Adam for allowing us to use his voice as background to some videos. Starting my PhD, I thought that getting the degree and graduating was the most important achievement. However, along the way I realised that what matters is not the destination but the journey itself and what you learn and achieve through this journey. Here in Cambridge, I had the opportunity to meet many distinguished scientists, to work in a highly professional environment and equally coordinate with them. These are opportunities of a lifetime. It was an honor for me to be member of this community and learn so much from them. Beside the academic part of my life here, I also had the chance to make new friends. There are so many that it is difficult to mention all of them without leaving somebody disgruntled. I owe a huge thanks to the Hellenic Society and Real Madra members for welcoming and helping me to easily adapt to the Cambridge life style. Special thanks to Zoe for her fertile discussions and to Themis for his invaluable help and for introducing me to Dr Lasenby. Finally, a deep thankyou to two special friends, Kantas and my housemate Bamies. I am also very grateful to my parents and sister, for their love and support, both psycho- logically and financially, throughout my life and for giving me the opportunity to follow my v –i ambitions. I owe my deepest gratitude to my godmother Nike and my uncle Tasos, as well as my cousins Peny and Marina, for their hospitality and support, for more than a month, at the time of my knee operation in Athens. Lastly, I offer my regards to all of those who believed on me, supported and inspired me in any respect during the completion of my studies. Contents Declyryfiion i Yzsfirycfi iii Ycknowledgemenfis – Confienfis –ii disfi of Figflres xi disfi of lyzles x– disfi of Ylgorifihms x–ii IF anfirodflcfiion I 1.1. Introduction and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2. Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.3. Outline of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 JF eyfihemyfiicyl Zyckgroflnd II 2.1. Geometric Algebra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.1.1. The products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.1.2. Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.1.3. The orthonormal basis for computation . . . . . . . . . . . . . . . . . . 16 2.1.4. Rotation using rotors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.1.5. Reflection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.2. Conformal Geometric Algebra . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.2.1. Rotation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.2.2. Translation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.2.3. Dilation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.2.4. Inversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.2.5. Blades in CGA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 vii –iii XdciEcih 2.2.6. Intersections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 2.3. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 KF Y fo–el an–erse cinemyfiics kol–er KO 3.1. Introduction and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 3.1.1. The articulated body model . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.2. Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.2.1. Jacobian inverse methods . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.2.2. Newton methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3.2.3. IK using Sequential Monte Carlo Methods . . . . . . . . . . . . . . . . . 50 3.2.4. Style or mesh-based Inverse Kinematics . . . . . . . . . . . . . . . . . . 50 3.2.5. Heuristic Inverse Kinematics algorithms . . . . . . . . . . . . . . . . . . 50 3.3. FABRIK: A New Heuristic IK Methodology . . . . . . . . . . . . . . . . . . . . 53 3.3.1. FABRIK with multiple end effectors . . . . . . . . . . . . . . . . . . . . 55 3.3.2. FABRIK within closed loops . . . . . . . . . . . . . . . . . . . . . . . . 57 3.4. Applying Joint Constraints to FABRIK . . . . . . . . . . . . . . . . . . . . . . 57 3.4.1. Restricting the motion to the allowed bounds . . . . . . . . . . . . . . . 58 3.5. The Experimental Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 3.6. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 3.6.1. A single end effector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 3.6.2. Multiple end effectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 3.6.3. Applying restrictions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 3.7. Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 3.8. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 LF eofiion Cypfiflre kolflfiions flnder eyrker gcclflsions OQ 4.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 4.2. Obtaining 3D Marker Positions . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 4.2.1. Marker-to-limb association . . . . . . . . . . . . . . . . . . . . . . . . . 81 4.2.2. Rigid body dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 4.3. Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 4.4. Calculating the Centre of Rotation . . . . . . . . . . . . . . . . . . . . . . . . . 86 4.4.1. Finding the rotor between 2 sets of vectors . . . . . . . . . . . . . . . . 86 4.4.2. Joint localisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 4.5. Marker Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 4.5.1. Variable Turn Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 4.5.2. Unscented Kalman Filter . . . . . . . . . . . . . . . . . . . . . . . . . . 93 4.6. Applying Marker Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 4.6.1. All markers are visible on a given limb . . . . . . . . . . . . . . . . . . . 96 4.6.2. One missing marker on a limb segment . . . . . . . . . . . . . . . . . . . 97 4.6.3. Two missing markers on a limb segment . . . . . . . . . . . . . . . . . . 98 4.6.4. All markers on a limb segment are missing . . . . . . . . . . . . . . . . 98 CgflEflk ix 4.6.5. Markers visible in only one camera . . . . . . . . . . . . . . . . . . . . . 99 4.7. Bone Length Constraints using Inverse Kinematics . . . . . . . . . . . . . . . . 100 4.7.1. Adjusting FABRIK for CoR correction in optical motion capture . . . . 101 4.8. Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 4.9. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 MF Hynd hose lrycker IIK 5.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 5.2. Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 5.3. Articulated Hand Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 5.4. Inverse Kinematics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 5.4.1. The hand model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 5.5. Physiological Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 5.5.1. Inertia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 5.5.2. Flexion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 5.5.3. Abduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 5.5.4. Intradigital correlation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 5.5.5. Transdigital correlation . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 5.5.6. Friction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 5.6. Missing Marker Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 5.7. Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 5.7.1. Mesh deformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 5.7.2. Analysis of results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 5.8. Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 NF Conclflsions ynd Fflfiflre oork IKI 6.1. Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 6.2. Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 6.3. Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 YF lrigonomefiric solflfiions flsing Geomefiric Ylgezry IKQ A.1. Nearest Point on a Circle from a Point in Space . . . . . . . . . . . . . . . . . . 139 A.2. Nearest Point on a Line from a Circle . . . . . . . . . . . . . . . . . . . . . . . 141 A.3. Nearest Point on a Line from a Sphere . . . . . . . . . . . . . . . . . . . . . . . 144 A.4. Nearest Point on a Sphere from a Point in Space . . . . . . . . . . . . . . . . . 145 A.5. IK Solutions using Conformal Geometric Algebra . . . . . . . . . . . . . . . . . 146 Glossyry ILQ kflpplemenfiyry eyfieriyls IMI Zizliogryphy IMK x XdciEcih andex INK aist of Figures 2.1. The outer product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.2. The trivector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.3. The rotation effect of bivector W3 over vectors z1 and z1 + z2. . . . . . . . . . . . . . 17 2.4. A reflection of the vector a in a plane perpendicular to m. . . . . . . . . . . . . 19 2.5. A line a = e ∧f∧n passing through the null vectors e and f and the ‘opposite’ line a′ = f ∧ e ∧ n . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.6. A circle X = e ∧f ∧R passing through the null vectors e , f and R. . . . . . 27 2.7. The unit circle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.8. The 3-plane Φ = e ∧f ∧R ∧ n . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 2.9. The solution for line to line intersection . . . . . . . . . . . . . . . . . . . . . . 32 3.1. An example of a skeletal structure of a human. . . . . . . . . . . . . . . . . . . 39 3.2. Part of a skeletal animation presenting joints of a human body and human legs. 40 3.3. Human joints with their available movements. . . . . . . . . . . . . . . . . . . . 41 3.4. Possible solutions of the IK problem. . . . . . . . . . . . . . . . . . . . . . . . . 42 3.5. The Jacobian solution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.6. An example of visual solution of the IK problem using the CCD algorithm. . . 51 3.7. An example of a full iteration of FABRIK for the case of a single target and 4 manipulator joints. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.8. An example of a model figure with multiple end effectors and multiple sub-bases. 57 3.9. The target is re-positioned within the allowed range of motion which is defined by the conic section. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 3.10. A graphical representation of the implemented constraints and the irregular cone describing the rotational motion bounds. . . . . . . . . . . . . . . . . . . . . . . 59 3.11. Solution for special joint restriction cases: (a) the original case and when the allowed range of motion is greater than 180 degrees, (b) when the target is located on a different hemisphere than the irregular cone. . . . . . . . . . . . . 61 3.12. Incorporating rotational and orientational constraints within FABRIK . . . . . 62 3.13. Incorporating constraints for a hinge joint. . . . . . . . . . . . . . . . . . . . . . 63 xi xii aIhi d[ [IGjgEh 3.14. The structure of the models used in our experimental examples. . . . . . . . . . 64 3.15. The stages illustrated in order that the end effector reaches the target. . . . . . 65 3.16. Unnatural joint angles exhibited by CCD. . . . . . . . . . . . . . . . . . . . . . 66 3.17. The iterations needed to reach the target against the distance between target and end effector. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 3.18. An example presenting the time needed for each methodology to achieve the solution with the degree of accuracy requested. . . . . . . . . . . . . . . . . . . 68 3.19. Experimental solutions using some of the most popular IK methods. . . . . . . 69 3.20. An example of the target tracking using different methods. . . . . . . . . . . . 70 3.21. FABRIK and CCD solution using the Kine application. . . . . . . . . . . . . . 71 3.22. Example of FABRIK implementation with multiple end effectors moving over time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 3.23. Example of FABRIK implementation with multiple end effectors moving over time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 3.24. A low rate body tracking example. . . . . . . . . . . . . . . . . . . . . . . . . . 74 3.25. Body reconstruction using different IK methodologies. . . . . . . . . . . . . . . 75 3.26. An example of FABRIK and CCD implementations with and without incorpo- rating constraints. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 3.27. An example of implementation with and without constraints. . . . . . . . . . . 78 4.1. Marker-to-limb association example with three markers on each limb. . . . . . . 81 4.2. Description of the rigid body. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 4.3. The motion of two limbs with a time-difference of m frames. . . . . . . . . . . . 88 4.4. A typical marker placement with relevant markers shown. . . . . . . . . . . . . 89 4.5. An example of CoR estimation of a human body model using [CL05]. . . . . . 90 4.6. Target prediction with variable acceleration using the VTM model compared to the linear prediction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 4.7. The observation vector in the case of 2 visible markers. . . . . . . . . . . . . . . 97 4.8. The observation vector in the case of only one visible marker. . . . . . . . . . . 98 4.9. The estimation procedure when all markers on a single limb segment are occluded. 99 4.10. The observation vector in the case of 2 visible markers and one marker visible only by one camera. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 4.11. The results produced using the UKF-VTM model. . . . . . . . . . . . . . . . . 101 4.12. A simple example of FABRIK implementation for the case where the estimated joints are positioned inbetween 2 observed joint positions. . . . . . . . . . . . . 103 4.13. A simple case where the estimated joints are located at the end of the chain. . 104 4.14. Two of the models used in our experiments. . . . . . . . . . . . . . . . . . . . . 106 4.15. Example showing comparison results for each methodology for the case of 1 missing marker on each limb segment. . . . . . . . . . . . . . . . . . . . . . . . 109 4.16. Examples of implementation on real data (lower Body). . . . . . . . . . . . . . 110 dakl gF FaGmjEk xiii 4.17. An example of marker prediction and CoR estimation using the proposed methodology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 4.18. An example of FABRIK implementation for CoR correction. . . . . . . . . . . . 111 4.19. An example of FABRIK implementation under extreme cases with extended data occlusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 5.1. The impulse glove designed by PhaseSpace for real-time finger-tracking solutions.114 5.2. The hand’s model geometry used in our implementation. . . . . . . . . . . . . . 116 5.3. The palm plane constraints. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 5.4. The joints’ positions at times k − 1 and k. . . . . . . . . . . . . . . . . . . . . 119 5.5. The current joint positions, after rotating them in order to lie on the current finger plane Φki . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 5.6. Graphical representation of the angles θ1P OOOP θ4 which define the rotational con- straints of each joint. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 5.7. Examples of unnatural hand postures. . . . . . . . . . . . . . . . . . . . . . . . 122 5.8. An example showing the intradigital correlation feature. . . . . . . . . . . . . . 123 5.9. The linked pairs of bones which share a transdigital correlation movement. . . 124 5.10. An example explaining the transdigital correlation of the hand. . . . . . . . . . 125 5.11. An example of the hand with attached markers and how the markers are seen from the motion capture system. . . . . . . . . . . . . . . . . . . . . . . . . . . 126 5.12. Example of a simple linear blend skinning scheme applied using the weights from bone-heat. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 5.13. An example of hand reconstruction using our methodology at a frame rate of 100Hz frame. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 5.14. An example of continuous hand pose tracking, at a frame rate of 10Hz. . . . . . 129 A.1. The nearest point on circle to point in space. . . . . . . . . . . . . . . . . . . . 140 A.2. Finding the nearest point on a line from a circle using CGA. Case where the distance from the centre of the circle X1 to the point n is smaller than the radius of the circle and point e is inside the circle area. . . . . . . . . . . . . . . . . . 142 A.3. Finding the nearest point on a line from a circle using CGA. Case where the distance from the centre of the circle X1 to the point n is smaller than the radius of the circle and point e is outside the circle area. . . . . . . . . . . . . . . . . 143 A.4. Finding the nearest point on a line from a circle using CGA. Case where the distance from the centre of the circle X1 to the point n is greater than the radius of the circle and point e . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 A.5. A full iteration of FABRIK for the case of a single target and 4 joints using CGA.148 aist of ivwles 2.1. The orthonormal basis for computation . . . . . . . . . . . . . . . . . . . . . . 16 3.1. Average results for a single kinematic chain with 10 joints. . . . . . . . . . . . . 67 3.2. Reconstruction comparison. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 3.3. Average results when joint constraints are incorporated. . . . . . . . . . . . . . 76 4.1. Average results under multiple cases of occlusion. . . . . . . . . . . . . . . . . . 105 4.2. Average results on real data with occlusions generated by deletions. . . . . . . 107 4.3. Error reduction using Inverse Kinematics. . . . . . . . . . . . . . . . . . . . . . 108 5.1. Hand joint configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 5.2. Average time needed at different frame rates. . . . . . . . . . . . . . . . . . . . 127 xv aist of Vlgorithms 1. A full iteration of the FABRIK algorithm. . . . . . . . . . . . . . . . . . . . . . . 56 2. The orientational constraints. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 3. The rotational constraints. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4. A full iteration of FABRIK for CoR correction. . . . . . . . . . . . . . . . . . . . 102 5. A full iteration of the FABRIK algorithm using CGA. . . . . . . . . . . . . . . . 147 xvii 1 Introyuxtion T hOs thesis addresses the problem of manipulating articulated figures in an interactiveand intuitive fashion for the design and control of their posture. It finds applications in the areas of robotics, computer animation, ergonomics and the computer games industry. A novel heuristic method, called FABRIK, is proposed, which solves the Inverse Kinematics problem in real-time and returns visually natural poses comparable with more complex and highly sophisticated approaches. We also derive a robust methodology for marker prediction under multiple marker occlusion for extended time periods, in order to drive real-time centre of rotation (CoR) estimations. Finally, an efficient real-time hand pose tracker is presented, using optical motion capture data. Physiological constraints are incorporated allowing motion within an anatomically feasible set. FCFC Introyuxtion vny botivvtion In computer graphics, articulated figures are a convenient model for humans, animals or other virtual creatures found commonly in films and video games. The most popular method for ani- mating such models is motion-capture; however, despite the availability of highly sophisticated techniques and expensive tools, many problems appear when dealing with complex figures. Most virtual character models are complicated; they are made up of many joints having a high number of degrees of freedom. Thus, it is often difficult to produce a realistic character animation. 1 J Introyuxtion A posturz is defined as the skeletal configuration of a figure; for a realistic posture a set of criteria should be satisfied: all character models have natural articulation limits and inter- penetration of the body with other objects or themselves is not permitted. In addition, physical laws should be considered as well as numerous personal factors. General constraints can be applied to most articulated figures, however special cases of posture control are needed when a large number of degrees of freedom exists. Inverse Kinematics (IK) is a method for computing the posture via estimating each indi- vidual degree of freedom in order to satisfy a given task; it plays an important role in the computer animation and simulation of articulated figures. Inverse Kinematics finds applica- tions in several areas; IK methods have been implemented in many computer graphics and robotics applications, aiming to animate or control different virtual creatures. They are also very popular in the video games industry. The field of computer-aided ergonomics is also concerned with articulated figures, especially human models developed for simulation and pre- diction purposes. The need for accurate biomechanical modelling and body sizing based on anthropometric data make IK methods a popular approach for fast and reliable solutions. In this work, the most popular Inverse Kinematics techniques are reviewed. A new heuristic iterative method, FABRIK, is also presented for solving the IK problem in different scenarios. FABRIK (Forward And Backward Reaching Inverse Kinematics) is an efficient method for solving the IK problem; it uses a forward and backward iterative approach, finding each joint position via locating a point on line. FABRIK is easy to implement and has been utilised in highly complex systems with single and multiple targets, with and without joint restric- tions. It can easily handle end effector orientations and support, to the best of our knowledge, all chain classes. A reliable method for incorporating constraints is also presented and used within FABRIK. The proposed method retains all the advantages of FABRIK, producing vi- sually smooth movements without oscillations and discontinuities, with low computational cost. Several experiments have been implemented for the sake of comparison between the most popular manipulator solvers, including multiple end effectors with multiple tasks, and highly constrained joints. The algorithms are tested for reliability, computational cost, realistic movement, reconstruction quality, conversion criteria and number of iterations needed. Inverse Kinematics solvers are very popular in the robotics, computer graphics and animation communities; they are extensively used to control and animate legged figures or articulated robots. The most popular method used to extract accurate data and animate such models is motion capture. Optical motion capture is a technology used to turn the observations of a moving subject (taken from a number of cameras) into 3D position and orientation information about that subject. Such information can be used for the following applications: to better analyse techniques in sports training and performance (e.g. posture, velocities, accelerations, angles, trajectories) [HNT+06, Gol]; to study human movements for medical reasons; to observe asymmetries and abnormalities in rehabilitation medicine (e.g. gait in stroke or prosthetic patients) [HDSB05, PTP+05, BSR07]; in the generation of virtual characters for films or computer games [Men99] and for human-computer interactions (HCI), including interaction with game consoles. IK has recently also been adopted in protein science for protein structure IFI anfirodflcfiion ynd eofii–yfiion K prediction and reconstruction [CD03]. There are two basic approaches used to capture such data, markered and markerless. Throughout this work, markered motion capture is used. The problem of establishing the motion of interest is simplified by attaching markers of some type to the subject being recorded. These markers can then be easily located in an image and their movement can be used to infer the complete movement of the person to whom they are attached. In general, to achieve accurate skeletal reconstruction of any legged body, three markers must be available on each limb segment at all times. However, even with many cameras, there are instances where occlusion of markers by elements of the scene leads to missing data. In order to establish its position without ambiguity, each marker must be visible to at least two cameras in each frame. Although many methods have been developed to handle the missing marker problem, most of them are not real-time, are usually limited to short time period occlusions and often require manual intervention. In this thesis, we investigate methodologies for real-time marker prediction, under multiple cases of occlusion, to drive centre of rotation (CoR) estimates and then to automatically es- tablish the skeleton model. A real-time integrated framework is presented, which predicts the occluded marker positions using a variable turn model within an Unscented Kalman filter. The previous marker positions are used within the framework in addition to information related to the missing markers of the current frame, inferred from an approximate rigid body assump- tion. The predicted marker positions are then used to locate the joints. Without assuming any skeleton model, we take advantage of the fact that for markers on a given limb segment, the inter-marker distance is approximately constant. The proposed marker constraint method- ology is simple, real-time implementable and very efficient. It can deal with all the cases of marker occlusion within a limb, resulting in accurate predictions even when all markers on a limb segment are missing for an extended period of time. The proposed approach also takes advantage of the special, but common, case where missing markers are still visible to one cam- era. With a continuous stream of accurate labelled 3D data, we can perform real-time CoR estimation; the CoR position is thereafter corrected via a real-time Inverse Kinematic tech- nique (FABRIK) which ensures that the inter-joint pairwise distances remain constant over time. A skeletal reconstruction is thereby achieved, producing information which can be used for visual performance feedback. Experiments demonstrate that our methodology effectively recovers good estimates of the true positions of the missing markers and CoRs, even if all the markers on a limb are occluded for a long period of time. The resulting motion is natural and smooth, and can be achieved in real-time. In recent years there has been a growing demand for reliable hand motion tracking systems, a technology used to turn the observations of a moving hand into 3D position and orientation information. However, building a fast and effective hand pose tracker remains challenging; the high dimensionality of the pose space, the ambiguities due to self-occlusions and the significant appearance variations due to shading make efficient tracking difficult. Marker-based motion capture has been demonstrated in several interactive systems (including but not limited to hand interaction); the results are highly accurate and easy to configure. There are, however, L Introyuxtion instances where we do not have many markers available or it is impossible to attach three markers on each limb segment; the large number of markers needed is often prohibitive. It may therefore be infeasible to track the object and reconstruct its skeletal model (i.e. the hand model). A new way of capturing the movement of these articulated models is therefore required, using the minimum possible number of markers. Instead of attaching three markers on each limb segment, we investigate a system in which a single marker is attached and captured on each finger (end effector), one marker at the chain base (root) and two markers at strategic positions to help us define the hand orientation. The markers’ positions are tracked using an optical motion capture system, such as [Pha]. However, prior knowledge about the geometry of the hand, the hand model and the restrictions of each joint are needed. Joint constraints are applied to ensure that finger motion is within a feasible set, giving a visually natural motion of the hand. An Inverse Kinematics solver (FABRIK) is incorporated to estimate the remaining joint positions and to fit them to the hand model. Physiological constraints related to the hand anatomy are then enforced to restrict the motion only to natural possible poses, without violating any model constraint. Finally, a mesh deformation algorithm has been applied to drive the animation of the underlying hand skeleton using a set of per-bone weights. The implemented mesh videos are compared with their true hand motions and the results verify that the suggested method is effective; the method is real-time implementable and tracks the hand motion smoothly, without oscillations, even with a low capture frame rate. FCGC aitervture geview The area of inverse kinematics has been extensively studied during the last decades. In this literature review we present the most popular methods which solve the IK problem and compute the poses of a manipulator. A joint localisation review is also given in addition to methods for marker prediction and centre of rotation estimation. We conclude this section with a review of recent hand pose tracking methods. Note that this section is not meant as an exhaustive literature review but rather a brief outline of the background literature. A more detailed review can be found separately in each chapter, dealing with the relevant subject. an–erse cinemyfiics re–iewR The production of realistic and plausible motions posed a problem for scholars for many years in the field of robotics technology and computer graph- ics. During recent decades, several models have been implemented for solving the IK problem from many different areas of study. [ZB94] treats the IK task as a problem of finding a local minimum of a set of non-linear equations, defining Cartesian space constraints. However, the most popular numerical approach is to use the Jacobian matrix to find a linear approxima- tion to the IK problem. The Jvxowivn solutions linearly model the end effectors’ movements relative to instantaneous system changes in link translation and joint angle. Several different methodologies have been presented for calculating or approximating the Jacobian inverse, such as the Jacobian Transpose, Damped Least Squares (DLS), Damped Least Squares with Singu- lar Value Decomposition (SVD-DLS), Selectively Damped Least Squares (SDLS) and several IFJ difieryfiflre je–iew M extensions [BDMS84, WE84, Bai85, Wam86, NH86, BK05]. Jacobian inverse solutions pro- duce smooth postures; however most of these approaches suffer from high computational cost, complex matrix calculations and singularity problems. An alternative approach is given by Pechev in [Pec08] where the inverse kinematics problem is solved from a control prospective. This approach is computationally more efficient than the pseudo-inverse based methods and does not suffer from singularity problems. The second family of IK solvers is based on Newton methods. These algorithms seek target configurations which are posed as solutions to a minimisation problem, hence they return smooth motion without erratic discontinuities. The most well known methods are Broyden’s method, Powell’s method and the Broyden, Fletcher, Goldfarb and Shanno (BFGS) method, see [Fle87] for a detailed review. However, the Newton methods are complex, difficult to implement and have high computational cost per iteration. A very popular IK method is the Cyclic Coordinate Descent (CCD) algorithm, which was first introduced by [WC91] and then biomechanically constrained by [Wel93]. CCD has been extensively used in the computer games industry [Lan98] and has recently been adapted for pro- tein structure prediction [CD03]. CCD is a heuristic iterative method with low computational cost for each joint per iteration, which can solve the IK problem without matrix manipula- tions; thus it formulates a solution very quickly. However, CCD has some disadvantages; it can suffer from unrealistic animation, even if manipulator constraints have been added, and often produces motion with erratic discontinuities. It is designed to handle serial chains, thus, it is difficult to extend to problems with multiple end effectors. [UPBS08] describes a Sequential IK solver (SIK), which is a direct extension of [BVU+06], in that its inputs are end effector positions, such as wrists, ankles, head and pelvis, which are used to find the human pose. The IK problem is then solved sequentially using simple analytic-iterative IK algorithms (CCD), in different parts of the body, in a specific order. [KM05] also adopted the CCD kinematic algorithm and solved its crucial problem of resulting unnatural poses. The proposed extension in [KM05] is able to solve problems with humanoid hierarchy, dividing the whole body into groups of joints near an end effector (typically head, trunk, arms and legs). In order to sat- isfy the desired centre of mass, the lightest group moves first, adjusting its centre of mass by changing the length of the limb and rotating it (assuming it is a rigid body). Recently, [CA08] and [HRE+08] proposed a Sequential Monte Carlo Method (SMCM) and particle filtering approach respectively. The proposed particle IK solver treats the character skeleton as a set of 3 degrees of freedom (DoF) particles having inter-length constraints. An iterative constrainer, with various pre-conditions and parameters, is then applied over the particles, tuning its behavior both statically and dynamically. The final particle positions and the length constraints are then used to reconstruct the resulting DoF of the body. Neither method suffers from matrix singularity problems and both perform reasonably well. However, these statistical methods have high computational cost. [GMHP04] presents a style-based IK method which is based on a learned model of human poses. Given a set of constraints, the proposed system can produce, in real-time, the most likely pose satisfying those constraints. The model has been trained on different input data that leads to different styles of IK; it can N Introyuxtion generate any pose, but poses are highly correlated with similar poses in the training data. In [SZGP05, DSP06], the authors used mesh-based IK techniques to configure the animated shapes. Mesh-based IK learns a space of natural deformations from example meshes. Using the learned space, they generate new shapes that respect the deformations exhibited by the examples, and satisfy vertex constraints imposed by the user. However, these methods require an off-line training procedure, their results are highly dependent on the training data and limited only to those models and movements on which the system has been trained. A detailed explanation of all these methods is given in chapter 3.2. [BLM04] present a real-time method which uses a ‘Follow-the-Leader’ (FTL) non-iterative technique which is similar to each individual iteration of FABRIK. FTL was specifically de- signed for rope simulation and has neither been used for IK solutions nor extended so that it can function as an IK solver. FTL does not use points and lines to estimate joint positions, does not work in an iterative fashion and manipulates the kinematic chain (ball-and-socket joints connected by rigid links) taking into account only that the end effector should move to a desired position. Although similar to FABRIK in its basic structure, the FTL algorithm has not been extended to support joint constraints and orientations (these are largely superfluous in rope simulation), nor has it been applied to cases where multiple end effectors exist. Inverse Kinematics is a method for manipulating articulated figures in an interactive and intuitive fashion for the design and control of their postures. However, in order to apply IK techniques for motion and body reconstruction, a joint localisation approach is needed to generate the CoRs and automatically establish legged skeletons from optical motion capture data. boinfi docylisyfiion re–iewR Many papers have focused on methods for localisation of the CoR. hphzrz fitting approaches are the most commonly used methods for calculating the CoR. This group of methods assumes that all markers remain a constant distance from the centre of rotation. In [SPB+98], the Levenberg-Marquardt method is used to optimise the CoR location and the radii of the marker spheres, via a cost function which sums a per marker cost over all markers and all frames. Halvorsen et al. describe a closed form solution using the geometric properties of the sphere [HLL99]. In [GL02], Gamage and Lasenby also introduce a closed form solution, using a cost function of the squared differences in the squared distance from the CoR to a marker and the radius of the sphere associated with that marker. An alternative approach provided by Halvorsen, [Hal03], gives a Bayesian analysis of the algorithm of [GL02], providing a first order approximation of the effect of isotropic Gaussian noise upon the algorithm. Another group of methods is that termed irvnsformvtionvl tzxhniquzs; they assume that markers are rigidly attached to limb segments. Such an approach was implemented in [Hol91], [OBBH00] and [ETDH06], where the limb orientation was obtained from sets of optical markers. In [CL05], a sequential algorithm was presented to locate the rotation centres of a human skeleton from marker data assuming that all markers are attached to a rigid body. The method is closed form, thus enabling real-time implementation. IFJ difieryfiflre je–iew O eyrker predicfiion ynd Coj esfiimyfiion re–iewR Whilst several methods to estimate the location of missing markers have been proposed, the performance of most is unsatisfactory in the presence of unusual motions or of many contiguous occlusion-affected frames. Indeed, there are instances, even with expensive motion capture systems, where occlusion of markers by elements of the scene leads to missing data. Several methods have been proposed to predict the occluded markers in order to drive CoR estimation and skeletal reconstruction. Interpolation of the data using linear or non-linear approaches is commonly used [WH97, RCB98]; this can produce accurate results, but it is a post-processing technique requiring data prior to and after the occlusion. Recently, [PLH+09] presented an extrapolation algorithm which assumes that the most common motion behaviors are circular or linear movements; however, this method can produce reliable predictions only for a limited number of occluded frames. While the discarding of frames with missing markers is another common technique, the omission of specific data could lead to the loss of useful information. Long-running occlusions leading to a large sequence of missing data can also cause complete failure of the system. In [RM06], Rhijn and Mulder proposed a model-based optical tracking and model estimation system for composite interaction devices; however, it is an off-line procedure unsuitable for real- time applications. Dorfmu¨ller in [DU03] used an extended Kalman filter (EKF) to predict the missing markers using previously available marker information, while Welch et al. in [WBV+99] used an EKF to resolve occlusions based on the skeletal model of the tracked person. Tak and Ko, [TK05], employed an Unscented Kalman Filter to ensure motion capture data remains in a feasible set. These methods require manual intervention or become ineffective in cases where markers are missing for an extended period of time. In our earlier work in [ACL08], we presented an EKF method using a constant velocity (CV) model with marker constraints from neighbouring1 markers. However, the CV model (2nx order Kalman Filter (KF)) limits its use to problems with constant marker velocity. These methods also do not take into consideration the fact that bones are rigid, thus the inter-joint pairwise distances remain constant over time. Herda et al., in [HFP+00] and [HFP+01], used a post-processing approach to increase the robustness of a motion capture system by using a sophisticated human model. The neighbour- ing markers that share kinematic relations with the occluded markers were used to help the estimation of the missing markers. However, the skeleton information must be known a priori in order to apply this method. [HSD05] also takes advantage of the fact that the markers on a limb have fixed inter-marker pairwise distances. This approach may become ineffective when all or a significant number of markers are missing so that no information on that limb can be inferred from the available neighbouring markers. Ringer and Lasenby, [RL02], also present an automatic method to identify indistinguishable markers based on cliques-. However, this re- quires an off-line procedure in order to determine marker cliques and parameters of the skeletal structure. In [GMHP04], a style-based inverse kinematic method has been developed where a Gaussian 1ceighwours ury thy murkyrs vylonging to thy sumy limv sygmyntB 2aurkyrs in u clique huvy wonstunt xistunwys vytwyyn yuwh othyrB P Introyuxtion Process Latent Variable Model (GPLVM) was used along with a pre-specified kinematic model. Although it is a real-time processing method, it requires knowledge of skeleton information, which severely restricts its use. Chai and Hodgins, [CH05], present a method that uses the neighbouring markers to estimate the missing marker in the current frame. They propose a local linear model from these neighbours and then reconstruct the full pose of the frame by conducting an optimisation in the space constrained by the model. Yu et al., in [YLD07], proposed an online motion capture labelling approach which also recovers missing markers. They cluster the markers into a number of rigid bodies based on the standard deviations of the marker-pair distances. If their fitting-rigid-bodies algorithm did not classify all the markers into rigid bodies, a missing marker auto-recovery method is applied assuming that the inter-marker distances are fixed over time. However, a training session is needed for the fitting-rigid-bodies algorithm, the auto-recovery method for marker estimation does not take into account the limb segment rotation, no information about markers visible to a single camera is considered and the CoR estimation is not investigated under marker occlusions. Recently, Liu et al., in [LZWM06, LM06], presented a piecewise linear approach for estimating human motions from a pre-selected set of informative markers (principal markers). A pre-trained classifier identifies an appropriate local linear model for each frame. Missing markers are then recovered using available marker positions and the principal components of the associated model. However, this data-driven family of methods requires an off-line training procedure and the results are highly dependent on training data and limited to those models and movements the system has been trained on. An extended literature review of these methods is also given in 4.3. Hynd firycking ynd reconsfirflcfiion re–iewR There are many approaches for tracking and configuring the hand model. The hand gesture identification algorithms can be classified into 2 major classes: glove-based and vision-based methods. In general, glove-based methods are real-time, however they are expensive (e.g. P5 Data glove) and only detect limited finger movements with low accuracy. Wang and Popovic [WP09] and Fredriksson et al [FRF08] proposed methods for hand tracking using a single camera and an ordinary cloth glove which was imprinted with a custom pattern; while this offers a simple, computationally cheap and promising solution, it is still not as reliable as the optical mocap systems. The vision-based methods, on the other hand, are more accurate but they have problems with occlusions, noise and spurious data. Lien and Huang [LH98] proposed a hand model together with a closed- form Inverse Kinematics solution for the finger fitting process. The 3D positions of the markers were obtained using colour markers and stereo vision, and the finger poses were chosen using a search method which finds the best solution amongst all possible positions. While this method is implementable in real-time, it is complex and can fail when different size models are used. De la Gorce et al [GPF08] also proposed a 3D hand tracking approach from monocular video. Stenger et al [SMC01, STTC06] proposed statistical methods, such as an Unscented Kalman Filter and a Hierarchical Bayesian Filter, to track hand motion. Such methods are IFK gflfiline of fihe lhesis Q still far from real-time thus limiting their use. Kaimakis and Lasenby [KL07] used a set of pre-calibrated cameras to extract the hand’s silhouette as a visual cue. The 2D silhouette data is then modelled as a conic field and physiological constraints are imposed to improve the reliability of the hand tracking [KL09]. A more detailed literature review of hand tracking and reconstruction methods is given in section 5.2. FCHC dutline of the ihesis The body of this thesis may be broadly divided into 4 main parts, each of which is described in chapters 2 to 5. The first part describes the mathematical and experimental framework. Within this first part, chapter 2 gives a brief introduction to Geometric Algebra (GA) and to those aspects of GA that have been used within this thesis. We briefly outline Conformal Geometric Algebra (CGA) and examine how the Conformal Model can represent geometric primitives such as points, lines, circles, planes and spheres. The second part introduces the Inverse Kinematics (IK) problem and discusses the most pop- ular recent IK techniques. It briefly describes the advantages and disadvantages of each family of methods including Inverse Jacobian methods, Newton methods, Sequential Monte Carlo Methods and heuristic methods. Many of the currently available methods suffer from high computational cost and production of unrealistic poses. Chapter 3 presents a novel heuristic approach, called Forward And Backward Reaching Inverse Kinematics (FABRIK), which solves the IK problem in an iterative fashion. FABRIK avoids the use of rotational angles or ma- trices, and instead finds each joint position via locating a point on a line. Thus, it converges in fewer iterations and has low computational cost while producing visually realistic poses. FABRIK has been compared against the most popular manipulator solvers under several con- ditions, including multiple end effectors with multiple tasks, and highly constrained joints. The algorithms are tested for reliability, computational cost, realistic movements, reconstruction quality, conversion criteria and number of iterations needed. The third part concerns the problem of fitting skeletal models to marker-based optical motion capture data. Section 4.4, studies the problem of estimating the centres of rotation (CoR) between every pair of limb segments and identifying the optimal skeleton in real-time. We present an efficient and accurate method for finding the rotors between 2 sets of vectors, and then we calculate the CoR by taking advantage of the approximation that all markers on a segment are attached to a rigid body. Part 3 also presents an integrated framework which predicts the occluded marker position and thereby maintains a continuous flow of data. Marker occlusion is a common phenomenon in motion capture systems due to camera system failure or marker occlusion by other limbs. In detail, section 4.2 describes the experimental environment including the cameras used for the experiments, the association between the markers and the limbs, and the clustering of markers into groups corresponding to the limb segments to which they are attached. Section 4.5 presents a real-time predicting approach using a variable turn model within an Unscented Kalman filter, in combination with inferred information from neighbouring markers in order to fill in the missing data when markers are IH Introyuxtion occluded. This approach takes advantage of the fact that markers located on the same limb of an articulated body have constant inter-marker distance and presents marker prediction solutions under 5 different scenarios; the proposed marker constraints are efficient, simple and real-time implementable. This work also imposes the common case that missing markers are often visible to only a single camera, resulting in more accurate predictions. Section 4.7 describes how the FABRIK Inverse Kinematics solver can be incorporated into the marker prediction and centre of rotation problem, re-positioning the joint locations in such a manner that the inter-joint distances remain constant over time. Finally, section 4.8 shows comparisons of the proposed methodology against some of the most popular methods for marker prediction and the results confirm that our approach outperforms these methods in estimating both marker and CoR positions in terms of absolute error and computational cost. The last part of the thesis describes a sophisticated hand model approach for real-time tracking and reconstruction. It presents a simple and efficient methodology for tracking and reconstructing 3D hand poses using a markered optical motion capture system. It is an ex- perimental use of the IK solver presented in Chapter 3, and the marker prediction technique described in Chapter 4. Markers were positioned at strategic points, and FABRIK was incor- porated to fit the rest of the joints to the hand model. The model is highly constrained with rotational and orientational constraints, allowing only natural finger motion. The hand model movements are also restricted by physiological constraints, allowing motion only to positions within an anatomically feasible set. In addition, a marker prediction system, similar to the one presented in Chapter 4, is applied to deal with cases where the markers are not visible to the motion capture cameras. The results were visualised using a mesh deformation algorithm, driving the animation of the hand according to the underlying hand skeleton. The method is real-time implementable and tracks the hand motion smoothly and without oscillations, even with a low capture frame rate. In Chapter 6 we conclude with some final remarks and propose directions for future work. Finally, in the Appendices we give the Geometric Algebra derivations for most of the imple- mented methods. 2 bvthemvtixvl Wvxkgrouny T hrUuMhUut this chapter, we present a brief overview of Geometric Algebra (GA) inorder to provide explanations of those elements used within this thesis. GA provides a convenient mathematical notation for representing orientations and rotations of objects in three dimensions. The Conformal model of GA give us the ability to describe algorithms in a geometrically intuitive and compact manner since basic entities, such as spheres, lines, planes and circles, are simply represented by algebraic objects. GA is also more numerically stable and more efficient than rotation matrices making it popular for applications in computer graphics and robotics. More detailed treatment of geometric algebra can be found in [DL03]. GCFC Geometrix Vlgewrv Classical vector algebra has a number of problems when we move from three dimensional space (Euclidean) to higher dimensional space. Hence, Hermann Grassmann (1809-77) and William Clifford (1845-79), attempted to create an ‘algebra of vectors’ in order to generalise conventional vector algebra to higher dimensions. The modern day extension of this work is now known as ‘Geometric Algebra’. GCFCFC ihe proyuxts The geometric product is the most fundamental product of Geometric Algebra. However, it is often useful first to define the innzr and outzr proyuxts for vectors and then to introduce the 11 IJ bvthzmvtixvl Wvxkgrouny geometric product of vectors. lhe inner prodflcfi A vector space alone is insufficient for describing Euclidean geometry as it lacks the concepts of distance and angles. Distances and angles are important in order to define entities like circles or perpendicular lines. Both can be defined through the introduction of a scalar product between vectors. This is known as the innzr proyuxt, is written as a · w and returns a scalar. In Euclidean space the scalar product is always positive, a- = a · a S 0 ∀ a ̸= 0 (2.1) Thus, the length of a vector |a| can be defined as: |a| = √ (a · a) (2.2) Hence, the inner product between a and w is defined via a · w = |a||w| cos(θ) (2.3) where θ is the angle between the vectors. The inner product can be also defined for higher dimensional generalisations of the vector (multivectors). lhe oflfier prodflcfi a b θ a ∧ b a b θ b ∧ a Figflre JFIFR ihz outzr proyuxtC ihz outzr or fizygz proyuxt of a vny w rzturns v yirzxtzy vrzv zlzmznt of vrzv |a||w| sin(θ)C The major failure of the cross product is that exists only in 3 dimensions. In 2D there is nowhere else to go, whereas in more than 3 dimensions this direction is not uniquely defined. The solution of this problem was solved by Grassmann ([Gra62]) encoding a plane geometri- cally, without relying on the notion of a vector perpendicular to it. Grassmann introduced a key feature of GA namely the outzr or zfltzrior proyuxt, which is written as a ∧ w. Unlike the cross product, which results in a perpendicular vector, the outer product of a pair of vectors results in a directed area, as shown in figure 2.1. This is the directed area swept out by a and JFI Geomefiric Ylgezry IK w and can be visualised as the parallelogram obtained by sweeping one vector along the other. Changing the order of the vectors reverses the orientation of the plane. The plane has area |a||w| sin(θ), which is defined to be the magnitude of a ∧ w. The outer product has the following key properties: • The outer product of vectors is antisymmetric a ∧ w = −w ∧ a (2.4) It follows that a ∧ a = 0. • Also the outer product is distributive over addition a ∧ (w+ x) = a ∧ w+ a ∧ x (2.5) • and associative a ∧ (w ∧ x) = (a ∧ w) ∧ x = a ∧ w ∧ x (2.6) • The outer product of a vector and a bivector defines a trivector that is an oriented volume. • Although the formation of a bivector is often illustrated as the result of sweeping one vector along a second to form a parallelogram, the use of any particular shape is mis- leading as it is easy to show that the outer product of many different pairs of vectors will result in the same bivector. lhe geomefiric prodflcfi William Clifford (1845-1879) made the next step and, investigating the work of Grassmann, he turned GA to a useful algebra. Clifford introduced the key feature of GA, the gzomztrix proyuxt, in order to define a product that could identify the roles of the terms in a complex product. The geometric product can be expressed in terms of the inner and outer products, and it is defined as: aw = a · w+ a ∧ w (2.7) The result of the geometric product seems strange, having two components lying in two different spaces, a scalar (a · w) and a bivector (a ∧ w). This combination is referred to as a multivzxtor and is analogous to complex numbers where real and imaginary numbers are combined to form the complex number. Since, wa = w · a+ w ∧ a = a · w− a ∧ w (2.8) IL bvthzmvtixvl Wvxkgrouny we can define the inner and outer product in terms of the symmetric and antisymmetric parts of the geometric product. Thus, for vectors a and w a · w = 1 2 (aw+ wa) a ∧ w = 1 2 (aw− wa) (2.9) Then, the geometric product can be extended to an arbitrary number of vectors with the following properties: • The geometric product is associative (aw)x = a(wx) = awx (2.10) • The geometric product is distributive over addition a (w+ x) = aw+ ax (2.11) • The symmetric part of the geometric product of two vectors is a scalar • The geometric product has a unique inverse aw ( w w- ) = a (2.12) hence, multiplying by w w2 is the inverse of multiplying by w. The geometric product can also be used to define the inner and outer product of elements with single grades, Vr and Ws, where r and s denote the grade of the vector. Such elements are called blades1. Vr ·Ws = 〈VrWs〉|r−s| if r,s S 0 (2.13) Vr ·Ws = 0 if r = 0 or s = 0 (2.14) Vr ∧Ws = 〈VrWs〉r+s (2.15) where 〈〉t denotes the grade extraction operator, which sets all multivector components of grade other than t to zero. For the extraction of the scalar part of a multivector, the subscript 0 is usually dropped, and it is formulated as 〈〉 instead of 〈〉+. GCFCGC dpervtors Geometric Algebra also has some other useful operators and elements. The most important of these are the rzvzrsz of a multivector, the yuvl and the pszuyosxvlvr. 1hhy tyrm wlvde in GA is usyx to ryfyr to quuntitiys thut wun vy writtyn us thy outyr proxuwt of rBvywtors, whyry r is tyrmyx thy grvdeB For yfiumply, un rAvluxy wun ulwuys vy writtyn us O1 ∧ O2 ∧ : : : ∧ OrB aory informution uvout gruxys unx thy nyw orthonormul vusis for womputution wun vy founx in sywtion FBEBGB JFI Geomefiric Ylgezry IM je–ersion Reversion substantially refers to the reversing of the order of any set of vectors that can be used to define a multivector, and it is symbolised as .˜ A general example of reversion for the case of a 3D GA is given below: b = α+ a+W + βI b˜ = α+ a−W − βI (2.16) where b is multivector, α is a scalar, a is a vector, W is a bivector and βI is a trivector. Using the fact that a ∧ w = −w ∧ a, the reversion operator can be expressed by changing the sign of some particular grades of elements (bivectors and trivectors in 3D) within the multivector. lhe psefldoscylyr The pseudoscalar refers to the highest grade basis element and it is symbolised as In, where n is the dimension of the space. For instance, in 3D GA the pseudoscalar can be written as: I. = z1z-z. (2.17) where z1P z-P z. are the orthonormal basis vectors. The reversion of I. is equal to I˜. = z.z-z1 = −z1z-z. = −I. (2.18) The pseudoscalar is also denoted as a simple I, because its grade is always the highest and it is unnecessary to repeat it. Since the pseudoscalar I is the unique right-handed unit trivector in the algebra, it gives us a number of new products. When we take the product of I with the vector zi, we produce a bivector, eg. Iz1 = z1z-z.z1 = −z1z-z1z. = z-z. (2.19) z1I = z1z1z-z. = z-z. = Iz1 (2.20) Vectors therefore commute with the 3D pseudoscalar, Ia = aIP for all a (2.21) Hence, all the basis bivectors can be expressed as the product of the pseudoscalar and a yuvl vector. Iz1 = z1z-z.z1 = z-z. Iz- = z1z-z.z- = −z1z. = z.z1 Iz. = z1z-z.z. = z1z- IN bvthzmvtixvl Wvxkgrouny The square of the pseudoscalar is equal to -1: I- = z1z-z.z1z-z. = z1z-z.z.z1z- = z1z-z1z- = −1 (2.22) Dflylifiy Duality is another frequently used concept of Geometric Algebra. The dual transformation is denoted as [ ]∗ and we map into the dual space by multiplying with the pseudoscalar as follows [b ]∗ =bI (2.23) GCFCHC ihe orthonormvl wvsis for xomputvtion Although many results in GA can be reached without resorting to a basis, it is helpful to have a basis specifying the numerical multivectors for the 3D GA. Hence, the multivector basis can be defined via outer products of the three orthonormal basis vectors z1, z- and z.. The outer product is usually visualised geometrically as the movement of one vector along the other to form a ‘directed area’. This is a new object, neither a vector nor a scalar. It is termed a wivzxtor. Similarly, the result of the outer product of this bivector and another vector is a trivzxtor (as already seen in section 2.1.2). Generally, an n-volume is termed an n-vector. Table 2.1 presents the multivector orthonormal basis. lyzle JFIFR ihz orthonormvl wvsis for xomputvtion Scalar: a Vector: z1 z- z. Bivector: z1- = z1 ∧ z- z1. = z1 ∧ z. z-. = z- ∧ z. Trivector: z1-. = z1 ∧ z- ∧ z. We can say that a scalar is grade 0, a vector is grade 1, a bivector is grade 2, as it is formed from 2 vectors, etc. Generally, an n-vector has grade n. Figure 2.2 illustrates an example of a trivector (sweeping a ∧ w along c). GCFCIC gotvtion using rotors Consider any three orthonormal basis vectors of R., {z1P z-P z.}, then the new three basis bivectors generated are W1 = z-z., W- = z.z1 and W. = z1z-. The basis bivectors all square to −1, and all anticommute as given below: W- = zizjzizj = −zizjzjzi = −1 for i ̸= j JFI Geomefiric Ylgezry IO a b c a ∧ b Figflre JFJFR ihz trivzxtor or yirzxtzy volumz is thz rzsult of sfizzping a ∧ w vlong xC An important feature of the bivectors is their effect on vectors. For example, the effect of the bivector W. on the vectors z1 and z1 + z- is to rotate them counter-clockwise by 90 degrees. This example is illustrated in figure 2.3. z1W. = z1z1z- = z- z-W. = z-z1z- = −z-z-z1 = −z1 (z1 + z-)W. = z1W. + z-W. = z- − z1 This is applicable for every bivector zizj over the plane defined by zi and zj . It is important here to mention that this method also works in higher-dimension spaces. e1 e2 e1 + e2e2 − e1 −e1 Figflre JFKFR ihz rotvtion zffzxt of wivzxtor W3 ovzr vzxtors z1 vny z1 + z2C Thus, any vector a in the plane defined by z1 and z- can be represented using the formula: a = r (z1 cos θ + z- sin θ) = z1r (cos θ +W. sin θ) (2.24) where r is the distance of the point a from the origin and θ is the angle between a and z1. This can be generalised if, instead of W., the unit bivector Wˆ = a∧w |a∧w| is used. Hence, an operator which performs rotation in the plane described by Wˆ by an angle 2θ, can be expressed as: R = exp ( θWˆ ) = 1 + θWˆ 1! + θ-Wˆ- 2! + θ.Wˆ. 3! + O O O = cos θ + Wˆ sin θ (2.25) IP bvthzmvtixvl Wvxkgrouny Then, any vector a which lies in the plane of the bivector Wˆ can be represented by: a = z1r exp ( θWˆ ) (2.26) Using this, it can be shown that a rotation by ϕ radians in the plane of Wˆ is accomplished by: a 7→ a′ = a exp ( ϕWˆ ) = a ( cos (ϕ) + Wˆ sin (ϕ) ) (2.27) It is important to mention that here the vector to be rotated a lies in the plane of rotation. In a different case, the vector should be decomposed into a component that lies in the plane, a‖, and one normal to the plane, a⊥. a = a‖ + a⊥ (2.28) Now, assuming that R = exp ( −ϕ- Wˆ ) and R˜ = exp ( ϕ - Wˆ ) , consider the following operation: a′ = R ( a‖ + a⊥ ) R˜ = ( cos ( ϕ 2 ) − Wˆ sin ( ϕ 2 ))( a‖ + a⊥ )( cos ( ϕ 2 ) + Wˆ sin ( ϕ 2 )) = a⊥ + ( cos- ( ϕ 2 ) − sin- ( ϕ 2 )) a‖ + 2 cos ( ϕ 2 ) sin ( ϕ 2 ) Wˆa‖ = a⊥ + a‖ ( cos(ϕ) + Wˆ sin(ϕ) ) (2.29) Equation 2.29 relies on Wˆa⊥ = a⊥Wˆ and Wˆa‖ = −a‖Wˆ, which states that a bivector commutes with a perpendicular vector and anticommutes with a parallel vector. Thus, the component of the vector that lies in the plane is rotated around an axis normal to the plane without affecting the component that is normal to the plane. This leads to a general method of rotation in any plane. Therefore, a given rotation ϕ in a plane specified by Wˆ can be performed as: a 7→ RaR˜ (2.30) using the element R = exp ( −Wˆ ϕ- ) . R is referred to as a rotor . The most important properties of rotors are that RR˜ = 1 and thus R˜ = R−1, and RaR˜ = (−R)a(−R˜), which means that there is a double covering of the space of rotations. Rotors in GA are simpler to manipulate than Euler angles and avoid the problem of gimbal lock. Gimbal lock is a common problem associated with Euler angles and occurs because two axes become aligned during rotational operations, producing unexpected behavior since one degree of freedom is lost. JFJ Conformyl Geomefiric Ylgezry IQ GCFCJC geflextion Consider a reflection of the vector a in the plane orthogonal to a unit 3-vectorm, wherem- = 1. aa ′ a‖ a⊥ m plane Figflre JFLFR V rzflzxtion of thz vzxtor v in v plvnz pzrpznyixulvr to mC It is owvious thvt a = a⊥+a‖ vny a′ = a⊥ − a‖ The component of a that is parallel to the plane mr changes sign, whereas the perpendicular components remains unaffected, as shown in figure 2.4. Hence, the parallel component is the projection onto m and the perpendicular component is the remainder. a‖ = (a ·m)m (2.31) a⊥ = a− a ·mm = (am− a ·m)m = (a ∧m)m (2.32) The result of the reflection is therefore: a′ = a⊥ − a‖ = −a ·mm+ a ∧mm = −(m · a+m ∧ a)m = −mam (2.33) This can be extended to higher dimensions as given in [HS84]. GCGC Conformvl Geometrix Vlgewrv This section is an introduction to the Conformal Model of Geometric Algebra (CGA) as first introduced by Hestenes and Sobczyk in 1984 [HS84]. CGA is a mathematical framework that offers a compact and geometrically intuitive formulation of algorithms and an easy and immediate computation of rotors; it is thus suitable for applications in engineering, computer vision and robotics. CGA extends 3D Euclidean space to 5 dimensions by adding two extra basis vectors and thereby providing tools with which to represent and manipulate geometry. The additional basis vectors z and z¯ are added to the 3 existing basis vectors z1, z- and z. of the 3D GA. z and z¯ have opposite signatures; z- = +1 z¯- = −1 (2.34) JH bvthzmvtixvl Wvxkgrouny The new basis vectors thus provide a space that allows null vzxtors, n and n¯, which are defined as: n = z+ z¯ n¯ = z− z¯ (2.35) where n is associated with the point at infinity and n¯ with the origin. It is very easy to prove that these vectors are null since: n- = (z+ z¯) · (z+ z¯) = z- + 2(z · z¯) + z¯- = 1 + 0− 1 = 0 (2.36) and n¯- = (z− z¯) · (z− z¯) = z- − 2(z · z¯) + z¯- = 1− 0− 1 = 0 (2.37) Two basic identities for n and n¯ are: n · n¯ = (z+ z¯) · (z− z¯) = z- − z¯- = 2 (2.38) x · n = x · n¯ = 0 (2.39) where x ∈ R.. Another observation is that the squared bivector E-, where E = n ∧ n¯ is equal to 4; E- = (n ∧ n¯) · (n ∧ n¯) = (n · n¯)(n · n¯)− n-n¯- = 4 (2.40) since n ·n¯ = 2 and n- = n¯- = 0 and using the identity (a∧w) ·(x∧y) = −(a ·w)(w ·y)+(a ·y)(w ·x). lhe Hesfienes’ mypping The mapping used in Hestenes’ illustration ([HS84], page 302) is used to take a spatial (con- ventional 3D) vector to the equivalent CGA representation. Thus, H (x) = 1 2 (x− z)n (x− z) (2.41) where x is a 3D Euclidean vector. Substituting for n = z + z¯ and using the fact that z¯ · x = z¯ · z = n · x = 0, it is possible to rewrite the equation in terms of null vectors as: H (x) = 1 2 ( x-n+ 2x− n¯) (2.42) JFJ Conformyl Geomefiric Ylgezry JI We can show that H(x) is always a null vector by evaluating: [H(x)]- = 1 4 (x-n+ 2x− n¯) · (x-n+ 2x− n¯) = −1 2 x-n · n¯+ x- = −x- + x- = 0 (2.43) as z¯ ·x = z¯ ·z = n ·x = 0. Another interesting observation is that, for any vector V and W ∈ R0 which represent the points a and w in R4, V · W is related to the Euclidean distance between the points a and w, as proved in equation 2.44. V ·W = H(a) ·H(w) = 1 4 (a-n+ 2a− n¯) · (w-n+ 2w− n¯) = −1 2 a- + a · w− 1 2 w- = −1 2 (a− w)- (2.44) Hence, the Euclidean distance between the two points a and w can be defined as: y (VPW) = √ −2 (V ·W) (2.45) More information about yistvnxz gzomztry can be found in [Hes01], [HS84] and [DH93]. GCGCFC gotvtion In Conformal Geometric Algebra rotations are also performed with the rotor elements as in usual Geometric Algebra (see equation 2.25). Thus, given that Rn¯R˜ = n¯ and RnR˜ = n, x 7→ RxR˜ can be replaced by H(x) 7→ H(RxR˜), as is shown below. RH(x)R˜ = R 1 2 ( x-n+ 2x− n¯) R˜ = 1 2 ( x-RnR˜+ 2RxR˜−Rn¯R˜ ) = 1 2 ( x-n+ 2RxR˜− n¯ ) = H(RxR˜) (2.46) GCGCGC irvnslvtion The translation along a vector a is defined as the mapping x 7→ x + a for x. Here we show that this can be performed by applying a rotor R = ia = exp( na - ). Using the power series expansion of the exponential, the rotor can be simplified to: i = exp (na 2 ) = 1 + na 2 + 1 2 (na 2 )- + O O O = 1 + na 2 (2.47) JJ bvthzmvtixvl Wvxkgrouny since n is null, an = −na and therefore the higher order terms are all zero. The rotor i acts over the vectors n, n¯ and x as below: ini˜ = ( 1 + na 2 ) n ( 1 + an 2 ) = n+ nan+ nanan 4 = n (2.48) Similarly, i n¯i˜ = ( 1 + na 2 ) n¯ ( 1 + an 2 ) = n¯+ na 2 n¯+ n¯ an 2 + na 2 n¯ an 2 = n¯− 2a− a-n (2.49) and ixi˜ = ( 1 + na 2 ) x ( 1 + an 2 ) = x+ n(a · x) (2.50) Therefore, the translation over the null vector H(x) will be: iH(x)i˜ = ( 1 + na 2 ) 1 2 ( x-n+ 2x− n¯) (1 + an 2 ) = 1 2 ( x-n+ 2(x+ n(a · x))− (n¯− 2a− a-n)) = 1 2 ( (x+ a)-n+ 2(x+ a)− n¯) = H(x+ a) (2.51) To summarise, the x 7→ x+ a can be replaced by H(x) 7→ iH(x)i˜ = H(x+ a). GCGCHC Dilvtion A dilation by a factor of α can be represented by the mapping x 7→ αx. This can be achieved by considering the rotor R = Yα = exp ( α - zz¯ ) . Since −zz¯n = nzz¯ = n and −n¯zz¯ = zz¯n¯ = n¯ and by expanding exp ( α - zz¯ ) , the dilation of the vector H(x) by a factor of exp(−α) about the origin returns: YH(x)Y˜ = exp (α 2 zz¯ ) 1 2 ( x-n+ 2x− n¯) exp(−α 2 zz¯ ) = 1 2 ( x- exp (α 2 zz¯ ) n+ 2x− exp (α 2 zz¯ ) n¯ ) = 1 2 ( x- exp (−α)n+ 2x− exp (α) n¯) = exp(α) 1 2 ( exp(−2α)x-n+ 2 exp(−α)x− n¯) = H(exp(−α)x) (2.52) JFJ Conformyl Geomefiric Ylgezry JK Hence, the dilation x 7→ exp(−α)x can be represented by H(x) 7→ YH(x)Y˜ = exp(α)H(exp(−α)x) (2.53) GCGCIC Inversion Inversion in the origin corresponds to the mapping x 7→ x x2 or for non-singular vectors x 7→ x−1. This can be performed by reflecting in z. The reflections in z of the vectors n, n¯ and x are given by: −znz = −zzn¯ = −n¯ −zn¯z = −zzn = −n −zxz = x The inversion of H(x) under the reflection in z then gives −zH(x)z = −z1 2 ( x-n+ 2x− n¯) z = 1 2 (−x-n¯+ 2x+ n) = x- 1 2 ( 1 x- n+ 2 x x- − n¯ ) = x-H ( x x- ) (2.54) Thus, the inversion x 7→ x x2 is replaced by H(x) 7→ − zH(x)z x2 = H( x x2 ). GCGCJC Wlvyes in CGV We have already seen that the term wlvyz in GA is used to refer to quantities that can be written as the wedge product of vectors. For example, an r-blade can always be written as V1∧V-∧O O O∧Vr, which differs from an r-vector that may be any linear combination of r-blades. Here we assumed that all null-vectors m are defined such that m · n = −1, unless otherwise stated. necfiors ynd JEzlydes A 2-blade is formed from the outer product V∧W of two null vectors V and W. Considering the differentiation between signs of null vectors in CGA, many different separations are possible; V ∧W = −(W ∧V) = −(W) ∧V = W ∧ (−V) (2.55) The outer product V∧W can be separated into a pair of individual null-vectors that are unique up to a scale, as shown below. JL bvthzmvtixvl Wvxkgrouny Exfirycfiing –ecfior V ynd –ecfior W from V ∧W We often need to extract the two vectors V and W from the bivector V ∧ W. This paragraph presents a solution of this problem using projzxtors. Assume that V and W are normalised so that V · n = −1 and W · n = −1. Let the 2-blade i = V ∧W and form F = 1 β V ∧W (2.56) where β S 0 and β- = i -, so that F - = 1 if β- ̸= 0. Thus, two projector operators can be defined as: e = 1 2 (1 + F ) e˜ = 1 2 (1− F ) (2.57) where e˜ denotes the normal reversion operation applied to e . Note that ee = e , which can be verified as follows ee = 1 4 (1 + F ) (1 + F ) = 1 4 (1 + 2F + 1) = 1 2 (1 + F ) (2.58) Similarly, it can be shown that e˜ e˜ = e˜ . An equally important property is that e˜e = ee˜ = 0 which, again, is easy to show ee˜ = 1 4 (1 + F ) (1− F ) = 1 4 ( 1− F -) = 1 4 (1− 1) = 0 (2.59) In the same way, it can be shown that e˜e = 0. It is important here to show what effect these projectors have on V and W. Hence, e and e˜ acting on V and W will return the following results: eV = 0P eW = WP e˜V = VP e˜W = 0P This can be easily verified as below: eV = 1 2 [ 1 + 1 β V ∧W ] V = 1 2 [ V+ 1 β (V ∧W)V ] = 1 2 [ V+ 1 β (V ·W)V ] = 1 2 (V−V) = 0 (2.60) JFJ Conformyl Geomefiric Ylgezry JM since (V ∧W)V = (V ∧W) ·V = −V-W + (V ·W)V = (V ·W)V , because V- = 0 and V ·W = −β. This follows from equation 2.45, where V ·W must be negative, the facts that V- = W- = 0 and also from: β- = (V ∧W) · (V ∧W) = −V-W- + (V ·W)- (2.61) Using similar manipulations, it can be shown that eW = W, e˜V = V and e˜W = W. The next step is to consider the vector obtained by dotting V ∧W with n. (V ∧W) · n = −n · (V ∧W) = − (n ·V)W + (n ·W)V = (W −V) (2.62) using the fact that V and W are normalised points such that V · n = W · n = −1. Thus, it follows that: e [(V ∧W) · n] = e (W −V) = W (2.63) −e˜ [(V ∧W) · n] = −e˜ (W −V) = V (2.64) It is also noted that since Ve = e˜V = V it follows that e˜Ve = e˜ e˜V = e˜V. In that way we have: e˜Ve = e˜V eVe˜ = 0 eWe˜ = eW e˜We = 0 which mean that the projectors can be written as two-sided operations. Hence, the two points V and W can be extracted from a 2-blade V ∧W as: V = −e˜ [(V ∧W) · n] ≡ −e˜ [(V ∧W) · n]e = −e˜ [〈(V ∧W)n〉1] (2.65) W = e [(V ∧W) · n] ≡ e [(V ∧W) · n] e˜ = e [〈(V ∧W)n〉1] (2.66) The use of projectors ensures that one projector, e , will always return the first point, V, and the second projector, e˜ , will return the second point, B. As a result, an orientation may JN bvthzmvtixvl Wvxkgrouny q p L = P ∧Q ∧ n q p L′ = Q ∧ P ∧ n Figflre JFMFR V linz a = e ∧f ∧ n pvssing through thz null vzxtors e vny f vny thz uoppositz’ linzA a′ = f ∧ e ∧ nC be inferred for this point pair, hence one can be considered to occur ‘before’ the other. Note that, it is not possible to extract the null vector from a bivector having one of its points as n (e.g. W = V ∧ n) using the 2-blade projector-based separation method. One approach to get round this problem was proposed in [LLW04] and in [Cam07] with minor corrections. A bivector W may be in the form V ∧ n or n ∧ V, where V is the null vector representing the spatial point a. Hence, a = 1 4 (W ∧ n¯) Oc ⇔ n is the first component (2.67) a = 1 4 (n¯ ∧W) Oc ⇔ n is the second component (2.68) where c = n ∧ n¯. The usefulness of these results will become palpable later when intersections are considered (section 2.2.6). KEnecfiors In this section we study trivectors. Consider the null vectors e , f and R in the 5D space representing the points p, q and r respectively in 3D space . dines ys firi–ecfiors In order to define a line with direction p to q, the trivector a needs to be introduced. The trivector a is formed as: a = e ∧f ∧ n (2.69) = f ∧ n ∧ e = n ∧ e ∧f and corresponds to a line that is passing through the points represented by the null vectors e and f. It can be shown algebraically that these 3 lines return the same blade. However, the line a′ = f ∧ e ∧ n = −a will have the opposite direction, differing only in sign, due to the anti-commuting nature of the outer product of the vectors. An example of the lines a and a′ is given in figure 2.5. JFJ Conformyl Geomefiric Ylgezry JO Circles ys firi–ecfiors A circle can be defined as a trivector X, where the point at infinity, n, is replaced by a third point on the circle, R. Hence, the equation of a circle in CGA is formed as: X = e ∧f ∧R (2.70) where X corresponds to a circle that is passing through the points represented by the null vectors e , f and R, as shown in figure 2.6. q p r C = P ∧Q ∧R Figflre JFNFR V xirxlz X = e ∧f ∧R pvssing through thz null vzxtors e A f vny RC The unit circle (figure 2.7) in the plane is formed as the circle passing through the three key points, z1, −z1 and z-. Hence, for any unit length vector x, we know that H(x) = 1 -(n+ 2x− n¯) = (x+ z¯). Thus, we have: H(z1) ∧H(z-) ∧H(−z1) = 2z1z1z¯ and hence the trivector X = 2z1z1z¯ represents the unit circle. The dual of the unit circle X ∗ is equal to X∗ = XI4 = 2z = (n+ n¯) (2.71) where I4 is the pseudoscalar given by I4 = z1z-zz¯. −e1 e1 e2 C = e1 ∧ e2 ∧ −e1 Figflre JFOFR ihz unit xirxlzC If m lies on the circle, we know that m ∧ X = 0. Since X∗ is the dual of a trivector in a 4D space, it is a vector. Hence, an alternative but useful representation of the circle is given by m · X∗ = 0 In the equation 2.44 we proved that for any two normalised point representations V and W, JP bvthzmvtixvl Wvxkgrouny V · W = −1-(a − w)-. Thus, consider a point on circle, m, and the circle centre, W, then the radius / of the circle can be calculated as: m ·W = −1 2 (x− w)- ≡ −1 2 /- (2.72) For a normalised point representation m this implies that m · ( W − 1 2 /-n ) = 0 (2.73) since m · n = −1. Comparing this with m · X∗ we see that our normalised X∗ is X∗ = W − 1 2 /-n (2.74) Therefore, the vector X∗ encodes in a neat fashion the centre and the radius of the circle in the plane. We also note that the radius of the circle can be easily calculated by squaring X∗ (X∗)- = ( W − 1 2 /-n )- = −/-W · n = /- (2.75) since W- = 0, n- = 0 and W · n = −1. Using equation 2.75 it is easy to show that: W = X∗ + 1 2 (X∗)-n (2.76) Note that the above relations assume that X∗ is normalised such that X∗ · n = −1 since X∗ · n = W · n = −1, as it is assumed that W is a normalised null vector. However, there is a more elegant way of calculating the centre of a circle in 3D, as proved in [LLW04]. The centre of a circle, X, is also given by reflecting the point at infinity, n, in the circle as: W = XnX (2.77) LEnecfiors This section studies 4-vectors. 4-vectors in 5D space reprsent 3-planes, Φ, and spheres, Σ. hlynes ys LE–ecfiors A 3-plane, Φ, passing through the three points defined by null vectors e , f and R is given by: Φ = e ∧f ∧R ∧ n (2.78) Figure 2.8 illustrates an example of a 3-plane passing through the points p, q and r. The physical quantities we might want to extract from such a 4-vector are clearly the normal to the plane and the perpendicular distance of the plane from the origin. This can be achieved JFJ Conformyl Geomefiric Ylgezry JQ b b b p q r Φ = P ∧Q ∧ R ∧ n Figflre JFPFR ihz HBplvnz Φ = e ∧f ∧R ∧ nC using the dual of the plane. Consider the plane z = y, which is parallel to the xy-plane and distance y from it. The plane Φ can be represented as Φ = F (yz.) ∧ F (z1 + yz.) ∧ F (z- + yz.) ∧ n = 1 8 {(2yz. − n¯) ∧ (2[z1 + yz.]− n¯) ∧ (2[z- + yz.]− n¯) ∧ n} = yz1 ∧ z- ∧ z. ∧ n− 1 2 z1 ∧ z- ∧ n¯ ∧ n = yz1 ∧ z- ∧ z. ∧ n− z1 ∧ z- ∧ z ∧ z¯ (2.79) Then it is simple to show that the dual of Φ is given by Φ∗ = ΦI = yn+ z. (2.80) This holds for any y. If the plane is rotated by R, such that Rz.R˜ = nˆ, the general equation for the dual of the new plane RΦ∗R˜ is given by Φ∗ = yn+ nˆ (2.81) Note that the above assumes that the dual is normalised such that [Φ∗]- = 1; this can be ensured by normalising the plane such that Φ- = 1. Therefore, given 3 points on the plane, e , f and R, we form the normalised plane Φ and its dual Φ∗, and we can then extract nˆ and y as follows y = 1 2 Φ∗ · n¯ (2.82) nˆ = Φ∗ − 1 2 (Φ∗ · n¯)n (2.83) kpheres ys LE–ecfiors Spheres can be defined as a 4-vector where the point at the infinity, n, is replaced by a point on the sphere. Hence, given any 4 points p, q, r and s whose 5D representations are the null vectors e , f, R and S respectively, the sphere passing through those points is defined by the 4-vector Σ = e ∧f ∧R ∧ S. KH bvthzmvtixvl Wvxkgrouny We know that m ∧ Σ = 0 for any m lying on the sphere. This can be rewritten as m · Σ∗ = 0 (2.84) where Σ∗ is the dual to Σ, hence is a vector. As for the circle solution, we can show that the dual representation of the sphere naturally encodes the centre and the radius of the sphere. Thus m · X = −1 2 (x− x)- ≡ −1 2 /- (2.85) where m is a point on a sphere, X is the centre of the sphere and / is the radius of the sphere. For a normalised point m, where m · n = −1 this means that m · ( X − 1 2 /-n ) = 0 (2.86) Comparing equation 2.86 with equation 2.84, we can find that Σ∗ = X − 1 2 /-n (2.87) Thus, the dual vector Σ∗ encodes the centre and radius of the sphere. The radius and the centre can immediately be calculated by squaring Σ∗ (Σ∗)- = ( X − 1 2 /-n )- = /- since X- = 0, n- = 0 and X · n = −1. Using this equation, we can easily show that: X = Σ∗ + 1 2 (Σ∗)-n (2.88) As for the circle case, the centre, X, of a sphere, Σ, is also given by reflecting the point at infinity, n, in the sphere: X = ΣnΣ (2.89) Note that we have used the idea of reflection, developed in section 2.1.5, to reflect geometric objects in other geometric objects [LLW04]. MEnecfiors 5-vectors, such as the pseudoscalar I0, have two orientations and these correspond to Stolfi’s concept [Sto91] of an oriented universe. The pseudoscalar is defined as I0 = z1∧z-∧z.∧z∧ z¯ = z1-.40, and it satisfies I - 0 = −1. JFJ Conformyl Geomefiric Ylgezry KI GCGCKC Intersextions This section outlines the various ways of intersecting objects within the conformal geometric algebra model. In this report, an operator termed thz mzzt is used [HS84], denoted by the symbol ∨, which given two objects V and W returns their intersection. For instance, the meet of 2 entities defined by blades Ym and Es of grades m and s, lying within an embedding entity of grade p is given by Y ∨ E = [ 〈YE〉-p−m−s ]∗ (2.90) anfiersecfiing lines wifih lines Here we consider the intersection of two lines. Let the lines be a1 and a-. The meet of these two lines is given by m = a1 ∨ a- = [〈a1a-〉-n−r−s]∗ (2.91) where n denotes the dimension of the embedding space, r is the grade of the first line and s the grade of the second. Thus, 2n − r − s = 10 − 3 − 3 = 4, so the dual object has grade 1. However, if the lines intersect at a point, the meet, m, will not return this intersection point. Instead, if the lines intersect, a1 ∨ a- = 0 and if the lines do not intersect, a1 ∨ a1 ∝ n. In order to find the intersection point, we use the following procedure. Assume that the lines a1 and a- intersect at point e . Reflect the line a1 in line a- and get a ′ 1 = a-a1a-. Then, find the line which is perpendicular to a-, passing through the point e , which can be shown to be equal to ae- = a1 − a-a1a- [LLW04]. Take any arbitrary point representation n and reflect in ae- via n ′ = ae- n ae- . Then, take the midpoint of n and n ′, n ′′ = 1 -(n + n ′), which must lie on the line ae- . Thereafter, reflect the point n ′′ in line a- to give n ′′′ = a-n ′′a- and again take the midpoint e ′ = 1-(n ′′+ n ′′′). The representation of the intersection point can then be extracted via equation 2.92; figure 2.9 illustrates the above. e = −(e ′ne ′) 2(e ′ · n)- (2.92) Note, n ′′ and e ′ as given here are not null vectors but are the null vectors representing the midpoints plus some multiples of n. In equation 2.92 these multiples of n are eliminated. anfiersecfiing circles wifih circles ynd lines We now consider the intersection of two circles or a circle and a line. Firstly, we will focus on the meet of two circles, X1 and X-. m = X1 ∨ X- = [〈X1X-〉-n−r−s]∗ (2.93) where 2n−r−s = 10−3−3 = 4, so that the dual object has grade 1. However, the intersection of two circles mostly returns two intersection points (when they lie in the same plane), and the KJ bvthzmvtixvl Wvxkgrouny L1L2 L ′ 1 L P 2 Y Y ′ Y ′′ Y ′′′ P Figflre JFQFR ihz solution for linz to linz intzrszxtionC 1-grade object m can not give us the two points. In that case, the meet does not return the intersection points but just an object that helps us to conclude whether the circles intersect at two points, a point or if they do not intersect at all. Hence, if circles have two intersections, X1 ∨ X- = 0P if circles have one intersection, X1 ∨ X- = mP where m- = 0 if circles have no intersection, X1 ∨ X- = mP where m- ̸= 0 If X1 ∨X- = m and m- = 0, then it is obvious that the intersection point is represented by m. If the meet gives zero, there are two intersections, and these can easily be found by intersecting the plane of one of the circles with the other circle as: W = X1 ∨ (X- ∧ n) = [〈X1(X- ∧ n)〉-n−r−s]∗ (2.94) where 2n− r − s = 10− 3− 4 = 3, so that the dual object has grade 2. Thus, the two points can be extracted from the bivector W using the projectors given in equation 2.65. Now, consider the intersection of a circle X1 and a line a1. The meet is again a grade 1 object, and it similarly holds that: if circle and line have two intersections, X1 ∨ a1 = 0P if circle and line have one intersection, X1 ∨ a1 = mP where m- = 0 if circle and line have no intersection, X1 ∨ a1 = mP where m- ̸= 0 Equally, if X1∨a1 = m and m- = 0, then the intersection point is represented by m and if the meet gives zero, the two intersections can be found by intersecting the plane of the circle with the line. It is also important to mention that, in the case where X1 ∨ a1 = m and m- ̸= 0, which means that no intersection exist, the sign of m- tell us whether the line passes through the circle (m- Q 0) or does not pass through the circle (m- S 0). JFJ Conformyl Geomefiric Ylgezry KK anfiersecfiing plynes wifih plynesD circles ynd lines This section studies the intersection between planes and planes, planes and circles, and planes and lines, Firstly, the plane to plane intersection will be studied. Consider two planes Φ1 and Φ-. The meet between two planes gives: a = Φ1 ∨ Φ- = [〈Φ1Φ-〉-n−r−s]∗ (2.95) where 2n − r − s = 10 − 4 − 4 = 2. Therefore, the dual object has grade 5 − 2 = 3, and represents a line. The sign of a- indicates whether the planes intersect. If a- S 0, then the planes intersect in the line a. On the other hand, when a- = 0, the intersection does not exist, which means that the two planes are parallel. The intersection of a plane Φ1 and a circle X1, when this exists, is a pair of points, or a single point and it is formulated as: W = Φ1 ∨ X1 = [〈Φ1X1〉-n−r−s]∗ (2.96) where 2n − r − s = 10 − 4 − 3 = 3, and so the dual object has grade 2. Looking at the sign of the resulting 2-blade, W, it is possible to tell if the intersection exists and whether the intersection is a single point or a pair of points. Therefore, if W- S 0, then the meet between Φ1 and X1 gives two points. Given the bivector, W, the two points of the intersection can be extracted via the projectors given in equation 2.65. In the case where W- = 0, the intersection is a single point, m. It is then trivial to find the representation of that point using the formula m = WnW. Finally, when W- Q 0 holds, the intersection between the plane and the circle does not exist. Replacing the circle X1 with a line a1, the meet will still give us a 2-blade, W. However, a line and a plane intersect at most in one single position. Thus, the bivector will be of the form W = m ∧ n, where m is the representation of the point of the intersection. If W- S 0 the line and plane intersect in a point, if W- = 0 the line and plane are parallel and never intersect and if W = 0 the line lies in the plane. anfiersecfiing spheres wifih circles or lines Now we consider, the intersection of a sphere, Σ1 (4-blade), with a circle, X1 (3-blade) or a single line, is the subject of this section. The intersection, where it exists, could be a single point or a pair of points (or the circle X1 for the case where the circle is exactly on the outline of the sphere). According to the meet formulation, the intersection of a sphere, Σ1, with a circle, X1 can be expressed as: W = Σ1 ∨ X1 = [〈Σ1X1〉-n−r−s]∗ (2.97) 2n − r − s = 10 − 4 − 3 = 3, hence, the dual quantity will have grade 5 − 3 = 2, which is a KL bvthzmvtixvl Wvxkgrouny bivector, and represents the 2 intersecting points. Once again, the sign of the resulting squared 2-blade, W-, gives us information on whether the intersection exists and if the sphere and circle intersect in a single point or a pair of points. In the case of two intersections the points can be extracted from W using the projectors, as before, and in the case of tangency, the single point of contact is obtained by taking WnW. Similarly, substituting the circle X1 with a line a1 and intersecting with a sphere Σ1, the meet again returns a 2-vector whose square denotes whether there are two, one or no intersection. The intersection points can be obtained easily in the same way as for the circle case. anfiersecfiing spheres wifih spheres or plynes Here we deal with the intersection of two spheres Σ1 and Σ- or the intersection of a sphere Σ1 and a plane Φ1. This intersection, where it exists, is a circle (or a single point). Firstly, we will consider the intersection between two spheres. Spheres do not intersect if the distance from their centres is less than the sum of their radii y S (/+ + /1). Also, if y Q |/1 − /-|, one of the two spheres is completely contained in the other, hence no intersection exists. If y Q |/1 − /-| and /1 Q /-, then the first sphere is contained in the second, otherwise the second sphere is contained in the first. In the case where y = |/1 − /-| = 0, then the two spheres are identical and the distance between them is trivially 0. The two spheres are intersecting only when |/1 − /-| ≤ y ≤ /1+ /-. The intersection will be a circle (or a point, if the two spheres merely touch one another) with normal plane Φ. This circle can be calculated using the formula for the meet ([LLW04]): X = Σ1 ∨ Σ- = [〈Σ1Σ-〉-n−r−s]∗ (2.98) 2n − r − s = 2 · 5 − 4 − 4 = 2, so that the dual quantity will have grade 5 − 2 = 3, which is a trivector, and generally represents the circle of intersection. The value of C can tell us whether the result is a circle ( X- S 0 ) , a single point ( X- = 0 ) or there is no intersection( X- Q 0 ) . In the case where ( X- S 0 ) , the centre and radius of the circle can be extracted according to section 2.2.5. Similarly, using the same extracting formula from C for the case where ( X- = 0 ) , we will find that the circle has zero radius and its centre will be the point of tangency of the two spheres. In the same way, an attempt to extract the radius and the centre for the case where ( X- Q 0 ) leads to an imaginary value for the radius, and a centre (because no intersection exist) which lies on the shortest line joining the surface of the spheres (i.e. that joining the centres). If the two spheres have the same radii, it is the midway point on this line. In the same way, instead of having a second sphere, Σ-, we can have an intersection with a plane, Φ1. The result of the meet between sphere and plane will also be a trivector, X, X = Σ1 ∨ Φ1 = [〈Σ1Φ-〉-]∗ (2.99) and the sign of the square of this trivector, X- indicates whether the two objects are tangent, intersect in a circle or do not intersect at all. JFK Conclflsions KM GCHC Conxlusions This chapter introduced Geometric Algebra and examined how the Conformal Model can be used to represent geometric primitives such as points pairs, lines, circles, planes and spheres. We also looked at how rotors can be applied to those objects within the Conformal Model. CGA is a mathematical framework that offers a compact and geometrically intuitive formulation of algorithms and an easy and immediate computation of rotors; it is thus suitable for applications in engineering, computer vision and robotics. The model described in this chapter will be the mathematical basis on which this thesis is based. 3 V covel Inverse Kinemvtixs holver I T this chapter, we address the problem of manipulating articulated figures in an inter-active and intuitive fashion for the design and control of their posture. This problem finds its application in the area of robotics, computer animation, ergonomics and the com- puter games industry. In the area of computer graphics, articulated figures are a convenient model for humans, animals or other virtual creatures from films and video games. The most popular method for animating such models is motion-capture; however, despite the availability of highly sophisticated techniques and expensive tools, many problems appear when dealing with complex figures. Most virtual character models are complicated; they are made up of many joints having a high number of degrees of freedom (DoF), thus, it is often difficult to produce a realistic character animation. HCFC Introyuxtion vny botivvtion A posturz is defined as the skeletal configuration of a figure; for a realistic posture a set of criteria should be satisfied. All character models have natural articulation limits and inter- penetration of the body with other objects or themselves is not permitted. In addition, physical laws should be considered as well as numerous personal factors. General constraints can be applied to most articulated figures, however special cases of posture control are needed when a large number of degrees of freedom exist. 37 KP V covzl Invzrsz ‘inzmvtixs holvzr Inverse Kinematics (IK) is a method for computing the posture via estimating each indi- vidual degree of freedom in order to satisfy a given task; it plays an important role in the computer animation and simulation of articulated figures. Inverse Kinematics finds applica- tions in several areas. IK methods have been implemented in many computer graphics and robotics applications, aiming to animate or control different virtual creatures. They are also very popular in the video games industry. The field of computer-aided ergonomics is also concerned with articulated figures, especially human models developed for simulation and pre- diction purposes. The need for accurate biomechanical modelling and body sizing based on anthropometric data make IK methods a popular approach for fast and reliable solution. IK has been used in rehabilitation medicine in order to observe asymmetries or abnormalities. Recently, IK techniques have also been applied in protein science for protein structure predic- tion [CD03]. However, the production of real-time IK solvers that are able to return realistic postures, without erratic discontinuities and singularities, posed a problem for researchers for many years; many algorithms have been implemented for computing skeletal poses but most suffer from unnatural poses, have difficulties in dealing with complex figures and are compu- tationally expensive. In this work, the most popular Inverse Kinematic techniques are reviewed. A new heuristic iterative method, FABRIK, is also presented for solving the IK problem in different scenarios. FABRIK (Forward And Backward Reaching Inverse Kinematics) is an efficient method for solving the IK problem; it uses a forward and backward iterative approach, finding each joint position via locating a point on line. FABRIK has been utilised in highly complex systems with single and multiple targets, with and without joint restrictions. It can easily handle end effector orientations and support, to the best of our knowledge, all chain classes. A reliable method for incorporating constraints is also presented and utilised within FABRIK. The proposed method retains all the advantages of FABRIK, producing visually smooth movements without oscil- lations and discontinuities, and with low computational cost. Several experiments have been implemented for comparison purposes between the most popular manipulator solvers, includ- ing multiple end effectors with multiple tasks, and highly constrained joints. The algorithms are tested for reliability, computational cost, realistic movements, reconstruction quality, con- version criteria and number of iterations. In this chapter, vectors will be designated in bold font to distinguish them from other symbols and avoid any confusion. HCFCFC ihe vrtixulvtey woyy moyel This section provides a brief introduction to the human skeleton and joint modelling. Before motion data can be edited by any system, it usually needs to be preprocessed to ensure that correct hierarchical connections and constraints are satisfied. Human body modelling is a problem that arises in ergonomics and in computer graphics applications. It is a complex hierarchical model consisting of many joints, each one having different degrees of freedom and various possible restrictions. In fact, the human body consists of more than 200 bones and joints. KFI anfirodflcfiion ynd eofii–yfiion KQ Joints Head & spine chain Right leg chain Left leg chain Right arm chain Left arm chain Right hand model Left hand model Right foot model Left foot model Figflre KFIFR Vn zflvmplz of v skzlztvl struxturz of v humvnC Hflmyn Zody eodelling A rigid multibody system consists of a set of rigid objects, called links, connected together by joints. A joint is the component concerned with motion; it permits some degree of relative motion between the connected segments. Virtual body modelling is important for human posture control. A well constrained model can restrict postures to a feasible set, therefore allowing a realistic motion. Most models assume that body parts are rigid, although this is just an assumption approximating reality. The skeletal structure is usually modeled as a hierarchy of rigid segments connected by joints, each defined by their length, shape, volume and mass properties. The skeletal structures are often defined using a parent-child system (see figure 3.1). The size, shape and proportions of the body and its segments are also essential in order to build models with realistic dimensions and proportions. Figure 3.2 shows examples of a model of a human body and human legs taken by a motion capture system (PhaseSpace Impulse System [Pha]) and graphically processed in Blender [Ble]. The joints are shown as spheres. A manipulator such as a robot arm or an animated graphics character is modeled as a xhvin composed of rigid links connected at their end by rotating joints. Any translation and/or rotation of the i-th joint affects the translation and rotation of any joint placed later in the chain. The chains are built under the assumption that all bones have at most one parent and any number of children. The chains can be formalised as follow: All bones (joints) with no children are marked as zny zffzxtors; a chain can be built for each end effector by moving back through the skeleton, going from parent to parent, until the root (the start of the chain) is reached. By definition, in the IK problem, the root joint is assumed fixed but methods can cope easily with translation of the root. There are a variety of possible joint types. For a well designed human model, it is essential to study these joint types. Each joint provides a local rotation (and each bone a local translation) with different degrees of freedom (DoF). Different rotation paradigms arise from different joint LH V covzl Invzrsz ‘inzmvtixs holvzr Figflre KFJFR evrt of v skzlztvl vnimvtion przsznting joints of v humvn woyy (lzft) vny humvn lzgs (right)C types. The main human joint types are enumerated below (see also figure 3.3): 1. ihz suturz joint moyzl (1 DoF): This is a fixed joint that allows very limited movement. Suture joints can be found in the skull. The bones in the skull are held together with fibrous connective tissue. 2. ihz hingz joint moyzl (1 DoF): The simplest type of joint; it can be found in the elbows, knees and the joints of the fingers and toes. Hinge joints allow movement in only one direction. 3. ihz gliying joint moyzl (2 DoF): Gliding joints permit a wide range of mostly sideways movements - as well as movements in one direction. 4. ihz svyylz joint moyzl (2 DoF): A saddle joint is more versatile than either a hinge joint or a gliding joint. It allows movement in two directions. 5. ihz pivot joint moyzl (2 DoF): The pivot joint is a 2 degree of freedom joint and it can be found in the neck allowing a side to side turn of the head. 6. ihz wvll vny soxkzt joint moyzl (3 DoF): This is the most mobile type of joint in the hu- man body; it allows 3 degrees of freedom. A limited (in the sense of restricted magnitude) version of the ball and socket joint is the Ellipsoidal joint. It is also possible to work with more general types of joints, and thereby simulate non-rigid objects. Kinematic joint models must be defined in order to formalise the relative motion of each joint. An analytically and anatomically correct model is necessary to control and constrain the available movements of the human body. These models are mainly characterised by the number of parameters which describe the motion space and are usually constrained by joint limits and joint structure [BPW93, Cra89]. Because of their complex nature, most of the KFI anfirodflcfiion ynd eofii–yfiion LI Figflre KFKFR ]umvn joints fiith thzir vvvilvwlz movzmzntsC ihz imvgzs hvvz wzzn tvkzn from thz bixrosoft Enxvrtv dnlinz Enxyxlopzyiv GEEM pEnxrC proposed joint models are simplified or approximated by more than one joint. There are many different models, each one performing different movements. Each specific model can be expressed via multiple joints of different types together with their movements and degrees of freedom. The most well-known models are: thz shoulyzr moyzl, a very complex model composed of 3 different joints [MT00, WV98, KTL07, HUHF03]; thz spinz moyzl, a complex arrangement of 24 vertebrae (usually, for simplicity, the spine is modelled as a simple chain of joints [BPW93, IP90, Kor85, MB91]); thz hvny moyzl, this is the most versatile part of the body comprising a large number of joints [RG91, MSZ94, KL07]; thz strzngth moyzl, which takes account of the forces applied from the skeletal muscles to the bones [BPW93]. A realistic body appearance is also very important in many graphical applications. Thus, data additional to the skeletal structure must be added for the generation of a more realistic human animation with skin, face, clothes etc [SPCM97]. Figure 3.3 shows an example of human joints with their available degrees of freedom. More details about human body and kinematic joint models can be found in [BPW93, Cra89, MT00, Kor85, MSZ94, WW91]. eofiion Once a body model has been defined, it can then be animated, manipulated or simply used for simulation purposes. Animating articulated figures is highly dependent on their allowed motion. botion is the change in position of an object with respect to a reference. A motion can be achieved when a rotational or translational transformation has been applied in order to move the end effector(s) of a chain to a desired position. There are two main issues related to motion and these are given below. LJ V covzl Invzrsz ‘inzmvtixs holvzr target base (u= target base (v= targetbase (w= Figflre KFLFR eossiwlz solutions of thz I‘ prowlzmO (v) ihz tvrgzt is unrzvxhvwlzP in mvny xvszs it is impossiwlz for thz linkzy struxturz to touxh thz tvrgztA (w) dnz solutionO thzrz vrz instvnxzs fihzrz thzrz is only onz solution to thz prowlzmA (x) bvny solutionsO most oftznA thz I‘ prowlzm hvs morz thvn v singlz solutionC • Forfivry Kinzmvtixs (FK): can be defined as the problem of locating the end effectors’ positions after applying known transformations to the chain. • Invzrsz Kinzmvtixs (IK): is described as the problem of determining an appropriate joint configuration for which the end effectors move to desired positions, named tvrgzt positions, as smoothly, rapidly, and as accurately as possible. During recent decades, many methods have been proposed to solve the IK problem. However, for a complete IK solver it is important to apply restrictions in order to control the joint configurations, according to the joint type. Moreover, we often have models with multiple end effectors and multiple targets. Performing single tasks sequentially is not a practical way of controlling complex figures. Therefore, it is desirable for a resolution technique to be able to manage multiple tasks with an appropriate strategy. The FK problem has a unique solution, and its success depends on whether the joints are allowed to do the desired transformation. In contrast, when dealing with IK, it is not always the case that a solution can be achieved. There are instances where the goal is unreachable or when two or more tasks conflict and cannot be satisfied simultaneously. Unreachable targets are the targets which can be further than the chain can reach or can be at a point where no pivoting of links can bend the chain to reach (see figure 3.4). These problems are known as ovzrBxonstrvinzy problems. On the other hand, there are instances where more than a single solution exists. It is up to the IK method to choose the best solution and the IK solver’s performance is ranked according to how realistic the solution is and the computational cost of choosing that solution. HCGC gelvtey lork The production of realistic and plausible motions remains an open challenge within the robotics and animation communities. Several algorithms have been implemented for computing the poses of a skeletal structure; the most popular techniques for solving the IK problem are presented in this section. KFJ jelyfied oork LK Let the complete joint configuration of the multibody be specified by the scalars θ1P OOOP θn, assuming that there are n joints and each θj value is called a joint vnglz (joint configurations may not always be in terms of angles), where θj is the angle in the plane of rotation assuming we also have knowledge of the rotation axis. Certain points on the links are identified as zny zffzxtors. To solve the IK problem, the joint angles must be settled so that the resulting configuration of the multibody places each end effector at, or as close as possible to, its target position. If there are k end effectors, let their positions be denoted as s1P OOOP sk relative to a fixed origin. Each end effector position si is a function of the joint angles. The column vector (s1P s-P OOOP sk) i can be written as s⃗; this can be viewed as a column vector either with m = 3k scalar entries or with k entries from R.. One way to control the multibody is to specify target positions, one for each end effector. The target positions are also defined by a vector f⃗i = (fi1P fi-P OOOP fik) i , where fii is the target position for the i -th end effector. Let ei = fii − si, be the desired change in position of the i -th end effector (moving to the desired i -th target). This equation can be rewritten as e⃗ = f⃗i− s⃗. The joint angles are also written as a column vector  = (θ1P OOOP θn) i . The end effector positions are functions of the joint angles; this fact can be expressed as s⃗ = f () (3.1) or, for i = 1P OOOP k, s⃗i = fi (). This is called the Forfivry Kinzmvtixs (FK) solution. The goal of Invzrsz Kinzmvtixs (IK) is to find a vector  such that s⃗ is equal to a given desired configuration s⃗y:  = f−1 (⃗sy) (3.2) where f is a highly non linear operator which is difficult to invert. However, there are instances where a solution to the Inverse Kinematics problem does not exist due to an unreachable target or where the (best) solution is not unique. Even in well- behaved situations, a closed-form equation cannot generally be achieved. Therefore, the use of iterative methods to approximate a good solution to the problem seems to be necessary. The most popular numerical approach is to use the Jacobian matrix to find a linear approximation to the IK problem. HCGCFC Jvxowivn inverse methoys The Jacobian J is a matrix of partial derivatives of the entire chain system relative to the end effectors s. The Jvxowivn solutions are a linear approximation of the IK problem (see figure 3.5); they linearly model the end effectors’ movements relative to instantaneous system changes in link translation and joint angle. The Jacobian matrix J is a function of the  values and is defined by J ()ij = ( Usi Uθj ) (3.3) LL V covzl Invzrsz ‘inzmvtixs holvzr t 1 Actual motion Linear approximation 2p 3 22 ! 33 ! 11 ! 1p 3p 1s 2 ss ! 1 Figflre KFMFR ihz Jvxowivn solution is v linzvr vpproflimvtion of thz vxtuvl motion of thz kinzmvtix xhvinC where i = 1P OOOP k and j = 1P OOOP n. Orin and Schrader in [OS84] discussed how to calculate the Jacobian matrix entries for different representations of joints and multibodies. The Jacobian matrix entries for the j-th rotational joint can be calculated as follows Usi Uθj = –j × (si − pj) (3.4) where pj is the position of the joint, and –j is the unit vector pointing along the current axis of rotation for the joint. Note that J can be viewed either as a k× n matrix whose entries are vectors in R., or as an m× n matrix with scalar entries (m = 3k). Equation 3.1 for forward dynamics can now be written as ˙⃗s = J () ˙ (3.5) where the dot notation specifies the first derivative with respect to time. Using the current values , s⃗ and f⃗i, the Jacobian J = J() can be computed. We then seek an update value ∆ for the purpose of incrementing the joint angles  by ∆:  :=  +∆ (3.6) The change in end effector positions caused by this change in joint angles can be estimated as ∆s⃗ ≈ J∆ (3.7) The idea is that the ∆ value should be chosen so that ∆s⃗ is approximately equal to e⃗, although it also common to choose ∆ so that the approximate movement ∆s⃗ in the end effectors (partially) matches the velocities of the target positions. Thus, the FK problem can be expressed as e⃗ = J∆ and the IK problem can be rewritten as ∆ = J−1e⃗. In most cases, the IK equation cannot be solved uniquely. Indeed, the Jacobian J may not be square or invertible, and even if it is invertible, J may work poorly as it may be KFJ jelyfied oork LM nearly singular1. Several approaches have been proposed to overcome these problems. Such methods are presented and discussed in the rest of this section. bycoziyn psefldoEin–erse The Jacobian Pseudo-inverse, also known as the boorzBeznrosz inverse of the Jacobian, sets the value ∆ equal to ∆ = J†e⃗ (3.8) where J† is an n×m matrix and is called the pszuyoBinvzrsz of J . It is defined for all matrices J , even ones which are not square or not of full row rank. The pseudo-inverse gives the best possible solution to the equation J∆ = e⃗ in the least squares sense. The pseudo-inverse has the property that the matrix (I − J†J) performs a projection onto the nullspace of J . Therefore, for all vectors <, J(I − J†J)< = 0. This means that we can set ∆ by ∆ = J†e⃗+ (I − J†J)< (3.9) for any vector < and still obtain a value for ∆ which minimises the value J∆ − e⃗. Several authors have used the nullspace method to help avoid singular configurations [Lie77, MK85]. A more sophisticated nullspace method, the Zfltznyzy Jvxowivn method, was introduced by Baillieul [Bai85]; in this version a local minimum value of a function is tracked as a secondary objective. The pseudo-inverse method can be derived as follows: JiJ∆ = Ji e⃗ (3.10) Then let z⃗ = Ji e⃗ and solve the equation JiJ∆ = z⃗ (3.11) It can be shown that z⃗ is always in the range of JiJ , hence the above equation always has a solution. When J is full row rank, JiJ or JJi is guaranteed to be invertible. In this case, the minimum magnitude solution ∆ can be expressed as ∆ = ( JiJ )−1 Ji e⃗ ≡ Ji (JJi )−1 e⃗ (3.12) The pseudo-inverse method is widely discussed in the literature, however it often performs poorly because of its instability near singularities. 1ginguluritiys owwur whyn no whungy in joint ungly wun uwhiyvy u xysiryx whungy in whuin ynx positionB LN V covzl Invzrsz ‘inzmvtixs holvzr bycoziyn firynspose The Jacobian transpose method was first used for inverse kinematics in [BDMS84, WE84]. The idea is to use the transpose of the Jacobian instead of its inverse. Hence, ∆ = αJi e⃗ (3.13) for some appropriate scalar α. Obviously the transpose of the Jacobian is not the same as the inverse; however, [BDMS84, WE84] justify the use of the transpose in terms of virtual forces. We can easily show that for all J and e⃗, 〈 JJi e⃗P e⃗ 〉 ≥ 0, where 〈yPz〉 indicates the dot product between vectors y and z, 〈 JJi e⃗P e⃗ 〉 = 〈 Ji e⃗P Ji e⃗ 〉 = ‖Ji e⃗‖- ≥ 0 (3.14) Therefore, if we update the angles ∆ in eq. 3.13 by a sufficiently small α ≥ 0, the end effector positions will be changed by αJJi e⃗. α can be calculated by minimising the new value of the error vector e⃗ after each update. Assuming that the end effector position change is equal to αJJi e⃗, α is chosen to make this value as close as possible to e⃗. Thus α is given by α = 〈 e⃗P JJi e⃗ 〉 〈JJi e⃗P JJi e⃗〉 (3.15) kingfllyr nylfle Decomposifiion The singular value decomposition (SVD) provides a powerful method for utilising the pseudo- inverse Jacobian. Let J be the Jacobian matrix. A singular value decomposition of J consists of expressing J in the form J = jYk i (3.16) where j and k are orthogonal matrices and Y is diagonal. For an m × n Jacobian matrix, j is m × m, Y is m × n, and k is n × n. The non-zero entries of the Y matrix are the values σi = yii along the diagonal. It is assumed that m ≤ n and, without loss of generality, σ1 ≥ σ- ≥ OOO ≥ σm ≥ 0. Note that there are cases where σi = 0, for some i. In fact, the rank of J is equal to the largest value r such that σr ̸= 0 and σi = 0 for i S r, . We use fli and –i to denote the i -th columns of j and k respectively. Their orthogonality implies that their columns form an orthonormal basis for Rm (respectively Rn). The vectors –r+1P OOOP–n are an orthonormal basis for the nullspace of J . The singular value decomposition of the Jacobian J always exists, and can be formed as J = r∑ i81 σifli– i i (3.17) The transpose, Yi , of Y is the n×m diagonal matrix. The product YYi is the m×m matrix with diagonal entries y-ii. The pseudo-inverse, Y † = ( y†ii ) , of Y is an n ×m diagonal matrix KFJ jelyfied oork LO with diagonal entries y†ii = { 1Ryii if yii ̸= 0 0 if yii = 0 (3.18) The pseudo-inverse of the Jacobian is thus equal to J† = k Y†ji and can be rewritten as J† = r∑ i81 σ−1i –ifl i i (3.19) Dymped deysfi kqflyres The Damped Least Squares method (DLS) was first used for inverse kinematics by [Wam86, NH86]. DLS avoids many of the pseudo-inverse method’s problems with singularities and can give a numerically stable method of selecting ∆. In the DLS method, instead of finding the minimum vector ∆ that gives a best solution to equation e⃗ = J∆, we find the value of ∆ that minimises the quantity ‖J∆ − e⃗‖- + λ-‖∆‖- (3.20) where λ ∈ R is a non-zero damping constant. This is given by ( JiJ + λ-I ) ∆ = Ji e⃗ (3.21) It is shown that JiJ + λ-I is non-singular, thus the DLS solution is equal to ∆ = ( JiJ + λ-I )−1 Ji e⃗ (3.22) Now JiJ is an n× n matrix, where n is the number of degrees of freedom. It is easily proven that ( JiJ + λ-I )−1 Ji = Ji ( JJi + λ-I )−1 ; the advantages of this transform over the one in eq. 3.22 is that the matrix being inverted is m ×m where m = 3k is the dimension of the space of the target positions, and m is often much less than n. Thus, ∆ = Ji ( JJi + λ-I )−1 e⃗ (3.23) The damping constant depends on the details of the multibody and the target positions and must be chosen carefully to make equation 3.23 numerically stable. The damping constant should be large enough so that the solutions for ∆ are well-behaved near singularities, but if it is too large, the convergence rate is slow. hsefldoEin–erse Dymped deysfi kqflyres The Pseudo-inverse Damped Least Squares uses the singular value decomposition (SVD) under the damped least squares method. Hence, the matrix JJi + λ-I can be rewritten as LP V covzl Invzrsz ‘inzmvtixs holvzr JJi + λ-I = ( jYk i ) ( k Yiji ) + λ-I = j ( YYi + λ-I ) ji (3.24) The matrix YYi + λ-I is a diagonal matrix with entries σ-i + λ -. It is clearly non-singular with inverse an m×m diagonal matrix with non-zero entries (σ-i + λ-)−1. Therefore, Ji ( JJi + λ-I )−1 = k Yi ( YYi + λ-I )−1 ji = k Eji (3.25) where E is an n×m diagonal matrix with entries ziPi = σi σ-i + λ - (3.26) Thus, the pseudo-inverse DLS solution can be expressed in the form Ji ( JJi + λ-I )−1 = r∑ i81 σi σ-i + λ - –ifl i i (3.27) Comparing the pseudo-inverse DLS with the simple pseudo-inverse method, we observe that in both cases the Jacobian is inverted by an expression ∑n i81 τi–ifl i i . In the case of the simple pseudo-inverse τi = σ −1 i , whereas for the pseudo-inverse DLS method, τi = σiR(σ - i + λ -). The simple pseudo-inverse method is unstable as σi approaches zero. Pseudo-inverse DLS acts similarly to the simple version away from singularities, but smooths out the performance of the simple pseudo-inverse method in areas close to singularities. kelecfii–ely Dymped deysfi kqflyres The Selectively Damped Least Squares (SDLS) method was presented by Buss and Kim in [BK05] and is an extension of the pseudo-inverse Damped Least Squares method. SDLS adjusts the damping factor separately for each singular vector of the Jacobian SVD based on the difficulty of reaching the target positions. The damping constants of SDLS depend not only on the current configuration of the articulated multibody, but also on the relative positions of the end effector and the target position. This method converges in fewer iterations and does not require ad hoc damping constants. SDLS also performs better than any other inverse Jacobian method when multiple end effectors exist. The DLS and pseudo-inverse DLS methods are computationally cheaper and easier to code than the SDLS method; however, SDLS offers improved performance for applications where runtime is not restricted and where it is difficult to choose a good damping constant. ancorporyfiing consfiryinfis There exist several ways to improve the performance and increase the realism of an animation; one of these is to incorporate constraints. However, implementing constraints in the Jacobian family of methods is not straightforward. A simple projection of the unconstrained solution KFJ jelyfied oork LQ onto a feasible posture has been proposed in [Wel93]. However, it is not guaranteed that the result will lie close to an optimal solution. A penalty-based method adding movement restrictions is presented in [F0ˆ3], with the drawback that this often converges to poor results. The simplest way of incorporating constraints can be achieved by weighting the moves of the individual joints [MM05]. Given an update vector p and a weight matrix l , where l = wi I and w is a vector of weights on the individual joints, the weighted update pw is given by pw =lp. Feedzyck an–erse cinemyfiics The Feedback Inverse Kinematics (FIK) method [Pec08] solves the inverse kinematics problem from a control prospective, minimising the difference between demanded and actual Cartesian velocities. Within the feedback loop, the required joint parameters are derived through a control sensitivity function. The algorithm operates as a filter and does not require matrix manipulations (inversion or singular value decomposition). Singularities are handled without the necessity of a damping factor and this makes it computationally more efficient than pseudo- inverse based methods. [Pec08] also describes how manipulator constraints can be applied, weighting both joints and end-effectors to a more feasible set of postures. As with the other Jacobian-based algorithms, it can easily handle problems with multiple end effectors. HCGCGC cewton methoys The Newton family of methods is based on a second order Taylor series expansion of the object function f(x): f(x+ σ) ≈ f(x) + [∇f(x)]i σ + 1 2 σiHf (x)σ (3.28) where Hf (x) is the Hessian matrix. However, the calculation of the Hessian matrix is very complex and results in high computational cost for each iteration. Hence, several approaches have been proposed which, instead of calculating the Hessian matrix, use an approximation of the Hessian matrix based on a function gradient value. The most well known methods are Broyden’s method, Powell’s method and the Broyden, Fletcher, Goldfarb and Shanno (BFGS) method [Fle87, CKM97]. Since the Newton methods are posed as a minimisation problem, they return smooth motion without erratic discontinuities. It is also straightforward to incorporate joint restrictions. The most obvious method for constraints is the gradient projection method proposed by Zhao in [ZB94]. The Newton methods also have the advantage that they do not suffer from singularity problems, such as that which occurs when finding the Jacobian Inverse; however they are complex, difficult to implement and have high computational cost per iteration. MH V covzl Invzrsz ‘inzmvtixs holvzr HCGCHC IK using hequentivl bonte Cvrlo bethoys Sequential Monte Carlo Methods (SMCM) have been recently introduced for solving IK prob- lems. Courty and Arnaud in [CA08] proposed such a solution based on the sampling principle. Using a sampling approach, the inverse kinematics problem can be solved with forward kine- matics, hence the numerical inversion of the forward operator can be avoided. The problem is cast as a hidden Markov model (HMM), whose hidden state is given by all the parameters that define the articulated figure. Hence, the state space consists of all the possible configura- tions of the state. The inverse kinematics is then reformulated in a filtering framework. The proposed SMCM IK solver does not require explicit numerical inversion and joint restrictions can be added to the system in an intuitive manner. These can be easily implemented without the need for complex optimisation algorithms. A particle IK solver has also been implemented in [HRE+08] which uses a body pose goals set and attempts to satisfy the goals by forming a system of constraints over the linked character bodies. HCGCIC htyle or meshBwvsey Inverse Kinemvtixs [GMHP04] presents a style-based IK method which is based on a learned model of human poses. Given a set of constraints, the proposed system can produce, in real-time, the most likely pose satisfying those constraints. The model has been trained on different input data that leads to different styles of IK; it can generate any pose, but poses are highly related to those which are most similar to the space of poses in the training data. In [SZGP05], a mesh- based Inverse kinematics (SKsh-IK) has been implemented which, instead of using human styles as training data, learns the space of meaningful shapes from example meshes. Using the learned space, SKsh-IK generates new shapes that respect the deformations exhibited by the examples, yet still satisfy vertex constraints imposed by the user. [DSP06] describes an extension of the SKsh-IK method which provides interactive control of reduced deformable models via an intuitive IK framework. The collection of transformations compactly represents articulated character movement that has been derived automatically from example data. The IK problem is formulated in a reduced space to achieve an independent resolution performance, meaning the speed of the posing task is a function of the model parameters rather than of character geometry. However, this family of methods requires an off-line training procedure and the results are highly dependent on the training data and limited only to those models and movements the system has been trained on. HCGCJC Heuristix Inverse Kinemvtixs vlgorithms Cyclic Coordinyfie Descenfi Cyclic Coordinate Descent (CCD), which was first introduced by [WC91] and then biomechan- ically constrained by [Wel93], is an iterative heuristic technique that is suitable for interactive control of an articulated body. CCD is one of the most popular IK iterative algorithms; it has been implemented in many computer graphic and robotics applications and is extensively KFJ jelyfied oork MI t 4 p 3 p 2 p 1 p (u= t 4 p 3 p 2 p 1 p (v= t 4 p 3 p 2 p 1 p (w= t4p 3 p 2 p 1 p (x= t 4 p 3 p 2 p 1 p (y= t 4 p 3 p 2 p 1 p (f= Figflre KFNFR Vn zflvmplz of visuvl solution of thz I‘ prowlzm using thz XXY vlgorithmC (v) ihz initivl position of thz mvnipulvtor vny thz tvrgztA (w) finy thz vnglz θ wztfizzn thz zny zffzxtorA joint p3 vny thz tvrgzt vny rotvtz thz joint p4 wy this vnglzA (x) finy thz vnglz θ wztfizzn thz zny zffzxtorA joint p2 vny thz tvrgzt vny rotvtz joints p4 vny p3 wy this vnglzA (y)A (z) vny (f) rzpzvt thz fiholz proxzss for vs mvny itzrvtions vs nzzyzyC htop fihzn thz zny zffzxtor rzvxhzs thz tvrgzt or gzts suffixizntly xloszC used for solving the inverse kinematic problem in the computer games industry (e.g. [Lan98]). CCD has also been effectively used in protein science for protein structure prediction and/or structure determination [CD03]. CCD provides a numerically stable solution and it has linear-time complexity in the number of degrees of freedom (DoF). The CCD method attempts to minimise position and orientation errors by transforming one joint variable at a time. The algorithm states that, starting from the end effector inward towards the manipulator base, each joint must be transformed in order to move the end effector as close as possible to the target. This procedure is repeated until a satisfactory solution is obtained. CCD is a heuristic iterative method with low computational cost for each joint per iteration, which can solve the IK problem without matrix manipulations; thus it formulates a solution very quickly. Figure 3.6 gives a visual solution of the IK problem using the CCD algorithm executing over a number of iterations. Like other inverse kinematics algorithms, CCD can generate many different resulting pos- tures for a given initial posture. It is then very difficult to choose a feasible posture among these many resulting postures. Therefore, manipulator constraints must be incorporated to restrict motions to a feasible set. In CCD it is easy to apply local constraints but it is more difficult to implement global manipulation restrictions. CCD is a very quick method but it is not free from problems; it suffers from unrealistic anima- MJ V covzl Invzrsz ‘inzmvtixs holvzr tion, even if manipulator constraints have been incorporated, and often produces motion with erratic discontinuities. CCD also tends to overemphasise the movements of the joints closer to the end effector of the kinematic chains, producing an unnatural movement, even if constraints have been incorporated. CCD is designed to handle serial chains; however multiple goals are necessary for most graphics and robotics applications. It is, however, non-straightforward to extend it to problems with multiple targets and end effectors. [MD04] describes such a tech- nique, which deals with tree articulated structures. The proposed multiple-chain CCD method can be applied successively over multiple articulated chains; it divides the articulated structure into smaller serial chains and treats each chain independently. [KM05] also adopted the CCD kinematic algorithm and solved its crucial problem of resulting unnatural poses. The proposed extension in [KM05] is able to solve problems with humanoid hierarchy, dividing the whole body into groups of joints near an end effector (typically head, trunk, arms and legs). In order to satisfy the desired centre of mass, the lightest group moves first, adjusting its centre of mass by changing the length of the limb and rotating it (assuming it as a rigid body). andflcfii–e an–erse cinemyfiics ylgorifihmR The Inductive Inverse Kinematics (IIK) al- gorithm [KLC+03] is an extension of the CCD algorithm; it uses a Uniform Posture Map (UPM) to control the posture of a human-like 3D character. The UPM is organised through the quantisation of various postures with an unsupervised learning algorithm, and the learning algorithm prevents the generating of invalid output neurons. The IIK algorithm can be formed by implementing a forward kinematic table containing the forward kinematics values of each output neuron. Thereafter, the forward kinematics table is searched to find the point with the smallest distance from the desired point, and to choose the posture vector associated with that point. If the current end point needs to be made closer to its target position, traditional CCD can be used in the final phase of the algorithm. It is guaranteed that the postures generated by the UPM are realistic postures which observe physical constraints. Hence it is possible to get a natural posture by finding a posture whose forward kinematics point is closest to the desired position. lriyngfllyfiion an–erse cinemyfiics Another method which does not use an iterative approach is presented in [MCM07]. The Triangulation algorithm uses the cosine rule to calculate each joint angle starting at the root of the kinematic chain moving outward towards the end effector. It is guaranteed to find a solution when used with unconstrained joints and when the target is in range. The Triangulation algorithm incurs a lower computational cost than the CCD algorithm, since it needs only 1 iteration to reach the target. However, the results are not realistic. The joints close to the end-effector are usually in a straight line, with the emphasis on rotation of the joints neighbouring the root. The Triangulation IK method can only be applied to problems with a single end effector; kinematic chains with multiple end effectors cannot be solved and it cannot therefore be used for complex character models. Another drawback of this algorithm is that, when constraints are applied, the end effector often cannot reach the target, even if there KFK FYZjacR Y few Heflrisfiic ac eefihodology MK is a solution. This happens because each joint position is calculated independently without considering the restrictions of the next joint. An improved version is given in [Muk09] where the n-link IK problem is reduced into a tfio-link problem, making sure that each link is rotated at most once in an attempt to reach the target position. keqflenfiiyl an–erse cinemyfiics Sequential Inverse Kinematics (SIK), which is presented in [UPBS08], is a direct extension of [BVU+06]. The SIK is an analytic-iterative IK method that reconstructs 3d human full- body movements in real-time. The inputs to this method are end effector positions, such as wrists, ankles, head and pelvis (the least possible input in order to be usable within a low-cost motion capture system in real-time), which are used to find the human pose. The IK problem is then solved sequentially using simple analytic-iterative IK algorithms (for instance CCD), in different parts of the body, in a specific order. The SIK, according to [UPBS08], outperforms many IK methods regarding the joint average position error, the joint average orientation error and the median processing time of each methodology. HCHC FVWgIKO V cew Heuristix IK bethoyology In this section, a new heuristic method for solving the IK problem, FABRIK [AL10d], is presented. It uses the previously calculated positions of the joints to find the updates in a forward and backward iterative mode. FABRIK involves minimising the system error by adjusting each joint angle one at a time. The proposed method starts from the last joint of the chain and works forwards, adjusting each joint along the way. Thereafter, it works backward in the same way, in order to complete a full iteration. This method, instead of using angle rotations, treats finding the joint locations as a problem of finding a point on a line; hence, time and computation can be saved. Assume p1P OOOPpn are the joint positions of a manipulator. Also, assume that p1 is the root joint and pn is the end effector, for the simple case where only a single end effector exists. The target is symbolised as fi and the initial base position by z. FABRIK is illustrated in pseudo-code in Algorithm 1 and a graphical representation of its full iteration with a single target and 4 joints is presented and explained in figure 3.7. First calculate the distances between each joint yi = |pi+1 − pi|, for i = 1P OOOP n − 1. Then, check whether the target is reachable or not; find the distance between the root and the target, yist, and if this distance is smaller than the total sum of all the inter-joint distances, yist Q ∑n−1 1 yi, the target is within reach, otherwise, it is unreachable. If the target is within reach, a full iteration is constituted by two stages. In the first stage, the algorithm estimates each joint position starting from the end-effector, pn, moving inwards to the manipulator base, p1. So, let the new position of the end-effector be the target position, p ′ n = fi. Find the line, ln−1, which passes through the joint positions pn−1 and p′n. The new position of the (n− 1)th joint, p′n−1, lies on that line with distance yn−1 from p′n. Similarly, the new position of the (n − 2)th joint, p′n−-, can be calculated using the line ln−-, which passes through pn−- and ML V covzl Invzrsz ‘inzmvtixs holvzr t d1 d2 d3 4 p 3 p 2 p 1 p (u= d1 d2 d3 4 p 3 p 2 p 1 p 4 p (v= l3 d1 d2 d3 3 p 2 p 1 p 4 p 3 p (w= l3 d1 d2 d3 l2 l1 4 p 3 p 2 p 1 p (x= d1 d2 d3 4 p 3 p 2 p 1 p 1 p (y= l3 d1 d2 d3 l2 l1 t 1 p 2 p 3 p 4 p (f= Figflre KFOFR Vn zflvmplz of v full itzrvtion of [VWgI‘ for thz xvsz of v singlz tvrgzt vny I mvnipulvtor jointsC (v) ihz initivl position of thz mvnipulvtor vny thz tvrgztA (w) movz thz zny zffzxtor p4 to thz tvrgztA (x) finy thz joint p′3 fihixh lizs on thz linz l3 thvt pvsszs through thz points p ′ 4 vny p3A vny hvs yistvnxz y3 from thz joint p ′ 4A (y) xontinuz thz vlgorithm for thz rzst of thz jointsA (z) thz szxony stvgz of thz vlgorithmO movz thz root joint p′1 to its initivl positionA (f) rzpzvt thz svmz proxzyurz wut this timz stvrt from thz wvsz vny movz outfivrys to thz zny zffzxtorC ihz vlgorithm is rzpzvtzy until thz position of thz zny zffzxtor rzvxhzs thz tvrgzt or gzts suffixizntly xloszC p′n−1, and has distance yn−- from p′n−1. The algorithm continues until all new joint positions are calculated, including the root, p′1. Having in mind that the new position of the manipulator base, p′1, should not be different from its initial position, a second stage of the algorithm is needed. A full iteration is completed when the same procedure is repeated but this time starting from the root joint and moving outwards to the end effector. Thus, let the new position for the 1st joint, p′′1, be its initial position z. Then, using the line l1 that passes through the points p ′′ 1 and p ′ -, we define the new position of the joint p′′- as the point on that line with distance y1 from p′′1. This procedure is repeated for all the remaining joints, including the end effector. In cases where the root joint has to be translated to a desired position, FABRIK works as described with the difference that in the backward phase of the algorithm, the new position of the root joint, p′′1, will be the desired and not the initial position. After one complete iteration, it is always the case (observed empirically) that the end effector is closer to the target. The procedure is then repeated, for as many iterations as needed, until the end effector is identical or close enough (to be defined) to the desired target. FABRIK always converges to any given chains/goal positions, when the target is within reach. If there are constraints which do not allow the chain to bend enough in order to reach the target or if the target is not within the reachable area, there is a termination condition which compares KFK FYZjacR Y few Heflrisfiic ac eefihodology MM the previous and the current position of the end effector, and if this distance is less than an indicated tolerance, FABRIK terminates its operation. Also, in the extreme case where the number of iterations has exceeded an indicated value and the target has not been reached, the algorithm is terminated (however, we have never encountered such a situation). Several optimisations can be achieved using Conformal Geometric Algebra (CGA) [HS84, DL03] to produce faster results and to converge to the final answer in fewer iterations; CGA has the advantage that basic entities, such as spheres, lines, planes and circles, are simply represented by algebraic objects. Therefore, a direct estimate of a missing joint, when it is between 2 true positions, can be achieved by intersecting 2 spheres with centres the true joint positions and radii the distances between the estimated and the true joints respectively; the new joint position will be taken as the point on the circle (created by the intersection of the 2 spheres) nearest to the previous joint position. Another simple optimisation is the direct construction of a line pointing towards the target, when the latter is unreachable. A similar solution of the algorithm using CGA is given in Appendix A.5 and [AL10a]. The proposed method has all the advantages of existing iterative heuristic algorithms. The computational cost for each joint per iteration is low, meaning the solution is arrived at very quickly. It is also very easy to implement, since it is simply a problem involving points, distances and lines and always returns a solution when the target is in range. It does not require complex calculations (e.g Jacobian or Hessian matrix) or matrix manipulations (inversion or singular value decomposition), it does not suffer from singularity problems and returns smooth motion without erratic discontinuities. A singularity problem might occur when the chain is completely straight and the target is located on that alignment but between two joints (on the bone). In such an instance, FABRIK does not converge to a solution but enters to an infinite loop (a similar problem is encountered in the CCD algorithm). The solution is to choose a very small angle in the beginning of the backward stage of the algorithm, in order to allow the chain to bend into a direction that satisfies the user constraints. HCHCFC FVWgIK with multiple eny effextors IK solvers are commonly used for solving the IK problem in many areas including computer graphics, gaming and protein science. In reality, most of the multibody models, such as hands, human or legged bodies etc, are comprised of several kinematic chains, and each chain generally has more than 1 end effector. Therefore, it is essential for an IK solver to be able to solve problems with multiple end effectors and targets. The proposed algorithm can be easily extended to process models with multiple end effectors. However, prior knowledge of the model, such as the sub-base- joints, and the number and structure of chains is needed. The algorithm is divided into two stages, as in the single end effector case. In the first stage, the normal algorithm is applied but this time starting from each end effector and moving inwards to the parent sub-base. This will produce as many different positions of the sub-base 2A suvAvusy joint is u joint whiwh wonnywts F or mory whuinsB A pryAunulysis of thy voxy wun xytyrminy yfiuwtly whyry thy suvAvusys ury lowutyxB MN V covzl Invzrsz ‘inzmvtixs holvzr Ylgorifihm IR A full iteration of the FABRIK algorithm. InputN hhy joint positions pi for i = E; :::; nB, thy turgyt position t unx thy xistunwys vytwyyn yuwh joint ri = |pi+1 − pi| for i = E; :::; n− EB eutputN hhy nyw joint positions pi for i = E; :::; nB 1.1 : ihe distvnce wetfieen root vnd tvrget 1.2 rist = |p1 − t| 1.3 : Check fihether the tvrget is fiithin revch 1.4 if rist L r1 + r2 + :::+ rn−1 then 1.5 : ihe tvrget is unrevchvwle 1.6 for i = E; :::; n− E do 1.7 : Find the distvnce ri wetfieen the tvrget t vnd the joint position pi 1.8 ri = |t− pi| 1.9 i = ri=ri 1.10 : Find the nefi joint positions piC 1.11 pi+1 = (E− i=pi + it 1.12 end 1.13 else 1.14 : ihe tvrget is revchvwle; thusA set vs b the initivl position of the joint p1 1.15 b = p1 1.16 : Check fihether the distvnce wetfieen the end effector pn vnd the tvrget t is grevter thvn v tolervnceC 1.17 ritA = |pn − t| 1.18 while ritA L tol do 1.19 : hiAGE FO FdglAgD gEACHIcG 1.20 : het the end effector pn vs tvrget t 1.21 pn = t 1.22 for i = n− E; :::; E do 1.23 : Find the distvnce ri wetfieen the nefi joint position pi+1 vnd the joint pi 1.24 ri = |pi+1 − pi| 1.25 i = ri=ri 1.26 : Find the nefi joint positions piC 1.27 pi = (E− i=pi+1 + ipi 1.28 end 1.29 : hiAGE GO WACKlAgD gEACHIcG 1.30 : het the root p1 its initivl positionC 1.31 p1 = b 1.32 for i = E; :::; n− E do 1.33 : Find the distvnce ri wetfieen the nefi joint position pi vnd the joint pi+1 1.34 ri = |pi+1 − pi| 1.35 i = ri=ri 1.36 : Find the nefi joint positions piC 1.37 pi+1 = (E− i=pi + ipi+1 1.38 end 1.39 ritA = |pn − t| 1.40 end 1.41 end as the number of end effectors connected with that specific sub-base. The new position of the sub-base will then be the centroid of all these positions. Thereafter, the normal algorithm should be applied inwards starting from the sub-base to the manipulator root. If there are more intermediate sub-bases, the same technique should be used. In the second stage, the normal algorithm is applied starting now from the root and moving outwards to the sub-base. Then, the algorithm should be applied separately for each chain until the end effector is reached; if more sub-bases exist, the same process is applied. The method is repeated until all end effectors reach the targets or there is no significant change between their previous and their KFL Ypplying boinfi Consfiryinfis fio FYZjac MO … root end effectors sub-bases … … … … … … … … … … … … …… … … … … … … … Figflre KFPFR Vn zflvmplz of v moyzl figurz fiith multiplz zny zffzxtors vny multiplz suwBwvszsC new positions. An example of a model figure having multiple end effectors and sub-bases is presented in figure 3.8. More sophisticated (and complex) models can be also tackled. Extending the proposed algorithm to take into account the figure’s shape, constraints and properties, will reduce the number of iterations needed to reach the target and will return more feasible postures. For example, FABRIK has been successfully applied to real-time hand tracking and reconstruction in motion capture, as it is presented in Chapter 5 and published in [AL10a, AL10b]. HCHCGC FVWgIK within xlosey loops FABRIK can also cope with cases where the “end effector” is not positioned at the end of the chain (i.e. it is a leaf) in the same way as for the sub-bases described in section 3.3.1. The whole model could be divided into groups of joints near the end effectors (such as head, trunk, arms and legs) and then sequentially adapt the body postures in a specific order, similarly to [UPBS08] and [KM05]. Obviously, the adaption hierarchy varies between models. An example where FABRIK has been successfully adjusted within closed loops of a humanoid, achieving real-time centre of rotation correction in motion capture, under marker occlusions is presented in Chapter 4 and been published in [AL10c]. HCIC Vpplying Joint Constrvints to FVWgIK Most legged body models are comprised of joints having biomechanical constraints, which provide natural restrictions on their motion. Such constraints are essential in physical simu- lations, IK techniques and tracking in motion capture systems to reduce visually unrealistic movements. A joint is defined by its position and orientation and, in the most general case, has 3 DoF. A bone rotation can be described by factoring it into two rotations: one “simple rotation”, named MP V covzl Invzrsz ‘inzmvtixs holvzr pi+1 pi t t' (u= tt' pi+1 pi (v= pi+1 pi tt' (w= Figflre KFQFR ihz tvrgzt is rzBpositionzy fiithin thz vllofizy rvngz of motion fihixh is yzfinzy wy thz xonix szxtionC ihzrz vrz H typzs of joint rzstrixtionA vs yzsxriwzy wy thz vnglzs θ1P OOOP θ4O (v) v xirxlzA (w) vn zllipsoiyvl shvpz vny (x) v pvrvwolix shvpzC here as rotvtionvl (2 DoF), that moves the bone to its final direction vector, and another which we call orizntvtionvl (1 DoF), which represents the twist around this final vector. Thus, the range of movement of a bone can be controlled by dividing the joint restriction procedure into two interconnected phases, a rotational and an orientational phase, contributing equally to the joint restrictions. The essential feature of a joint is that it permits a relative motion between the two limbs it connects. Most of the existing structure models, such as those described above, use techniques which restrict the bone to lie within the rotational and orientational limits of the joint. Blow [Blo02] proposes a loop hung in space, limiting the range of motion of the bone to “reach windows” described by star polygons. Wilhelms and Van Gelder [WG01] present a 3D “reach cone” methodology using planes, treating the joint limits in the same way as [Blo02]. [Kor85, BB01] parameterise realistic joint boundaries of the ball-and-socket joint by decomposing the arbitrary orientation into two components and controlling the rotational joint limits so they do not exceed their bounds. Once a proper parametrisation is defined for each joint of the articulated body, an animation engine is utilised. HCICFC gestrixting the motion to the vllowey wounys In this section, a reliable methodology for incorporating manipulator constraints is described using FABRIK. Since FABRIK is iterative, the joint restrictions can be enforced at each step just by taking the resultant orientation and forcing it to stay within the valid range. FABRIK’s ability to converge to an answer, if the target is within reach, is not affected by any imposed joint limits. The main idea behind this methodology is the re-positioning and re-orientation of the target to be within the allowed range bounds; ensuring that these restrictions are always satisfied means a more feasible posture can be achieved. This can be accomplished by checking if the target is within the valid bounds, at each step of FABRIK, and if it is not, to guarantee that it will be moved accordingly. In contrast to most existing techniques for joint constraints, the proposed methodology simplifies the 3D problem into a 2D problem, meaning that the complexity and the required processing time is reduced. In this chapter, a joint restriction KFL Ypplying boinfi Consfiryinfis fio FYZjac MQ q1 q3 q2q4 1 3 4 2S S q1 q3 q2 q4 O L1 L1L1 pi+1 pi (u= Ground plan – 2D t t t! q1 q2 q3 q4Constrained joint Mapping from 3D to 2D x y Range of motion O (v= Figflre KFIHFR V grvphixvl rzprzszntvtion of thz implzmzntzy xonstrvints vny thz irrzgulvr xonz yzsxriwing thz rotvtionvl motion wounysC (v) ihz wvllBvnyBsoxkzt jointA piA fiith its vssoxivtzy irrzgulvr xonz fihixh yzfinzs thz vllofizy rvngz of motionC (w) hhofis thz xompositz zllipsoiyvl shvpz xrzvtzy wy thz yistvnxzs qj mvppzy from HY to GYC Ylgorifihm JR The orientational constraints. InputN hhy rotor h yfipryssing thy rotution vytwyyn thy oriyntution frumys ut joints pi unx pi−1B eutputN hhy nyw ryAoriyntyx joint p′i−1B 2.1 Chywk whythyr thy rotor h is within thy motion rungy vounxs 2.2 if fiithin the wounds then 2.3 xo nothing unx yfiit 2.4 else 2.5 ryoriynt thy joint pi−1 in suwh u wuy thut thy rotor will vy within thy limits 2.6 end methodology is presented for the general case of a ball and socket joint; this example should be considered an illustration of how joint or model constraints can be incorporated within FABRIK. Assume we have a ball-and-socket joint with orientational limits described by the rotor j and rotational limits described by the angles θ1P OOOP θ4. A graphical representation of a joint limit boundary could be an irregular cone which is defined by these angles. The rotational limits are enforced by re-positioning the target point as the nearest point on a conic section from the target position; this procedure is described in detail later. There are 3 possible conic sections, according to the angles defining the irregular cone: if all θs are equal, the conic section is a circle; if all θs are greater or smaller than 90o and are not equal, the conic section has an ellipsoidal shape; finally, if there are θs both greater and smaller than 90o, then the joint boundary limits are defined by a parabolic shape, as illustrated in figure 3.9. In the subsequent analysis shown here, the joint limits are assumed to be defined by an ellipsoidal shape, since this is the most common case, but similar procedures apply for different conic sections. Figure 3.10 gives a graphical representation of the implemented constraints and the irregular cone describing the rotational motion bounds for the case of an ellipsoidal shape. The orientation of the joint can be assigned as follows: Assume we are in the first stage of the algorithm, i.e. we have just calculated the new position of joint p′i, and we want to find the new position of the (i − 1)th joint, p′i−1. Find the rotor expressing the rotation between NH V covzl Invzrsz ‘inzmvtixs holvzr Ylgorifihm KR The rotational constraints. InputN hhy turgyt position t unx thy unglys xy ning thy rotution wonstruints j for x = E; :::; 4B eutputN hhy nyw turgyt position t′B 3.1 Finx thy liny yquution Z1 3.2 Finx thy projywtion O of thy turgyt t on liny Z1 3.3 Finx thy xistunwy vytwyyn thy point O unx thy joint position 3.4 aup thy turgyt (rotuty unx trunsluty= in suwh u wuy thut O is now lowutyx ut thy ufiis origin unx oriyntyx uwworxing to thy x unx yAufiis ⇒ bow it is u FD simpli yx provlym 3.5 Finx in whiwh quuxrunt thy turgyt vylongs 3.6 Finx whut woniw sywtion xyswrivys thy ullowyx rungy of motion 3.7 Finx thy woniw sywtion whiwh is ussowiutyx with thut quuxrunt using thy xistunwys qj = S tun j , whyry x = E; ::; 4 3.8 Chywk whythyr thy turgyt is within thy woniw sywtion or not 3.9 if fiithin the conic section then 3.10 usy thy truy turgyt position t 3.11 else 3.12 Finx thy nyuryst point on thut woniw sywtion from thy turgyt 3.13 aup (rotuty unx trunsluty= thut point on thy woniw sywtion viu ryvyrsy of I:4 unx usy thut point us thy nyw turgyt position 3.14 end the orientation frames at joints p′i and pi−1 and if this rotor represents a rotation greater than a limit, reorient the joint pi−1 in such a way that the rotation will be within the limits. Repeat the procedure for all the joints on both stages of the algorithm. The methodology is also described in pseudo-code in Algorithm 2. Once the joint orientation is established, the rotational (2 DoF) limits, described by angles θ1P OOOP θ4, can be applied as follows. Firstly, we find the projection d of the target fi on line a1, where a1 is the line passing through the joint under consideration, pi, and the previous joint of the chain, pi+1. Then determine the distance S from the point d to the joint position pi and calculate the distances qj = S tan(θj), for j = 1P OOOP 4, as shown in figure 3.10. We then apply a rotation and translation which takes d to the origin and the axes defining the constraints to the x and y axes, as in figure 3.10(b). Working in this 2D plane, we locate the target in a particular quadrant and find the ellipse defined on that quadrant using the associated distances qj ; for example, in figure 3.10(b) we are working with the ellipse which is defined by the angles θ- and θ. (or the distances q- and q.). Finally, find the nearest point on that ellipse from the target, if the latter is not in the allowed motion range. The nearest point on an ellipse from a point can be found by simultaneously solving the ellipse equation and the equation of the tangent line at the orthogonal contacting point on the ellipse using the Newton-Raphson method, as described in [ARW01]. Obviously, it is not necessary to calculate all the ellipses which define the composite ellipsoidal shape of figure 3.10(b), but only the ellipse related to the quadrant in which the target is located. It is important here to recall that, if the constraints which define the allowed range of motion are described by a different conic section (circle or parabola), the target should be re-positioned as the nearest point on that conic section, similarly to the ellipsoidal case. The last step is to undo the initial transformation which mapped d to the origin. This procedure is illustrated in pseudo-code in Algorithm 3 and a demonstration of the process is given in figure 3.12. This is a versatile, and easily visualisable, method of restricting where the bone can go. KFL Ypplying boinfi Consfiryinfis fio FYZjac NI tt' t t' pi+1 pi pi pi+1 (u= t t' t pi+1 pi (v= Figflre KFIIFR holution for spzxivl joint rzstrixtion xvszsO (v) thz originvl xvsz vny fihzn thz vllofizy rvngz of motion is grzvtzr thvn FME yzgrzzsA (w) fihzn thz tvrgzt is loxvtzy on v yiffzrznt hzmisphzrz thvn thz irrzgulvr xonzC Incorporating this methodology within an IK solver, such as FABRIK, will give us the oppor- tunity to reconstruct or track animated figures with high accuracy. IK algorithms are generally more effective if the constraints are applied at each step (not at the end of the algorithm), ensuring that the rotational and orientational restrictions are satisfied at each iteration. Thus, the proposed joint constraints can be applied within FABRIK by ensuring that the target, at each step, is moved to be within the allowed orientational and rotational bounds. Hence, as- sume that we are in the first stage of the algorithm, and have just calculated the new positions of the joints, p′i+1 and p ′ i, and we want to find the new position of the (i − 1)th joint, p′i−1. Check if the joint pi−1 satisfies the orientational limits and if so, check whether it is within the composite ellipsoidal shape that describes the allowed range bounds, as illustrated above. If it is not, then pi−1 should be re-oriented and/or re-positioned within the allowed bounds (vpi−1). Thereafter, p′i−1 can be defined as the point on the line li−1, which passes through the joint positions vpi−1 and p′i and has yi−1 distance from p ′ i, as is illustrated in figure 3.12. The same technique for constraining joints is applied in the second stage of the algorithm and for each iteration until the target is reached or there is no significant change in the end effectors’ positions. The algorithm copes with joints and limbs having 3 DoF, and it can handle cases of joint and limb twist. It is important to recall here that the inter-joint lengths are not changing over time since these distances are implicity kept constant by FABRIK. The proposed restriction methodology can be easily extended to manage joints limits greater than 180 degrees. For instance, when the angle which defines the allowed range of motion is greater than 180 degrees, the associated irregular cone will define the area which is outside NJ V covzl Invzrsz ‘inzmvtixs holvzr d1 d2 d3 2p 1 p 3 p 4 p t (u= 2p 1 p 3 p d1 d2 4p (v= 2p 1 p 4p d1 d3 3 p (w= 2p 1 p 4p d1 d3 3 p (x= 2p 1 p 4p d1 d3 3 p (y= 2p 1 p 4p d1 d3 3 p (f= 1 p 4p d1 d3 2 p 3 p d2 (g= 1 p 4p d1 d3 3 p 2 p d2 (h= Figflre KFIJFR Inxorporvting rotvtionvl vny orizntvtionvl xonstrvints fiithin [VWgI‘C (v) ihz initivl xonfigurvtion of thz mvnipulvtor vny thz tvrgztA (w) rzloxvtz vny rzoriznt joint p4 to tvrgzt fiA (x) movz joint p3 to p ′ 3A fihixh lizs on thz linz thvt pvsszs through thz points p ′ 4 vny p3 vny hvs yistvnxz y3 from p′4A (y) rzoriznt joint p ′ 3 in suxh v fivy thvt thz rotor zflprzssing thz rotvtion wztfizzn thz orizntvtion frvmzs vt joints p′3 vny p ′ 4 is fiithin thz motion rvngz wounysA (z) thz rotvtionvl xonstrvintsO thz vllofizy rzgions shofin vs v shvyzy xompositz zllipsoiyvl shvpzA (f) thz joint position p2 is rzloxvtzy to v nzfi positionA vp2A fihixh is thz nzvrzst point on thvt xompositz zllipsoiyvl shvpz from p2A znsuring thvt thz nzfi joint position p′2 fiill wz fiithin thz vllofizy rotvtionvl rvngzC (g) movz vp2 to p ′ 2A to xonszrvz wonz lzngthA (h) rzoriznt thz joint p′2 in oryzr to svtisfy thz orizntvtion limitsC ihis proxzyurz is rzpzvtzy for vll thz rzmvining joints in v forfivry vny wvxkfivry fvshionC the limits. In that case, the joint restriction methodology will work in a reverse fashion; if the target is within the irregular cone area, meaning it is outside the limits, it will be projected to the cone surface, as is demonstrated in figure 3.11(a). Another special case of joint restriction occurs when the target is located in such a position that is not in the same hemisphere (in figure 3.11(b), the upper hemisphere) as the irregular cone. The limits of motion are defined as the irregular cone in the upper hemisphere and a reflection of the cone in the lower hemisphere; the target in the lower hemisphere is projected onto the limit boundary by first projecting its position onto the reflected cone and taking the associated point on the regular cone, as shown in figure 3.11(b). Obviously, the algorithm works in a similar way for different conic sections. One big advantage of the proposed methodology is that no bone requires rotation to lie in any cone or polygon window, such as those described in [Blo02, WG01]; it is only necessary to check whether the target is within the composite ellipsoidal shape defined by the restrictions on the motion. It loses none of the advantages of the FABRIK algorithm, incorporating joint limits via only points, lines and basic 2D entities; no rotational matrices need to be calculated, resulting in large savings in computational time. It also produces visually smooth and natural movements without oscillations and discontinuities, and requires low processing time per iteration. If it is desirable to retrieve the joint angles, all necessary information is of KFL Ypplying boinfi Consfiryinfis fio FYZjac NK d1 d2 2p 1 p 3 p 1 2 t (u= d2 1 p 3 p 1 2 2 p 2pˆ d1 2p l2 3 p (v= d1 d2 1 p 3 p 1 2 3 p 1p 2 p 2p (w= d1 d2 1 p 3 p 1 2 2 p 3 p 2p (x= Figflre KFIKFR Inxorporvting xonstrvints for v hingz jointC (v) ihz initivl xonfigurvtion of thz mvnipB ulvtor vny thz tvrgztC hinxz p2 is v hingz joint fiith G Yo[A vll joints liz in thz plvnz Φ1C ihz root vny thz tvrgztA fihixh is orizntzyA vlso yzfinz thz plvnz Φ2C (w) gzloxvtz vny rzoriznt joint p ′ 3 to tvrgzt fiC ihznA projzxt p2 onto thz plvnz Φ2 to givz v nzfi point vp2A vny finy thz point p ′ 2 on linz l2 thvt pvsszs from thz joint position p′3 vny thz projzxtzy joint position vp2 vny hvs yistvnxz y2 from p ′ 3C gzoriznt thz nzfi joint using thz orizntvtion xonstrvintsC (x) movz vny rzoriznt joint p1 to p ′ 1A fihixh lizs on thz linz thvt pvsszs through thz points p′2 vny p1 vny hvs yistvnxz y3 from p ′ 4A (y) ihz prowlzm is nofi vgvin v GY prowlzm vny vll joints liz on plvnz Φ2C ihusA thz prototypz vzrsion of [VWgI‘ xvn wz vpplizy to vll thz rzmvining joints in v forfivry vny wvxkfivry fvshionC course avialable (position and orientation of each joint). If more information about the allowed range of motion is available, the proposed methodology can be extended to include increased sophistication, supporting more complex joint types. Thus, instead of having an ellipsoidal entity to describe the sub-area in which the target can be placed, a polygonal area can be implemented. If the target is out of range, we would look for the nearest point on the polygon. The constraining methodology can also be easily modified to support other IK solvers. There are, however, some limitations on what joint types this prototype version can support, since it is assumed that the inter-joint distance remains constant over time. Prismatic, sliding or shifting joints (joint types more usually discussed in robotics) are not supported. Self-collisions can be handled using existing techniques, such as [LG98]; but more work is needed to ascertain if the FABRIK framework gives any advantages when dealing with self occlusions. Having in mind that the target should be reoriented and repositioned in such a way as it satisfy the user or the model constraints, different joint models can be formulated using NL V covzl Invzrsz ‘inzmvtixs holvzr (u= (v= (w= (x= Figflre KFILFR ihz struxturz of thz moyzls uszy in our zflpzrimzntvl zflvmplzsC (v) V kinzmvtix xhvin xonsisting of FE joints vny F zny zffzxtorC ihzrz vrz G kinzmvtix xhvin moyzlsA vn unxonstrvinzy vny v xonstrvinzy vzrsionA (w) v kinzmvtix moyzl fiith FE unxonstrvinzy joints vny G zny zffzxtorsA (x) v hvny moyzl fiith GK unxonstrvinzy joints vny J zny zffzxtorsA (y) v FH joint humvnoiy moyzlA in v xonstrvinzy vny unxonstrvinzy vzrsionA fiith I zny zffzxtorsC ihz tvrgzt joint positions (zny zffzxtors) vrz shofin in rzy vny thz joint positions thvt thz I‘ solvzrs hvvz to zstimvtz vrz shofin in grzznC similar techniques. For instance, the joint limitations for simple 2D joint models, such as the hingz joint, can be simplified using alternative approaches. Since FABRIK operates on the joint coordinates by adjusting the positions in an iterative fashion, the 2D restrictions can be enforced by projecting the joint onto the plane of orientation. That plane is defined by the root and the (oriented) target position. An illustration showing how restrictions can be enforced for a hinge manipulation is given in figure 3.13. Similar techniques can be applied to incorporate constraints for different types of joint, in a variety of motions. HCJC ihe Eflperimentvl Environment A target database has been created for the validation and testing of the IK methods. The database consists of reachable and unreachable targets, targets with different distances from the end effectors and targets that move smoothly in space with end effectors tracking their position. The tests also consist of reconstructing sequences with different classes of motion in order to process different swivel angles and axial orientations of the root joint. The examples are demonstrated in 6 different kinematic models; a chain with 10 unconstrained joints allowing 3 DoF on each joint; a chain with 10 constrained joints allowing limited angle rotations with 3 DoF; a model containing a ‘Y-shape’ having 10 unconstrained joints and 2 end effectors; a fully unconstrained and un-modelled hand with 26 joints, 3 DoF on each joint and 5 end effectors; and a 13 joint humanoid model, in a constrained and unconstrained version, with 3 DoF on each joint and 4 end effectors. Figure 3.14 shows the different kinematic models used within this work. IK techniques will mostly work with specified positions and orientations of specific joints, usually the end effectors, since they are more easily specified by the animator and tracked by the motion capture system; thereby, they automatically configure the remaining joints according to different criteria that depend on the model variant and joint type restrictions. KFN jesfllfis NM (u= (v= Figflre KFIMFR ihz stvgzs illustrvtzy in oryzr thvt thz zny zffzxtor rzvxhzs thz tvrgztC (v) ihz [VWgI‘ solution vny (w) thz XXY solutionC HCKC gesults Some of the most popular IK methods have been tested against FABRIK, such as CCD, Jacobian Transpose, Jacobian DLS and Jacobian pseudo-inverse DLS (SVD-DLS). In some of our experiments, we implemented examples with large distances between target and end- effectors; hence, some methods tend to require more iterations to reach the target and thus the convergence differences are more obvious. The Jacobian and DLS parameter values used in our experiments are the parameter values suggested by [BK05]; the damping constant was set to λ = 1O1. Several tests and comparisons have been implemented between the proposed algorithms in respect of their computational cost, processing time, convergence, the number of iterations needed to reach the target and the reconstruction quality. All experiments were run using MATLAB [MAT] on a computer with a Pentium 2 Duo 2.2 GHz processor. The operating system used is Microsoft Windows Vista service pack 1.0. The results have been animated in video sequences using Blender [Ble]. HCKCFC V single eny effextor In this section, the methods have been tested on problems with a single end effector and fixed target positions. These experiments did not include any joint constraints, but all methods could be enhanced to enforce rotational and orientational limits. Examples with the resulting postures for each methodology are presented in figure 3.19. FABRIK produces results significantly faster than all IK methods tested. It is approximately 10 times faster than the CCD method and a thousand times faster than the Jacobian-based methods, for these examples with large end effector movements; FABRIK has the lowest com- putational cost and, at the same time, produces visually the smoothest and most natural movements. It needs the fewest iterations to reach the target, it converges faster to the desired position and, when the target is unreachable, it keeps the end effector pointing to the target. Figure 3.15 shows an example of an IK solution using FABRIK and CCD; the figure presents NN V covzl Invzrsz ‘inzmvtixs holvzr Figflre KFINFR jnnvturvl joint vnglzs zflhiwitzy wy XXYP thz kinzmvtix xhvin rolls itszlf wzforz rzvxhing thz tvrgztA proyuxing unrzvlistix poszsC all the stages before the kinematic chain reaches the target for both cases. It is clear that FABRIK needs fewer iterations and has a more natural movement to the target. On average, FABRIK needs 15.4 iterations and just 13O2ms to attain a reachable target and 67 iterations and 62ms for an unreachable target. The time and iterations needed to converge to a final answer, when the target is unreachable, can be reduced dramatically when algorithm optimi- sations are applied (see Alg.1); using optimisations, FABRIK needs just 1 iteration and 0O2ms to return the final chain pose. Obviously, as the target gets closer to the end effector, fewer iterations will be needed to reach the target. CCD can also be applied in real-time. It is much faster than any Jacobian-based method; it needs, on average, 26 iterations and 123ms to reach the target when it is within reach. On the other hand, when the target is not reachable, it needs almost 400 iterations and 4sec to converge to its final solution (using the default algorithm without optimisations). However, CCD can often generate unrealistic postures since it can roll and unroll itself before reaching the target (figure 3.15 and 3.16). CCD also tends to overemphasise the movements of the joints closer to the end effector of the kinematic chain. Another drawback of CCD is that it is designed to handle problems with serial chains; it has to be modified in order to solve problems with multiple end effectors [KM05]. The Jacobian methods return reasonable results; the reconstructed chain poses are visually more natural than CCD. Nevertheless, the biggest advantage of the Jacobian methods over all other methods is that, by default, they can treat problems with multiple end effectors very easily. Constraints can be applied within the Jacobian algorithms, but the way in which these restrictions are incorporated is not straightforward. Some Jacobian methods also suffer from singularity problems, since matrix inverses need to be calculated. The Transpose and DLS methods do not suffer in this way since they do not use the matrix inverse. The Jacobian methods also incur high computational cost making this family of methods non-ideal for real- time applications. For the examples considered here, the Jacobian Transpose method needs on average more than 1300 iterations and 13sec to reach the target when it is within reach, the DLS needs more than 990 iterations and 10sec and SVD-DLS more than 800 and 9sec. The Jacobian methods generally converge very slowly to their final solutions since they use a linear approximation with a small step. This is more obvious in figure 3.17, where the number of iterations needed to reduce the distance between target and end effector as this changes KFN jesfllfis NO lyzle KFIFR Vvzrvgz rzsults (ovzr GE runs) for v singlz kinzmvtix xhvin fiith FE jointsC Meaccabge Oambeo Pimeaccabge Oambeo Iphbem ja Maogab ese) Odhe kem doemaodji Doemaodjin Iphbem ja Maogab ese) Doemaodjin odhe (syw) (di mszx) kem necjid Doemaodjin odhe (syw) AA=MDF 10)411 +)+1.-3 +)3 1114 12)014 +)+1-+2 CCD -1).+3 +)1-.01 3)3 -1. .4+)1.0 .)4-314 Eacjbdai Omainkjne 1.11)14+ 1-)43442 4)4 1+1 1044)+++ ..)4+42. Eacjbdai DLN 443)143 1+)43+01 1+)0 40 -331)112 14)32413 Eacjbdai NVD(DLN 3+3)242 4)-410- 11)0 32 -3+3)40- 10)42041 over time is presented for each methodology. In this example, the original chain is 9000mm long, the distance between target and end effector is 6000mm, and the termination tolerance is 1× 10−.mm. The Triangulation algorithm also incurs lower computational cost than the CCD algorithm and it is substantially faster than the Jacobian methods. However, Triangulation returns the poorest results from the methods used within this report. The kinematic chain does not have a realistic shape; the joints close to the end-effector are usually in a straight line, with the emphasis on rotation of the joints neighbouring the root. Another important drawback of the Triangulation algorithm is that it cannot be adapted for multiple end effectors, it is thus useless for complex character models. The Triangulation algorithm also suffers from an inability to reach a feasible solution when constraints are applied; the end effector often cannot reach the target, even if there is a solution, since each joint position is calculated independently without considering the restrictions of the next joint. Table 3.1 presents the average runtimes of each of the methods, as well as the number of iterations needed to reach the target, for both cases of a reachable and an unreachable target. It also indicates on average the time needed per iteration for each method and how many iterations per second each methodology can support for the case of a single chain with 10 joints and 1 end effector. Runtimes are in seconds and were measured with custom MATLAB code on a Pentium 2 Duo 2.2 GHz. No optimisations were used for any method reported in the table. It also indicates the time needed per iteration for each method and how many iterations per second each methodology can support. An iteration of FABRIK has the lowest computational cost since, instead of using angle rotations, it treats finding the joint locations as a problem of finding a point on a line. Thus, it can process up to 1164 iterations in one second, requiring 0.85ms per iteration. The time required for a full iteration using CCD is 8.8ms, where the Jacobian Transpose, DLS and SCD-DLS methods need 9.9ms, 10.5ms and 11.5ms per iteration respectively. Figure 3.19 compares the performance of each algorithm for solving inverse kinematic prob- lems; it shows the initial configuration and the goal solution obtained with each methodology. The manipulator is fully unconstrained and has no limits on the range of allowed movement for each joint. In each case a position goal is specified for the end effector and the inverse kinematic problem is solved to varying degrees of accuracy. Figure 3.18 plots the convergence of each method, meaning the time taken to achieve the solution with the requested degree of accuracy. It is clearly observed that FABRIK converges to the target faster than any other NP V covzl Invzrsz ‘inzmvtixs holvzr 0 5 10 15 20 25 30 0 1 2 3 4 5 6 x 10 4 Number of iterations D is ta nc e be tw ee n ta rg et a nd e nd e ffe ct or (m m) FABRIK CCD 0 200 400 600 800 1000 1200 0 1 2 3 4 5 6 x 10 4 Number of Iterations D is ta nc e be tw ee n ta rg et a nd e nd e ffe ct or (m m) J.Transpose J.DLS J.SVD−DLS Figflre KFIOFR ihz itzrvtions nzzyzy to rzvxh thz tvrgzt vgvinst thz yistvnxz wztfizzn tvrgzt vny zny zffzxtorC 5 1 0.5 0.1 0.05 0.01 0.005 0.001 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 Ti m e (se c) Tolerance (x10−3 mm) FABRIK CCD 5 1 0.5 0.1 0.05 0.01 0.005 0.001 0 10 20 30 40 50 60 Tolerance (x10−3 mm) Ti m e (se c) J. Transpose J. DLS J. SVD−DLS Figflre KFIPFR Vn zflvmplz przsznting thz timz nzzyzy for zvxh mzthoyology to vxhizvz thz solution fiith thz yzgrzz of vxxurvxy rzquzstzyC implemented methodology. An iteration of FABRIK has the lowest computational cost and figure 3.18 verifies that FABRIK always converges to the target, if the latter is reachable. The FABRIK, CCD, DLS and SVD-DLS methods have also been tested when the target is moving in a sinusoidal trajectory and the end-effector is tracking its position when it is within reach, and keeping the end effector pointing at the target when it is unreachable. The accuracy of the tracking was measured over a period of a thousand simulation steps. FABRIK tracks the target in real-time producing smooth and visually natural motion without erratic discontinuities. CCD produces reasonable results within the real-time constraints; however there are instances where the motion produced is not visually realistic. It is important to mention that CCD’s performance improves when the target is a small distance from the end effector’s position or the frame rate is high. This happens because the kinematic chain does not roll and unroll itself. On the other hand, the Jacobian-based methods can still produce oscillating motion with discontinuities. Their biggest drawback however is the time needed to track the target; only under some circumstances, eg using fast C++ matrix libraries, can the Jacobian-based methods reach the target of real-time application. Although Triangulation is a real-time methodology, it produces the most unrealistic poses for kinematic animations. Figure 3.20 presents the performance of each method on selected frames over time. KFN jesfllfis NQ (u= (v= (w= (x= (y= (f= (g= Figflre KFIQFR Eflpzrimzntvl solutions using somz of thz most populvr I‘ mzthoysC ihz kinzmvtix xhvins xonsistzy of FE unxonstrvinzy jointsA vllofiing H yzgrzzs of frzzyom on zvxh jointC (v) Initivl positionA (w) [VWgI‘A (x) XXYA (y) JC irvnsposzA (z) JC YahA (f) JC hkYBYahA (g) irivngulvtionC eyking cine more flexizle In this section we implemented the FABRIK algorithm within the Kine [Lan98] application; Kine is a 2D real-time gaming application that initially has a kinematic chain with six joints. Kine allows you to interact with the IK solver; you click on the screen and the snake (the kinematic chain is drawn as a snake) solves the IK problem. There is also an option where you click and drag on the screen and the snake attempts track your mouse. You are also able to add more links, optimise orientation, and modify the application to a 3D environment. Figure 3.21 shows examples of the FABRIK and CCD methods implemented within the Kine environment when the end effector moves through large distances. It is clearly observed that FABRIK out-performs CCD in producing smoother poses. The environment presented in this section has been adopted from the work of Jeff Lander; we would like to express our enormous thanks to Jeff for giving us permission to use his code and application. The Kine application OH V covzl Invzrsz ‘inzmvtixs holvzr (u= (v= (w= (x= (y= Figflre KFJHFR Vn zflvmplz of thz tvrgzt trvxking using yiffzrznt mzthoysC ihz frvmzs przszntzy hzrz vrz thz svmz for zvxh mzthoyologyC (a) [VWgI‘A (b) XXYA (c) YahA (d) hkYBYahA (e) irivngulvtion is included in the supplementary materials, offering the opportunity to interact and evaluate the results in real-time. KFN jesfllfis OI (u= (v= Figflre KFJIFR [VWgI‘ vny XXY solution using thz ‘inz vpplixvtionC (v) [VWgI‘A (w) XXYC HCKCGC bultiple eny effextors Most real models, such as the hand, legged bodies etc, consist of multiple chains, each chain having at least one end effector. Hence, it is essential to test our methodology in cases where more than one end effector exists. To test FABRIK under these conditions, we implemented the ‘Y-shaped’ multibody pictured in figure 3.22, also used in [BK05], and a hand multibody presented in figure 3.23. The ‘Y-shape’ multibody has 10 joints with 2 end effectors. The target positions (the red balls in the figures) moved in sinusoidally varying curves in and out of reach of the multibody. The target positions moved in small increments and in each time step the joint positions were updated. The simulations were visually inspected for oscillations and tracking abilities. The end-effectors can successfully track the target positions when they are within reach, and remain pointing at the targets when these are out of reach. Figure 3.22 presents a simple example of how FABRIK performs with multiple end effectors; although it is hard to show in images, FABRIK can easily track both targets with a smooth motion and without oscillations, shaking or erratic discontinuities. Figure 3.23 shows another example of implementing FABRIK into a multiple end effector model. This is a fully unconstrained hand example with 5 end effectors and 26 joints in total, allowing 3 DoF on each joint. Incorporating a highly constrained model, such as [KL07], and restricting the motion of each joint to a feasible set, the hand will have even more natural movement. Such a model has been implemented and presented in Chapter 5. Figure 3.24 shows an example of a tracking animation of a humanoid with 13 joints, 5 of which are treated as end effectors. In this demonstration, the frame rate was low (3 frames per second); the 3Hz frame rate selection increases the distance between target and end effector, thus the performance of each method is more obvious. FABRIK can easily track the animated OJ V covzl Invzrsz ‘inzmvtixs holvzr Figflre KFJJFR Eflvmplz of [VWgI‘ implzmzntvtion fiith multiplz zny zffzxtors moving ovzr timzP v kinzmvtix xhvin fiith FE unxonstrvinzy jointsA G zny zffzxtors vny G tvrgztsC lyzle KFJFR gzxonstruxtion xompvrisonC Vvzrvgz rzsults (ovzr GE runs)C FAVfIK CCD JBhrunsposy JBDLg JBgjDADLg bumvyr of Ityrutions JI JK EGIF LD4 KFG ayxiun timy† (mssq= EBJ FDBI EMFL EIGG E4M4 himy pyr ityrution† (mssq= DBDF4J DBGDJD EB4GG4 EBMDJK FBDJJ4 ayxiun Error (mm= ILBJL JMBMM EGKB4F L4BL4 LGBKG † Ocdn dn a MAOLA= esecpoabge odhe) humanoid in real-time, producing very reasonable results. Figure 3.25 shows the reconstruction quality of different methodologies over the same humanoid model. The differences between the implemented methodologies, on these unconstrained humanoid examples, are more obvious on shoulders, elbows and hips. FABRIK produces visually natural postures, having the smaller reconstruction error compared to the original sequences. These animations have been obtained from an optical markered motion capture system and have not been filtered; thus, the algorithm is shown to be robust in noisy environments. Selected internal joints have been artificially deleted in order to examine the reconstruction quality of each methodology. These humanoids do not have a mesh that defines their external shape, so self collisions are not considered within these reconstruction examples. Table 3.2 shows the performance (over 20 runs) of each methodology for the case of a dancing humanoid model. The computational cost and the reconstruction quality for tracking the animated model is also presented. FABRIK gives the best results with respect to computational KFN jesfllfis OK Figflre KFJKFR Eflvmplz of [VWgI‘ implzmzntvtion fiith multiplz zny zffzxtors ovzr timzC ihis is v fully unxonstrvinzy hvny zflvmplzA vllofiing H Yo[ on zvxh jointC cost and reconstruction quality; it requires the fewest iterations to achieve the desired posture and produces visually the smoothest poses. The median error presented in table 3.2 refers to the difference between the estimated joint positions and the true joint positions. A video included in the supplementary materials demonstrates FABRIK and compares its performance against other state of the art methods. HCKCHC Vpplying restrixtions Most IK problems have rotational and orientational restrictions since most real world joints have limitations on their movements. Joint constraints can be easily added to our proposed methodology (see subsection 3.4.1). The experimental dataset used to test the reconstruction quality of the constrained FABRIK is made up of 10 joints, each having angle rotational restrictions allowing movements only within a range. The same humanoid model, as described in section 3.6.2, is used to examine the reconstruction quality of the proposed methodology with and without constraints. FABRIK can be easily constrained producing visually realistic postures without oscillations and discontinuities. The constrained version is slightly slower than its unconstrained coun- terpart, requiring now almost 3O0ms to reach the target. Nevertheless, it is still much faster than other IK methods and approximately 10 times faster than constrained CCD. The recon- struction quality is high, producing postures with an average error of just over 30mm, almost half the average error of the unconstrained version. On the other hand, while it is not difficult OL V covzl Invzrsz ‘inzmvtixs holvzr (u= (v= Figflre KFJLFR V lofi rvtz woyy trvxking zflvmplzC ihz joints in rzy vrz thz knofin positions of thz zny zffzxtors vny thosz in wluz vrz thz zstimvtzy joint positionsC (v) shofis thz truz woyy poszs vny (w) thz zstimvtzy poszs using [VWgI‘C to apply manipulator constraints to CCD, the resulting animation often still has unnatural movements, especially when the target is at a significant distance from the end effector. The unconstrained version of CCD produces different joint poses compared to its constrained ver- sion, even if the latter is not violating the angle restrictions. It is interesting to note that there are instances where the constrained version of CCD needs fewer iterations and therefore performs slightly faster than its unconstrained version. This happens because the constraints prevent the chain from rolling and unrolling itself before reaching the target. Figure 3.26, shows examples of FABRIK and CCD implementations with and without joint restrictions. On this example, rotational limits have been applied restricting the allowed bending of each joint angle to a maximum value. Figure 3.27 shows the reconstruction improvement between the unconstrained and constrained versions of FABRIK applied to the humanoid model; rota- tional and orientational constraints have been employed on each joint limiting the angle and the twist between limbs to a feasible set. Finally, table 3.3 shows the number of iterations and the time needed to reach the target for both the unconstrained and constrained FABRIK and CCD approaches. KFO Ypplicyfiions OM (u= (v= (w= (x= (y= (f= (g= Figflre KFJMFR Woyy rzxonstruxtion using yiffzrznt I‘ mzthoyologizsC ihz joints in rzy vrz thz knofin positions of thz zny zffzxtors vny thosz in wluz vrz thz zstimvtzy joint positionsC (v) shofis thz initivl position vny (w) thz truz finvl positionC (x) shofis thz [VWgI‘ solutionA (y) thz XXY solutionA (z) thz JC irvnsposz solutionA (f) thz JC Yah solutionA (g) thz JC hkYBYah solutionC HCLC Vpplixvtions FABRIK has been successfully used for real-time marker prediction and centre of rotation (CoR) estimation [AL10c]. The joint positions of the estimated markers are re-positioned assuming that the inter-joint distance is constant over time. Incorporating bone length con- straints using FABRIK ensures that the model will have a more feasible motion. The proposed approach predicts the missing markers and estimates the joint positions reliably even if large sequences with occluded data exist, in which more than 1 marker is occluded on each limb, even if the limb rapidly changes direction. FABRIK’s performance has been also tested for hand tracking and reconstruction [AL10b]. FABRIK captures the movements of the hand model in real-time, using the minimum possible number of markers. Needing only prior knowledge about the geometry of the hand, the hand model and the restrictions of each joint, it reproduces good estimates of the captured motion. Joint constraints are applied to ensure that the hand motion is within a feasible set, giving a visually natural motion of the hand. This method is effective and real-time implementable. ON V covzl Invzrsz ‘inzmvtixs holvzr lyzle KFKFR Vvzrvgz rzsults fihzn joint xonstrvints vrz inxorporvtzyC Number of Matlab exe. Frames per Iterations time (szx) second FABRIK 15.461 0.01328 75.301 CCD 26.308 0.12359 8.091 FABRIK Constrained 17.142 0.03110 32.154 CCD Constrained 26.857 0.29281 3.415 HCMC Conxlusions IK methods are used to control the postures of articulated bodies in frame animation produc- tion. However, most of the currently available methods suffer from high computational cost and production of unrealistic poses. In this report, FABRIK, a simple, fast and reliable IK solver is presented. This is the first algorithm to use an iterative method with points and lines to solve the IK problem. It divides the problem into 2 phases, a forward and backward reaching approach, and it supports (to the best of our knowledge) all the rotational joint limits and joint orientations by repositioning and re-orienting the target at each step. It does not suffer from singularity problems and it is fast and computationally efficient. No pre-recorded motion database is necessary, thereby avoiding the need for extra memory. Also, a reliable methodology for applying joint restrictions, which supports and utilises all the advantages of FABRIK, is presented. FABRIK is a novel methodology for solving the IK problem which does not suffer from singularity problems and which is fast and computationally efficient. Our experiments show that FABRIK requires on average fewer iterations to reach the target than any other IK method tested, both with constrained and unconstrained kinematic chains. At the same time, it produces visually smooth postures, with and without constraints, reaching the desired position with very low computational cost. FABRIK can also be extended to a multiple end effector version supporting multiple kinematic chains. CCD is also a real-time IK solver. It is much faster than any Jacobian-based method but it is 10 times slower than FABRIK. The bigger drawback of CCD is the generation of unrealistic postures since it often rolls and unrolls itself before reaching the target. This rolling tends to overemphasise the movements of the joints closer to the end effector of the kinematic chain, thus producing unnatural movements. The CCD algorithm performs better when it is tracking a moving target (with small step-size) or when the distance between end effector and target is significantly small. In the case where the initial distance between target and end effector is large, CCD can produce unrealistic animation. Angle constraints can be easily added to the CCD methodology, controlling the movement of the manipulator. There are however, instances where the animation produced still has unnatural movements, even if manipulator constraints have been applied, especially when the target is at a significant distance from the end effector. Another limitation of CCD is that handling problems with multiple end effectors is not straightforward, since it is designed to solve problems with serial chains. The Jacobian methods return reasonable results; the chain poses, at most times, are more KFP Conclflsions OO (u= (v= (w= Figflre KFJNFR Vn zflvmplz of [VWgI‘ vny XXY implzmzntvtions fiith vny fiithout inxorporvting xonstrvintsC ihz top linz shofis thz [VWgI‘ solution vny thz wottom linz thz XXY solutionC (v) thz initivl position of thz kinzmvtix xhvinA (w) thz unxonstrvinzy solutionA (x) thz xonstrvinzy solutionC realistic than CCD, especially when the target is positioned at a significant distance from the end effector. The biggest advantage of the Jacobian methods over all other methods is that, by default, they can treat problems with multiple end effectors very easily. Manipulator constraints can be incorporated within the Jacobian algorithm, but the way in which these restrictions are applied is not straightforward. The Jacobian-based methods also perform poorly when the target is moving in a sinusoidal trajectory and the end-effector must track its position. They can produce unrealistic movements and motion with oscillation, shaking and discontinuities. There are also instances where the Jacobian methods suffer from singularity problems, since matrix inversions need to be calculated. Their biggest drawback however is that they converge very slowly to their final solutions since they use a linear approximation with a small step; only under some circumstances, eg using fast C++ matrix libraries, can these kinds of methods reach the target of real-time application. Triangulation returns the poorest results from the methods used within this report. The kinematic chain does not have a realistic shape; the joints close to the end-effector are usually in a straight line, emphasising the rotation of the joints neighbouring the root. Use of the Triangulation algorithm is limited to problems with a single end effector, making it unsuitable for complex character models with multiple end effectors. By definition, the Triangulation algorithm does not support manipulator restrictions. In this report, angle constraints have been incorporated confirming that the Triangulation algorithm often suffers from an inability to find a feasible solution, even if there is a solution, since each joint position is calculated independently without considering the restrictions of the next joint. The FABRIK algorithm is a powerful tool that can be used to provide reliable solutions to the IK problem both in robotics and computer vision. Chapters 4 and 5 demonstrate examples of OP V covzl Invzrsz ‘inzmvtixs holvzr (u= (v= (w= (x= Figflre KFJOFR Vn zflvmplz of implzmzntvtionC (v) ihz initivl positionA (w) thz rzvl posturzA (x) thz solution using unxonstrvinzy [VWgI‘A (y) thz solution vftzr inxorporvting joint rzstrixtionsC implementing FABRIK in motion capture problems for real-time marker prediction and centre of rotation estimation as well as hand tracking and reconstruction. 4 botion Cvpture holutions unyer bvrker dxxlusions T hOs chapter addresses the problem of real-time joint localisation of legged skeletons in thepresence of missing data. The data is assumed to be labelled 3D marker positions from a motion capture system. An integrated framework is presented which predicts the occluded marker positions using a variable turn model within an Unscented Kalman filter. Inferred information from neighbouring markers is used as observation states; these constraints are efficient, simple and real-time implementable. This work also takes advantage of the common case that missing markers are often still visible to a single camera, resulting in more accurate predictions. An Inverse Kinematics technique is then applied ensuring that the bone lengths remain constant over time; the system can thereby maintain a continuous data-flow. The marker and Centre of Rotation (CoR) positions can be calculated with high accuracy even in cases where markers are occluded for a long period of time. Our methodology is tested against some of the most popular methods for marker prediction and the results confirm that our approach outperforms these methods in estimating both marker and CoR positions. ICFC Introyuxtion Optical motion capture is a technology used to turn the observations of a moving subject (taken from a number of cameras) into 3D position and orientation information about that 79 PH botion Xvpturz holutions unyzr bvrkzr dxxlusions subject. It is commonly used to better analyse techniques for sports training [HNT+06]; to observe asymmetries and abnormalities in rehabilitation medicine (clinical analysis) [HDSB05, PTP+05, BSR07]; for biomechanics (prosthetics, ergonomics); to study the person’s movements for medical reasons or sport performance [Gol]; for gait labs; or for visualisation of virtual characters for films and computer games [Men99]. In general, the problem of automatic skeleton generation can be separated into three stages. First is the marker clustering, then the problem of finding the joint location and finally, the identification of the full body. In order to achieve accurate skeletal reconstruction of any legged body using optical motion capture systems, 3 markers must be available on each limb segment at all times. However, even with many cameras, there are instances where occlusion of markers by elements of the scene leads to missing data. In order to unambiguously establish its position, each marker must be visible to at least two cameras in each frame. Although many methods have been developed to handle the missing marker problem, most of them are not applicable in real-time, are usually limited to short time period occlusions and often require manual intervention. This chapter investigates methodologies for real-time marker prediction, under multiple cases of occlusion, to drive CoR estimates and then to automatically establish the skeleton model. A real-time integrated framework is presented, which predicts the occluded marker positions using a variable turn model within an Unscented Kalman filter. The previous marker positions are used within the framework in addition to information related to the missing markers of the current frame, inferred from an approximate rigid body assumption. The predicted marker positions are then used to locate the joints. Without assuming any skeleton model, we take advantage of the fact that for markers on a given limb segment, the inter-marker distance is approximately constant. The proposed marker constraint methodology is simple, real-time implementable and very efficient. Our method is automatic and scalable, without requiring any parameters to be set by the user. It considers all the cases of marker occlusion within a limb resulting in accurate predictions even in cases where all markers on a limb segment are missing for an extended period of time. The proposed approach also takes advantage of the special, but common, case where missing markers are still visible to one camera. With a continuous stream of accurate labelled 3D data, we can perform real-time CoR estimation; the CoR position is thereafter corrected via a real-time Inverse Kinematic technique which ensures that the inter-joint pairwise distances remain constant over time. A skeletal reconstruction is thereby achieved, producing information which can be used for visual performance feedback. Experiments demonstrate that our methodology effectively recovers good estimates of the true positions of the missing markers and CoRs, agreeing with human intuition, even if all the markers on a limb are occluded for a long period of time. The movements produced are smooth, without abnormalities or oscillations, resulting in natural motion. ICGC dwtvining HD bvrker eositions Motion capture hardware, such as that provided by PhaseSpace [Pha], CodaMotion [Cod] and Vicon [Vic], is under constant development, providing real-time acquisition of labelled LFJ gzfiyining KD eyrker hosifiions PI Figflre LFIFR bvrkzrBtoBlimw vssoxivtion zflvmplz fiith thrzz mvrkzrs on zvxh limwC ihis zflvmplz shofis thz uppzr woyy fiith thz mvrkzrBtoBlimw vssoxivtionC 3D marker data. These data can be used for reconstruction of the human skeleton allowing accurate real-time feedback via tracking and modelling of human motion. In order to overcome the tracking problem, we used a 16 camera and 480 frvmzs pzr szxony motion capture system using modulated LEDs, provided by PhaseSpace [Pha]. These cameras contain a pair of linear scanner arrays operating at high frequency each of which can capture the position of any number of bright spots of light as generated by the LEDs. The markers consist of a control circuit and bright red LEDs which are wired to a synchronisation box. For reconstruction it is necessary for each marker to be visible by at least two cameras in each frame. The system offers a fast rate of capture (480Hz) and allows the individual markers to be identified by combining the information from several frames and hence identifying the marker from its unique modulation. The markers are placed at strategic points on the articulated body so that these points can be easily and accurately located by the cameras. It is desirable to allocate the markers such that there is the same number of markers on each limb, if possible, in order to obtain the same quality of data for all parts of the body we are interested in. The subject moves in a specified space that can be tracked by the cameras and the markers attached to its body are tracked over time and used to reconstruct the three-dimensional pose of the subject at each instant of time. ICGCFC bvrkerBtoBlimw vssoxivtion Since the three-dimensional trajectory of each marker is available we are able to determine which markers are attached to which limb on the body. This is done in two steps: 1. Firstly, markers attached to the same limb are grouped. This is done by finding which markers maintain the same distance from each other throughout the motion. Since the data is noisy we expect that the distance between the markers does not remain constant. Therefore, the variances of the distances between markers are calculated and the markers are clustered as belonging to the same limb if the variance of the distance between them is less than a certain threshold. [JMF99] gives a survey on clustering techniques. PJ botion Xvpturz holutions unyzr bvrkzr dxxlusions Frame Frame mk k )(tr k x mk x )(tR Figflre LFJFR Yzsxription of thz rigiy woyyC ihz vzxtor r(t) spzxifizs thz position of thz xzntrz of mvssA rzlvtivz to thz originC ihz rotor R(t) yzfinzs thz orizntvtion of thz woyyA rzlvtivz to v fiflzy xopy plvxzy vt thz originC ihz xk is v vzxtor in v rzfzrznxz woyy (in this zflvmplz in frvmz k)A vny xk+m is v vzxtor in spvxz of thz zquivvlznt point of thz moving woyy (in this zflvmplz vftzr m frvmzs)C 2. Afterwards, association of the groups of markers to specific limbs of the body is necessary. This is done either manually or automatically by first calculating the distance between the centroid of each group of markers in a given time frame and then, according to the model used, the limbs are identified. It is assumed that the skeleton consists of rigid limbs connected with ball joints. An example of associated markers in an upper body model is presented in figure 4.1. ICGCGC gigiy woyy yynvmixs A rigid body can be viewed as a system of particles moving subject to the constraint that all inter-particle distances are fixed. The final body position can be expressed in terms of a rotation and translation from a fixed “reference” body on to the body in space (figure 4.2). We let r(t) denotes the position of the centre of mass and xk+m denote the position in space of a point in the body. These are related by: xk+m = RxkR˜+ r(t) (4.1) where xk is a fixed constant vector in the reference copy of the body. In this manner we have placed all of the rotational motion in the time-dependent rotor R(t); rotors will be regularly used later on this chapter. ICHC gelvtey lork Several papers have focused on methods for localisation of the centre of rotation (CoR) using MoCap data. These methods can be separated into two major groups, the sphzrz fitting methods and the trvnsformvtionvl methods. Both families of methods assume that at least 3 markers are available on each limb segment; there are, although, instances where marker data are unavailable due to marker occlusions. LFK jelyfied oork PK kphere fififiingR Sphere fitting methods are the most commonly used methods for calculating the CoR. This group of methods assumes that all markers remain a constant distance from the centre of rotation. A first step for solving this problem is to find a rigid body transformation moving the problem into a reference frame (e.g. frvmz F) so that the limb segment on one side of the joint is viewed as stationary and then solve a simpler one-sided problem. Dorst, in [Dor05], presents a first order approximation which also illustrates the dependence of the attitude estimation error on the distribution of the point cloud (in this case the marker locations on the limb segments). In [SPB+98] the Levenberg-Marquardt method was implemented in order to optimise the location of the centre of rotation and the radii of the marker spheres. A cost function S = (|xkm − X| − rm)- was introduced, where xkm is the marker position of the marker m at frame k, X is the centre of the sphere and rm is the radius of the sphere associated with the marker m. The overall cost is the sum of the individual costs over all markers and all frames. This algorithm requires a series of weights to be set, such as perceived accuracies of markers and a spatial reweighting of data points via a voxel grid. This method also has disadvantages; the inaccuracy of the heuristics used to set the weight parameters and the non-linear nature of the solver, making it susceptible to problems with local minima. Halvorsen et al., [HLL99], describe a closed form solution using the geometric properties of the sphere. They used the fact that the perpendicular bisectors of chords of a sphere intersect at the sphere origin, and every pair of frames provide an approximate perpendicular bisector for each marker. However, this method is dependent on which data points are used to form the chords, affecting the accuracy and effectiveness of the algorithm. In [GL02], Gamage and Lasenby also introduced a closed form solution, using a cost function of the squared differences in the squared distance from the CoR X, to a marker m, at position xkm in frame k, and the radius of the sphere associated with the marker rm. That is( (X − xkm)- − (rm)- )- (4.2) An alternative approach provided by Halvorsen, [Hal03], gives a Bayesian analysis of the algorithm of [GL02], providing a first order approximation of the effect of isotropic Gaussian noise upon the algorithm. An extension to [Hal03] can be achieved by assuming that the measured points are the result of Gaussian noise only in the axial direction1. lrynsformyfiionylR The transformational method assumes that all positions of the mark- ers are rigidly attached to limb segments. Such an approach was implemented in [Hol91]. A technique for using magnetic motion capture data to determine the joint parameters of an articulated hierarchy was described in [OBBH00]. This technique makes it possible to deter- mine limb lengths, joint locations, and sensor placement for a human subject without external measurements, just from the motion data acquired during the capture session. The parameters are computed by performing a linear least squares fit of a rotary joint model to the input data. 1hhy ufiiul xirywtion is thut purullyl to u liny from thy wyntry of thy sphyry to thy truy murkyr lowutionB PL botion Xvpturz holutions unyzr bvrkzr dxxlusions In many recent papers, such as [ETDH06], the same techniques are applied but the orientation is obtained from sets of optical markers. In [CL05] a sequential algorithm was presented to locate the rotation centres of a human skeleton from marker data assuming that all markers on a body segment are attached to a rigid body. This method does not suffer from optimisation steps with computational requirements that grow with the amount of data supplied ([GL02], [KOF05]) and no user feedback to set marker weights, as in [SPB+98], are needed. The method is closed form, thus enabling real-time implementation. Whilst several methods to estimate the location of centre of rotation have been proposed, the performance of most is unsatisfactory in the presence of unusual motions or of many contiguous occlusion-affected frames. Indeed, there are instances, even with expensive mo- tion capture systems, where occlusion of markers by elements of the scene leads to missing data. Many of these methods behave suboptimally with diverse motions, a high percentage of missing markers and/or external occlusions. Several methods have been proposed to deal with the problem of occluded markers in order to drive CoR estimation and skeletal recon- struction, but these do not generally run in real-time and often require manual intervention. Interpolation of the missing data using linear or non-linear approaches is commonly used [WH97, RCB98, Neb99a, AW00, Neb99b]; this can produce accurate results, but it is a post- processing technique requiring data prior to and after the occlusion. Some motion capture (MoCap) systems also provide missing markers recovery solutions using interpolation tech- niques in combination with kinematic information, but they do not reliably work in real-time. Recently, [PLH+09] presented an extrapolation algorithm which assumes that the most com- mon motion behaviors are circular or linear movements; however, this method can produce reliable predictions only for a limited number of occluded frames. While the discarding of frames with missing markers is another common technique, the omission of specific data could lead to the loss of useful information. Long-running occlusions leading to a large sequence of missing data can also cause complete failure of the system. Rhijn and Mulder, [RM06], proposed a model-based optical tracking and model estimation system for composite interaction devices. The proposed system automatically constructs the geometric skeleton structure, degrees of freedom (DoF) relations, and DoF constraints between segments, and thus pre-defined models are not required. The system supports segments with only a single marker, so that interaction devices can be small with a low number of markers. However, it is an off-line procedure unsuitable for real-time applications. Dorfmu¨ller in [DU03] used an extended Kalman filter (EKF) to predict the missing markers using previously available marker information, while Welch et al. in [WBV+99] used an EKF to resolve occlusions based on the skeletal model of the tracked person. Tak and Ko, [TK05], employed an Unscented Kalman Filter to ensure motion capture data remains in a feasible set. Again, these methods require manual intervention or become ineffective in cases where markers are missing for an extended period of time. In our earlier work, [ACL08], we presented an EKF method using a constant velocity (CV) model with marker constraints from neighbouring- markers. However, the CV model (2nx order Kalman Filter (KF)) limits its use to problems with constant marker 2ceighwours ury thy murkyrs vylonging to thy sumy limv sygmyntB LFK jelyfied oork PM velocity. These methods also do not take into consideration the fact that bones are rigid, thus the inter-joint pairwise distances remain constant over time. Li et al [LMPF09] propose DynaMMo, an approach that uses a Linear Dynamical System (LDS) to model motion capture data under sequences with missing values; in [LMPF10], the same authors introduce BoLeRo, a similar technique which also takes into consideration bone length constraints. The suggested algorithm has a ‘hard’ and ‘soft’ version of bone constraints assuming rigidity limitations of the distance between markers on a given segment limb. Although their algorithm results in smooth motion, the method is complex and expensive in terms of computational cost. Herda et al., in [HFP+00] and [HFP+01], used a post-processing approach to increase the robustness of a motion capture system by using a sophisticated human model. They can predict the 3D location and visibility of the markers increasing the robustness of the marker tracking and reducing the need for human intervention during the reconstruction process. The neighbouring markers that share kinematic relations with the occluded markers were used to help the estimation of the isolated markers. However, the skeleton information must be known a priori in order to apply this method. [HSD05] also takes advantage of the fact that the markers on a limb have fixed inter-marker pairwise distances. Thus, in the case where a marker is missing, its position can be recovered through the distance constraints imposed by the markers of the same limb. This approach may become ineffective when all or a significant number of markers are missing so that no information on that limb can be inferred from the available neighbouring markers. Ringer and Lasenby, [RL02], also present an automatic method to identify indistinguishable markers based on cliques.. However, this requires an off- line procedure in order to determine marker cliques and parameters of the skeletal structure. Zordan and Van Der Horst in [ZVDH03] mapped 3D marker position data to joint trajectories for a fixed limb-length skeleton, by attaching virtual springs and controllers, to follow the Cartesian-based marker data. In general, these skeleton methods could work well for short time occlusions but fail to track the missing markers for large occluded sequences. In [GMHP04], a style-based inverse kinematic method has been developed where a Gaussian Process Latent Variable Model (GPLVM) was used along with a pre-specified kinematic model. Wang et al, [WFH08], presented a Gaussian process dynamical model (GPDM) in order to learn the human pose and motion models. They observed the motion using a chain of latent variables and nonlinear mapping from the latent space; the proposed learned model was also able to cope with marker occlusions. Although these are real-time processing methods, they require knowledge of skeleton information which severely restricts their use. Chai and Hodgins, [CH05], present a method that uses the neighbouring markers to estimate the missing marker in the current frame. They propose a local linear model from these neighbours and then reconstruct the full pose of the frame by conducting an optimisation in the space constrained by a pre- recorded database. Yu et al., [YLD07], proposed an online motion capture labelling approach which also recovers missing markers. They cluster the markers into a number of rigid bodies based on the standard deviations of the marker-pair distances and if their fitting-rigid-bodies algorithm did not classify all the markers into rigid bodies, a missing marker auto-recovery 3aurkyrs in u clique huvy wonstunt xistunwys vytwyyn yuwh othyrB PN botion Xvpturz holutions unyzr bvrkzr dxxlusions method is applied assuming that the inter-marker distances are fixed over time. However, a training session is needed, the auto-recovery method for marker estimation does not take into account the limb segment rotation, no information about markers visible to a single camera is considered and the CoR estimation is not investigated under marker occlusions. Park and Hodgins, [PH06], fill the missing data by learning a statistical model of the spa- tial relationship between each marker and its neighbours; they use a Principal Component Analysis (PCA) on each marker position and its neighbours throughout the entire motion. In [THR07], the authors modelled the human motion and filled the gaps in the data using a Conditional Restricted Boltzmann Machine (CRMB) with discrete hidden states. Their ap- proach was trained using non-linear binary representations, conditioned on previous frames; at the same time, they took into consideration the correlation between joint angles, to produce more accurate results. Liu et al., [LZWM06, LM06], presented a piecewise linear approach for estimating human motions from a pre-selected set of informative markers (principal markers). A pre-trained classifier identifies an appropriate local linear model for each frame. Missing markers are then recovered using available marker positions and the principal components of the associated model. In [HGP04], the data were mapped onto a target motion by searching over patterns in existing databases. However, this data-driven family of methods requires an off-line training procedure and the results are highly dependent on training data and limited to those models and movements the system has been trained on. So far, no existing method operates inter-joint constraints to maintain a constant bone length over time. Also, no method- ology currently exploits the useful information we can extract from the special, but frequent, cases where markers are visible to only a single camera. ICIC Cvlxulvting the Centre of gotvtion Locating the CoRs is important in both computer graphics and rehabilitation medicine. During capture, markers must be carefully placed on the body in order to obtain good results. Results using markers placed too close to the CoR are more susceptible to errors since a small error may cause large deviations in the estimated rotation, leading to erroneous calculation of the model parameters. The data discussed here are labelled from an active marker system (PhaseSpace [Pha]) where no tracking is necessary. In general, 3 markers per bone segment are required to estimate the CoR for joints with 3 DoF; for simpler problems having fewer DoF, such as knees and elbows, the CoR can be calculated with fewer markers [CP07].In this report, we consider the general case of joints with 3 DoF since no prior knowledge of the model or joint-type is assumed to be known. ICICFC Finying the rotor wetween 2 sets of vextors The CoR estimation is a crucial step in acquiring a skeleton from raw motion capture data. To calculate the joints between two sets of markers, it is helpful to have the rotation of a limb at any given time. We can estimate the orientation of a limb at time k relative to a reference frame using the eroxrustzs formulation, as described by Horn in [Hor87]. LFL Cylcfllyfiing fihe Cenfire of jofiyfiion PO Assume a set of n-labelled points xi and the same set of points after an unknown rotation R(t), wi; the problem of finding a least square solution of the unknown rotor (unit quaternion) R can be formulated as, R = argmax n∑ i81 ( RxiR˜ ) · wi (4.3) In order to avoid introducing quaternion terminology and methods, we can express the solution in terms of Geometric Algebra. To clarify the following equations, the marker index i will temporally be suppressed. Let: R = a+W1z-. +W-z1. +W.z1- (4.4) xi = x1z1 + x-z- + x.z. wi = w1z1 + w-z- + w.z. (4.5) where a is a scalar, z1, z-, z. are orthogonal vectors in the 3-D space, and z-., z.1 and z1- are the unit bivectors, as described in chapter 2. Each of these encodes a distinct plane, and there are 3 of them to match the 3 independent orthogonal planes in the 3-D space. Then: (RxiR˜) · wi = a-(x · w) + 2aW1(x.w- − x-w.) + 2aW-(x.w1 − x1w.) + 2aW.(2x-w1 − 2x1w-) +W-1(x1w1 − x-w- − x.w.) + 2W1W-(−x-w1 − x1w-) + 2W1W.(x.w1 + x1w.) +W--(−x1w1 + x-w- − x.w.) + 2W-W.(−x.w- + x-w.) +W-.(−x1w1 − x-w- + x.w.) Eq. 4.3 can now be expressed as follows: R = argmax n∑ i81 ( RiυcRυ ) (4.6) where: Rυ =  a W1 W- W.  c =  x1w1 + x-w- + x.w. x.w- − x-w. x.w1 − x1w. x-w1 − x1w- x.w- − x-w. x1w1 − x-w- − x.w. −(x-w1 + x1w-) x.w1 + x1w. x.w1 − x1w. −(x-w1 + x1w-) −x1w1 + x-w- − x.w. x.w- − x-w. x-w1 − x1w- x.w1 + x1w. x.w- + x-w. −x1w1 − x-w- + x.w.  PP botion Xvpturz holutions unyzr bvrkzr dxxlusions ky 1 ky 2 ky 3 kx 1 kx 3 kx 2 CoR Frame Frame mk k k yc k xc mky 1 mky 2 mky 3 mkx 1 mkx 3 mkx 2 CoR mk yc mk xc mkk yR , mkk xR , Figflre LFKFR ihz motion of tfio limws fiith v timzByiffzrznxz of m frvmzsC In terms of limb motion, it is noted that x1, x-, x. and w1, w-, w. correspond to values given below, with xki , y k i , x k x and x k y be the centre of mass of the 3 markers, as shown in figure 4.3. x1 = x k 1 − xkx w1 = xk+m1 − xk+mx x- = x k - − xkx w- = xk+m- − xk+mx (4.7) x. = x k . − xkx w. = xk+m. − xk+mx xkx = xk1 + x k - + x k . 3 xk+mx = xk+m1 + x k+m - + x k+m . 3 (4.8) As a valid rotor must always obey the property RR˜ = 1 it must be the case that RiυRυ = 1. If this condition is relaxed and normalisation is introduced into the above matrix expression we have: R = argmax ( Riυ ∑n i81 (c)Rυ RiυRυ ) (4.9) This expression is now in the form of the Rayleight Quotient; ∑n i81 (c) is clearly Hermitian and the value of R at the maximum value of the expression to be maximised. Thus, provides a least squares estimate of the required rotor, where the eigenvector associated with the greatest eigenvalue maximises the matrix product. Therefore the rotor Rυ which corresponds to the rotation is equal to that eigenvector. Hereafter, in this thesis, vectors will be symbolised in bold font to distinguish them from other symbols and avoid any confusion. ICICGC Joint loxvlisvtion After extracting the rotation of two limbs between the current and a reference frame, the location of the joints can be calculated using the approach in [CL05]. This takes advantage LFL Cylcfllyfiing fihe Cenfire of jofiyfiion PQ k 1 y k 2 y k 3 y k 1 x k 3 x Limb x Limb y 1 ya 2 ya 3 ya 2 xa 3 xa 1 xa k 13 b k 11 b k 12 b k 2 x k C Figflre LFLFR V typixvl mvrkzr plvxzmznt fiith rzlzvvnt mvrkzrs shofinC of the approximation that all markers on a segment are attached to a rigid body. Suppose the markers are placed on two segments (x and y) joined by a CoR. Let the CoR location in frame k be Ck. The vectors from the CoR to markers in the reference frame are denoted by yix and y j y for limbs x and y respectively, where i and j are marker labels. Figure 4.4 presents a typical marker placement with the zkij , y i x and y j y as shown. The positions of the markers in frame k are given by: xki = C k +Rkxy i xR˜ k x y k j = C k +Rkyy j yR˜ k y (4.10) where Rx and Ry are the rotors expressing the rotation of the joint limbs x and y respectively. R˜ is the reverse of R. Let zkij be the vector from x k i to y k j , that is: zkij = x k i − ykj = RkxyixR˜kx −RkyyjyR˜ky (4.11) A cost function S can be constructed that has a global minimum at the correct values of yix and yjy if the data is noise free, and returns a good estimate in the presence of moderate noise. S = m∑ k81 nx∑ i81 ny∑ j81 [ zkij − ( Rkxy i xR˜ k x −RkyyjyR˜ky )]- (4.12) where nx, ny are the number of markers on limbs x and y respectively, and m is the number of frames used for the calculations. The minimum is given by the solution of the simultaneous linear equations, obtainable by differentiation [CL05]: yix = 1 m m∑ k81 R˜kx.z kRkx + 1 m m∑ k81 R˜kxR k y.yyR˜ k yR k x (4.13) yjy = 1 m m∑ k81 R˜ky.z kRky + 1 m m∑ k81 R˜kyR k x.yxR˜ k xR k y (4.14) QH botion Xvpturz holutions unyzr bvrkzr dxxlusions (u= (v= Figflre LFMFR Vn zflvmplz of Xog zstimvtion of v humvn woyy moyzl using pXaEJrC (v) ihz mvrkzr positions vs rzturnzy wy thz motion xvpturz systzmA (w) thz xvlxulvtzy Xogs vny thz skzlztvl rzxonB struxtionC where .zk = 1 nxny nx∑ i81 ny∑ j81 zkij .yw = 1 nw nw∑ i81 yiw w = {xP y} Having calculated the Rkw and .yw, we can locate the CoR. Figure 4.5 demonstrates an example of CoR estimation and skeletal reconstruction in real-time using the above method. However, due to occlusions, there are instances where not all marker positions are available. If all markers are available on one limb segment, w, the CoR may be estimated using only the current Rkw and .yw as estimated in the previous frame, when all markers were visible, via (4.10). If there are markers occluded on both limb segments, a marker prediction methodology is needed. ICJC bvrker ereyixtion The marker position estimates can by predicted using filtering, with each single marker tracked individually and incorporating constraints from the neighbouring markers. Most tracking problems require a dynamic model for accurate estimation of the trajectory of a maneuverable object. The key for an efficient target tracking algorithm is being able to extract information about the target’s state from the observations. Hence, a model that takes account of velocity and acceleration changes of the target state over time is needed. During the last decades various mathematical models of target motions have been developed. Singer ([Sin70, SB71]) proposes a model which assumes that the target acceleration is a zero- mean first-order stationary Markov process. Based on hingzr’s assumption, many papers have proposed a constant or variable acceleration (e.g [LJ03]). The general state-space model can be generally divided into two models, the state transition LFM eyrker hredicfiion QI and the state measurement model p(xt|xt−1) p(yt|xt) (4.15) where xt ∈ Rnx denotes the state of the system at time t and yt ∈ Rny the observations. The states follow a first order Markov process and the observations are assumed to be independent given the states. Therefore, the dynamic model can be expressed as follows xt = f(xt−1P–t−1) yt = h(fltPxtPnt) (4.16) where, yt ∈ Rny is the output observation, flt ∈ Rnu is the input observation, xt ∈ Rnx is the state of the system, –t ∈ Rnv is the process noise and nt ∈ Rnn the measurement noise. The mapping f : Rnx × Rnv 7→ Rnx and h : Rnx × Rnn 7→ Rny represent the deterministic process and measurement models. A simple and popular model used to track the motion of a target is the nzvrly xonstvnt turn (NCT) model [LJ03], which assumes that the target moves with nearly constant speed and turn rate. The altitude changes are most often modeled independently by a nearly xonstvnt vzloxity (CV) model or a random walk model along the z direction, leading to an acceptable accuracy in practice. ICJCFC kvrivwle iurn boyel The NCT and CV models were based on a constant-speed condition and constant turn-rate assumption which restricts the variety of possible supported maneuvers. The Variable Turn Model (VTM), [LJ03], is a more sophisticated model which assumes that a target’s velocity and acceleration are not constant over time. Figure 4.6 compares target prediction with variable acceleration using the VTM model against a linear prediction. It is easily seen that a constant speed (i.e ,˙ = 0) motion corresponds to y · – = 0, where y and – are the target acceleration and velocity vectors. This relationship is described by y = Ω× – (4.17) where Ω is the angular velocity vector of the target and is equal4 to Ω = – × y ,- (4.18) where Ω · – = 0 and , = ‖–‖ is the target speed. Asseo and Ardila in [AA82] describe a general motion of a rigid body in space. They show that, under the orthogonal velocity condition Ω ⊥ –, the target acceleration can be expressed as a second-order Markov process with state dependence wy = −2αy− (2α- + ω-)– +w w = –¨B + wΩ× – (4.19) 4v× w = v× uv = v× (Ω× v= = (v · v=Ω− (Ω · v=v = 2Ω− (Ω · v=v, if unx only if Ω · v = D, thut is Ω ⊥ vB QJ botion Xvpturz holutions unyzr bvrkzr dxxlusions xk−4|k−4 xk−3|k−3 xk−2|k−2 xk−1|k−1 xˆk|k linear prediction aˆ Figflre LFNFR ivrgzt przyixtion fiith vvrivwlz vxxzlzrvtion using thz kib moyzl xompvrzy to thz linzvr przyixtionC where if – = ,izi, then – B = ,˙izi (sum over i assumed) and ω , ‖Ω‖ = ‖– × y‖ ,- α = −– · y ,- (4.20) Thus, the target equations of a 3D discrete motion of the VTM with state xk = [xkP x˙kP x¨kP ykP y˙kP y¨kP zkP z˙kP z¨k] i are now represented by xk = diag [Fx(ωxPxP αx)P Fy(ωxPyP αy)P Fz(ωxPzP αz)]xk−1 +Wflk−1 +wk−1 where Fi(ωxPiP αi) for each component i = xP yP z is given by: f i11 = 1; f i -1 = f i .1 = 0 f i1- = V−W ωxPi(α-i + ω - xPi) where V = 2αiωxPi and W = z−αii (2αiωxPi cos(ωxPii ) + (α-i − ω-xPi) sin(ωxPii )) f i1. = ωxPi − z−αii (ωxPi cos(ωxPii ) + αi sin(ωxPii )) ωxPi(α-i + ω - xPi) f i-- = z−αii (ωxPi cos(ωxPii ) + αi sin(ωxPii )) ωxPi f i-. = z−αii sin(ωxPii ) ωxPi f i.- = − (α-i + ω - xPi)z −αii sin(ωxPii ) ωxPi f i.. = z−αii (ωxPi cos(ωxPii ) + αi sin(ωxPii )) ωxPi LFM eyrker hredicfiion QK and f = cov (wk−1) = diag [Sxfx(ωxPxP αx)P Syfy(ωxPyP αy)P Szfz(ωxPzP αz)] where ω-xPi = α - i + ω - i , Sx, Sy and Sz are the power spectral densities of the white noise w of each component xP yP z. fi(ωxPiP αi) for i = xP yP z is given by: qi11 = V+W + X 4αiω-xPi(α - i + ω - xPi) . V = z−-αii [(α-i − 3ω-xPi)x+ (ω-xPi − 3α-i )s− (α-i + ω-xPi)-] W = 8z−αiiαiωxPi[2αiωxPix+ + (α-i − ω-xPi)s+] X = α-iω - xPi(4αii − 11) + ω4xPi(1 + 4αii ) qi1- = z−-αii (ωxPix+ + αis+ − z−αiiωxPi)- 2ω-xPi(α - i + ω - xPi) - qi1. = z−-αii (x− s− αi + ω-xPi) + 4z−αiiαiωxPis+ − ω-xPi 4αiω-xPi(α - i + ω - xPi) - qi-- = z−-αii (x− s− αi − ω-xPi) + ω-xPi 4αiω-xPi(α - i + ω - xPi) - qi-. = z−-αii s-+ 2ω-xPi qi.. = z−-αii (x+ s− αi − ω-xPi) + ω-xPi 4αiω-xPi qi-1 = q i .1 = q i .- = 0 x = α-i cos(2ωxPii )P s = αiωxPi sin(2ωxPii ) x+ = cos(ωxPii )P s+ = sin(ωxPii ) ICJCGC jnsxentey Kvlmvn Filter Kalman filtering has been extensively used for real-time estimation of linear dynamic systems. However, the traditional Kalman filter [Kal60] is not suitable for use with non-linear dynamical systems, even if Gaussian approximations to the joint distribution of state x and measurement y are made. The Extended Kalman Filter (EKF) [Jaz70] is a minimum mean-square-error (MMSE) estimator which extends the scope of the Kalman filter to nonlinear optimal filtering problems. It forms a Gaussian approximation to the joint distribution of state and mea- QL botion Xvpturz holutions unyzr bvrkzr dxxlusions surement using a Taylor series-based transformation. Nevertheless, EKF implementation is complex (Jacobian and Hessian matrices with second order filters are required), difficult to tune, and only reliable for systems that are almost linear on the timescale of the updates. The Unscented Kalman Filter (UKF), [JU97], propagates mean and covariance information through nonlinear transformations providing more accurate results than the EKF, for a similar computational cost. Consider propagating an nx-dimensional random variable x and assume x has mean .x and covariance hx. First, a set of 2nx + 1 weighted samples or sigmv points Si = {liPXi} are deterministically chosen so that the true mean and covariance of the random variable x can be completely recovered from them. A set of scaled sigma points S = {oPX} can be calculated by setting: λ = α-(nx + κ)− nx (4.21) and selecting the sigma point set by: l (m) + = λR (nx + λ) i = 0 l (x) + = λR (nx + λ) + (1− α- + β) i = 0 l (m) i =l (x) i = 1R {2 (nx + λ)} i = 1P OOOP 2nx where α is a positive scaling parameter which controls the size of the sigma point distribution. κ ≥ 0 is also a scaling parameter and li is the weight associated with the ith point such that ∑-nx i8+li = 1. [MDFW00] proposed κ = 0, to guarantee positive semidefiniteness of the covariance matrix and 0 ≤ α ≤ 1 to avoid sampling non-local effects when the nonlinearities are strong. β ≥ 0 is a weighting term which incorporates knowledge of the higher order moments of the distribution. β = 2 is the optimal choice for a Gaussian prior. Let the original state and noise variables at time k be xαk = [x i k P– i k Pn i k ]. The sigma point selection scheme is applied to this augmented state Random Variable (RV) to calculate the corresponding sigma matrix, Xαk . The mean .x and covariance h of the Gaussian approximation is updated to the posterior distribution of the states as follows: .x+ = E[x+] h+ = E[(x+ − .x+) (x+ − .x+)i ] (4.22) .xα+ = E[x α] = [xi+ 0 0] i hα+ = E[(x α + − .xα+ ) (xα+ − .xα+ )i ] = h+ 0 01 i 0 1 0 j  For k ∈ {1P OOOP∞} the sigma points are equal to: X ak−1 = [ .xαk−1 .x α k−1 ± √ (nα + λ)hαk−1 ] (4.23) LFM eyrker hredicfiion QM and the time update is given by: X xk|k−1 = f (X xk−1PX ,k−1) .xk|k−1 = -na∑ i8+ l (m) i X xiPk|k−1 hk|k−1 = -na∑ i8+ l (x) i [X xiPk|k−1 − .xk|k−1][X xiPk|k−1 − .xk|k−1]i Zk|k−1 = h ( X xk|k−1PX nk−1 ) .zk|k−1 = -na∑ i8+ l (m) i ZxiPk|k−1 where f(O) is the transition and h(O) the observation function. The measurement update equa- tions are: h~zk~zk = -na∑ i8+ l (x) i [ZiPk|k−1 − .zk|k−1][ZiPk|k−1 − .zk|k−1]i hxkzk = -na∑ i8+ l (x) i [XiPk|k−1 − .xk|k−1][ZiPk|k−1 − .zk|k−1]i ck = hxkykh −1 ~yk~yk .xk = .xk|k−1 +ck(rk − .zk|k−1) hk = hk|k−1 −ckh~zk~zkcik where, xα = [xi –i ni ]i , Xα = [(X x)i (X ,)i (X n)i ]i , λ is the composite scaling parameter, nα = nx+n,+nn, i is the process and j the measurement noise covariance, c is the Kalman gain, li are the Unscented Transform weights and rk is the observation vector. The transition function f(O) and the observation function h(O) are very important for im- plementing efficient UKF filtering. In this model the transition function is taken to be a variable turn model (VTM) with target velocity and acceleration the velocity and acceleration of the relevant marker. However, a detailed look at real marker data indicates that estimated velocities are significantly noisy. Many factors contribute to marker position noise, such as optical measurement noise, miscalibration of the optical systems, reflections, motion of mark- ers relative to the skin and motion of the skin relative to the rigid body (underlying bone). As a result, the target acceleration is mostly noisy. One method of measuring accelerations would be to attach accelerometers to the markers placed on the limb segments. Such a system, which synchronises measurements from the accelerometer and active markers was trialled and described in [Bac09]. While the system shows some promise, we are faced with dealing with both the noise on the accelerometer readings and the fact that the reference frame of those readings must be determined. Given these difficulties, we propose to use estimates of accelera- tions obtained from the position data. A real-time median filter has been applied to the target QN botion Xvpturz holutions unyzr bvrkzr dxxlusions velocity (more details in [ALC09]). Having a smoothed velocity, an adequate acceleration can be calculated and the proposed model can then be used straightforwardly. On the other hand, for the observation function we use a simple model which assumes that the rotation between two consecutive frames is constant. Thus, the time update of the observation state is equal to Zk|k−1 = Rk−-Pk−1X xk|k−1R˜k−-Pk−1 (4.24) where RpPq is the rotor for the rotation between frames p and q, assuming that the rotation of the markers between two consecutive frames remains constant. It is important at this point to recall that the rotation between the previous and the current frame is re-calculated every frame just after the marker prediction using the estimated marker positions. Early numerical implementations using Particle Filtering [DDFG01] showed that, although it could improve the performance of our state estimates, this improvement was marginal com- pared to the UKF-VTM and therefore not worth the added computational effort. Particle filtering has high computational cost thus ruling out its use for real-time applications. ICKC Vpplying bvrker Constrvints The observation vector, rjk, gives the true (observed by the mocap system) position of the tracked marker j when available, otherwise it represents estimated position. The state vector represents true position and velocity as given above. To cope with cases where markers are missing for long periods of time, a tracker that uses information from both the previous frames and the current positions of neighbouring visible markers has been implemented, taking into account the rigidity of the body. Assuming that there are three markers on each limb and the inter-marker distance is constant over time, the observation vector can be updated as given below for 5 different scenarios. The proposed marker constraints methodology, from inferred neighbouring markers, was first published in [ACL08]; the approach is simple, real-time implementable and does not assume prior knowledge other than fixed inter-marker distance. This assumption is true in a noise free environment, however since limb segments rapidly change direction, there is noise on marker motion relative to the skin and motion of the skin relative to the rigid body (underlying bone). The solutions to the mathematical problems discussed in this section are given in details, using CGA, in the Appendix A. ICKCFC Vll mvrkers vre visiwle on v given limw Where all markers are visible on a given limb, then: rjk = Hx k j + – k j (4.25) where xkj is the current state of a tracked marker j on the limb, H is the owszrvvtion moyzl (in this case the identity) and –kj is the observation noise. – is assumed to be zero mean multivariate normal with covariance j. LFN Ypplying eyrker Consfiryinfis QO Frame k − 1 x k−1 2 D k−1 1,2 x k−1 1 x k−1 3 D k−1 1,3 Frame k x k 2 Dˆ k 1,2 x k 3 Dˆ k 1,3 x˜ k 1 xˆ k 1 Figflre LFOFR ihz owszrvvtion vzxtor in thz xvsz of G visiwlz mvrkzrsC ihz rzy yotA x˜k1 A rzprzsznts thz vvzrvgz vvluz vs givzn in zquvtion ICGNC ihz grzzn yotA xˆk1 A is thz point on thz intzrszxtion of thz G sphzrzs fihixh is xloszst to x˜k1 C ICKCGC dne missing mvrker on v limw segment In the case where two markers are visible on the limb, r1k = Hxˆ k 1 + – k 1 (4.26) where xˆk1 is the estimated position of the occluded marker m1 in frame k (assuming, in this example, that m1 is the missing marker). xˆ k 1 can be calculated as given below. Firstly we calculate Dk−11P- and D k−1 1P. which correspond to the vectors between marker m1 and markers m-, m. in frame k − 1 respectively. These vectors are given by: Dk−1iPj = x k−1 j − xk−1i (4.27) Thereafter, these vectors are rotated as DˆkiPj = R k−-Pk−1Dk−1iPj R˜ k−-Pk−1 (4.28) assuming that the rotation between the current and the previous frame is the same as that between the previous 2 frames. Predicting the current rotation using previous rotations offers marginal improvement to the system performance, and such a small improvement means that it is not worth the additional computational cost [ALC09]. One obvious way to proceed is to calculate the point x˜k1 which is an average of the estimated positions in frame k using the Dˆ vectors; x˜k1 = ( xk- − Dˆk1P- ) + ( xk. − Dˆk1P. ) 2 (4.29) where xki is the position of marker i in frame k. We now improve on this estimate by finding the solution of the intersection of the two spheres in frame k with centres xk-, x k . and radii |Dˆk1P-| and |Dˆk1P.| respectively. xˆk1 is assigned as the closest point on the circle of intersection to x˜k1. Figure 4.7 illustrates this process. QP botion Xvpturz holutions unyzr bvrkzr dxxlusions Frame k − 1 x k−1 2 D k−1 1,2 x k−1 1 x k−1 3 D k−1 3,2 xˆ k 1 xˆ k 3 Frame k x k 2 Dˆ k 1,2 Dˆ k 3,2 Figflre LFPFR ihz owszrvvtion vzxtor in thz xvsz of only onz visiwlz mvrkzrC ihz rzy yotsA xˆkj A fihzrz j = {1P 3} rzprzsznt thz zstimvtzy position of thz missing mvrkzr mj vs givzn in zquvtion ICHFC ICKCHC iwo missing mvrkers on v limw segment In the case of only one marker visible (for example m-) on a given limb, the observation vector is given as: rjk = Hxˆ k j + – k j (4.30) where xˆkj is the estimated position of the occluded marker mj (j = 1P 3) in frame k. xˆ k j is given by: xˆkj = x k - − DˆkjP- (4.31) where xk- is the position of the visible marker m- on the limb in the current frame and Dˆ k jP- is as described above. In that case, it is assumed that the rotation around the axis joining the two remaining markers on the limb does not exist. Figure 4.8 demonstrates this process. ICKCIC Vll mvrkers on v limw segment vre missing When all markers on a limb are occluded, we consider two possible subcases; the case where the other limb segment has some markers visible and the case where both limb segments have all of their markers occluded. If some markers on the other limb segment are visible (assume the y limb), the missing marker positions can be calculated using the CoR estimate, Cˆk = y k i − Rky.yyR˜ky where i = {1P 2P 3}. Our methodology takes advantage of the approach described in section 4.4.2 for CoR estimation and provides better approximations of the CoR using information from the adjacent limbs. The observation vector of the Unscented Kalman filter is then updated as: rjk = Hxˆ k j + – k j (4.32) where xˆkj is the estimated position of the occluded marker mj (j = 1P 2P 3) in frame k. xˆ k j is given by; xˆkj = Cˆk + Dˆ k jPx (4.33) LFN Ypplying eyrker Consfiryinfis QQ Ck−1 y k−1 2 y k−1 1 y k−1 3 x k−1 2 x k−1 1 x k−1 3 a¯y a¯x Cˆk yk 2 yk 2 yk 3 xˆk 2 xˆk 1 xˆk 3 a¯y Dˆk 2,c Dˆk 1,c Dˆk 3,c Frame k − 1 Frame k Figflre LFQFR ihz zstimvtion proxzyurz fihzn vll mvrkzrs on v singlz limw szgmznt vrz oxxluyzyC ihz rzy yots rzprzsznts thz zstimvtzy position of thz XogA Cˆk = yki −Rky.yyR˜ky fihzrz i = {1P 2P 3}A vny thz zstimvtzy mvrkzr positions on limw szgmznt xA xˆkj = Cˆk + Dˆ k j;cA fihzrz j = {1P 2P 3}C .yx vny .yy vrz upyvtzy using thz przyixtzy mvrkzr positions in thz xurrznt frvmzC where DˆkjPx is an estimate of the vector between marker mj and the CoR. This approach takes advantage of the fact that the distance between markers and the CoR is constant. This vector is approximated by DˆkjPx = R k−-Pk−1Dk−1jPx R˜ k−-Pk−1 where Dk−1jPx = x k−1 j −Ck−1. This assumes that the rotation of the markers between two consecutive frames remains constant. Figure 4.9 illustrates this procedure. If both limb segments have all markers occluded, only information from previous frames can be used. The observation vector, rjk, in this instance is calculated using a rotor based method. This method also assumes that the segment rotation between two consecutive frames is constant. The observation vector can now be expressed as rjk = Hxˆ k j + – k j (4.34) where xˆk1 is equal to xˆ k 1 = R k−-Pk−1xk−11 R˜ k−-Pk−1. This method performs better than a simple variable velocity model since it ensures that all markers on a limb do not move independently [ALC09]. ICKCJC bvrkers visiwle in only one xvmerv Each marker can be reconstructed by the motion capture system if it is visible in at least two cameras. Indeed, looking at numerous real datasets, we have observed a high probability that the missing markers are not entirely occluded; information about position is often returned by a single camera. This information identifies a line, a1, starting from the camera and passing through the position of the missing marker. By relaxing the constraints that the inter-marker distance is constant and accepting that the real position of the marker is on that line, we may be able to obtain a more accurate estimate of the position of the relevant marker. This IHH botion Xvpturz holutions unyzr bvrkzr dxxlusions Camera L1 Frame k x k 2 Dˆ k 1,2 x k 3 Dˆ k 1,3 xˆ k 1 x´ k 1 Figflre LFIHFR ihz owszrvvtion vzxtor in thz xvsz of G visiwlz mvrkzrs vny onz mvrkzr visiwlz only wy onz xvmzrvC ihz rzy yotA x´k1 A is nofi uszy for thz xvlxulvtion of thz owszrvvtion vzxtorA r 1 k = Hx´ k 1 +– k 1 C position, x´k1, corresponds to the projection from the point xˆ k 1 onto the line a1, as in figure 4.10. This is applicable for the cases in which the motion capture system fully reconstructs one or two marker locations and another marker is visible in just one camera. If a limb segment has only one known and one partially visible marker, the system is more reliable when it first predicts the partially visible marker and then the entirely occluded marker. ICLC Wone aength Constrvints using Inverse Kinemvtixs The UKF-VTM model for marker prediction returns good estimates of the marker positions, even if the markers are occluded for an extended period of time. It is, however, easily observable (see fig. 4.11) that there are instances where the reconstructed markers may break the rigid body assumption (limbs may not have constant lengths over time), resulting in a violation of the model’s structure. This is more obvious in extreme cases where many markers from the same limb segment are occluded for an extended time period (see fig. 4.19); the limb segment lengths drift (as the inter-joint distances were not fixed to be constant) even if the estimated marker positions do not significantly differ from their true positions. Therefore, it seems sensible to constrain the bone lengths to be constant over time, taking into account the rigidity if the body, as the skeleton methods does; this extension does not presuppose any knowledge of the model. For such unconstrained models we can preserve bone lengths using Inverse Kinematics (IK) techniques (see Chapter 3). Ishigaki et al. [IWZL09] implement a real-time IK control interface for character animation which translates the performance into corresponding actions; that was achieved by integrating prerecorded motions with online performance and dynamic simulation. If the input motion does not match the conditions, a kinematic process is applied to match users’ motion with the example interactions. However, the proposed inter-joint constraint approach requires offline training, meaning that the results depend on the training data, and prior knowledge of the model is required. To the best of our knowledge, this is the first time that a real-time IK technique has been used for CoR correction under multiple marker occlusion in optical motion capture. LFO Zone dengfih Consfiryinfis flsing an–erse cinemyfiics IHI Figflre LFIIFR ihz rzsults proyuxzy using thz j‘[Bkib moyzlC It is szzn thvt thz yistvnxzs wztfizzn hips vny knzzs xhvngzA sinxz it is not guvrvntzzy thvt thzsz yistvnxzs vrz xonstvntC ihz owszrvzy positions vrz xolourzy wluzA fihilz thz przyixtzy positions vrz rzyC ICLCFC Vyjusting FVWgIK for Cog xorrextion in optixvl motion xvpture In order to achieve a real-time framework, the proposed IK solver must process a large number of frames per second and must return natural poses which satisfy the user constraints. FAB- RIK, as described in Chapter 3 and [AL10d], is a real-time IK solver which returns smooth postures in an iterative fashion. FABRIK simply repositions the CoRs in order to ensure the constant inter-joint distance over time, without considering the limb segment rotation; the limb segment orientation is known since there are 3 markers attached to each limb segment, thus time and computational cost can be saved. Nevertheless, FABRIK can also treat more complex cases where it is desirable to estimated the bone rotation. hrozlems wifih keriyl Chyin models FABRIK uses the positions of the joints that have been estimated using predicted marker positions to find updates that meet the fixed inter-joint distance assumption. In the most general case, where the estimated CoRs are not positioned at the end of the chain, the solution can be achieved using a forward and backward iterative mode, very similar to the basic FABRIK algorithm. The modified method starts from a joint of the chain which was calculated using observed marker positions and works forwards, adjusting each estimated joint along the way until the next joint which was calculated using observed data. Thereafter, it works backward in the same way, in order to complete a full iteration. A graphical representation of the algorithm in action is given in figure 4.12. Assume p1P OOOPpn are the joint positions of a chain, where p1 and pn are joint positions calculated using observed data, and the joints inbetween are joints estimated using predicted marker positions. Set the distances between each joint to be yi = |pi+1 − pi|uvg, for i = 1P OOOP n− 1, z = p1 and fi = pn; these distances can be established by averaging over time from the frames where the markers, and the joints, are available. Then, check whether the target is reachable or not; find the IHJ botion Xvpturz holutions unyzr bvrkzr dxxlusions Ylgorifihm LR A full iteration of FABRIK for CoR correction. InputN hhy joint positions pi for i = E; :::; n, unx thy uvyrugy xistunwys vytwyyn yuwh joint ri = |pi+1 − pi|uvg whyry n is thy numvyr of joints of thy whuinB eutputN hhy nyw joint positions pi for i = E; :::; nB 3.1 b = p1; t = pn; 3.2 : Initivlisvtion of the ritA vvlueC 3.3 ritA = E 3.4 while ritA L tol do 3.5 : hiAGE FO FdglAgD gEACHIcG 3.6 : het p1 to its initivl positionC 3.7 p1 = b 3.8 for i = E; :::; n− E do 3.9 : Find the distvnce ri wetfieen the estimvted joint position pi+1 vnd the joint pi 3.10 ri = |pi+1 − pi| 3.11 i = ri=ri 3.12 : Find the nefi joint positions piC 3.13 pi+1 = (E− i=pi + ipi+1 3.14 end 3.15 : hiAGE GO WACKlAgD gEACHIcG 3.16 : het pn to its initivl positionC 3.17 pn = t 3.18 for i = n− E; :::; E do 3.19 : Find the distvnce ri wetfieen the joint position pi+1 vnd the estimvted joint pi 3.20 ri = |pi+1 − pi| 3.21 i = ri=ri 3.22 : Find the nefi joint positions piC 3.23 pi = (E− i=pi+1 + ipi 3.24 end 3.25 ritA = |p1 − b| 3.26 end distance between p1 and pn, yist, and if this distance is smaller than the total sum of all the inter-joint distances, yist Q ∑n−1 1 yi, the target is within reach, otherwise, it is unreachable. If the target is within reach, a full iteration is performed in two stages. In the first stage, the algorithm estimates each joint position starting from p1, moving forward to pn. Thus, initialise p1 = z and find the line, l1, which passes through the joint positions p1 and p-. The new position of the 2ny joint, p′-, lies on that line with distance y1 from p1. Similarly, the new position of the 3ry joint, p′., can be calculated using the line l-, which passes through the p. and p′-, and has distance y- from p′-. The algorithm continues until all new joint positions are calculated, including p′n. Having in mind that initially pn was the observed position of the n th joint, a second stage of the algorithm is needed. A full iteration is completed when the same procedure is repeated but this time starting from the observed position of the nth joint and moving backwards to the 1st joint. Therefore, let the new position for the nth joint, p′′n, be its initial position fi. Then, using the line ln−1 that passes through the points p′′n and p′n−1, we define the new position of the joint p′′n−1 as the point on that line with distance yn−1 from p′′n. This procedure is repeated for all the remaining joints, including p1. The procedure is then repeated, for as many iterations as needed, until p1 and pn are close enough (to be defined) to their initial, observed positions. FABRIK always converges to any given chains/goal positions, when this is possible. If there are constraints which do not allow the chain to bend enough or if the LFO Zone dengfih Consfiryinfis flsing an–erse cinemyfiics IHK 3 pˆ 2 pˆ 1 p 4 pˆ 5 p (u= 3 pˆ 2 pˆ 1 p 4 pˆ d3 2 p 4 p 3 p d1d2 d4 l2 l3 l1 5 p 5 p l4 (v= 1 p 5 p d3 2p 4p 3 p d1 d2 d4 l2 l3 l1 4 p l4 3 p 2 p 1 p (w= 1 p 5 p d3 2pˆ 3 pˆ d1 d2 d4 4 pˆ (x= Figflre LFIJFR V simplz zflvmplz of [VWgI‘ implzmzntvtion for thz xvsz fihzrz thz zstimvtzy joints vrz positionzy inwztfizzn G owszrvzy joint positionsC (v) ihz initivl position of thz xhvinA fihzrz {pi} vrz owszrvzy vny {vpi} vrz zstimvtzy joint positionsC (w) ihz first phvsz of thz vlgorithmP thz joint p′i hvs wzzn vyjustzy vs thz point on linz li−1 fiith yistvnxz yi−1 from pi−1C (x) ihz szxony phvsz of thz vlgorithmO lzt thz nzfi position of p′5 wz its initivl position p5P rzpzvt thz proxzyurz stvrting this timz from thz othzr siyz of thz xhvinC (y) ihz finvl posturzP thz vlgorithm is rzpzvtzy for vs mvny itzrvtions vs nzzyzy until thz yiffzrznxz wztfizzn thz owszrvzy vny thz rzturnzy positions of thz joints p1 vny p5 is lzss thvn v givzn tolzrvnxzC target is not within the reachable area, there is a termination condition which compares the previous and the current position of the joint p1, and if this distance is less than a specified tolerance, FABRIK terminates its operation. Similarly to the prototype FABRIK algorithm, several optimisations can be achieved using CGA to produce faster results and to converge to the final answer in fewer iterations. Another simple optimisation is the direct construction of a line pointing towards pn, when the target is unreachable, adjusting the distances in such a way that each distance has changed uniformly. The adjusted FABRIK for CoR correction is illustrated in pseudo-code in Algorithm 4. There are, indeed, some special instances where FABRIK does not necessarily work in an iterative fashion. Such a special case is when the joints, which have been estimated using predicted markers, are located at the end of the chain. This is a simplified case of the general solution, since the answer is given directly in one single iteration. In that case, it is ensured that the inter-joint distance remains constant and the bone lengths are not stretched out. The new simplified algorithm is given here: assume pi is a true joint position. The joint update p′i+1 is set as the point on line li that passes through pi and pi+1 and has yi distance from pi. This procedure is repeated for all the remaining estimated joints until the end of the chain. A graphical representation of an implemented example is given in figure 4.13. IHL botion Xvpturz holutions unyzr bvrkzr dxxlusions d1d2d3 3 pˆ 2 p 1 p 4 p 3 p 4pˆ l3 l2 Figflre LFIKFR V simplz xvsz fihzrz thz zstimvtzy joints vrz loxvtzy vt thz zny of thz xhvinC In this zflvmplz thz szrivl xhvin hvs I joints fihzrz p1 vny p2 hvvz wzzn xvlxulvtzy using owszrvzy mvrkzr positions vny thz vp3 vny vp4 using przyixtzy mvrkzrs positionsC ihusA szt p ′ 3 to wz thz point on linz l2 fihixh hvs yistvnxz y2 from p2 vny p ′ 4 thz point on linz l3 thvt hvs y3 yistvnxz from joint p ′ 3C hrozlems wifih firee models In reality, most of the legged models are comprised of several kinematic chains in a tree fashion. The proposed algorithm can be easily extended to process tree models with multiple serial chains. The solution is similar to the original version of FABRIK for multiple end effectors, given in Chapter 3. Hence, if the sub-base. joint is an observed position, FABRIK is used on each chain with estimated CoR positions individually, starting from the sub-base, in a forward and backward iterative fashion. If the sub-base is an estimate, the same procedure is applied but this time starting from the observed joints of the connected chains and moving towards to the sub-base. This will produce as many different positions of the sub-base as the number of the connected chains. The new position of the sub-base will then be the centroid of all these positions. In the second stage, the normal algorithm is applied separately for each chain, starting now from the sub-base and moving outwards to the starting joints. The method is repeated until all observed joints have no significant change between their initial and updated positions. Ypplying Consfiryinfis The main aim of the work in this Chapter is to describe a general solution, where each single joint is treated as ball joint, allowing 3 degrees of freedom. In this way, we can ensure the generality of the proposed method, producing solutions without prior knowledge of the model. Nevertheless, in cases where it is desirable to incorporate joint restrictions, FABRIK can be easily adapted to support joint limitations by readjusting the target position and orientation, on each step, to satisfy the joint biomechanical limits, as described in Chapter 3, [AL10d] and [AL09]. Obviously, the more information available regarding the model’s structure and joint con- straints, the more accurate and efficient will be the results. This information can help to give a visually more realistic motion within a feasible set; however, it will limit universal use of the proposed methodology and fails to satisfy our aim of no prior knowledge of the model. The Inverse Kinematic system ensures that, even if the marker prediction system fails to track the marker positions, the derived CoRs will be good estimates of their true positions. 3A suvAvusy joint is u joint whiwh wonnywts F or mory whuinsB LFP jesfllfis ynd Discflssion IHM lyzle LFIFR Vvzrvgz rzsults (ovzr GE runs) unyzr multiplz xvszs of oxxlusion (lvrgz oxxlusions of GJEE frvmzs in totvl)C Meocjd Mdnndib hamfemn Mamfem'n Rjmno Cane CjM'n Rjmno Cane Kmjc) (gdhbn ji a gdhb nebheio Emmjm (ch) nceiamdj (ch) Emmjm (ch) nceiamdj (ch) kem necjid) Jie hamfem 1)-403 .)0122 1)+3-+ 1)2333 PFA(VOM Orj hamfemn .)42.2 2)1034 1)4312 -)2340 .10 Agg hamfemn 3)4+1- 1-)3244 2)3041 1.)113. Jie hamfem 4)4132 10)-00- 2)3222 1)1134 Dioemkjgaodji Orj hamfemn 1+)0-41 11)3324 4)..4+ 3)0324 4++ Agg hamfemn 11)++1+ 12)--3- 1-)-014 -+)-111 Jie hamfem 1)3024 4)3311 1)0324 .)1142 Esomakjgaodji Orj hamfemn 4)0-14 4)+444 -)0142 .)0003 -2+ Agg hamfemn 1+)0411 1.)3041 11)03-4 10)-033 Jie hamfem 1).43- .)3421 1)-.01 -)1441 EFA Orj hamfemn .)0..1 3)+140 -)+.+- .)++43 .2+ Agg hamfemn 3)4-+0 1.)++14 2)4+00 1.)1143 This happens since the IK procedure restricts the limb segment (bone) length to a constant value over time. The length of each limb segment is calculated when all joints have been estimated using observed marker positions in previous frames. The inter-joint distances in reality are not constant since marker positions are noisy, violating the assumption of a fixed inter-marker pairwise distance. The effectiveness of the proposed method is strongly related to the stability of the inter-joint distance. The proposed bone length constraint is independent and can be easily adapted to most marker prediction methods for CoR estimation. kelfECollision DefierminyfiionR Collision detection has been a fundamental problem in computer animation, physically-based modeling, geometric modeling, and robotics. Since the data used in these examples are captured from a markered optical motion capture system and since the 3D animated humanoids do not have a mesh that defines their external shape, self-collisions are not considered. Nevertheless, self-collisions can be handled using existing techniques, such as [LG98]. ICMC gesults vny Disxussion Experiments were carried out using a 24 camera PhaseSpace motion capture system capable of capturing data at 480Hz [Pha]. The algorithm described in this chapter can process up to 300 limw szgmznt pvirs pzr szxony (using MATLAB). Our datasets comprise real data (i.e. captured data with natural occlusions or occlusions generated by artificial deletion) with more than 10000 frvmzs in each. Our methodology has been tested on various motions including dancing (14 segment body datasets), fast walking (7 segment leg datasets) and boxing (5 segment arm datasets). Figure 4.14 shows 2 of the models that have been used in our experiments. Using real data with occlusions generated by artificial deletion, we are able to calculate the error of the proposed methodologies; the error is measured as the average distance between the observed and the estimated position (for marker and the CoRs) after artificial deletions. The error varies between different instances of marker occlusion. As more markers become available, more information relative to the limb segment is available and thus, a higher accuracy is achieved. IHN botion Xvpturz holutions unyzr bvrkzr dxxlusions (u= (v= Figflre LFILFR ifio of thz moyzls uszy in our zflpzrimzntsC (v) v FI szgmznt humvn woyyA (w) v L szgmznt lofizr humvn woyyC The magnitude of the error reported in the results is given in terms of real world distances. Within this work, our methodology (referred as UKF-VTM) was tested and compared against some of the most popular marker prediction approaches: an EKF model, such as [ACL08], using similar marker constraints as the ones proposed here, a cubic spline interpolation and a real-time extrapolation method with marker position constraints, as reported in [PLH+09]. Using the VTM we took into consideration the velocity, direction and acceleration changes of the markers’ trajectories over time. Certain drawbacks of the simple EKF were overcome by using a more evolved and sophisticated method; the use of a variable turn model gives significant improvements in cases where the trajectories of the markers are variable and have abrupt fluctuations in speed and direction. The proposed system (UKF-VTM), without ap- plying bone constraints, returns an average error (over 20 runs) of 1O296cm in the case of one missing marker, 3O474cm when 2 markers are missing and 8O401cm when all markers are oc- cluded. The corresponding CoR estimation error is 1O082cm in the case of one missing marker, 1O9867cm in the case of two markers and 7O859cm in the case where all markers are not visible to the cameras. Table 4.1 lists and compares the results between each method implemented here under several occlusion scenarios. In general, if the methods are not constrained with the proposed inter-marker constant distance assumption, they fail to track the missing marker paths when the occlusion lasts longer than a time threshold. Table 4.1 also tabulates the worst case scenario (distance between the observed and predicted positions) of marker prediction and CoR estimation. The worst case scenarios usually appear on abrupt changes in velocity and direction of the missing marker during the occlusion period, where the UKF-VTM model requires some time to efficiently track the target. Obviously, the interpolation method has the lowest computation cost, however it is an off-line application. The UKF-VTM framework increased the processing time by 20% compared to the simple EKF, processing 315 limw szgB mznt pvirs pzr szxony in MATLAB; which does, however, still allow real-time implementation. The UKF-VTM method performs better than the other methods looked at here, resulting in the most accurate results; our methodology efficiently recovers good estimates of the missing markers and accurate real-time CoR estimation. A further error reduction was observed when LFP jesfllfis ynd Discflssion IHO lyzle LFJFR Vvzrvgz rzsults (ovzr GE runs) on rzvl yvtv fiith oxxlusions gznzrvtzy wy yzlztionsC Xvsz of onz missing mvrkzr on zvxh limw szgmznt for morz thvn FJEE frvmzsC Entiryly owwluxyx durtiully visivly Chungy (Error wm= (Error wm= aurkyr position EBG4IL DBFII4 ALEBDF 9 Cof position† FBFF4K DBIJFL AK4BKD 9 † whyn aw is upxutyx using thy pryxiwtyx xutuB the missing markers were partially visible to one camera. The error is significantly decreased, by 80% for marker prediction and 75% for CoR estimation, compared to the case where this information was ignored. Table 4.2 states the prediction error for the case of one missing marker on each limb segment when the missing markers are both entirely occluded and visible in just one camera. The difference between true and estimated positions is further reduced when the fixed inter- joint pairwise assumption was taken into consideration. The CoR error, in our methodology, is decreased on average (over 30 runs) by 11O9% in total, 2O96% when the estimated joints are located at the end of the chain and 12O7% when they are positioned between 2 observed positions. FABRIK ensures that the inter-joint pairwise distances are constant over time, thus eliminating the error in the CoR estimation due to unnatural bone extensions. The error reduction is larger in cases where the estimated joints are between 2 observed CoR positions, since both observed joints, in an iterative fashion, constrain the inter-joint distance from the estimated joints. In the case where the estimated joints are positioned at the end of the chain, only one observed joint contributes in the final solution. Table 4.3 presents the average error and achieved error reduction after applying IK to the case of a humanoid model with artificial deletions on 6 markers (over 30 markers in total) on more than 2500 frames out of 8000. At the same time, the processing time was increased by only 5O27%, processing now approximately 300 limw szgmznt pvirs pzr szxony. FABRIK has been applied to all methodologies ensuring that the inter-joint distances will remain unchanged over time; the average CoR error was reduced on average by 12O92% for the EKF, 45O33% for the extrapolation method and 62O8% for the interpolation approach. Figure 4.15(a) shows zoomed examples of the true and predicted x-positions of an occluded marker and the CoR, after incorporating FABRIK, for the case of a single occlusion and a marker visible to just one camera, respectively. It is clear that the occluded marker can be tracked with high accuracy when it is visible in at least one camera and its CoR position can be reconstructed efficiently even if the occlusion period exceeds 1500 frames. Figure 4.15(b) also shows an example of the error variation of all methodologies due to occlusion for the case of 1 missing marker on each limb segment. Clearly, UKF-VTM has the lowest error both on average and in the worst case scenario; this error is further decreased when the marker is visible to just one camera. Figure 4.15(c) reinforces the above showing the cumulative distribution function (CDF) of the estimation error for the case of one missing marker on each limb segment. The x−axis IHP botion Xvpturz holutions unyzr bvrkzr dxxlusions lyzle LFKFR Error rzyuxtion using Invzrsz ‘inzmvtixsC Meocjd Jvemagg Menpgon Ejdion ame gjcaoed ao Ejdion ame gjcaoed di oce eid ja oce ccadi beoreei - ompe ejdio kjndodjin Emmjm (ch) Ccaibe Emmjm (ch) Ccaibe Emmjm (ch) Ccaibe CjM† -)+-13 -)1+1. -)+1.1 PFA(VOM CjM‡ 1)2301 (11)4+% -)+.41 (-)41% 1)2024 (1-)2+% CjM† 2)2144 3).144 2)-10+ Dioemkjgaodji CjM‡ -)3143 (1-)3+% .)240+ (04).0% -)-4.2 (13)4-% CjM† 4)-4.2 4)4..4 .)2-1. Esomakjgaodji CjM‡ -).-++ (40)..% -)3440 (4-)-0% 1)421- (42)1+% CjM† -)0+22 -)3134 -).0+4 EFA CjM‡ -)13.1 (1-)4-% -)0424 (11)14% -)+-1- (14)+-% † hhy CoR wus wulwulutyx using only pryxiwtyx murkyr positionsB ‡ hhy CoR wus worrywtyx using FABRIK, ussuming thut vonys huvy fiyx lyngth ovyr timyB shows the estimation error and the y−axis shows the probability, for y = a, of having an error less than or equal to a. Hence, for example, the probability the estimation error is less than or equal to 1.2cm is approximately 0.57 for the extrapolation and less than 0.4 for interpolation, while it is 0.62 for the case of the UKF-VTM. The median estimation error of the markers using the EKF model is approximately 1.12cm (0.83cm for the CoR), where the corresponding median error for the case of UKF-VTM is approximately 1.09cm (0.75cm for the CoR and 0.12cm when markers are visible to just one camera). Obviously, interpolation produces smooth results since it uses previous and future positions. Our method shows small oscillations or discontinuities between the last predicted position and the first frame when the marker become available. This is expected since we do not use future positions and as the markers become available, we use the observed positions. This phenomenon can be avoided if, instead of using the observed marker position as the final result (after the occlusion), we continue using the UKF framework, having observation states as the true marker positions. In that way, although the average errors of the marker and CoR estimates will show a marginal increase, we will obtain smooth results without oscillations. Extrapolation and interpolation return useful predictions for short-time occlusions but fail to track the marker positions when the occlusion is maintained for extended time periods, es- pecially if markers change rapidly in direction and velocity. In particular, the path followed by the interpolation method does not reflect the actual state of the missing marker, but only con- nects the available positions using a cubic spline shape. In contrast, the UKF-VTM performs fairly well even if markers are missing for long time periods. Figure 4.16 and 4.17 show examples of using our algorithm on real data; in blue are the true (observed by the mocap system) positions of the markers and CoRs and in red the predicted positions. Figure 4.18 is another example that compares the results on the leg model; the left picture presents the results when only the integrated UKF-VTM model is used, and the right picture shows the results when bone length control was incorporated using FABRIK. Note the bone length violation, which is more obvious on the hips. LFP jesfllfis ynd Discflssion IHQ 3000 3100 3200 3300 3400 3500 3600 440 460 480 500 520 540 560 580 Number of frames x− co o rd in at e Marker position over time True positions UKF−VTM Visible to 1 camera EKF Interpolation Extrapolation 3000 3100 3200 3300 3400 3500 800 850 900 950 1000 Number of frames x− co o rd in at e CoR position over time True positions UKF−VTM Visible to 1 camera EKF Interpolation Extrapolation (u= 3000 3100 3200 3300 3400 3500 0 1.0 2.0 3.0 4.0 5.0 Number of frames Er ro r ( cm ) Error Variation (Markers) UKF−VTM Visible to 1 camera EKF Interpolation Extrapolation 3000 3100 3200 3300 3400 3500 0.5 1.0 1.5 2.0 2.5 Number of frames Er ro r ( cm ) Error Variation (CoRs) UKF−VTM Visible to 1 camera EKF Interpolation Extrapolation (v= 0 2 4 6 8 10 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 The estimation error (cm) Th e pr ob ab ilit y Pr (e< =a ) The CDF of the estimation error (Markers) UKF−VTM Visible to 1 camera EKF Interpolation Extrapolation 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 The estimation error (cm) Th e pr ob ab ilit y Pr (e< =a ) The CDF of the estimation error (CoRs) UKF−VTM Visible to 1 camera EKF Interpolation Extrapolation (w= Figflre LFIMFR Eflvmplz shofiing xompvrison rzsults for zvxh mzthoyology for thz xvsz of F missing mvrkzr on zvxh limw szgmzntP oxxlusions fizrz introyuxzy for zfltznyzy timz pzrioys of up to FJEE xonszxutivz frvmzsC ihz lzft xolumn shofis thz rzsults for mvrkzr przyixtions vny thz right xolumn for Xog zstimvtzsA vftzr vpplying [VWgI‘C (v) thz zoomzy xBxooryinvtz pvth ovzr timz (in this zflvmplz notz thvt thz wlvxk vny grzzn xurvzs for thz mvrkzrs vrz zflvxtly ovzrlvyzy)A (w) thz zoomzy zrror vvrivtion ovzr timzA (x) thz XY[ of thz zstimvtion zrror (ovzr FE runs)C Figure 4.19 shows another example of implementation under an extreme case with extended marker occlusions; the top picture shows the results when the fixed inter-joint distance was not imposed and the bottom picture when FABRIK was applied. Obviously, in the first case the skeletal structure was violated since the bones were not restricted to their original lengths; in the second case the results have been improved to a visually more natural shape. IIH botion Xvpturz holutions unyzr bvrkzr dxxlusions Figflre LFINFR Eflvmplzs of implzmzntvtion on rzvl yvtv (aofizr woyy)C dn thz lzft siyz of thz pixturz thzrz vrz mvrkzr oxxlusionsP on thz right siyz of thz imvgz thzsz mvrkzrs hvvz wzzn xorrzxtly zstimvtzyC ihz mvrkzrsA on zvxh limw szgmzntA vrz xlustzrzy vny xolourzy using yiffzrznt xoloursC Experiments demonstrate that the proposed methodology can effectively track the occluded markers with high accuracy, even if they are occluded for extended periods of time, recovering in real-time good estimates of the true joint positions. FABRIK controls and corrects the CoR estimates decreasing the error between the estimated and true positions, thereby enabling real- time skeletal reconstruction. The resulting motion is natural and smooth, without oscillations and discontinuities, resembling that of true human movements. A video included in the supplementary materials demonstrates our methodology and shows its performance with and without incorporating the FABRIK algorithm. LFQ Conclflsion III (u= (v= Figflre LFIOFR Vn zflvmplz of mvrkzr przyixtion vny Xog zstimvtion using thz proposzy mzthoyologyC ihz owszrvzy positions vrz xolourzy wluz vny thz przyixtzy positions rzyC (v) ihz vrtifixivlly yzlztzy yvtvA (w) thz przyixtzy yvtvC (u= (v= Figflre LFIPFR Vn zflvmplz of [VWgI‘ implzmzntvtion for Xog xorrzxtionC Wluz rzprzsznts thz owszrvzy positions vny rzy thz przyixtzy positionsO (v) using only thz intzgrvtzy j‘[Bkib frvmzfiorkA (w) using [VWgI‘ for wonz lzngth xontrolC ICNC Conxlusion This chapter describes a methodology related to the problem of using marker-based optical motion capture data to automatically establish a skeleton model to which the markers are attached. It presents a real-time prediction method that estimates the missing markers and reconstructs the skeletal motion. An Unscented Kalman Filter framework with a variable turn model has been deployed for marker tracking. Information about the missing markers in the current frame inferred from an approximate rigid body assumption has been used for the obser- IIJ botion Xvpturz holutions unyzr bvrkzr dxxlusions (u= (v= Figflre LFIQFR Vn zflvmplz of [VWgI‘ implzmzntvtion unyzr zfltrzmz xvszs fiith zfltznyzy yvtv oxxlusionC (v) shofis rzsults using thz intzgrvtzy j‘[BkibA (w) shofis thz rzsults fihzn [VWgI‘ fivs vpplizy in oryzr to mvintvin thz fiflzy intzrBjoint yistvnxz vssumptionC ihz owszrvzy positions vrz xolourzy in wluz vny thz przyixtzy in rzyC vation states. The proposed marker constraint model is simple and real-time implementable. At the same time, the system takes advantage of the information returned by each single cam- era, regarding the position of the missing markers which are visible to just one camera. The predicted data is then used for real-time joint localisation. Thereafter, the joint positions are re-positioned using Inverse Kinematics taking into account the fact that the inter-joint dis- tance is constant over time. FABRIK is a real-time iterative IK solver which ensures that the bones lengths remain constant over time, resulting in more feasible motion. Our methodology outperforms in accuracy the methods used for comparison in this work. It is able to maintain real-time marker predictions, thus enabling good estimates of the CoR, even in the presence of several marker occlusions on each limb segment. It is reliable even if the limb rapidly changes direction and the occlusions exist for large sequences. The proposed technique could also be used to isolate bad inputs from single cameras; it is a common phenomenon that even one error in large marker inputs can result in several errors per second, pulling the marker position out of the expected path. Thus, this method can be utilised to eliminate pops and jumps from camera switching and errors as they fall outside the predicted positions. Future work will introduce biomechanical constraints to restrict motions to those from a feasible set; however, prior knowledge of the model and joints will then be needed. Also, a hybrid system with low cost inertial measurement units could be used to validate the method proposed here. 5 Hvny eose irvxker A rtOIuRGtKd hand-tracking systems have been widely used in virtual reality systems andthe computer games industry, but due to their complexity and high computational cost, accurately tracking hands remains challenging. In this chapter, we focus on real-time hand tracking and reconstruction using optical motion capture technology. The IK system developed in Chapter 3 is adapted to control the postures of the hand, subject to physiological constraints that restrict the allowed movements to a feasible and natural set. JCFC Introyuxtion In recent years, there has been a growing demand for reliable hand motion tracking systems, a technology used to turn the observations of a moving hand into 3D position and orientation information. Such information can be used for hand gesture recognition; to generate virtual figures for films or computer games; and for human-computer interaction (HCI) including interaction with game consoles. However, building a fast and effective hand pose tracker remains challenging; the high dimensionality of the pose space, the ambiguities due to self- occlusions and the significant appearance variations due to shading, make efficient tracking difficult. Marker-based motion capture has been demonstrated in several interactive systems (includ- ing but not limited to hand interaction) producing results which are highly accurate and easily configurable. There are, however, instances where we do not have many markers available or it 113 IIL ]vny eosz irvxkzr Figflre MFIFR ihz impulsz glovz yzsignzy wy ehvszhpvxz for rzvlBtimz fingzrBtrvxking solutionsC is impossible to attach 3 markers on each limb segment; the large number of markers needed is often prohibitive. It may therefore be infeasible to track the object and reconstruct its skeleton model (i.e. the hand model). Hence, it would be useful to find a new way of capturing the movement of these articulated models using the minimum possible number of markers. Such a method is presented in this chapter, reproducing good estimates of real captured hand motion. Instead of attaching 3 markers on each limb segment, a single marker is attached and captured on each finger (end effector), 1 marker at the chain base (root) and 2 markers at strategic positions to help us define the hand orientation. The markers’ positions are tracked using an optical motion capture system, such as [Pha]. Physiological joint constraints are applied to find the hand palm and to ensure that the hand motion is within a feasible set, giving a smooth and visually natural motion of the hand. An Inverse Kinematics solver (FABRIK) is then incorporated to estimate the remaining joint positions and to fit them to the hand model. In addition, a marker prediction system, similar to the one presented in chapter 4, is applied to deal with cases where the markers are not visible to the motion capture cameras. The results were visualised using a mesh deformation algorithm, driving the animation of the hand according to the underlying hand skeleton. The skeletal fitting and the incorporation of constraints are real-time implementable and the hand motion is tracked smoothly and without oscillations, even with a low frame rate. JCGC gelvtey lork There are many approaches for tracking and configuring the hand model. The hand gesture identification algorithms can be classified into two major classes: glove-based and vision-based methods. In general, glove-based methods are real-time, they are, however, expensive (e.g. P5 Data glove) and detect only a limited set of finger movements with low accuracy. Wang and Popovic [WP09] and Fredriksson et al [FRF08] proposed methods for hand tracking using a single camera and an ordinary cloth glove which was imprinted with a custom pattern; the pose corresponding to the nearest database match was then retrieved. Although this offers a simple, computationally cheap and promising solution, the resulting poses are highly correlated with the training data making these methods less reliable than optical mocap systems. The vision-based methods, on the other hand, are more accurate, but they have problems with occlusions, noise and spurious data. Lien and Huang [LH98] proposed a hand model MFJ jelyfied oork IIM together with a closed-form Inverse Kinematics solution for the finger fitting process. The 3D positions were obtained using colour markers and stereo vision, and the finger poses were chosen using a search method which finds the best solution amongst all possible positions. While this method is implementable in real-time, it is complex and can fail when different size models are used. Park and Yoon, [PY06], also used a marker-based motion capture system; a LED-glove has been employed to produce interactions in multi-model display environments where the gestures were recognised and classified using hidden Markov chains. De la Gorce et al [GPF08] proposed a 3D hand tracking approach from monocular video. A formulation for exploiting both shading and texture is presented, which is able to handle the problem of self- occlusions. However, due to the high dimensionality of the human hand, this method might suffer from high computational complexity and singularities. Several papers [RK94, SMC01, MTHC03, STTC06] focus on statistical methods, such as an Unscented Kalman Filter and a Hierarchical Bayesian Filter, to track the hand motion. These statistical methods approximate the posterior by a single Gaussian and update these approximations via a linearisation of the measurement process. [STW07] employed a mean shift embedded particle filter for visual tracking; they incorporate the mean shift optimisation into particle filtering to move the particles to local peaks in the likelihood. However, such methods are still far from real-time, therefore limiting their use. In [SMFW04, DDHF06], a bare-hand tracking approach is implemented using edge data detection and silhouettes to identify the pose of the hand. Sudderth et al, [SMFW04], used a nonparametric belief propagation (NBP) algorithm for tracking the hand poses and kinematic constraints; in that way, they were able to handle cases with self-interactions and self-collisions. Nevertheless, these methods have high computational cost making it difficult to use them for real-time interactions in control applications. In contrast, [LGPW98, DMR06, SK07] achieve interactive speeds using bare-hand tracking systems at the cost of resolution and scope. Cerveri et al, [CDML+07], utilised a kinematic model using a multi-camera system and markers, which consists of a hierarchical chain and rigid body segments. Limitations relative to the joint rotational and orientational constraints were taken into consideration to restrict the motion to natural postures. In [FAT05] and [CFAT07], vision-based hand shape estimation methods were introduced using shape features acquiring from camera images. The extracted features were then used to approximate the hand’s state and the local state of each finger was estimated using Inverse Kinematics and physical hand constraints. Finally, Kaimakis and Lasenby [KL07] used a set of pre-calibrated cameras to extract the hand’s silhouette as a visual cue. The 2D silhouette data is then modelled as a conic field and physiological constraints are imposed to improve the reliability of the hand tracking [KL09]. A prototype version of the hand model methodology proposed in this work was published in [AL10b]. That was a first step towards an effective real-time hand motion tracking; however, most of the currently implemented physiological constraints were ignored, reducing the final reconstruction quality. In addition, the hand was not visualised using a mesh deformation algorithm, thus it was difficult to observe the differences in shape as well as the error on the resulting pose. IIN ]vny eosz irvxkzr R End effector Joint Base root Palm plane 1,iF 2,iF 3,iF 4,iF 5,iF 1 i 2 i 3 i 4 i 5 i Extra Markers P Q Figflre MFJFR ihz hvny’s moyzl gzomztry uszy in our implzmzntvtionC PhaseSpace, [Pha], have designed an impulsz glove with attached markers (see figure 5.1) for real-time finger-tracking solutions. This is similar to the configuration we will consider in this chapter. JCHC Vrtixulvtey Hvny boyel Human motion is typically represented as a series of different configurations of a rigid multibody mechanism consisting of a set of segments connected by joints. These joints are hierarchically ordered and have one or more degrees of freedom (DoF). The DoFs describe the rotations relative to their parent joints up to the root joint, for which the position and orientation are represented with respect to a reference coordinate system. Most motion capture systems reconstruct figures by tracking several markers placed over the body of the performer. Hence, in order to configure a human pose, it is important to locate the end effectors in strategic positions as they are more easily specified by an animator and tracked by mocap systems. It is assumed that the hand geometry, meaning the initial joint configuration of the hand, is known a priori. An example of a hand model is graphically represented in figure 5.2. The proposed hand model consists of 25 joints and has in total 25 DoFs. The end effector positions are captured using an optical motion capture system, such as PhaseSpace [Pha]. Using inverse kinematics, we then tracked and reconstructed the hand poses over time. The markers are identified (e.g. in PhaseSpace, each LED marker is pulsed at a different frequency) so that it is known a priori on which finger each marker is placed. It is also important to know the orientation of the hand in order to efficiently incorporate constraints. This can be achieved by attaching 2 extra markers at specific positions, p and q, on the back of the hand (reverse palm). Assuming that the palm is always flat, we can find the plane describing the orientation of the hand using p, q and the position of the base root, r, which also lies on the palm plane. For simplicity, markers p and q can be placed at the joint positions F1P- and F4P- respectively. A factor which should also be considered is the noise on the marker data. The motion of MFL an–erse cinemyfiics IIO the markers relative to the skin and the motion of the skin relative to the underlying bone constitute the most important causes of noise. In addition, noise due to miscalibration of the optical system and the optical measurement noise might affect the accuracy of the marker positions. The precision of the system is also highly related to the markers’ positioning; the markers should be carefully placed at the end positions of the finger as well as at the root in order to fit to the hand geometry. JCIC Inverse Kinemvtixs It is time-consuming for an animator to manually set all the DoFs of a virtual character. It is therefore more sensible to use a simulation mechanism, such as an Inverse Kinematics (IK) solver, to situate limbs according to their known end effector positions. The IK techniques require only positions and orientations of certain joints, usually named end effectors, to be specified by the animator and the remaining DoFs are automatically determined according to criteria that depend on the IK variant. For the purpose of this study, FABRIK is chosen due to its efficiency, implementation sim- plicity and low computational cost. As we have seen in Chapter 3, FABRIK is a flexible IK solver which supports most of the existing joint types, is able to solve problems with multiple end effectors and targets, and can handle cases with closed loops. JCICFC ihe hvny moyel Before employing the IK solver, it is crucial to find the fingers’ orientations, the chain roots and the end effectors for each chain; the target positions are assumed to be known since they are tracked by the motion capture system. The procedure is simple. Firstly, we estimate the hand orientation; thereafter, we calculate the palm joints and the finger orientations at each time step. When each finger orientation is known, the finger joints at the previous time step are translated and rotated in such a way that all joints belong to the current finger plane. Finally, a constrained version of FABRIK, with rotational limitations, is incorporated to fit the joints of each finger. This procedure is given in detail in the following paragraphs. The first step is to find the hand orientation; hence, by accepting that the hand plane Φx is similar to the palm plane and that the markers p, q and r are lying on that plane, the hand orientation, meaning the plane Φx, can be estimated. Therefore, e = 1 2 ( p-n+ 2p+−n˜) f = 1 2 ( q-n+ 2q +−n˜) (5.1) R = 1 2 ( r-n+ 2r +−n˜) where e , f, and R are the 5y null vectors representing points p, q and r respectively, and n IIP ]vny eosz irvxkzr Marker positions Joint P Q R p q 1 ! 2 ! nRQPx """#$ Figflre MFKFR ihz pvlm plvnz xonstrvintsO thz hvny plvnz xvn wz xvlxulvtzy using thz mvrkzr positions e A f vny RA vxxzpting thvt thz mvrkzrs liz on thvt plvnz vny thvt thz hvny vny pvlm plvnzs vrz similvrC ihz rzst of thz pvlm joints xvn wz zstimvtzyA vssuming thvt thz intzrBjoint yistvnxzs rzmvin xonstvnt ovzr timzA wy intzrszxting thz sphzrzs Σp vny Σq fiith xzntrzs vt thz mvrkzr positions e vny f vny rvyii of thz yistvnxz wztfizzn thzir xzntrz vny thz joint position fiz vrz looking forC and n¯ are the null vectors in CGA, as described in section 2.2. The plane Φx is equal to Φx = e ∧f ∧R ∧ n = 〈〈〈ef〉-R〉. n〉4 (5.2) Note that the form given on the right hand side of 5.2, and other relevant equations, is useful for implementation purposes. Cylcfllyfiing fihe pylm joinfis The next step is to incorporate constraints to obtain other palm joints. Thus, by assuming that the inter-joint distances (for the joints FiP1 where i = 1P O O O P 5 and FjP- where j = 1P O O O P 4) are fixed over time and that all these joints lie on the palm plane, we can easily locate them using basic geometric entities such as planes, circles and spheres. An example of palm constraints is given in figure 5.3. For instance, the joint position we are working on can be estimated by intersecting the spheres with centres being the marker positions p and q and radii being the distance between the marker and that joint position (taken from the model). Therefore, find the sphere with its centre at the marker position e and radius equal to the distance between the marker e and the joint we are working on Σp = ( e − 1 2 /-1n ) I (5.3) where / is the sphere radii. Similarly, find the sphere with centre the marker position f and radius equal to the distance between the marker f and the joint we are working on Σq = ( f− 1 2 /--n ) I (5.4) MFL an–erse cinemyfiics IIQ 1 2, k i F 1 3, k i F 1 4, k i F 1 5, k i F 1 ! k i k i F 2, k i F 5, k i Frame 1 k Frame k Figflre MFLFR ihz joints’ positions vt timzs k − 1 vny kC Evxh fingzr joint vt timz k − 1 nzzys to wz rotvtzy in suxh v fivy thvt vll joints of thvt fingzr liz on thz plvnz of thz xurrznt frvmz kC The intersection of the two spheres gives a circle or a single point or no intersection. Thus, the meet between the two spheres is given by, X = Σp ∨ Σq = [〈ΣpΣq〉-]∗ (5.5) • if X- S 0, then X is a circle. In that case, the possible solutions are given by intersecting the circle X and the palm plane Φx W = X ∨ Φx = [〈XΦx〉.]∗ (5.6) – if W- S 0, the meet between X and Φx gives two points which can be extracted via projectors, as given in section 2.2.5. The new joint position is assigned as the point that is closer to the previous joint position (at time k − 1). – if W- = 0, the intersection is a single point m = WnW. – if W- Q 0, the intersection does not exist. For that instance, the new joint position is then taken as the nearest point on circle, X, from the previous joint position (at time k − 1, see Appx A.1 for the solution). • if X- = 0, the intersection is a single point m = XnX. • if X- Q 0, the two spheres do not intersect. In that case, the final joint position is given by averaging the distance between the two markers x = (p+ q)R2. Cylcfllyfiing fihe finger joinfis In order to estimate the finger joints, we need to find the finger planes Φi, for i = 1P O O O P 4. Each Φi can be calculated using the known joint positions FiP-, the marker positions FiP0 and by assuming that they are perpendicular to the palm plane Φx (note that this does not hold for the thumb plane Φ0). Since both points from each finger are known (the motion capture system tracks the end effector positions FiP0 and the finger roots FiP- lie on the palm plane with constant distance from the attached markers p and q, as explained in previous paragraphs), each finger plane can be estimated at the current time frame. The vector that is perpendicular IJH ]vny eosz irvxkzr k i F 2, k i F 5, k i k i F 5, ˆ k i F 3, ˆ k i F 4, ˆ Frame k Figflre MFMFR ihz xurrznt joint positionsA vftzr rotvting thzm in oryzr to liz on thz xurrznt fingzr plvnz Φki C ihz prowlzm of orizntvtion is thzrzforz solvzy vny [VWgI‘ xvn thzn wz utiliszy vssuming thvt thz root of thz xhvin is F ki;2A thz zny zffzxtor is thz point Fˆ k i;5 vny thz tvrgzt is thz xurrznt mvrkzr position F ki;5C to the hand plane Φx is given by nˆ = Φ∗x − 1 2 (Φ∗x · n¯)n (5.7) as explained in section 2.2.5. The finger planes can then be calculated as Φi = FiP- ∧ FiP0 ∧ nˆ ∧ n = 〈〈〈FiP-FiP0〉- nˆ〉. n〉4 for i = 1P O O O P 4 (5.8) The thumb orientation Φ0 can be estimated using the marker position F0P4, and the joint positions F1P- and F0P- that lie on the palm, assuming that when the thumb bends to the ventral side of the palm, it always points at the joint F1P- (approximately true in practice). The next step is to estimate the rotation between the previous and the current frame of each finger plane. This can be done using CGA and rotors; the rotor R which expresses the rotation between the plane in the previous frame and the plane in the current frame, for each finger, can be found using either the eroxrustzs formulation [Hor87] or the GA equivalent given in [LFLD98]. Then each finger joint at time k − 1 is translated and rotated in such a way that all joints of a given finger lie on the plane of the current frame k, as demonstrated in figures 5.4 and 5.5. Hence, Fˆ kiPj = RF k−1 iPj R˜ (5.9) where i = 1P O O O P 4 and j = 3P 4P 5 (except for the thumb where i = 5 and j = 2P 3P 4). All joints now lie on plane Φki . Lastly, FABRIK is applied to each finger chain, assuming that the root of the chain is F kiP-, the end effector is the rotated point Fˆ k iP0 and the target is the current marker position F kiP0, as shown in figure 5.5. The inter-joint distances are constant over time, thus, for computational efficiency, they can be calculated and stored at the first frame. The resulting posture can be further improved in accuracy and naturalness by incorporating constraints subject to the physiological model of the hand, taking into account the hand, fingers, muscle, skin and individual joint properties. MFM hhysiologicyl Consfiryinfis IJI lyzle MFIFR ]vny joint xonfigurvtion DjA Mjoaodjiag(fi Mjoaodjiag(y Jmdeioaodjiag (debmeen) (debmeen) θ1 θ3 θ2 θ4 Fi;1 i 8 1P ...P 4 1 ( ( ( ( Ij ordno Fi;1 i 8 0 - -+ -+ .+ 4+ Ij ordno Fi;2 i 8 1P ...P 4 - 1+ 30 10 10 Ij ordno Fi;2 i 8 0 - 0 .+ 1+ 1+ Ij ordno Fi;3 i 8 1P ...P 0 1 1+ 40 ( ( Ij ordno Fi;4 i 8 1P ...P 4 1 1+ 4+ ( ( Ij ordno Ojoag -0 2 4 1 3 Figflre MFNFR Grvphixvl rzprzszntvtion of thz vnglzs θ1P OOOP θ4 fihixh yzfinz thz rotvtionvl xonstrvints of zvxh jointC JCJC ehysiologixvl Constrvints In addition to the basic rotational and orientational joint constraints, presented in [AL10b], it is important to incorporate motion limitations based on the model properties. Kaimakis and Lasenby, in [KL09], proposed a physiological model to restrict the hand poses according to the hand anatomy. That was the first paper which utilised physiological constraints taking into account the hand, finger and joint properties. In this study, the physiological hand constraints are defined using the same terminology as in [KL09], and each of these is outlined below in the order in which they appear in [KL09]. JCJCFC Inertiv The first physiological constraint discussed in [KL09] is inzrtiv, a limitation correlated with the dynamics of the articulated structure. For implementation, it is assumed that all moving parts of the hand skeleton have similar velocity and acceleration at different time periods. Obviously, the kinematic movement can be divided into two classes, the spatial velocity of the hand’s root giving the translation of the hand, and the local angular velocity of each bone. The problem of inertia, in this study, is considered as solved since the marker positions are tracked by the optical motion capture system; however, this information can be useful in predicting the missing marker positions when the latter are occluded by elements of the scene. When the hand moves, the finger with the missing marker will have similar direction, velocity and acceleration to that of the hand. Thus, a better approximation of the filtering observation state can be achieved. IJJ ]vny eosz irvxkzr (u= (v= Figflre MFOFR Eflvmplzs of unnvturvl hvny posturzs yuz to violvtion of thz (v) flzflion vny vwyuxtionA (w) rigiyity propzrtizs of thz fingzrC JCJCGC Fleflion The design and physical restrictions of the human hand mean that fingers can move to the ventral side, but cannot move in any other direction. The movements that a hand can undergo are therefore restricted in terms of flzflion and zfltznsion. JCJCHC Vwyuxtion Another family of limitations, caused by hand physiology, are the vwyuxtion and vyyuxtion constraints. These constraints control and limit the amount of sideways motion. In this study it is assumed that finger orientation is highly correlated with that of the hand palm. The hands posture constraints, related to flzflion and vwyuxtion, can be incorporated directly into FABRIK as rotational and orientational constraints, in the same way as described in section 3.4.1. For instance, a bone rotation can be limited by factorising it into two rotations: one “simple rotation” that moves the bone to its final direction vector and one that represents the twist around that final vector. The hand model studied here consists of 25 joints and has in total 25 DoFs. Table 5.1 lists the degrees of freedom for each joint as well as its rotational and orientational limits. Figure 5.6 presents the angles θ1P OOOP θ4 which define the rotational limits of each joint FiPj . Fingers do not twist, thus only rotational constraints are applied, locking the joint orientation to be identical to that of the palm (apart from the thumb). FABRIK easily supports rotational and orientational constraints; the main idea is the re- positioning and re-orientation of the target to be within an allowed range bound. This can be accomplished by checking whether the target is within the valid bounds, at each step of FABRIK, and if it is not, to guarantee that it will be moved accordingly. The allowed range of motion is defined by the angles θ1P OOOP θ4, which represent the minimum and maximum allowed MFM hhysiologicyl Consfiryinfis IJK (u= (v= Figflre MFPFR Vn zflvmplz shofiing thz intrvyigitvl xorrzlvtion fzvturzC (v) ihz intrvyigitvl xorrzlvtion xonstrvint is violvtzyP zvzn if thz rotvtionvl vny orizntvtionvl xonstrvints vrz svtisfizyA thz posturz of thz hvny is not nvturvl sinxz it is impossiwlz to wzny thz yistvl phvlvngzs fiithout flzfling thz intzrmzyivtz vny proflimvl phvlvngzsA (w) thz xorrzxt posturz of thz hvny fihzn thz intrvyigitvl xorrzlvtion of thz fingzr hvs wzzn tvkzn into vxxountC rotation of each joint about the x and y-axes, respectively. More information on how FABRIK works, how joint limitations can be incorporated and how it can be extended to problems with multiple end effectors and targets, can be found in chapter 3 and [AL10d]. Figure 5.7(a) shows an example where the flexion and abduction constraints are not satisfied; the forefinger was erroneously rotated where the little finger was bent in an inappropriate direction and angle. JCJCIC Intrvyigitvl xorrelvtion As well as the limitations due to inertia, flexion and abduction, several posture restrictions are caused by the muscles of the hand. For instance, the phalangeal flexion in particular fingers is influenced by tendinous synapses with more than one phalanx of that finger. Therefore, it is clear that the muscle contraction and phalangeal flexion are not fully independent, but there is an inter-connection between them. [KL09] introduced the intrvyigitvl xorrzlvtion constraint that is responsible for the inter-finger connections caused by certain tendons. In order to cope with these motion restrictions, we add an extra step; after the completion of the IK operation, a check mechanism is activated to identify whether the finger has a natural posture. If a violation of the transdigital correlation constraint is detected, the IK solver is repeated but this time with rotational constraints being applied to ensure that the flexion is uniformly distributed to all finger joints. However, in order to apply this constraint, the joint angles must be calculated, resulting in an increase in computational cost. Figure 5.8 shows an example where the intradigital correlation constraint is not satisfied; an unnatural pose of the hand is produced, even if the rotational and orientational constraints for individual joints are satisfied. IJL ]vny eosz irvxkzr (u= (v= (w= (x= No Correlation Low Correlation High Correlation Moving Finger (y= Figflre MFQFR ihz linkzy pvirs of wonzs fihixh shvrz v trvnsyigitvl xorrzlvtion movzmzntC (v) thz littlz fingzr vny thz vffzxtzy nzighwouring fingzrsA (w)A (x) vny (y) thz ringA miyylz vny inyzfl fingzrsA rzspzxtivzlyA fiith thz zffzxt of thzir flzflion on thz nzighwouring fingzrsA (z) thz thumwC ihz thumw’s movzmznt is inyzpznyznt of thz othzr fingzrs sinxz it is yirzxtly xonnzxtzy to thz trvpzziumC ihz moving fingzrs vrz highlightzy in wluzA thz highly xorrzlvtzy fingzrs in rzyA thz fingzrs hvving lofi xorrzlvtion to thz moving fingzr in grzzn vny finvllyA thz fingzrs fiith no xorrzlvtion vrz xolourzy in light grvyC JCJCJC irvnsyigitvl xorrelvtion Beyond the fingers’ intrvyigitvl xorrzlvtion constraint discussed in 5.5.4, the fingers also share trvnsyigitvl xorrzlvtions. In particular, [KL09] argues that certain ligaments and muscles in- teract to cause an amount of flexion to be transmitted across neighbouring fingers, as shown in figure 5.9. An exception to the transdigital correlation is the thumb, which moves indepen- dently of other fingers since it is directly connected to the trapezium. Figure 5.9 indicates the linked pairs of bones which share a correlation movement, subject to transdigital correlation. An example showing violation of the transdigital correlation is given in figure 5.10(a) where even if both individual rotational and orientational constraints are satisfied, the resulting hand posture remains abnormal. Figure 5.10(b) shows the physiologically correct hand configuration for this specific pose. The transdigital correlation constraint is assumed to be solved by the optical motion capture system, since each finger is tracked individually by labelled markers. However, this information MFN eissing eyrker hredicfiion IJM (u= (v= Figflre MFIHFR Vn zflvmplz zflplvining thz trvnsyigitvl xorrzlvtion of thz hvnyC (v) thz fingzr flzflzs fiithout vffzxting its nzighwouring fingzrsA wrzvxhing thz trvnsyigitvl xorrzlvtion fzvturz vny proyuxing vn unnvturvl posturzA (w) v rzvlistix hvny posz vftzr tvking into vxxount thz trvnsyigitvl xorrzlvtion wztfizzn nzighwouring fingzrsC finds applications in cases where the marker position is missing due to occlusions by elements of the scene, thus contributing to better estimates of the observation vectors. JCJCKC Frixtion Finally, [KL09] introduces frixtion, a hand property associated with the nature of the skin that restricts hand movements. For instance, when frictional forces are applied to a finger, they cause motion that is transmitted to other fingers. A clear example of the friction feature is given during the formation of the fist. Since, in this work, each finger position is tracked using a mocap system, this feature finds application only during the marker prediction algorithm returning better estimates of the missing marker positions. JCKC bissing bvrker ereyixtion During motion capture, we encountered cases where markers were occluded or just not visible to the cameras. That is a common problem in optical motion capture since markers are self- occluded or occluded by other fingers or elements in the capture environment. To cope with the missing marker problem, a marker prediction mechanism is employed using a VTM-UKF model, similar to the one described in chapter 4 and [AL10c], where the observation vector is constrained subject to inferred information from the hand geometry and the available marker positions. Physiological hand constraints have been incorporated within the filter to give good estimates of possible positions of the observation state at each time period, such as those described in section 5.5. The VTM-UKF filter can also handle the negative effects of the IJN ]vny eosz irvxkzr (u= (v= Figflre MFIIFR (v) ihz truz hvny poszA (w) thz mvrkzrs vs szzn from thz motion xvpturz systzmC marker data noise. JCLC Eflperimentvl gesults Experiments were carried out using a 10 camera PhaseSpace motion capture system, capturing data at 100Hz [Pha]. The implemented methodology was able to process up to 70 frames per second. Runtimes were measured with custom MATLAB [MAT] code on a Pentium 2 Duo 2.2GHz. Our dataset comprises marker motion capture data. Data captured using colour video cameras are also used to compare the reconstruction quality between the estimated and the true hand postures. The reconstructed hand postures were visualised using a mesh deformation algorithm. Figure 5.11 shows an example of the hand with attached markers and how the markers are seen from the motion capture system. JCLCFC besh yeformvtion A mesh deformation algorithm is employed to visualise the movements of the underlying hand skeleton in order to compare the resulting animations with the true hand poses. Animating an articulated 3D character requires manual rigging to specify its internal skeletal structure and to define how the input motion deforms its surface. [BP07] and [WL08] proposed mesh deformation algorithms driven by animation of an underlying skeleton, named bone-heat and bone-glow respectively. The articulated hand is automatically assigned a per-vertex and per- bone weighting given only by an underlying skeleton. In this work, we used the version of bone-heat implemented in Blender [Ble]. Figure 5.12 provides a representative illustration of implementing the mesh deformation algorithm; the mesh and armature representing the hand is automatically associated using the bone-heat algorithm. JCLCGC Vnvlysis of results The proposed method is simple and has low computational cost, meaning it is real-time imple- mentable. It requires 1.43msec per frame for tracking and fitting 25 joints, hence processing MFO Experimenfiyl jesfllfis IJO Figflre MFIJFR Eflvmplz of simplz linzvr wlzny skinning sxhzmz vpplizy using thz fizights from wonzB hzvtC ihz fizight vssignzy to zvxh vzrtzfl hvs wzzn inyixvtzy using v grvyvtion from wluz to rzy to inyixvtz thz rvngz [0P 1]C lyzle MFJFR Vvzrvgz timz nzzyzy vt yiffzrznt frvmz rvtzsC Time per frame Frames per (msec) second Frame rate at 100Hz 1.4311 69.8 Frame rate at 10Hz 1.6251 61.5 Frame rate at 3Hz 2.2287 44.8 on average 70 frames per second when the data frame rate is high, and more than 40 frames per second for a low frame rate dataset. The rotational and orientational constraints ensure that each finger movement remains normal without showing asymmetries, or irregular bends and rotations. In addition, the physiological constraints restrict the results to anatomically correct postures, subject to the hand, muscle and skin properties. The implemented system can smoothly track the hand movements, resulting in visually natural motion without abnormalities, oscillations and discontinuities. The reconstruction quality can be checked visually by comparing the generated 3D hand animations with the data captured using a colour video camera, as seen in figure 5.13; our system is precise, producing postures which meet the hand’s physiological model restrictions and are very close to the true hand poses. Figure 5.14 shows an example of continuous hand pose tracking and reconstruction using a dataset captured at a frame rate of 10Hz. In this example, the hand flexes to its ventral side, to form a fist. It is difficult to illustrate the reconstruction quality in still images, but the resulting motion does not suffer from oscillation or discontinuities and each finger smoothly moves to the target. We also investigate the performance of the system under different frame rates; clearly, the reconstruction quality is better when the frame rate becomes higher. However, even at a low frame rate (3 frames per second) the reconstruction quality of our methodology remains IJP ]vny eosz irvxkzr (u= (v= (w= (x= Figflre MFIKFR Vn zflvmplz of hvny rzxonstruxtion using our mzthoyology vt v frvmz rvtz of FEE]z frvmzC (v) kizfi of thz hvny from gGW xvmzrv 1A (w) v yiffzrznt vizfi of thz hvny from gGW xvmzrv 2A (x) thz rzxonstruxtzy skzlzton vny (y) thz finvl visuvliszy posturzC ihz rzsulting poszs vrz visuvlly nvturvl vny wiomzxhvnixvlly xorrzxtC sufficient, delivering smooth and visually natural results. The time needed for the IK solver to fit the joints to the model also varies for data captured at different frame rates; by reducing the frame rate, the distance between the target and end effectors is increased, thus more computational time is required from FABRIK to track the target positions. Table 5.2 lists the average time needed per frame to track the hand at different frame rates. Despite the apparent accuracy in performance, the resulting postures of our approach are not unique; several possible poses could result from the 3D articulated hand tracking. However, the advantages of this method are its efficiency and ability to return natural and feasible motion, which meets the user constraints, with low computational cost. It is also important to note here that FABRIK results in poses which are closely related to previous states. Therefore, the final joint configuration might be different when the IK problem is solved with the end effectors in different initial positions but with similar final states. Nevertheless, these differences are minimal causing only a small decrease in performance. A video showing examples of implementation using different frame rates is included in the supplementary materials. MFP Conclflsions ynd Fflfiflre oork IJQ Figflre MFILFR Vn zflvmplz of xontinuous hvny posz trvxkingA vt v frvmz rvtz of FE]zP in this zflvmplz thz hvny flzflzs to its vzntrvl siyz to form v fistC JCMC Conxlusions vny Future lork In this chapter we have presented a system that can track a human hand of 25 DoFs relying on optical motion capture data; one labelled optical marker is attached on each finger, treated as end effectors, and 3 more markers are placed at strategic positions on the hand reverse-palm to help us identify the root and orientation of the hand. Physiological constraints relevant to the hand model, skin and muscle properties are incorporated, ensuring that the resulting movements remain within a feasible set. FABRIK, a real-time inverse kinematics solver, is employed to fit the remaining joint positions subject to the proposed physiological restrictions. The problem of missing data, due to marker occlusions, is solved using a marker prediction methodology similar to the one presented in chapter 4, recovering the missing positions in real-time. Finally, bone-heat, a mesh deformation algorithm, is used to visualise the results for evaluation and comparison. The proposed methodology produces smooth and natural hand postures over time; the required processing time remains low enabling an effective real-time hand motion tracking and reconstruction system. The results are precise, producing visually natural and biomechanically correct movements. The system can process up to 70 frames per second; even with a low capture frame rate, the proposed methodology tracks the hand motion smoothly, without oscillations or discontinuities and with high reconstruction quality. In future work, a more sophisticated model will be implemented which takes into considera- tion, in addition to the joint rotational and orientational restrictions, skin related constraints and constraints related to the rigidity of the hand model. The rigiyity feature of the hand was not investigated in this work since the data was captured from a markered optical motion capture system and the 3D animated hand does not automatically have a mesh that defines its external shape. As the hand is the most mobile part of the human body, we expect a consider- IKH ]vny eosz irvxkzr able degree of interaction between fingers, despite the limitations already discussed. Rigidity concerns such interactions, where different fingers may self-intersect, thus causing unnatural postures. While some of the constraints already discussed will limit such self-intersections, there are instances where extra movement restrictions must be applied. This problem is also known as self-collision and has been tackled in several papers, such as [KL09] and [LG98]. Figure 5.7(b) shows an example where two fingers collide disproportionately; by taking into account the rigidity of each finger, problems similar to hand collisions can be avoided. 6 Conxlusions vny Future lork M UtOvGtKd by the need to develop effective real-time inverse kinematics solutions, thisthesis intends to provide insights into existing computer vision techniques and presents a novel iterative IK solver. Also, we investigate real-time marker prediction and centre of rotation estimation techniques as well as a hand pose tracker for real-time visual interactions. In this chapter, we summarise the findings from previous chapters, we describe the main contributions of this thesis and we propose future directions and applications. KCFC Contriwutions CGA is a mathematical framework that offers a compact and geometrically intuitive formula- tion of algorithms and an easy and immediate computation of rotors. Rotors are simpler to manipulate than Euler angles and avoid the problem of gimbal lock1. It also simplifies the mathematical model since basic entities, such as spheres, lines, planes and circles, are simply represented by algebraic objects. Thus, CGA gives us the ability to describe algorithms in a geometrically intuitive and compact manner, making it suitable for applications in engineering, computer vision and robotics. 1Gimvul lowk is u wommon provlym ussowiutyx with Eulyr unglys unx owwurs vywuusy two ufiys vywomy ulignyx xuring rotutionul opyrutions, proxuwing unyfipywtyx vyhuvior sinwy ony xygryy of fryyxom is lostB 131 IKJ Xonxlusions vny [uturz lork Forwyrd Ynd Zyckwyrd jeyching an–erse cinemyfiicsR Inverse Kinematics meth- ods are used to control the postures of articulated bodies in frame animation production. IK finds applications in several areas such as robotics, computer animation, ergonomics and the computer games industry. However, most of the currently available methods suffer from high computational cost and production of unrealistic poses. Chapter 3 presents a review of algorithms related to the IK problem; it also introduces a novel iterative IK solver, named For- ward And Backward Reaching Inverse Kinematics (FABRIK). Firstly, this chapter concerns the articulated body model describing the human joint types, and gives a brief introduction to human models and motion; it also considers the most popular numerical solutions to the IK problems, such as the Jacobian family of methods (Pseudo-inverse, Transpose, DLS, SVD- DLS, SDLS), the Newton family of methods (e.g. BFGS), methods that solve the IK problem from a control prospective (FIK), sequential monte carlo methods and methods which learn the space of meaningful shapes from example meshes. Heuristic iterative approaches, such as CCD and its extension (IIK), as well as IK solvers that solve the IK problem in a sequential fashion, are discussed. Lastly, the FABRIK algorithm is compared against the most popular IK solvers and under several different conditions: (a) with and without manipulator constraints, (b) with single and multiple end effectors, (c) tracking moving targets and (d) when the tar- get has significant distance from the end effector. Several tests and comparisons have been implemented between the IK algorithms regarding their computational cost, processing time, conversion error, the number of iterations needed to reach the target as well as how visually realistic the resulting postures are. FABRIK is the first algorithm that uses an iterative method with points and lines to solve the IK problem. Thus, no rotational angles or matrix calculations are needed. It divides the problem into 2 phases, a forward and backward reaching approach, and it supports most of the joint types by repositioning and re-orienting the target at each step. It does not suffer from singularity problems, it is easy to implement and computationally efficient. No pre-recorded motion database is necessary, thereby avoiding the need for extra memory. An efficient approach of how joint constraints can be embedded to the FABRIK algorithm is also illustrated, restricting the resulting poses to a naturally feasible set. FABRIK is able to solve problems with closed loops and can be easily expanded to treat problems with multiple end effectors, meaning that it is able to support complex models with multiple kinematic chains. It is experimentally shown that FABRIK requires on average fewer iterations to reach the target than any other IK method considered here, both with constrained and unconstrained kinematic chains. It produces visually realistic postures, with and without constraints, reaching the desired position with the lowest computational cost. It executes in real-time and has the best performance on tracking a moving target and produces natural movements without oscillations and discontinuities. The FABRIK algorithm can be applied to a variety of problems, such as visual modelling, virtual creature animation, game consoles, rehabilitation medicine etc. Some examples of FABRIK applications are demonstrated in chapters 4 and 5. NFI Confirizflfiions IKK eyrker hredicfiion ynd Cenfire of jofiyfiion EsfiimyfiionR Chapter 4 includes algo- rithms related to the problem of using marker-based optical motion capture data in order to automatically establish a skeleton model to which the markers are attached. The experimen- tal framework was described, including the cameras used for the experiments, the association between the markers and the limbs, and how the marker data was clustered into groups corre- sponding to the limb segments to which they were attached. We also investigated the problem of fitting skeletal models to marker-based optical motion capture data. Using the well known eroxrustzs formulation, we showed how to establish estimates of the relative orientation of a limb in each frame relative to a reference frame. Thereafter, a method for real-time estima- tion of the joint locations and identification of the optimal skeleton was implemented. That method takes advantage of the approximation that all markers on a segment were attached to a rigid body, acquiring the skeleton model from a stream of motion capture data. However, a common issue for motion capture, even when multiple cameras are used, is marker occlusion by elements of the scene, leading to missing data. In order to unambiguously establish the marker position, each marker must be visible to at least two cameras in each frame. Section 4.5 introduces a prediction method which copes with such instances. An Unscented Kalman Filter with a variable turn model has been deployed for marker tracking. Information on the missing markers in the current frame were inferred from an approximate rigid body assump- tion, which has been used for the observation states. With a continuous stream of accurate 3D data, we were able to perform real-time CoR estimation and produce skeletal information for visual performance feedback. Without assuming any skeleton model, we took advantage of the fact that, for markers on a given limb segment, the inter-marker distance is approximately constant. Also, the proposed system uses information about the missing markers which are visible to a single camera, reducing the marker position estimation error to a minimum. To the best of our knowledge, this is the first time that such information has been used for real-time marker prediction in optical motion capture. The joint estimations were further improved by employing the FABRIK algorithm to ensure that the inter-joint distances remain constant over time. FABRIK simply re-positions the CoRs and guaranteeing that the bones lengths will not change during the marker prediction period. This was done with an added computational cost of only 5O27%. Experiments demonstrate that our methodology outperforms, in terms of accuracy, the methods used for comparison and effectively recovers good estimates of the true positions of the missing markers. It is able to maintain real-time marker predictions for extended time periods, even in the presence of several marker occlusions, thus enabling good estimates of the CoR. The method is reliable even if the limb rapidly changes direction and provides precise estimates of the markers and CoR locations even on data with large sequences of missing markers. The resulting motion is natural and smooth, without discontinuities or oscillations. This is the first work which uses an IK methodology to restrict, in real-time, the inter-joint distances in optical motion capture under marker occlusions. IKL Xonxlusions vny [uturz lork Hynd hose lryckerR Analytically and anatomically correct models are necessary to control and constrain the movements of any legged body. Chapter 5 introduces a hand pose tracker that requires a minimum number of available markers, one on each finger; the labelled markers’ positions are tracked using an optical motion capture system, such as [Pha]. The hand model is highly constrained using a sophisticated physiology model, which takes into account not only the joint rotational and orientational restrictions, but also constraints related to the hand anatomy, such as self-collisions, inertia, flexion etc. This extension provides more accurate results, ensuring that the hand adopts smoother and more natural poses. FABRIK was also employed to fit the other joint positions and manipulate the finger movements. Physiological constraints were used to ensure that the states of the articulated body conform to the data, minimise the misfits and do not violate the hand shape. Finally, a mesh deformation algorithm was applied to visualise the movements of the underlying hand skeleton and to compare the resulting animations with the true hand poses. Our experimental results demonstrate high precision and reliability in tracking, while also achieving real-time performance. KCGC euwlixvtions Several papers have been published based on the content of this thesis; these papers are enumerated in inverse chronological order below: Yndreys Yrisfiidofl and Joan Lasenby, ”Real-Time Marker Prediction and CoR Estimation in Optical Motion Capture”, huwmittzy to thz Imvgz vny kision Computing, February 2010. Yndreys Yrisfiidofl and Joan Lasenby, FABRIK: a fast, iterative solver for the Inverse Kinematics problem, huwmittzy to Grvphixvl boyzls, February 2010. Yndreys Yrisfiidofl and Joan Lasenby, Inverse Kinematics solutions using Conformal Geometric Algebra, In eroxzzyings of thz Ith xonfzrznxz on Vpplizy Gzomztrix Vlgzwrvs in Computzr hxiznxz vny Znginzzring (VGVChZ’FE), The Netherlands, June 14-16, 2010. Yndreys Yrisfiidofl and Joan Lasenby, Motion Capture with Constrained Inverse Kinematics for Real-Time Hand Tracking, In IZZZ eroxzzyings of thz Ith Intzrnvtionvl hymposium on CommuB nixvtionsA Control vny hignvl eroxzssing (IhCChe’FE), Cyprus, May 3-5, 2010. Yndreys Yrisfiidofl and Joan Lasenby, Inverse Kinematics: a review of existing techniques and intro- duction of a new fast iterative solver, izxhnixvl gzportA CjZDFBIcFZcGA igBKHG, Cambridge, September 2009. Yndreys Yrisfiidofl, Joan Lasenby and Jonathan Cameron, Methods for Real-time Restoration and Estimation in Optical Motion Capture, izxhnixvl gzportA CjZDFBIcFZcGA igBKFN, Cambridge, January 2009. Yndreys Yrisfiidofl, Jonathan Cameron and Joan Lasenby, Predicting Missing Markers to Drive Real-Time Centre of Rotation Estimation, In eroxzzyings of thz Jth Intzrnvtionvl Confzrznxz on Vrtixulvtzy botion vny Dzformvwlz dwjzxtsA VbDd’EM, pages 239-247, Spain, July 9-11, 2008. Yndreys Yrisfiidofl, Jonathan Cameron and Joan Lasenby, Real-Time Estimation of Missing Markers in Human Motion Capture, In IZZZ eroxzzyings of thz Gny Intzrnvtionvl Confzrznxz on WioinB formvtixs vny Wiomzyixvl ZnginzzringA iCWWZ’EM, pages 1343-1346, China, May 16-18, 2008. NFK Fflfiflre Direcfiions IKM KCHC Future Dirextions Forwyrd ynd Zyckwyrd jeyching an–erse cinemyfiicsR Future work will see the intro- duction of FABRIK within a larger variety of analytically and anatomically correct models. A further study of the collision problem, with simultaneous study of more sophisticated joint types, is also essential for the production of smoother and more natural movements. FABRIK could also find applications in markerless motion capture, e.g. problems with silhouette data, to solve the data-fitting optimisation problem. eyrker predicfiion ynd Coj esfiimyfiionR The marker prediction and CoR accuracy could be further improved by incorporating biomechanical constraints to control the postures of ani- mated characters, in optical motion capture under marker occlusion; however, prior knowledge of the model and joints is needed, reducing its generality to specific models. In addition, a hybrid system with low cost inertial measurement units could be employed allowing extra val- idation of the data. Hynd pose firyckerR A hand model which takes into consideration constraints relevant to the skin properties and the hand’s rigidity will be studied. The data in this thesis is captured from an optical motion capture system, thus there is no mesh to define the hand’s shape; the mesh presented in the examples was driven by the underlying skeleton using bone-heat, a mesh deformation algorithm. Problems relevant to the mobility of the hand, such as self-collisions, were not investigated. AppKTdOIKs A irigonometrix solutions using Geometrix Vlgewrv A ppKTdOx A illustrates the CGA mathematical solutions described in this thesis. Mostof these CGA solutions are applied to the marker constraint problem, as discussed in chapter 4.6. An alternative implementation of the FABRIK algorithm using CGA is also presented. VCFC cevrest eoint on v Cirxle from v eoint in hpvxe This section considers the problem of constraining the estimated marker positions to give a better approximation of the observation vector used in the UKF filter (section 4.5.2). Using information from the rigid body, and also keeping in mind the fact that markers have a constant distance between each other, we can find estimates of the non-visible marker. The observation vector describing the occluded marker m1 in frame k, xˆ k 1, can be assigned as the shortest distance between the intersection of the two spheres with centre the positions of the markers m- andm., radii the distances Y1- and Y1., and the point in space, x˜ k 1. The solution presented here is solved using Conformal Geometric Algebra (CGA, see [LLW04]) as shown in the next paragraphs. The intersection of 2 spheres is discussed in section 2.2.6. The distance from the point x˜k1 (which can be considered as any point in space) to xˆ k 1 can be solved as the problem of finding the nearest point on a circle to a point in space. 139 ILH irigonomztrix solutions using Gzomztrix Vlgzwrv C X ! X P X 1 X 2 X min d Figflre YFIFR ihz nzvrzst point on xirxlz to point in spvxzC ihz point m is projzxtzy on thz xirxlz’s plvnz ΦC V linz is thzn formzy through thz miypoint of m vny its projzxtzy xountzrpvrt vny thz xzntrz of thz xirxlzC ihz intzrszxtion wztfizzn thz linz vny thz xirxlz rzturns tfio possiwlz solutionsA thz onz thvt is shortzr to thz point m is xhosznC The minimum distance on a circle from a point in space is related to the projection of that point onto the plane Φ of the circle. This can be achieved by reflecting the point in the plane and finding the mid-point of the reflected and the original point. Hence, let the circle X = H(w) ∧H(x) ∧H(y), where w, x and y are points that lie on the circle and the operation of H(x) is the Hestenes’ mapping in 2.2. The centre x of the circle X can be calculated as, x = XnX (A.1) and the plane Φ of the circle can be formulated as: Φ = X ∧ n = 〈Xn〉4 (A.2) Having the plane Φ and the point m = H(x) in space, the nearest point on circle can be found by reflecting that point in the plane Φ. m ′ = ΦmΦ (A.3) The mid-point me is then calculated as: m ′e = me + αn = H ( 1 2 ( H−1 ( m ′ ) + x )) (A.4) Then, a line, a, is formed through this midpoint and the centre of the circle, a = me ∧ x ∧ n (A.5) The intersection between line a and circle X will return a bivector, V ∧ W, which represents the shortest and longest distances on the circle from the point in space. The vectors m1 and m- can be extracted from m1 ∧m- using projectors (see equation 2.65). The nearest point is then selected using a simple distance comparison method. This method is also illustrated in YFJ feyresfi hoinfi on y dine from y Circle ILI figure A.1. (m1P m-) = a ∨ X m = arg (max (m1 ·mPm- ·m)) (A.6) VCGC cevrest eoint on v aine from v Cirxle Section A.2 considers the problem of finding the nearest point on a line from a circle in order to obtain a better estimate of the missing marker position. In section 4.6.2 we presented an efficient model for estimating the missing marker on a limb segment, in the case where 2 markers are visible on the same limb segment. Also, in section 4.6.5 we talked about instances where missing markers are visible in a single camera. This information identifies a line, a1, starting from the camera and passing through the position of the missing marker. Here we present an alternative way of estimating the missing marker which is visible to one camera, by finding the nearest point on line a1 from circle X1. Circle X1 is the circle created after the intersection of the spheres Σ1 and Σ-, as described in section 4.6.2. Consider a line a1 passing through the points a and w, which are represented in CGA as: V = 1 2 ( a-n+ 2a− n¯) (A.7) W = 1 2 ( w-n+ 2w− n¯) (A.8) Line a1 will then be equal to: a1 = V ∧W ∧ n = 〈〈VW〉- n〉. (A.9) Assume that circle X1 lies on the plane Φ. Line a1 intersects with the plane Φ, otherwise a1 and Φ are parallel. Thus, the bivector W is given by W = Φ ∨ a1 = [〈Φa1〉.]∗ = 〈Φa1〉. I (A.10) If W- S 0, then a1 intersects with plane Φ, while if W - = 0, then a1 and plane Φ are parallel. Firstly, consider the case where line a1 and plane Φ intersect. In such a case the bivector W will be equal to W = e ∧ n or W = n ∧ e . The vector e can be extracted from bivector W as discussed in section 2.2.5. Thereafter, we need to find the projection of the centre of the circle, X = H (x), on line a1. This is achieved by reflecting the centre of the circle in the line and then finding the mid-point, E, of point X and its reflection X ′. X ′ = a1Xa1 (A.11) E = H ( 1 2 ( H−1 ( X ′ ) +H−1 (X) )) (A.12) We then need to find the point n which is the projection of point E onto the plane Φ. A ILJ irigonomztrix solutions using Gzomztrix Vlgzwrv 1 C E 1 x 1 L C C! 2 L 2 x 1 x! n x Y E! P Figflre YFJFR [inying thz nzvrzst point on v linz from v xirxlz using XGVC Xvsz fihzrz thz yistvnxz from thz xzntrz of thz xirxlz X1 to thz point n is smvllzr thvn thz rvyius of thz xirxlz vny point e is insiyz thz xirxlz vrzvC similar procedure as above is used; E′ = ΦEΦ (A.13) n = H ( 1 2 ( H−1 ( E′ ) +H−1 (E) )) (A.14) After finding the point n , the algorithm can be separated into three possible subcases: • If the distance from the centre of the circle, X, to the point n , is smaller than the radius of the circle, then a line a- = e ∧ n ∧ n will be established. The intersection of line a- with the circle X1 will return 2 points, x1 and x-, i.e. those points on the circle with the shortest and longest distances from point e . The selection of the appropriate point can be achieved using a simple distance comparison. (x1P x-) = a- ∨ X = [〈a-X〉4]∗ (A.15) m = arg (max (m1 · ePm- · e )) (A.16) where m1 and m- are the CGA representations of x1 and x- respectively. Assume that the point with the shortest distance from e is point x1. Obviously, the point on a1 which is closest to x1 is the projection of that point onto the line. Hence, the point we are looking for, xn, is equal to: m ′1 = a1m1a1 (A.17) xn = H ( 1 2 ( H−1 ( m ′1 ) +H−1 (m1) )) (A.18) Figures A.2 and A.3 illustrate the above. Point xn is given in red. However, there are some extreme instances to be considered: – In the case where point e and n are identical (Line a1 is perpendicular to plane Φ), then a- = X ∧ n ∧ n. The rest of the algorithm remains the same. – In the case where line a1 passes through the centre of the circle, line a- cannot be established using the above procedure. Hence, we consider line a- as the projection YFJ feyresfi hoinfi on y dine from y Circle ILK 1 C E 1 x 1 L C C! 2 L 2 x 1 x! n x Y E! P Figflre YFKFR [inying thz nzvrzst point on v linz from v xirxlz using XGVC Xvsz fihzrz thz yistvnxz from thz xzntrz of thz xirxlz X1 to thz point n is smvllzr thvn thz rvyius of thz xirxlz vny point e is outsiyz thz xirxlz vrzvC of line a1 onto the plane Φ. The rest of the algorithm remains the same. – In the case where line a1 is perpendicular to plane Φ and also passes through the centre of the circle, then any point on circle X1 has the same distance from line a1. • If the distance from the centre of the circle, X, to the point n , is equal to the radius of the circle, then x1 = n . In such a case, the point on line we are looking for, xn can be achieved by projecting the point x1 onto the line a1 as above. • If the distance from the centre of the circle, X, to the point n is greater than the radius of the circle, then a line a- will be introduced which is equal to a- = X ∧ n ∧ n. Hence, we have a similar situation as above but line a- is different. Again, the intersection of line a- with the circle X1 will return 2 points, x1 and x-, and the appropriate point can be selected as the point with the shortest distance from point e . Thus, point xn is related to the projection of point x1 onto line a1. Figure A.4 illustrates the above. In the second case, where W- = 0, plane Φ and line a1 are parallel. We can check if the circle and the line intersect by calculating the mzzt between them. G = X ∨ a1 = [〈Xa1〉4]∗ = 〈Xa1〉4 I (A.19) There are three further possible subcases: • If G = 0 then line a1 belongs to plane Φ and intersects with the circle at two points. In such a case we can extract the two points as in section 2.2.5. Both points are on line a1 and on circle X. • If G = m and m- = 0 then line a1 lies in the plane Φ and intersects with the circle at one point. That is the point we are looking for. • If G = m and m- ̸= 0 then line a1 and circle do not intersect at any point. This case can be further separated in two subcases; the instance where line a1 lies in the plane Φ and the instance where it does not. – In the case where line a1 lies in the plane Φ, then the nearest point on the line from the circle can be achieved by finding the projection of the centre of the circle onto the line. ILL irigonomztrix solutions using Gzomztrix Vlgzwrv 1 C E 1 x 1 L C C 2 L 2 x 1 x n x Y E P Figflre YFLFR [inying thz nzvrzst point on v linz from v xirxlz using XGVC Xvsz fihzrz thz yistvnxz from thz xzntrz of thz xirxlz X1 to thz point n is grzvtzr thvn thz rvyius of thz xirxlz vny point e C – In the case where line a1 does not lie in the plane Φ, then the problem can be solved by finding the projection of line a1 onto the plane Φ. Hence, we can continue as above, having a line on plane Φ. VCHC cevrest eoint on v aine from v hphere This section focuses on the problem of finding the nearest point on a line from a sphere. In section 4.6.3, an efficient model for estimating missing markers has been proposed for the case where 2 markers on a limb segment are occluded. However, there are instances where the missing marker is visible in a single camera, giving us more information about the position of this missing marker. This information identifies a line, a1, starting from the camera and passing through the position of the missing marker. Section 4.6.5 presents a reliable approach of estimating the missing markers for such instances. Here we discus an alternative way of estimating the occluded marker, which is visible to one camera, by finding the nearest point on line a1 from sphere Σ1. Σ1 has as its centre the position of the visible marker m- and radius the distance |DˆkjP-|, as calculated in section 4.6.3. Consider a line a1 passing through the points a and w, which are represented in CGA as: V = 1 2 ( a-n+ 2a− n¯) (A.20) W = 1 2 ( w-n+ 2w− n¯) (A.21) Line a1 is then: a1 = V ∧W ∧ n = 〈〈VW〉- n〉. (A.22) We now need to find the projection of the centre of the sphere, X, onto line a1. This can be achieved by reflecting the centre of the circle in the line and then finding the mid-point, n , YFL feyresfi hoinfi on y kphere from y hoinfi in kpyce ILM between the original, X, and the reflected point X ′. X ′ = a1Xa1 (A.23) n = H ( 1 2 ( H−1 ( X ′ ) +H−1 (X) )) (A.24) The problem can now be separated into three possible subcases; the case where line a1 intersects the sphere at two points, the case where line a1 touches the sphere and the case where the line a1 and the sphere have no intersection points. The solutions for all cases is given below: • If the distance between the centre of the sphere and the point n is greater than the radius of the sphere, then line a1 does not intersect with sphere Σ1. Hence, n represents the point on line we are looking for. • If the distance between the centre of the sphere and the point n is equal to the radius of the sphere, then line a1 intersect with sphere Σ1 at exactly one point, the point n . Obviously, n represents the point on the line with minimum distance from the sphere. • If the distance between the centre of the sphere and the point n is smaller than the radius of the sphere, then line a1 intersect with sphere Σ1 at two points. Both points could represent the point we are looking for. However, we select as the position of the missing marker the point which returns minimum distance from the position in the previous frame. VCIC cevrest eoint on v hphere from v eoint in hpvxe This section presents how to calculate the nearest point on a sphere from a point in space using CGA. Assume that a sphere has centre x and radius /. The sphere Σ1 can be expressed as blades in CGA as follows: Σ1 = ( x− 1 2 /-n ) I where x = 1 2 x-n+ x− n¯ 2 Assume a point in space q. In order to find the nearest point on the sphere from that point, we need to find the intersection of the line a1, that passes through the point q and the sphere centre x. Thus, a1 = f ∧ x ∧ n = 〈〈fx〉- n〉. (A.25) where f = H(q) is the Hestens mapping of q. The intersection between the line a1 and the ILN irigonomztrix solutions using Gzomztrix Vlgzwrv sphere Σ1 always returns two possible solutions, which are given by the bivector, V ∧W. V ∧W = a1 ∨ Σ1 (A.26) Finally, the vectors V and W can be extracted from V ∧ W using equations 2.65. Then, the nearest point on the sphere is assigned as the point that returns the minimum distance from the point in space. VCJC IK holutions using Conformvl Geometrix Vlgewrv In general, CGA gives us the ability to describe algorithms in a geometrically intuitive and compact manner; it simplifies the mathematical model of the inverse kinematics solver, since basic entities, such as spheres, lines, planes and circles, are simply represented by algebraic objects. It is worth noting that FABRIK performs three times faster when it is implemented by simply taking distances along lines rather than intersecting with spheres, as described in chapter 3.3. However, when we wish to incorporate constraints, we often need the sphere-line information, so in this appendix we present the entire work in this unified framework. In addition, several optimisations can be applied using CGA; for instance, a direct estimation of a missing joint, when it is between 2 true positions, can be achieved by intersecting the 2 spheres rather than finding an approximation using the normal iterative algorithm. Another simple optimisation is the direct construction of a line pointing towards the target, when the latter is unreachable. The FABRIK algorithm is still real-time implementable and is described in pseudocode in Alg. 5. The FABRIK solution in CGA was also presented in [AL10a]. The implementation of FABRIK in CGA is similar to the normal algorithm. Thus, assume that p1P OOOPpn are the joint positions of a manipulator. For the simple case where only a single end effector exists, take p1 as the root joint and pn as the end effector. The target is symbolised as fi and the initial base position by z. First calculate the distances between each joint yi = |pi+1 − pi|, for i = 1P OOOP n−1. Then, to check whether the target is reachable or not, find the distance between the root and the target, yist, and if this distance is smaller than the total sum of all the inter-joint distances, yist Q ∑n−1 1 yi, the target is within reach, otherwise, it is unreachable. If the target is within reach, a full iteration is constituted by two stages. In the first stage, the algorithm estimates each joint position starting from the end-effector, pn, moving inwards to the manipulator base, p1. So, let the new position of the end-effector be the target position, p′n = fi. The new position of the (n − 1)th joint, p′n−1, is assigned as the nearest point on the sphere Σn−1, with centre the joint position p′n and radii the distance yn−1, from the joint position pn−1. Similarly, the new position of the (n− 2)th joint, p′n−-, is selected as the nearest point on sphere Σn−-, with centre the joint position p′n−1 and radii the distance yn−-, from the joint pn−-. The algorithm continues until all new joint positions are calculated, including the root, p′1. The trigonometric solution of the nearest point on a sphere from a point in space in given in section A.4. A full iteration is completed when the same procedure is repeated but this time starting YFM ac kolflfiions flsing Conformyl Geomefiric Ylgezry ILO Ylgorifihm MR A full iteration of the FABRIK algorithm using CGA. InputN hhy joint positions pi for i = E; :::; nB, thy turgyt position t unx thy xistunwys vytwyyn yuwh joint ri = |pi+1 − pi| for i = E; :::; n− EB eutputN hhy nyw joint positions pi for i = E; :::; nB 5.1 : ihe distvnce wetfieen root vnd tvrget 5.2 rist = |p1 − t| 5.3 : Check fihether the tvrget is fiithin revch 5.4 if rist L r1 + r2 + :::+ rn−1 then 5.5 : ihe tvrget is unrevchvwle 5.6 for i = E; :::; n− E do 5.7 : Find the nevrest point on sphereA fiith centre the joint position pi vnd rvdius the distvnce riA from v point is spvceA t 5.8 pi+1 = dewrestfointiphere(pi,ri,t=; 5.9 end 5.10 else 5.11 : ihe tvrget is revchvwle; thusA set vs b the initivl position of the joint p1 5.12 b = p1 5.13 : Check fihether the distvnce wetfieen the end effector pn vnd the tvrget t is grevter thvn v tolervnceC 5.14 ritA = |pn − t| 5.15 while ritA L tol do 5.16 : hiAGE FO FdglAgD gEACHIcG 5.17 : het the end effector pn vs tvrget t 5.18 pn = t 5.19 for i = n− E; :::; E do 5.20 : Find the nevrest point on sphereA fiith centre the joint position pi+1 vnd rvdius the distvnce riA from v point is spvceA pi 5.21 pi = dewrestfointiphere(pi+1,ri,pi=; 5.22 end 5.23 : hiAGE GO WACKlAgD gEACHIcG 5.24 : het the root p1 its initivl positionC 5.25 p1 = b 5.26 for i = E; :::; n− E do 5.27 : Find the nevrest point on sphereA fiith centre the joint position pi vnd rvdius the distvnce riA from v point is spvceA pi+1 5.28 pi = dewrestfointiphere(pi,ri,pi+1=; 5.29 end 5.30 ritA = |pn − t| 5.31 end 5.32 end 5.33 5.34 : lhere the function dewrestfointiphere(f,Y ,Z=, finds the nevrest point on v sphere from v point in spvceC f is the sphere’s centreA Y is the sphere’s rvdii vnd Z is the point in spvceC from the root joint and moving outwards to the end effector. Thus, let the new position for the 1st joint, p′′1, be its initial position z. Then, the new joint position p′′- is assigned as the nearest point on the sphere Σ1, with centre the p ′′ 1 and radii the distance y1, from the joint p ′ -. This procedure is repeated for all the remaining joints, including the end effector. FABRIK is illustrated in pseudo-code in Algorithm 5 and a graphical representation of a full iteration of the algorithm is demonstrated in figure A.5. The forward and backward procedure is then repeated, for as many iterations as needed, until the end effector is identical or close enough (to be defined) to the desired target. Finally, the CGA framework offers several algorithm optimisations such as for cases where the ‘end effectors’ are not positioned at the end of the chains (i.e. it is a leaf). For instance, assume ILP irigonomztrix solutions using Gzomztrix Vlgzwrv t d1 d2 d3 4 p 3 p 2 p 1 p (u= d1 d2 d3 4 p 3 p 2 p 1 p 4 p (v= l3 d1 d2 d3 3 p 2 p 1 p 4 p 3 p 3 (w= d1 d2 d3 l2 l1 4 p 3 p 2 p 1 p 2 1 (x= d1 d2 d3 4 p 3 p 2 p 1 p 1 p (y= l3 d1 d2 d3 l2 l1 t 1 p 2 p 3 p 4 p 2 1 3 (f= Figflre YFMFR V full itzrvtion of [VWgI‘ for thz xvsz of v singlz tvrgzt vny I joints using XGVC (v) ihz initivl position of thz mvnipulvtor vny thz tvrgztA (w) movz thz zny zffzxtor p4 to thz tvrgztA (x) finy thz joint p′3 fihixh is thz intzrszxtion of thz sphzrz Σ3 vny thz linz l3 fihixh pvsszs through thz points p′4 vny p3A (y) xontinuz thz vlgorithm for thz rzst of thz jointsA (z) thz szxony stvgz of thz vlgorithmO movz thz root joint p′1 to its initivl positionA (f) rzpzvt thz svmz proxzyurz wut this timz stvrt from thz wvsz vny movz outfivrys to thz zny zffzxtorC ihz vlgorithm is rzpzvtzy until thz position of thz zny zffzxtor rzvxhzs thz tvrgzt or gzts suffixizntly xloszC that the joint positions pi and pi−- are known, and that we want to estimate the joint position pi−1. This can be done by finding the intersection of the spheres Σ1 and Σ- (as is described in section 2.2.6) with centres the known joint positions pi and pi−- and radii the distances yi = |pi − pi−1| and yi−- = |pi−- − pi−1|, respectively. If the intersection is a circle, then the estimated joint position can be assigned as the nearest point on that circle from its previous position (as described in section A.1). If the intersection is a single point, the estimated joint position is assumed to be that point, otherwise, if the two spheres do not intersect, the estimated joint position is equal to pi−1 = pi+pi−2 - . Another simple optimisation is the direct construction of a line pointing towards the target, when the latter is unreachable. In that case, each joint pi is assigned to be the nearest point on the sphere, with centre the previous joint pi−1 and radii the distance yi−1, from the target. Glossvry BFGS - Broyden, Fletcher, Goldfarb, Shanno method CCD - Cyclic Coordinate Descent CDF - Cumulative Distribution Function CGA - Conformal Geometric Algebra CoR - Centre of Rotation CRBM - Conditional Restricted Boltzmann Machine CV - Constant Velocity DLS - Damped Least Squares DoF - Degrees of Freedom EKF - Extended Kalman Filter FABRIK - Forward And Backward Reaching Inverse Kinematics FIK - Feedback Inverse Kinematics FK - Forward Kinematics FTL - Follow The Leader GA - Geometric Algebra GPDM - Gaussian process dynamical model GPLVM - Gaussian Process Latent Variable Model HCI - Human Computer Interaction HMM - Hidden Markov Model IIK - Inductive Inverse Kinematics IK - Inverse Kinematics KF - Kalman Filter LDS - Linear Dynamical System MMSE - Minimum Mean Square Error Mocap - Motion Capture NBP - Nonparametric Belief Propagation NCT - Nearly Constant Turn PCA - Principal Component Analysis SDLS - Selectively Damped Least Squares 149 IMH irigonomztrix solutions using Gzomztrix Vlgzwrv SIK - Sequential Inverse Kinematics SMCM - Sequential Monte Carlo Methods SVD - Singular Value Decomposition VTM - Variable Turn Model UKF - Unscented Kalman Filter UPM - Uniform Posture Map hupplementvry bvterivls A DVD containing supplementary materials accompanies this thesis. The DVD includes videos demonstrating our algorithms and comparing their performance against other state of the art methods. In addition, the Kine application (see section 3.6.1) is included, offering the opportunity to interact and evaluate the results obtained in this work. The supplementary materials included in the DVD are: 1. FYZjacFy–i A video that demonstrates FABRIK and compares its performance against other methods and under different scenarios. 2. eyrkerhredicfiionFy–i A video presenting experimental examples of our marker pre- diction and CoR methodology, with and without incorporating FABRIK for length con- straints. It also compares its performance against other methods, such as a simple EKF model, interpolation and extrapolation. 3. HyndhoselryckerFy–i A video showing examples of our hand pose tracker under different frame rates. 4. lhe cine Ypplicyfiion Kine is a 2D real-time gaming application which offers to the user the opportunity to interact and evaluate the performance of the FABRIK algorithm against CCD. There is also an option where the user clicks and drags on the screen, and the snake (the kinematic chain is drawn as a snake) will track the mouse. 151 Wiwliogrvphy [AA82] S. J. Asseo and R. J. Ardila. Sensor independent target state estimator design and evalu- ation. In eroxzzyings of thz cvtionvl Vzrospvxz vny Zlzxtronixs Confzrznxz (cVZCdc), pages 916–924, 1982. 91 [ACL08] Andreas Aristidou, Jonathan Cameron, and Joan Lasenby. Predicting missing markers to drive real-time centre of rotation estimation. In eroxzzyings of thz k Confzrznxz on Vrtixulvtzy botion vny Dzformvwlz dwjzxts, volume 5098, pages 238–247, Mallorca, Spain, 2008. LNCS. 7, 84, 96, 106 [AL09] Andreas Aristidou and Joan Lasenby. Inverse kinematics: A review of existing techniques and introduction of a new fast iterative solver. Technical Report F-INFENG/TR. 632, CUED, 2009. 104 [AL10a] Andreas Aristidou and Joan Lasenby. Inverse kinematics solutions using conformal ge- ometric algebra. In eroxzzyings of thz Ith xonfzrznxz on Vpplizy Gzomztrix Vlgzwrvs in Computzr hxiznxz vny Znginzzring (VGVChZ’FE), Amsterdam, The Netherlands, June 14-16 2010. 55, 57, 146 [AL10b] Andreas Aristidou and Joan Lasenby. Motion capture with constrained inverse kinematics for real-time hand tracking. In eroxzzyings of thz Ith Intzrnvtionvl hymposium on ComB munixvtionsA Control vny hignvl eroxzssing (IhCChe GEFE), pages 1–5, Limassol, Cyprus, March 3-5 2010. IEEE. 57, 75, 115, 121 [AL10c] Andreas Aristidou and Joan Lasenby. Real-time marker prediction and CoR estimation in optical motion capture. huwmittzy to Imvgz vny kision Computing, 2010. 57, 75, 125 [AL10d] Andreas Aristidou and Joan Lasenby. FABRIK: a fast, iterative solver for the inverse kinematics problem. huwmittzy to Grvphixvl boyzls, 2010. 53, 101, 104, 123 [ALC09] Andreas Aristidou, Joan Lasenby, and Jonathan Cameron. Methods for real-time restora- tion and estimation in optical motion capture. Technical Report F-INFENG/TR. 619, CUED, 2009. 96, 97, 99 [ARW01] Sung Joon Ahn, Wolfgang Rauh, and Hans-Jrgen Warnecke. Least-squares orthogonal distances fitting of circle, sphere, ellipse, hyperbola, and parabola. evttzrn gzxognition, 34(12):2283 – 2303, 2001. 60 [AW00] Golam Ashraf and Kok Cheong Wong. Dynamic time warp based framespace interpolation for motion editing. In Sidney Fels and Pierre Poulin, editors, Grvphixs Intzrfvxz, pages 45–52. Canadian Human-Computer Communications Society, 2000. 84 153 IML WIWaIdGgVe]n [Bac09] Adam S. Backer. Estimating Missing Motion Capture Data with Accelerometers. bphil thzsisA Cvmwriygz jnivzrsity Znginzzring Dzpvrtmznt. Cambridge, UK, August 2009. 95 [Bai85] J. Baillieul. Kinematic programming alternatives for redundant manipulators. In eroxzzyB ings of thz IZZZ Intzrnvtionvl Confzrznxz on gowotixs vny Vutomvtion, volume 2, pages 722–728, March 1985. 5, 45 [BB01] P. Baerlocher and B. Boulic. Parametrization and range of motion of the ball-and-socket joint. In Dzformvwlz Vvvtvrs, pages 180–190. Kluwer Academic Publishers, 2001. 58 [BDMS84] A. Balestrino, G. De Maria, and L. Sciavicco. Robust control of robotic manipulators. In eroxzzyings of thz Nth IFVC lorly Congrzss, volume 5, pages 2435–2440, 1984. 5, 46 [BK05] Samuel R. Buss and Jin-Su Kim. Selectively damped least squares for inverse kinematics. Journvl of Grvphixs iools, 10(3):37–49, 2005. 5, 48, 65, 71 [Ble] Blender foundation: 3d content creation suite program. hffb://iii.blended.adg. 39, 65, 126 [BLM04] Joel Brown, Jean-Claude Latombe, and Kevin Montgomery. Real-time knot-tying simu- lation. ihz kisuvl Computzr, 20(2):165–179, 2004. 6 [Blo02] Jonathan Blow. Inverse kinematics with quaternion joint limits. Gvmz Dzvzlopzr, April 2002. 58, 62 [BP07] Ilya Baran and Jovan Popovic´. Automatic rigging and animation of 3d characters. VCb irvnsvxtion on Grvphixs (idG), 26(3):72, 2007. 126 [BPW93] N. I. Badler, C. B. Phillips, and B. L. Webber. himulvting HumvnsO Computzr Grvphixs Vnimvtion vny Control. Oxford University Press, New York, Oxford, 1993. 40, 41 [BSR07] Jurgen Broeren, Katharina S. Sunnerhagen, and Martin Rydmark. A kinematic analysis of a haptic handheld stylus in a virtual environment: a study in healthy subjects. Journvl of czuroZnginzzring vny gzhvwilitvtion, 4:13, May 2007. 2, 80 [BVU+06] R Boulic, J Varona, L Unzueta, M Peinado, A Suescun, and F Perales. Evaluation of on-line analytic and numeric inverse kinematics approaches driven by partial vision input. kirtuvl gzvlity, 10(1):48–61, 2006. 5, 53 [CA08] Nicolas Courty and Elise Arnaud. Inverse kinematics using sequential monte carlo meth- ods. In eroxzzyings of thz k Confzrznxz on Vrtixulvtzy botion vny Dzformvwlz dwjzxts, volume 5098, pages 1–10, Mallorca, Spain, 2008. LNCS. 5, 50 [Cam07] Jonathan Cameron. Vspzxts of Gzomztrix Vlgzwrv fiith vpplixvtions in motion xvpturz. PhD thesis, Cambridge University Engineering Department, Cambridge, UK, 2007. 26 [CD03] Adrian A. Canutescu and Roland L. Dunbrack. Cyclic coordinate descent: A robotics algorithm for protein loop closure. erotzin hxiznxz, 12(5):963–972, May 2003. 3, 5, 38, 51 [CDML+07] P. Cerveri, E. De Momi, N. Lopomo, G. Baud-Bovy, R. M. L. Barros, and G. Ferrigno. Finger kinematic modeling and real-time hand motion estimation. Vnnvls of Wiomzyixvl Znginzzring, 35(11):1989–2002, November 2007. 115 [CFAT07] Weiying Chen, Ryuji Fujiki, Daisaku Arita, and Rin-ichiro Taniguchi. Real-time 3d hand shape estimation based on image feature analysis and inverse kinematics. In eroxzzyings of thz FIth Intzrnvtionvl Confzrznxz on Imvgz Vnvlysis vny eroxzssing (ICIVe ’EL), pages 247–252, Washington, DC, USA, 2007. IEEE Computer Society. 115 ZaZdagGjYhHq IMM [CH05] Jinxiang Chai and Jessica K. Hodgins. Performance animation from low-dimensional control signals. VCb irvnsvxtions on Grvphixs (idG), 24(3):686–696, 2005. 8, 85 [CKM97] Kwan W. Chin, B. R. von Konsky, and A. Marriott. Closed-form and generalized inverse kinematics solutions for the analysis of human motion. In eroxzzyings of thz FNth IZZZ Intzrnvtionvl Confzrznxz of Znginzzring in bzyixinz vny Wiology soxizty, volume 5, pages 1911–1914, 1997. 49 [CL05] Jonathan Cameron and Joan Lasenby. A real-time sequential algorithm for human joint localization. In VCb hIGGgVeH eostzrs, page 107, New York, USA, 2005. ACM Press. xii, 6, 84, 88, 89, 90 [Cod] CodaMotion: Optical motion capture systems. hffb://iii.cadamafian.cam. 80 [CP07] Lillian Y. Chang and Nancy Pollard. Constrained least-squares optimization for robust estimation of center of rotation. Journvl of Wiomzxhvnixs, 40(1):1392 – 1400, 2007. 86 [Cra89] John J. Craig. Introyuxtion to gowotixsO bzxhvnixs vny Control. Addison-Wesley Longman Publishing Co., Inc., 1989. 40, 41 [DDFG01] Arnaud Doucet, Nando De Freitas, and Neil Gordon. hzquzntivl bontz Cvrlo mzthoys in prvxtixz. Springer, New York, 2001. 96 [DDHF06] Guillaume Dewaele, Fre´de´ric Devernay, Radu P. Horaud, and Florence Forbes. The align- ment between 3-d data and articulated shapes with bending surfaces. In eroxzzyings of thz Nth Zuropzvn Confzrznxz on Computzr kisionA GrvzA Vustriv, volume III of acCh, pages 578–591. Springer, May 2006. 115 [DH93] Andreas W. M. Dress and Timothy F. Havel. Distance geometry and geometric algebra. Founyvtions of ehysixs, 23(10):1357 – 1374, October 1993. 21 [DL03] Chris Doran and Anthony Lasenby. Gzomztrix Vlgzwrv for ehysixists. Cambridge Univer- sity Press, Cambridge UK, 2003. 11, 55 [DMR06] Pushkar Dhawale, Masood Masoodian, and Bill Rogers. Bare-hand 3d gesture input to interactive systems. In eroxzzyings of thz Lth VCb hIGCHI czfi ozvlvny xhvptzr’s inB tzrnvtionvl xonfzrznxz on ComputzrBhumvn intzrvxtion (CHIco ’EK), pages 25–32, New York, NY, USA, 2006. ACM. 115 [Dor05] Leo Dorst. First order error propagation of the Procrustes method for 3d attitude estima- tion. IZZZ irvnsvxtions on evttzrn Vnvlvlysis vny bvxhinz Intzlligznxz, 27(2):221–229, February 2005. 83 [DSP06] Kevin G. Der, Robert W. Sumner, and Jovan Popovic´. Inverse kinematics for reduced deformable models. In VCb hIGGgVeH evpzrs, pages 1174–1179, New York, NY, USA, 2006. ACM. 6, 50 [DU03] Klaus Dorfmu¨ller-Ulhaas. Robust optical user motion tracking using a Kalman filter. Technical Report TR-2003-6, Institut fuer Informatik, Universitatsstr. 2, 86159 Augsburg, May 2003. 7, 84 [Enc] Microsoft Encarta Online Encyclopedia. Types of joints in the human body. c©1997-2008 Microsoft Corporation, hffb://encadfa.men.cam. 41 IMN WIWaIdGgVe]n [ETDH06] Rainald M. Ehrig, William R. Taylor, Georg N. Duda, and Markus O. Heller. A sur- vey of formal methods for determining the centre of rotation of ball joints. Journvl of Wiomzxhvnixs, 39(15):2798–2809, 2006. 6, 84 [F0ˆ3] Martin Feˆdor. Application of inverse kinematics for skeleton manipulation in real-time. In eroxzzyings of thz FNth spring xonfzrznxz on Computzr grvphixsA hCCG ’EH, pages 203–212, New York, NY, USA, 2003. ACM. 49 [FAT05] Ryuji Fujiki, Daisaku Arita, and Rin-ichiro Taniguchi. Real-time 3d hand shape esti- mation based on inverse kinematics and physical constraints. In eroxzzyings of thz FHth Intzrnvtionvl Confzrznxz on Imvgz Vnvlysis vny eroxzssing (ICIVe ’EJ), volume 3617 of azxturz cotzs in Computzr hxiznxz, pages 850–858. Springer, 2005. 115 [Fle87] R. Fletcher. ervxtixvl mzthoys of optimizvtionP (Gny ZyC). Wiley-Interscience, New York, NY, USA, 1987. 5, 49 [FRF08] Jonas Fredriksson, Sven Berg Ryen, and Morten Fjeld. Real-time 3d hand-computer interaction: optimization and complexity reduction. In eroxzzyings of thz Jth coryix xonfzrznxz on HumvnBxomputzr intzrvxtion, pages 133–141, New York, NY, USA, 2008. ACM. 8, 114 [GL02] Sahan S. Hiniduma Udugama Gamage and Joan Lasenby. New least squares solutions for estimating the average centre of rotation and the axis of rotation. Journvl of Wiomzxhvnixs, 35(1):87–93, January 2002. 6, 83, 84 [GMHP04] Keith Grochow, Steven L. Martin, Aaron Hertzmann, and Zoran Popovic´. Style-based inverse kinematics. In VCb irvnsvxtions on Grvphixs (idG), pages 522–531, New York, NY, USA, August 2004. ACM. 5, 7, 50, 85 [Gol] Motion reality golf systems. hffb://iii.mafiandealifkgalf.cam. 2, 80 [GPF08] Martin de La Gorce, Nikos Paragios, and David J. Fleet. Model-based hand tracking with texture, shading and self-occlusions. In IZZZ Confzrznxz on Computzr kision vny evttzrn gzxognition, pages 1–8, June 2008. 8, 115 [Gra62] Hermann Grassmann. Vusyzhnungslzhrz. Enslin, 1862. 12 [Hal03] K. Halvorsen. Bias compensated least squares estimate of the center of rotation. Journvl of Wiomzxhvnixs, 36:999–1008(10), July 2003. 6, 83 [HDSB05] Maureen K. Holden, Thomas A. Dyar, Lee Schwamm, and Emilio Bizzi. Virtual- environment-based telerehabilitation in patients with stroke. erzsznxzO izlzopzrC kirtuvl ZnvironC, 14(2):214–233, 2005. 2, 80 [Hes01] D. Hestenes. Old wine in new bottles: a new algebraic framework for computational geometry. In Gzomztrix Vlgzwrv fiith Vpplixvtions in hxiznxz vny Znginzzring, pages 1– 16. Birkhauser, 2001. 21 [HFP+00] Lorna Herda, Pascal Fua, Ralf Pla¨nkers, Ronan Boulic, and Daniel Thalmann. Skeleton- based motion capture for robust reconstruction of human motion. In eroxzzyings of thz IZZZ Computzr Vnimvtion (CV’EE), pages 77–86, Pennsylvania, USA, May 3-5 2000. 7, 85 [HFP+01] Lorna Herda, Pascal Fua, Ralf Pla¨nkers, Ronan Boulic, and Daniel Thalmann. Using skeleton-based tracking to increase the reliability of optical motion capture. Humvn bovzB mznt hxiznxz Journvl, 20(3):313–341, 2001. 7, 85 ZaZdagGjYhHq IMO [HGP04] Eugene Hsu, Sommer Gentry, and Jovan Popovic´. Example-based control of human mo- tion. In eroxzzyings of thz GEEI VCb hIGGgVeHDZurogrvphixs symposium on Computzr vnimvtion (hCV ’EI), pages 69–77, Grenoble, France, 2004. Eurographics Association. 86 [HLL99] K. Halvorsen, M. Lesser, and A. Lundberg. A new method for estimating the axis of rotation and the center of rotation. Journvl of Wiomzxhvnixs, 32:1221–1227, 1999. 6, 83 [HNT+06] Junichi Hashiguchi, Hiroki Nivomiya, Hiroshi Tanaka, Mari Nakamura, and Katsuya Nobuhara. Biomechanical analysis of a golf swing using motion capture system. In eroB xzzyings of Vnnuvl bzzting of Jvpvnzsz hoxizty for drthopvzyix Wiomzxhvnixs, volume 27, pages 325–330, 2006. 2, 80 [Hol91] S. Holzreiter. Calculation of the instantaneous centre of rotation for a rigid body. Journvl of Wiomzxhvnixs, 24(7):643–647, 1991. 6, 83 [Hor87] Berthold Horn. Closed-form solution of absolute orientation using unit quaternions. JourB nvl of thz dptixvl hoxizty of Vmzrixv V, 4:629–642, April 1987. 86, 120 [HRE+08] Chris Hecker, Bernd Raabe, Ryan W. Enslow, John Deweese, Jordan Maynard, and Kees van Prooijen. Real-time motion retargeting to highly varied user-created morphologies. VCb irvnsvxtions on Grvphixs (idG), 27(3):1–11, 2008. 5, 50 [HS84] D. Hestenes and G. Sobczyk. Cliffory Vlgzwrv to Gzomztrix CvlxulusO V unifizy lvnguvgz for mvthzmvtixs vny physixs. D. Reidel, 1984. 19, 20, 21, 31, 55 [HSD05] Alexander Hornung and Sandip Sar-Dessai. Self-calibrating optical motion tracking for articulated bodies. In eroxzzyings of thz IZZZ Confzrznxz on kirtuvl gzvlityA kg ’EJ, pages 75–82, Washington, DC, USA, 2005. IEEE Computer Society. 7, 85 [HUHF03] L. Herda, R. Urtasun, A. Hanson, and P. Fua. Automatic Determination of Shoulder Joint Limits using Experimentally Determined Quaternion Field Boundaries. Intzrnvtionvl Journvl of gowotixs gzszvrxh, 22(6), 2003. 41 [IP90] Augustus A. White III and Manohar M. Panjabi. Clinixvl wiomzxhvnixs of thz spinz. J.B. Lippincott Company, Second Edition, 1990. 41 [IWZL09] Satoru Ishigaki, Timothy White, Victor B. Zordan, and C. Karen Liu. Performance-based control interface for character animation. VCb irvnsvxtion on Grvphixs (idG), 28(3):1– 8, 2009. 100 [Jaz70] Andrew H. Jazwinski. htoxhvstix eroxzsszs vny Filtzring ihzory. Academic Press, April 1970. 93 [JMF99] A. K. Jain, M. N. Murty, and P. J. Flynn. Data clustering: A review. VCb Computing hurvzys, 31(3):264–323, 1999. 81 [JU97] Simon J. Julier and Jeffrey K. Uhlmann. A new extension of the kalman filter to nonlinear systems. In eroxzzyings of thz Intzrnvtionvl hymposium on VzrospvxzDDzfznsz hznsingA himululvtion vny Controls, volume Acquisition, Tracking and Pointing XI, pages 182–193, Orlando, Florida, USA, 1997. 94 [Kal60] Rudolph E. Kalman. A new approach to linear filtering and prediction problems. irvnsB vxtion of thz VhbZ B Journvl of Wvsix Znginzzring, pages 35–45, 1960. 93 IMP WIWaIdGgVe]n [KL07] Paris Kaimakis and Joan Lasenby. Gradient-based hand tracking using silhouette data. In eroxzzying of thz Hry Intzrnvtionvl hymposium on kisuvl Computing (IhkC), volume 1, pages 24–35, Lake Tahoe, NV/CA, USA, November 26-28 2007. 9, 41, 71, 115 [KL09] Paris Kaimakis and Joan Lasenby. Physiological modelling for improved reliability in silhouette-driven gradient-based hand tracking. In eroxC IZZZ Intzrnvtionvl Confzrznxz on Computzr kision vny evttzrn gzxognition (Ckeg), pages 19–26, Miami, FL, USA, June 25 2009. 9, 115, 121, 123, 124, 125, 130 [KLC+03] Jin Ok Kim, Bum Ro Lee, Chin Hyun Chung, Jun Hwang, and Woongjae Lee. The inductive inverse kinematics algorithm to manipulate the posture of an articulated body. In eroxzzyings of thz Intzrnvtionvl Confzrznxz on Computvtionvl hxiznxzA ICCh GEEH, volume 2657, pages 305–313, St. Petersburg, Russia, June 2-4 2003. 52 [KM05] Richard Kulpa and Franck Multon. Fast inverse kinematics and kinetics solver for human- like figures. In Intzrnvtionvl Confzrznxz on Humvnoiy gowotsA IZZZBgVh, pages 38–43, December 2005. 5, 52, 57, 66 [KOF05] Adam G. Kirk, James F. O’Brien, and David A. Forsyth. Skeletal parameter estimation from optical motion capture data. In eroxzzyings of thz IZZZ Confzrznxz on Computzr kision vny evttzrn gzxognition, pages 782–788, June 2005. 84 [Kor85] James Urey Korein. V gzomztrix invzstigvtion of rzvxh. MIT Press, Cambridge, MA, USA, 1985. 41, 58 [KTL07] N. Klopˇcar, M. Tomsˇicˇ, and J. Lenarcˇicˇ. A kinematic model of the shoulder complex to evaluate the arm-reachable workspace. Journvl of Wiomzxhvnixs, 40(1):86 – 91, 2007. 41 [Lan98] Jeff Lander. Making kine more flexible. Gvmz Dzvzlopzr, 5(3):15–22, 1998. 5, 51, 69 [LFLD98] Joan Lasenby, William J. Fitzgerald, Anthony N. Lasenby, and Chris J. L. Doran. New ge- ometric methods for computer vision: An application to structure and motion estimation. Intzrnvtionvl Journvl of Computzr kision, 26(3):191–213, 1998. 120 [LG98] Ming C. Lin and Stefan Gottschalk. Collision detection between geometric models: A survey. In eroxzzyings of IbV Confzrznxz on bvthzmvtixs of hurfvxzs, pages 37–56, 1998. 63, 105, 130 [LGPW98] ChanSu Lee, SangWon Ghyme, ChanJong Park, and KwangYun Wohn. The control of avatar motion using hand gesture. In eroxzzyings of thz VCb symposium on kirtuvl rzvlity softfivrz vny tzxhnology (kghi ’NM), pages 59–65, Taipei, Taiwan, 1998. ACM. 115 [LH98] Cheng-Chang Lien and Chung-Lin Huang. Model-based articulated hand motion tracking for gesture recognition. Imvgz vny kision Computing, 16(2):121–134, 1998. 8, 114 [Lie77] A. Liegeois. Automatic supervisory control of the configuration and behavior of multi- body mechanisms. IZZZ irvnsvxtions on hystzmsA bvn vny Cywzrnztixs, 7(12):868–871, December 1977. 45 [LJ03] X. Rong Li and Vesselin P. Jilkov. Survey of maneuvering target tracking. Part I: Dynamic models. IZZZ irvnsvxtions on Vzrospvxz vny Zlzxtronix hystzms, 39(4):1333–1364, Oc- tober 2003. 90, 91 [LLW04] Anthony N. Lasenby, Joan Lasenby, and Richard Wareham. A covariant approach to geom- etry using geometric algebra. Technical Report F-INFENG/TR-483, Cambridge University Engineering Department, 2004. 26, 28, 30, 31, 34, 139 ZaZdagGjYhHq IMQ [LM06] Guodong Liu and Leonard McMillan. Estimation of missing markers in human motion capture. ihz kisuvl Computzr, 22(9-11):721–728, September 2006. 8, 86 [LMPF09] Lei Li, James McCann, Nancy S. Pollard, and Christos Faloutsos. Dynammo: mining and summarization of coevolving sequences with missing values. In eroxzzyings of thz FJth Intzrnvtionvl Confzrznxz on Knofilzygz Disxovzry vny Dvtv mining, pages 507–516, Paris, France, 2009. ACM. 85 [LMPF10] Lei Li, James McCann, Nancy S. Pollard, and Christos Faloutsos. Bolero: A principled technique for including bone length constraints in motion capture occlusion filling. In eroxzzyings of thz VCb symposium on Computzr vnimvtion, Madrid, Spain, 2010. 85 [LZWM06] Guodong Liu, Jingdan Zhang, Wei Wang, and Leonard McMillan. Human motion estima- tion from a reduced marker set. In IHD ’EKO eroxzzyings of thz hymposium on Intzrvxtivz HD grvphixs vny gvmzs, pages 35–42, New York, NY, USA, 2006. ACM. 8, 86 [MAT] The Mathworks - MA:LAB and Simulink for technical computing. hffb://iii. mafhiadke.cam. 65, 126 [MB91] G. Monheit and N.I. Badler. A kinematic model of the human spine and torso. IZZZ Computzr Grvphixs vny Vpplixvtions, 11(2):29–38, Mar 1991. 41 [MCM07] R. Mu¨ller-Cajar and R. Mukundan. Triangulation: A new algorithm for inverse kinematics. In eroxzzyings of thz Imvgz vny kision Computing czfi ozvlvny GEEL, pages 181–186, New Zealand, December 2007. 52 [MD04] Damian Merrick and Tim Dwyer. Skeletal animation for the exploration of graphs. In Vustrvlvsivn hymposium on Informvtion kisuvlisvtionA (invisCvu’EI), volume 35, pages 61–70, Christchurch, New Zealand, 2004. ACS. 52 [MDFW00] Rudolph V. D. Merwe, Arnaud Doucet, Nando D. Freitas, and Eric Wan. The unscented particle filter. Technical Report F-INFENG/TR. 380, CUED, 2000. 94 [Men99] Alberto Menache. jnyzrstvnying botion Cvpturz for Computzr Vnimvtion vny kiyzo Gvmzs. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1999. 2, 80 [MK85] Anthony A. Maciejewski and Charles A. Klein. Obstacle avoidance for kinematically redundant manipulators in dynamically varying environments. ihz Intzrnvtionvl Journvl of gowotixs gzszvrxh, 4(3):109–117, 1985. 45 [MM05] Michael Meredith and Steve Maddock. Adapting motion capture data using weighted real-time inverse kinematics. Computzrs in Zntzrtvinmznt (CIZ), 3(1):1–15, 2005. 49 [MSZ94] Richard M. Murray, S. Shankar Sastry, and Li Zexiang. V bvthzmvtixvl Introyuxtion to gowotix bvnipulvtion. CRC Press, Inc., Boca Raton, FL, USA, 1994. 41 [MT00] W. Maurel and D. Thalmann. Human shoulder modeling including scapulo-thoracic con- straint and joint sinus cones. Computzrs ; Grvphixs, 24(2):203–18, 2000. 41 [MTHC03] Ivana Mikic´, Mohan Trivedi, Edward Hunter, and Pamela Cosman. Human body model acquisition and tracking using voxel data. Intzrnvtionvl Journvl of Computzr kision, 53(3):199–223, 2003. 115 [Muk09] R. Mukundan. A robust inverse kinematics algorithm for animating a joint chain. IntzrB nvtionvl Journvl of Computzr Vpplixvtions in izxhnology, 34(4):303–308, 2009. 53 INH WIWaIdGgVe]n [Neb99a] Jean-Christophe Nebel. Keyframe animation of articulated figures using autocollision-free interpolation. In eroxzzyings of thz FLth Zurogrvphixs jK Confzrznxz, 13-15 April 1999. 84 [Neb99b] Jean-Christophe Nebel. Keyframe interpolation with self-collision avoidance. In eroxzzyB ings of thz Zurogrvphixs lorkshop on Computzr Vnimvtion vny himulvtion, pages 77–86. Springer, September 1999. 84 [NH86] Y. Nakamura and H. Hanafusa. Inverse kinematic solutions with singularity robustness for robot manipulator control. irvnsC VhbZA Journvl of Dynvmix hystzmsA bzvsurzmzntA vny Control, 108(3):163–171, September 1986. 5, 47 [OBBH00] James F. O’Brien, Robert E. Bodenheimer, Gabriel J. Brostow, and Jessica K. Hodgins. Automatic joint parameter estimation from magnetic motion capture data. In eroxzzyings of Grvphix Intzrfvxz, pages 53–60, 2000. 6, 83 [OS84] David E. Orin and William W. Schrader. Efficient computation of the jacobian for robot manipulators. ihz Intzrnvtionvl Journvl of gowotixs gzszvrxh, 3(4):66–75, 1984. 44 [Pec08] Alexandre N. Pechev. Inverse kinematics without matrix invertion. In eroxzzyings of thz IZZZ Intzrnvtionvl Confzrznxz on gowotixs vny Vutomvtion (ICgV ’EM), pages 2005– 2012, Pasadena, CA, USA, May 19-23 2008. 5, 49 [PH06] Sang Il Park and Jessica K. Hodgins. Capturing and animating skin deformation in human motion. VCb irvnsvxtion on Grvphixs (idG), 25(3):881–889, 2006. 86 [Pha] PhaseSpace inc: Optical motion capture systems. hffb://iii.bhaeeebace.cam. 4, 39, 80, 81, 86, 105, 114, 116, 126, 134 [PLH+09] Tommaso Piazza, Johan Lundstro¨m, Alexander Hugestrand, Andreas Kunz, and Morten Fjeld. Towards solving the missing marker problem in realtime motion capture. In eroB xzzyings of thz Intzrnvtionvl Dzsign Znginzzring izxhnixvl Confzrznxz, 2009. 7, 84, 106 [PTP+05] Lamberto Piron, Paolo Tonin, Francesco Piccione, Vincenzo Iaia, Elena Trivello, and Mauro Dam. Virtual environment training therapy for arm motor rehabilitation. izlzopB zrvtors vny kirtuvl Znvironmznts, 14(6):732–740, 2005. 2, 80 [PY06] Jun Park and Yeo-Lip Yoon. LED-glove based interactions in multi-modal displays for teleconferencing. In eroxzzyings of thz FKth Intzrnvtionvl Confzrznxz on Vrtifixivl gzvlity vny izlzflistznxz (ICVi ’EK), pages 395–399, Washington, DC, USA, 2006. 115 [RCB98] Charles Rose, Michael Cohen, and Bobby Bodenheimer. Verbs and adverbs: Multidi- mensional motion interpolation. IZZZ Computzr Grvphixs vny Vpplixvtions, 18(5):32–40, 1998. 7, 84 [RG91] Hans Rijpkema and Michael Girard. Computer animation of knowledge-based human grasping. In eroxzzyings of thz FMth vnnuvl xonfzrznxz on Computzr grvphixs vny intzrB vxtivz tzxhniquzs, pages 339–348, New York, NY, USA, 1991. ACM. 41 [RK94] James Rehg and Takeo Kanade. Digiteyes: Vision-based hand tracking for human- computer interaction. In In eroxzzyings of thz fiorkshop on botion of conBgigiy vny Vrtixulvtzy Woyizs, pages 16–22. IEEE Computer Society Press, 1994. 115 [RL02] Maurice Ringer and Joan Lasenby. A procedure for automatically estimating model param- eters in optical motion capture. In eroxzzyings of thz Writish bvxhinz kision Confzrznxz, pages 747–756, 2002. 7, 85 ZaZdagGjYhHq INI [RM06] Arjen Van Rhijn and Jurriaan D. Mulder. Optical tracking and automatic model es- timation of composite interaction devices. IZZZ kirtuvl gzvlity Confzrznxz (kg’EK), 00:135–142, 2006. 7, 84 [SB71] Robert A. Singer and Kenneth W. Behnke. Real-time tracking filter evaluation and selec- tion for tactical applications. IZZZ irvnsvxtions on Vzrospvxz vny Zlzxtronix hystzms, AES-7(1):100–110, January 1971. 90 [Sin70] Robert A. Singer. Estimating optimal tracking filter performance for manned maneuvering targets. IZZZ irvnsvxtions on Vzrospvxz vny Zlzxtronix hystzms, AES-6(4):473–483, July 1970. 90 [SK07] Markus Schlattman and Reinhard Klein. Simultaneous 4 gestures 6 dof real-time two-hand tracking without any markers. In eroxzzyings of thz VCb symposium on kirtuvl rzvlity softfivrz vny tzxhnology (kghi ’EL), pages 39–42, California, USA, 2007. 115 [SMC01] Bjo¨rn Stenger, Paulo R. S. Mendonc¸a, and Roberto Cipolla. Model-based 3d tracking of an articulated hand. In IZZZ Confzrznxz on Computzr kision vny evttzrn gzxognition, volume 2, pages 310–315, Los Alamitos, CA, USA, 2001. IEEE Computer Society. 8, 115 [SMFW04] Erik B. Sudderth, Michael I. M, William T. Freeman, and Alan S. Willsky. Distributed occlusion reasoning for tracking with nonparametric belief propagation. In In czurvl Informvtion eroxzssing hystzms (cIeh), pages 1369–1376. MIT Press, 2004. 115 [SPB+98] Marius-Calin Silaghi, Ralf Pla¨nkers, Ronan Boulic, Pascal Fua, and Daniel Thalmann. Local and global skeleton fitting techniques for optical motion capture. In eroxzzyings of thz Intzrnvtionvl lorkshop on boyzlling vny botion Cvpturz izxhniquzs for kirtuvl Znvironmznts, pages 26–40, London, UK, 1998. 6, 83, 84 [SPCM97] Ferdi Scheepers, Richard E. Parent, Wayne E. Carlson, and Stephen F. May. Anatomy- based modeling of the human musculature. In eroxzzyings of thz GIth vnnuvl xonfzrznxz on Computzr grvphixs vny intzrvxtivz tzxhniquzs, pages 163–172, 1997. 41 [Sto91] Jorge Stolfi. drizntzy projzxtivz gzomztryO v frvmzfiork for gzomztrix xomputvtions. Aca- demic Press, Boston, 1991. 30 [STTC06] Bjo¨rn Stenger, Arasanathan Thayananthan, Philip H.S. Torr, and Roberto Cipolla. Model- based hand tracking using a hierarchical bayesian filter. IZZZ irvnsC on evttzrn Vnvlysis vny bvxhinz Intzlligznxz, 28(9):1372–1384, Sept. 2006. 8, 115 [STW07] Caifeng Shan, Tieniu Tan, and Yucheng Wei. Real-time hand tracking using a mean shift embedded particle filter. evttzrn gzxognition, 40(7):1958–1970, 2007. 115 [SZGP05] Robert W. Sumner, Matthias Zwicker, Craig Gotsman, and Jovan Popovic´. Mesh-based inverse kinematics. VCb irvnsvxtions on Grvphixs (idG), 24(3):488–495, 2005. 6, 50 [THR07] Graham W. Taylor, Geoffrey E. Hinton, and Sam T. Roweis. Modeling human motion using binary latent variables. In Vyvvnxzs in czurvl Informvtion eroxzssing hystzms, pages 1345–1352, Cambridge, MA, 2007. MIT Press. 86 [TK05] Seyoon Tak and Hyeong-Seok Ko. A physically-based motion retargeting filter. VCb irvnsvxtions on Grvphixs (idG), 24(1):98–117, 2005. 7, 84 [UPBS08] Luis Unzueta, Manuel Peinado, Ronan Boulic, and A´ngel Suescun. Full-body performance animation with sequential inverse kinematics. Grvphixvl boyzls, 70(5):87–104, 2008. 5, 53, 57 INJ WIWaIdGgVe]n [Vic] Vicon motion system and peak performance. hffb://iii.hican.cam. 80 [Wam86] C. W. Wampler. Manipulator inverse kinematic solutions based on vector formulations and damped least-squares methods. eroxzzying of thz IZZZ irvnsvxtions on hystzmsA bvn vny Cywzrnztixs, 16(1):93–101, 1986. 5, 47 [WBV+99] Greg Welch, Gary Bishop, Leandra Vicci, Stephen Brumback, Kurtis Keller, and D’nardo Colucci. The HiBall tracker: High-performance wide-area tracking for virtual and aug- mented environments. In kirtuvl gzvlity hoftfivrz vny izxhnologyA kghiA VCb, pages 1–10, December 20-22 1999. 7, 84 [WC91] Li-Chun Tommy Wang and Chih Cheng Chen. A combined optimization method for solving the inverse kinematics problems of mechanical manipulators. IZZZ irvnsvxtions on gowotixs vny Vutomvtion, 7(4):489–499, 1991. 5, 50 [WE84] W. A. Wolovich and H. Elliott. A computational technique for inverse kinematics. ihz GHry IZZZ Confzrznxz on Dzxision vny Control, 23:1359–1363, December 1984. 5, 46 [Wel93] Chris Welman. Inverse kinematics and geometric constraints for articulated figure manip- ulation. bvstzr DisszrtvtionA himon Frvszr jnivzrsity, 1993. 5, 49, 50 [WFH08] Jack M. Wang, David J. Fleet, and Aaron Hertzmann. Gaussian process dynamical models for human motion. IZZZ irvnsvxtions on evttzrn Vnvlysis vny bvxhinz Intzlligznxz, 30(2):283–298, 2008. 85 [WG01] Jane Wilhelms and Allen Van Gelder. Fast and easy reach-cone joint limits. Journvl of Grvphix iools, 6(2):27–41, 2001. 58, 62 [WH97] Douglas J. Wiley and James K. Hahn. Interpolation synthesis of articulated figure motion. IZZZ Computzr Grvphixs vny Vpplixvtions, 17(6):39–45, 1997. 7, 84 [WL08] Rich Wareham and Joan Lasenby. Bone glow: An improved method for the assignment of weights for mesh deformation. In eroxzzyings of thz k Confzrznxz on Vrtixulvtzy botion vny Dzformvwlz dwjzxts, volume 5098, pages 63–71, Mallorca, Spain, 2008. LNCS. 126 [WP09] Robert Y. Wang and Jovan Popovic´. Real-time hand-tracking with a color glove. In VCb hIGGgVeH evpzrs, pages 1–8, New York, NY, USA, 2009. ACM. 8, 114 [WV98] Xuguang Wang and Jean Pierre Verriest. A geometric algorithm to predict the arm reach posture for computer-aided ergonomic evaluation. Journvl of kisuvlizvtion vny Computzr Vnimvtion, 9(1):33–47, 1998. 41 [WW91] Alan Watt and Mark Watt. Vyvvnxzy vnimvtion vny rznyzring tzxhniquzs. Addison - Wisley, ACM Press, New York, NY, USA, 1991. 41 [YLD07] Qian Yu, Qing Li, , and Zhigang Deng. Online motion capture marker labeling for multiple interacting articulated targets. Computzr Grvphixs Forum (eroxzzyings of Zurogrvphixs GEEL), 27(7):477–483, 2007. 8, 85 [ZB94] Jianmin Zhao and Norman I. Badler. Inverse kinematics positioning using nonlinear programming for highly articulated figures. VCb irvnsvxtions on Grvphixs (idG), 13(4):313–336, 1994. 4, 49 [ZVDH03] Victor Brian Zordan and Nicholas C. Van Der Horst. Mapping optical motion capture data to skeletal motion using a physical model. In eroxzzyings of thz GEEH VCb hIGB GgVeHDZurogrvphixs symposium on Computzr vnimvtion (hCV ’EH), pages 245–250, San Diego, California, 2003. Eurographics Association. 85 Inyefl Conformal Geometric Algebra, 19 Dilation, 22 Hestenes’ mapping, 20 Inversion, 23 Rotation, 21 Translation, 21 End effectors, 39 Forward Kinematics, 42 Geometric Algebra, 11 Bivector, 16 Blades, 14 Duality, 16 Geometric Product, 13 Inner Product, 12 Multivector, 16 Null vector, 20 Outer Product, 12 Pseudoscalar, 15 Reflection, 19 Reversion, 15 Rotor, 18 Trivector, 16 Inverse Kinematics, 2, 38, 42 Cyclic Coordinate Descent, 50 FABRIK, 53 Feedback Inverse Kinematics, 49 Inductive Inverse Kinematics, 52 Jacobian Inverse Methods, 43 Mesh-IK, 50 Newton Methods, 49 Sequential Inverse Kinematics, 53 Sequential Monte Carlo Methods, 50 Triangulation, 52 Jacobian Inverse Methods Damped Least Squares, 47 Pseudo-inverse, 45 Pseudo-inverse DLS, 47 Selectively Damped Least Squares, 48 Singular Value Decomposition, 46 Transpose, 46 Joint, 39 Joint models ball and socket, 40 hinge, 40 pivot, 40 saddle, 40 suture, 40 Kinematic chain, 39 Links, 39 Motion, 41 Motion capture, 80 Optical motion capture, 2, 79 Physiological Constraints, 121 Abduction, 122 Flexion, 122 Friction, 125 Inertia, 121 Intradigital Correlation, 123 Rigidity, 129 Transdigital Correlation, 124 Posture, 2, 37 Unscented Kalman Filter, 93 163 INL IcYEm Kalman gain, 95 Measurement noise covariance, 95 Observation function, 95 Observation state, 96 Observation vector, 95 Process noise covariance, 95 Transition function, 95 Unscented Transform weights, 95 Variable Turn Model, 91