Expressive Personalized 3D Face Models from 3D Face Scans

Produktbeschreibung

In this work, different methods are presented to create 3D face models from databases of 3D face scans. The challenge in this endeavour is to balance the limited training data with the high demands of various applications.
The 3D scans stem from various persons showing different expressions, with varying number of points per 3D scan and different numbers of scans per person. This data of posed facial expressions revealed substructures, which are utilised to improve the proposed model. In the process of creating and using the models, for each specifc application objective quality criteria are carefully designed tailored to the task to quantify the quality.
In total four face models built from three databases are compared based on: 3D face synthesis, 3D approximation, person and expression transfer, and 3D reconstruction from 2D.

Contents

1.1 The Difficulty of Quality Assessment . . . . . . . . . . . . . . 2
1.2 Face Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Data Preprocessing and Alignment . . . . . . . . . . . . . . . 8
1.4 Summary of Contributions . . . . . . . . . . . . . . . . . . . . 10
1.5 Thesis Overview . . . . . . . . . . . . . . . . . . . . . . . . . 11
2 Fundamentals 15
2.1 Camera Models . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.1.1 Orthographic Camera Model . . . . . . . . . . . . . . 15
2.1.2 Weak-Perspective Camera Model . . . . . . . . . . . . 15
2.1.3 Projective Camera Model . . . . . . . . . . . . . . . . 16
2.2 Estimation of Camera Parameters . . . . . . . . . . . . . . . 17
2.3 Factorization . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.3.1 Principal Component Analysis . . . . . . . . . . . . . 20
2.3.2 Whitening . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.3.3 Correlation vs. Dependence . . . . . . . . . . . . . . . 22
2.3.4 Independent Component Analysis . . . . . . . . . . . 23
2.3.5 Projection Pursuit . . . . . . . . . . . . . . . . . . . . 26
2.4 Tensor Algebra . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.4.1 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.4.2 High-Order Singular Value Decomposition . . . . . . . 30
2.5 Numerical Optimization . . . . . . . . . . . . . . . . . . . . . 31
2.5.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . 31
2.5.2 Line-Search based Methods . . . . . . . . . . . . . . . 33
2.6 Generalized Canonical Time Warping . . . . . . . . . . . . . 37
3 Face Databases 40
3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.2 Selected Databases . . . . . . . . . . . . . . . . . . . . . . . . 42
3.2.1 BU3DFE . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.2.2 BU4DFE . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.2.3 Bosphorus . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.2.4 Facewarehouse . . . . . . . . . . . . . . . . . . . . . . 54
3.2.5 MMI . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.2.6 ADFES . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4 From 3D Face Scans to Aligned Faces 59
4.1 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
4.1.1 Rigid Global Alignment . . . . . . . . . . . . . . . . . 60
4.1.2 Detection of Outliers . . . . . . . . . . . . . . . . . . . 60
4.1.3 Removing Points outside of the Face Region . . . . . . 61
4.2 Spatial Alignment by nonrigid Registration . . . . . . . . . . 66
4.2.1 Correspondence between Point Sets . . . . . . . . . . 66
4.2.2 Nonrigid 3D Registration . . . . . . . . . . . . . . . . 68
4.2.3 Quantifying Quality . . . . . . . . . . . . . . . . . . . 78
4.2.4 Experiments and Evaluation . . . . . . . . . . . . . . 84
4.3 Temporal Alignment . . . . . . . . . . . . . . . . . . . . . . . 100
4.3.1 Quantifying Expression Intensity . . . . . . . . . . . . 101
4.3.2 Alignment of Expression Intensities . . . . . . . . . . . 107
4.3.3 Applications for Proposed Expression Intensities . . . 110
5 Face Models 114
5.1 Surrey’s 3D Morphable Face Model . . . . . . . . . . . . . . . 114
5.2 Sela’s Neural Network for detailed 3D Face Reconstruction . 115
5.3 Proposed Tensor Face Models . . . . . . . . . . . . . . . . . . 116
5.3.1 The Expression Space and the Apathy Mode . . . . . 117
5.3.2 Model 1: Basic Model . . . . . . . . . . . . . . . . . . 124
5.3.3 Model 2: Subspace-aware Parameterization . . . . . . 128
5.3.4 Model 3: Projection Pursuit in Expression Space . . . 132
5.3.5 Model 4: Four-Way Model including Expression Strength134
5.3.6 Overview of Presented Tensor Face Models . . . . . . 141
5.4 Quality of Face Models . . . . . . . . . . . . . . . . . . . . . . 142
6 Experiments 146
6.1 Facial Animation by Improved Synthesis Using Apathy . . . . 146

6.2 3D Approximation, Person and Expression Transfer . . . . . 149
6.2.1 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 153
6.3 Dense 3D Reconstruction from sparse 2D . . . . . . . . . . . 156
6.3.1 3D Reconstruction With Ground Truth . . . . . . . . 156
6.3.2 3D Reconstruction Without Ground Truth . . . . . . 170
6.3.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . 170
7 Summary and Conclusions 174
7.1 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
Appendix 179
A 3D Rotations and Computing Optimal Angles . . . . . . . . . 179
B Normal Vector of 3D Points . . . . . . . . . . . . . . . . . . . 182
C Parameterization of Lines along Principal Axis . . . . . . . . 182
D Apathy Estimation – How to Find the Closest Point . . . . . 183
E Examples of Dense 3D Reconstruction of Bosphorus Database 185
Literature 189
Index 199

Keywords: Informatik, Kommunikation, 3D-Gesichts-Scan, nicht-rigide Registrierung, Korrespondenzschätzung, Intensität von Gesichtsausdrücken, Tensor, statistische Modelle, Transfer von Gesichtsausdrücken, 3D-Rekonstruktion, 3D face scans, nonrigid registration, correspondence estimation, expression intensity, tensor, factorization, statistical models.expression transfer, 3D reconstruction, TNT, Universität Hannover,