# Model-Based 3D Hand Pose Estimation from Monocular Video Martin de La Gorce, Student Member, IEEE, David J. Fleet, Senior Member, IEEE, and Nikos Paragios, Fellow, IEEE Abstract—A novel model-based approach to 3D hand tracking from monocular video is presented. The 3D hand pose, the hand texture, and the illuminant are dynamically estimated through minimization of an objective function. Derived from an inverse problem formulation, the objective function enables explicit use of temporal texture continuity and shading information while handling important self-occlusions and time-varying illumination. The minimization is done efficiently using a quasi-Newton method, for which we provide a rigorous derivation of the objective function gradient. Particular attention is given to terms related to the change of visibility near self-occlusion boundaries that are neglected in existing formulations. To this end, we introduce new occlusion forces and show that using all gradient terms greatly improves the performance of the method. Qualitative and quantitative experimental results demonstrate the potential of the approach. Index Terms—Hand tracking, model based shape from shading, generative modeling, pose estimation, variational formulation, gradient descent. ## 1 INTRODUCTION H $_{AND}$ gestures provide a rich form of nonverbal human communication for man-machine interaction. To this end, hand tracking and gesture recognition are central enabling technologies. Data gloves are commonly used as input devices, but they are expensive and often inhibit free movement. As an alternative, vision-based tracking is an attractive, nonintrusive approach. Fast, effective, vision-based hand pose tracking is, however, challenging. The hand has approximately 30 degrees of freedom (DOF), so the state space of possible hand poses is large. Searching for the pose that is maximally consistent with an image is computationally demanding. One way to improve state space search in tracking is to exploit predictions from recently estimated poses, but this is often ineffective for hands as they can move quickly and in complex ways. Thus, predictions have large variances. The monocular tracking problem is exacerbated by inherent depth uncertainties and reflection ambiguities that produce multiple local optima. Another challenge concerns the availability of useful visual cues. Hands are usually skin colored and it is difficult to discriminate one part of the hand from another based solely on color. The silhouette of the hand, if available, provides only weak information about the relative positions of different fingers. Optical flow estimates are not reliable as hands have little surface texture and are often self-occluding. Edge information is often ambiguous due to clutter. For example, the first column in Fig. 1 shows images of a hand with different poses of its index finger. The other columns show corresponding edge maps and silhouettes which remain unchanged as the finger moves; as such, they do not reliably constrain the hand pose. For objects like the hand which are relatively uniform in color, we posit that shading is a crucial visual cue. Nevertheless, shading has not been used widely for articulated tracking (but see [^1], [^2]). The main reason is that shading constraints require an accurate model of surface shape. Simple models of hand geometry where hands are approximated as a small number of ellipsoidal or cylindrical solids may not be detailed enough to obtain useful shading constraints. Surface occlusions also complicate shading cues. Two complementary approaches have been suggested for monocular hand tracking. Discriminative methods aim to recover hand pose from a single frame through classification or regression techniques (e.g., [^3], [^4], [^5], [^6]). The classifier is learned from training data that is generated offline with a synthetic model or acquired by a camera from a small set of known poses. Due to the large number of hand DOFs, it is impractical to densely sample the entire state space. As a consequence, these methods are perhaps best suited for rough initialization or recognition of a limited set of predefined poses. Generative methods use a 3D articulated hand model whose projection is aligned with the observed image for pose recovery (e.g., [^7], [^8], [^9], [^10], [^11]). The model projection is synthesized online and the registration of the model to the image can be done using local search. A variety of cues such as the distance to edges, segmented silhouettes [^12], [^13], or optical flow [^1] can be used to guide --- ![Image](images/0345b1ee0186b118316ab43722b57250ab1e74c85d8ef5563727b4802b82ae55.jpg)
Fig. 1. Two hand pictures in the first column and their corresponding edge map and segmented silhouette in the two other columns. Edge and silhouette provide little information to disambiguate the two different index poses.
the registration. The method in [^14] combines discriminative and generative approaches but does not use online synthesis. A set of possible poses is generated in advance which restrict the set of pose that can be tracked. Not surprisingly, similar challenges exist for the related problem of full body human pose tracking. For full body tracking, it is particularly difficult to formulate good likelihood models due to significant variability in shape and appearance (e.g., see [^15]). Interestingly, with the exception of recent work in [^16] on human body shape and pose estimation for unclothed people, shading has not been used extensively for human pose tracking. This paper advocates the use of richer generative models of the hand, both in terms of geometry and appearance, along with carefully formulated gradient-based optimization. We introduce a new analysis-by-synthesis formulation of the hand tracking problem that incorporates both shading and texture information while handling self-occlusion. Given a parametric hand model and a well-defined image formation process, we seek the hand pose parameters which produce the synthetic image that is as similar as possible to the observed image. Our similarity measure (i.e., our objective function) is simply the sum of residual errors, taken over the image domain. The use of a triangulated mesh-based model allows for a good shading model. By also modeling the texture of the hand, we obtain a method that naturally captures the key visual cues without the need to add new ad hoc terms to the objective function. During the tracking process, we determine, for each frame, the hand and illumination parameters by minimizing an objective function. The hand texture is then updated in each frame, and then remains static while fitting the model pose and illumination in the next frame. In contrast to the approach described in [^1], which relies on optical flow, our objective function does not assume small interframe displacements in its formulation. It therefore allows large displacements and discontinuities. The optimal hand pose is determined through a quasi-Newton descent using the gradient of the objective function. In particular, we provide a novel, detailed derivation of the gradient in the vicinity of depth discontinuities, showing that there are important terms in the gradient due to occlusions. This analysis of continuity in the vicinity of occlusion boundaries is also relevant, but yet unexplored, for the estimation of full body human pose in 3D. ![Image](images/a67abf66f4303e5d832e0e29df83fd3722ef80f928c3900b78ca773406b58441.jpg)
Fig. 2. (a) The skeleton. (b) The deformed hand triangulated surface.
Finally, we test our method on challenging sequences involving large self-occlusion and out-of-plane rotations. We also introduce sequences with 3D ground truth to allow for quantitative performance analysis. ## 2 GENERATIVE MODEL ### 2.1 Synthesis Our analysis-by-synthesis approach requires an image formation model, given a 3D hand pose, surface texture, and an illuminant. The model is derived from well-known computer animation and graphics concepts. Following [^17], we model the hand surface by a 3D, closed and orientable, triangulated surface. The surface mesh is comprised of 1,000 facets (Fig. 2b). It is deformed according to pose changes of an underlying articulated skeleton using Skeleton Subspace Deformation [^18], [^19]. The skeleton is comprised of 18 bones with 22 degrees of freedom. Each DOF corresponds to an articulation angle whose range is bounded to avoid unrealistic poses. The pose is represented by a vector $\theta$, comprised of 22 articulation parameters plus three translational parameters and a quaternion to define the global position and orientation of the wrist with respect to the camera. To accommodate different hand shapes and sizes, we also add 54 scaling parameters (3 per bone), called morphological parameters. These parameters are estimated during the calibration process (see Section 4.1), subject to linear constraints that restrict the relative lengths of the parts within each finger. Since hands have relatively little texture, shading is essential to modeling hand appearance. Here, we adopt the Gouraud shading model and assume Lambertian reflectance. We also include an adaptive albedo function to model texture and miscellaneous, otherwise unmodeled, appearance properties. The illuminant model includes ambient light and a distant point source, and is specified by a 4D vector denoted by L, comprised of three elements for a directional component and one for an ambient component. The irradiance at each vertex of the surface mesh is obtained by the sum of the ambient coefficient and the scalar product between the surface normal at the vertex and the light source direction. The irradiance across each --- ![Image](images/bfac8898065be56b5ce1cc7451efedc8fc41f5e1328df7aebc100e393430de5d.jpg)
Fig. 3. Adjacent facets of the triangulated surface mesh project to two triangles in the 2D texture map T. Since the shared edge projects to distinct segments in the texture map, one must specify constraints that the texture elements along the shared edge be consistent. This is done here using seven linear equality constraints.
face is then obtained through bilinear interpolation. Multiplying the reflectance and the irradiance yields the appearance for points on the surface. Texture (albedo variation) can be handled in two ways. The first associates an RGB triplet with each vertex of the surface mesh, from which one can linearly interpolate over mesh facets. This approach is conceptually simple but computationally inefficient as it requires many small facets to accurately model smooth surface radiance. The second approach, widely used in computer graphics, involves mapping an RGB reflectance (texture) image onto the surface. This technique preserves detail with a reasonably small number of faces. In contrast with previous methods in computer vision that used textured models (e.g., [^20]), our formulation (Section 3) requires that surface reflectance be continuous over the surface. Using bilinear interpolation of the discretized texture, we ensure continuity of the reflectance within each face. However, since the hand is a closed surface, it is impossible to define a continuous bijective mapping between the whole surface and a 2D planar surface. Hence, there is no simple way to ensure continuity of the texture over the entire surface of the hand. Following [^21] and [^22], we use patch-based texture mapping. Each facet is associated with a triangular region in the texture map. The triangles are uniform in size and have integer vertex coordinates. As depicted in Fig. 3, each facet with an odd (respectively, even) index is mapped onto a triangle that is the left-upper-half (respectively, right-lower-half) of a square divided along its diagonal. Because we use bilinear interpolation, we need to reserve some texture pixels (texels) outside the diagonal edges for points with noninteger coordinates (see Fig. 3). This representation is therefore redundant for points along edges of the 3D mesh, i.e., 3D edges of the mesh belong to two adjacent facets and therefore occur twice in the texture map, while each vertex occurs an average of six times (according to the Eulerian properties of the mesh). We therefore introduce constraints to ensure consistency between RGB values at different points in the texture map that map to the same edge or vertex of the 3D mesh. By enforcing consistency we also ensure the continuity of the texture across edges of the mesh. With bilinear texture mapping, consistency can be enforced with two sets of linear constraints on the texture map. The first set specifies that the intensities of points on mesh edges with integer-coordinates in the texture map must be consistent. The second set enforces the interpolation of the texture to be linear along edges of the triangular patches in the texture map. As long as the texture interpolation is linear along each edge and the intensities of the texture map are consistent at points with integer texture coordinates, the mapping will be consistent for all points along each edge. The bilinear interpolation is already linear along vertical and horizontal edges, so we only need to consider the diagonal edges while defining the second set of constraints. Let T denote the discrete texture image. Let $(i+1, j)$ and $(i, j+1)$ denote two successive points with integer coordinates along a diagonal edge of a triangular patch in the texture map. Using bilinear texture mapping, the texture intensities of points with noninteger coordinates $(i+1-\lambda, j+\lambda), \lambda \in (0, 1)$ along the edge are given by $$ \lambda(1-\lambda)(T_{i,j}+T_{i+1,j+1})+\lambda^{2}T_{i,j+1}+(1-\lambda)^{2}T_{i+1,j}. $$ By twice differentiating this expression with respect to $\lambda$, we find that it is linear with respect to $\lambda$ if and only if the following linear constraint is satisfied: $$ T_{i,j}+T_{i+1,j+1}=T_{i+1,j}+T_{i,j+1}. $$ The second set of constraints are for those points along the diagonal edges. Finally, let T denote the linear subspace of valid textures, i.e., whose RGB values satisfy the set of linear constraints to ensure continuity over the surface. Given the mesh geometry, the illuminant, and the texture map, the formation of the model image at each 2D image point x can be determined. First, as in ray-tracing, we determine the first intersection between the triangulated surface mesh and the ray starting at the camera center and passing through x. If no intersection exists, the image at x is determined by the background. The appearance of each visible intersection point is computed using the illuminant and information at the vertices of the visible face. In practice, the image is computed on a discrete grid and the image synthesis can be done efficiently using the triangle rasterization technique in combination with a depth buffer. When the background is static, we simply assume that an image of the background is readily available. Otherwise, we assume a probabilistic model for which background pixel colors are drawn from a background density $p_{bg}(\cdot)$ (e.g., it can be learned in the first frame with some user interaction). This completes the definition of the generative process for hand images. Let $I_{\text{syn}}(\mathbf{x}; \theta, L, T)$ denote the synthetic image comprising the hand and background, at the point x for a given pose $\theta$, texture T, and illuminant L. ### 2.2 Objective Function Our task is to estimate the pose parameters $\theta$, the texture T, and the illuminant L that produce the synthesized image $I_{syn}()$ that best matches the observed image, denoted $I_{obs}()$. Our objective function is based on a simple discrepancy measure between these two images. First, let the residual image R be defined as --- ![Image](images/ae3e528234a4e6a1554b1753c794f97fb7eaf0c83b6e2c121d94c7edee8c8829.jpg)
Fig. 4. The observed image $I_{obs}$, the synthetic image $I_{syn}$, and the residual image R.
$$ R(\mathbf{x};\theta,T,L)=\rho(I_{s y n}(\mathbf{x};\theta,L,T)-I_{o b s}(\mathbf{x})), $$ where $\rho$ is either the classical squared-error function or a robust error function such as the Huber function used in [^23]. Fig. 4 depicts an example of a pair of observed and synthesized images with the corresponding residual image. For our experiments, we choose $\rho$ to be the conventional squared-error function, and therefore the pixel errors are implicitly assumed to be IID Gaussian. Contrary to this IID assumption, nearby errors often tend to be highly correlated in practice. This is mainly due to simplifications of the model. Nevertheless, as the quality of the generative model improves, as we aim to do in this paper, these correlations become less significant. When the background is defined probabilistically, through an hypothesized color distribution $p_{bg}$, we separate the image domain $\Omega$ (a continuous subset of $\mathbb{R}^2$) into the region $S(\theta)$ covered by the hand, given the pose, and the background $\Omega\backslash S(\theta)$. Then the residual can be expressed using the background color density as $$ R(\mathbf{x};\theta,L,T)=\left\{\begin{aligned}&\rho\big(I_{s y n}(\mathbf{x};\theta,L,T)-I_{o b s}(\mathbf{x})\big),\forall\mathbf{x}\in S(\theta)\\ &-\log p_{b g}\big(I_{o b s}(\mathbf{x})\big),\forall\mathbf{x}\in\Omega\backslash S(\theta).\end{aligned}\right. $$ Fig. 11 (rows 3 and 6) depicts an example of the residual function for the likelihood computed in this way. Our discrepancy measure, E, is defined to be the integral of the residual error over the image domain $\Omega$: $$ E(\theta,T,L)=\int_{\Omega}R(\mathbf{x};\theta,L,T)\;d\mathbf{x}. $$ This simple discrepancy measure is preferable to more sophisticated measures that combine heterogeneous cues like optical flow, silhouettes, or chamfer distances between detected and synthetic edges. First, it avoids the tuning of weights associated with different cues, which is often problematic; in particular, a simple weighted sum of errors from different cues implicitly assumes (usually incorrectly) that errors in different cues are statistically independent. Second, the measure in (5) avoids early decisions about the relevance of edges through thresholding, about the area of the silhouette by segmentation, and about the position of discontinuities in optical flow. Third, (5) is a continuous function of $\theta$; this is not the case for measures based on distances between edges like the symmetric chamfer distance. By changing the domain of integration, the integral of the residual within the hand region can be reexpressed as a continuous integral over the visible part of the surface. It can then be approximated by a finite weighted sum over centers of visible faces. Much of the literature on 3D deformable models adopts this approach and assumes that the visibility of each face is binary (fully visible or fully hidden) and can be obtained from a depth buffer (e.g., see [^20]). Unfortunately, such discretizations produce discontinuities in the approximate functional as $\theta$ varies. When the surface moves or deforms, the visibility state of a face near an occlusion boundary is likely to change, causing a discontinuity in the sum of residuals. This is problematic for gradient-based optimization. To preserve continuity of the discretized functional with respect to $\theta$, the visibility state should not be binary. Rather, it should be real-valued and smooth as the surface becomes occluded or unoccluded. In practice, this is cumbersome to implement and the derivation of the gradient is complicated. Therefore, in contrast with much of the literature on 3D deformable models, we keep the formulation in the continuous image domain when deriving the expression of functional gradient. To estimate the pose $\theta$ and the lighting parameters L for each new image frame or to update the texture T, we look for minima of the objective function. During tracking, the optimization involves two steps. First, we minimize (5) with respect to $\theta$ and L, to register the model with respect to the new image. Then we minimize the error function with respect to the texture T to find the optimal texture update. ## 3 PARAMETER ESTIMATION The simultaneous estimation of the pose parameters $\theta$ and the illuminant L is a challenging nonlinear, nonconvex, high-dimensional optimization problem. Global optimization is impractical and therefore we resort to an efficient quasi-Newton, local optimization. This requires that we are able to efficiently compute the gradient of E in (5) with respect to $\theta$ and L. The gradient of E with respect to lighting L is straightforward to derive. The synthetic image $I_{syn}()$ and the residual $R()$ are smooth functions of the lighting parameters L. As a consequence, we can commute differentiation and integration to obtain $$ \begin{align*}\frac{\partial E}{\partial L}(\theta,L,T)&=\frac{\partial}{\partial L}\int_{\Omega}R(\mathbf{x};\theta,L,T)d\mathbf{x}\\&=\int_{\Omega}\frac{\partial R}{\partial L}(\mathbf{x};\theta,L,T)d\mathbf{x}.\end{align*} $$ Computing $\frac{\partial R}{\partial L}(\mathbf{x};\theta,L)$ is straightforward with application of the chain rule to the generative process. Formulating the gradient of E with respect to pose $\theta$ is not straightforward. This is the case along occlusion boundaries where the residual is a discontinuous function. As a consequence, for this subset of points, $R(x;\theta,L,T)$ is not differentiable and therefore we cannot commute differentiation and integration. Nevertheless, while the residual is discontinuous, the energy function remains continuous in $\theta$. The gradient of E with respect to $\theta$ can therefore be specified analytically. Its derivation is the focus of the next section. --- ![Image](images/ad079933363fe71817886bfa72bbd8327c11da953b12a35f01ee76f568f32ab2.jpg) ![Image](images/993122051a9ff9172d41e88032ba6ad18246fdacf122f2fc2276502be2e8efa1.jpg)
(a)
(b)
Fig. 5. (a) Example of a segment crossing the occlusions boundary $\Gamma_{\theta}$. (b) A curve representing the residual along this segment.
### 3.1 Gradient with Respect to Pose and Lighting The generative process above was carefully formulated so that scene radiance is continuous over the hand. We further assume that the background and the observed image are known and continuous. Therefore the residual error is spatially continuous everywhere except at self-occlusion and occlusions boundaries denoted by $\Gamma_{\theta}$. #### 3.1.1 1D Illustration To sketch the main idea behind the gradient derivation, we first consider a 1D residual function on a line segment that crosses a self-occlusion boundary, as shown in Fig. 5. Along this line segment, the residual function is continuous everywhere except at the self-occlusion location, $\beta$. Accordingly, we express the residual on the line in two parts, namely, the residual to the left of the boundary $r_0$ and the residual on the right $r_1$: $$ r(x,\theta)=\left\{\begin{array}{l}r_{0}(x,\theta),x\in[0,\beta(\theta)]\\ r_{1}(x,\theta),x\in(\beta(\theta),1].\end{array}\right. $$ Accordingly, the energy is then the sum of the integral of $r_{0}$ on the left and the integral of $r_{1}$ on the right: $$ E=\int_{0}^{\beta(\theta)}r_{0}(x,\theta)d x+\int_{\beta(\theta)}^{1}r_{1}(x,\theta)d x. $$ Note that $\beta$ is a function of the pose parameter $\theta$. Intuitively, when $\theta$ varies the integral E is affected in two ways. First, the residual functions $r_0$ and $r_1$ vary, for example, due to shading changes. Second, the boundary location $\beta$ also moves. Mathematically, the total derivative of E with respect to $\theta$ is the sum of two terms: $$ \frac{d E}{d\theta}=\frac{\partial E}{\partial\theta}+\frac{\partial E}{\partial\beta}\frac{\partial\beta}{\partial\theta}. $$ The first term, the partial derivative of E with respect to $\theta$, with $\beta$ fixed, corresponds to the integration of the residual derivative everywhere except the discontinuity: $$ \frac{\partial E}{\partial\theta}=\int_{[0,1]\backslash\{\beta\}}\frac{\partial r}{\partial\theta}(x,\theta)\;d x. $$ The second term in (9) depends on the partial derivative of E with respect to the boundary location $\beta$. Using the fundamental theorem of calculus, this term reduces to the difference between the residuals at both sides of the occlusion boundary multiplied by the derivative of the boundary location $\beta$ with respect to the pose parameters $\theta$. From (8), it follows that $$ \frac{\partial E}{\partial\beta}\frac{\partial\beta}{\partial\theta}=\left[r_{0}(\beta,\theta)-r_{1}(\beta,\theta)\right]\frac{\partial\beta}{\partial\theta}. $$ While the residual $r(x,\theta)$ is a discontinuous function of $\theta$ at $\beta$, the energy E is still a differentiable function of $\theta$. #### 3.1.2 General 2D Case The 2D case is a generalization of the 1D example. Like the 1D case, the residual image is spatially continuous almost everywhere, except for points at occlusion boundaries. At such points the hand occludes the background or other parts of the hand (see Fig. 5a). Let $\Gamma_{\theta}$ be the set of boundary points, i.e., the support of depth discontinuities in the image domain. Because we are working with a triangulated surface mesh, $\Gamma_{\theta}$ can be decomposed into a set line segments. More precisely, $\Gamma_{\theta}$ is the union of the projections of all visible portions of edges of the triangulated surface that separate front-facing facets from back-facing facets. For any point x in $\Gamma_{\theta}$, the corresponding edge projection locally separates the image domain into two subregions of $\Omega\backslash\Gamma_{\theta}$, namely, the occluding side and the near-occluded side (see Fig. 5). Along $\Gamma_\theta$, the residual image $R()$ is discontinuous with respect to x and $\theta$. To derive $\nabla E$, we need to define a new residual $R^+(\theta,x)$, which extends $R()$ by continuity onto the occluding points $\Gamma_\theta$. This is done by replicating the content of $R()$ in $\Omega\backslash\Gamma_\theta$ from the near-occluded side. In forming $R^+$, we are recovering the residual of occluded faces in the vicinity of the boundary points. Given a point x on a boundary segment with outward normal $\hat{n}_{\Gamma_\theta}(\mathbf{x})$ (i.e., facing the near-occluded side as in Fig. 5a), we define $$ R^{+}(\theta,\mathbf{x})=\lim_{k\to\infty}\biggl(R\biggl(\theta,\mathbf{x}+\frac{\hat{n}_{\Gamma_{\theta}}(\mathbf{x})}{k}\biggr)\biggr). $$ One can interpret $R^+(\theta, \mathbf{x})$ on $\Gamma_\theta$ as the residual we would have obtained if the near-occluded surface had been visible instead of the occluding surface. When a pose parameter $\theta_j$ is changed with an infinitesimal step $d\theta_j$, the occluding contour $\Gamma_\theta$ in the neighborhood of x moves with an infinitesimal step $v_j(\mathbf{x})$ $d\theta_j$ (see (7) of the Appendix, which can be found in the Computer Society Digital Library at for a definition of the boundary speed $v_j$). Then, the residual in the vicinity of x is discontinuous, jumping between $R^+(\theta,\mathbf{x})$ and $R(\theta,\mathbf{x})$. However, the surface area where this jump occurs is also infinitesimal and proportional to $v_j(\mathbf{x})\hat{n}_{\Gamma_\theta}(\mathbf{x})$ $d\theta_j$. The jump causes an infinitesimal change in E after integration over the image domain $\Omega$. Like the 1D case, one can express the functional gradient $\nabla_{\theta}E \equiv \left( \frac{\partial E}{\partial \theta_1}, \ldots, \frac{\partial E}{\partial \theta_n} \right)$ using two terms: $$ \begin{align*}\frac{\partial E}{\partial\theta_{j}}=&\int_{\Omega\backslash\Gamma_{\theta}}\frac{\partial R}{\partial\theta_{i}}(\mathbf{x};\theta,L,T)d\mathbf{x}\\&-\int_{\Gamma^{\theta}}\underbrace{\left[R^{+}(\mathbf{x};\theta,T,L)-R(\mathbf{x};\theta,L,T)\right]\hat{n}_{\Gamma_{\theta}}(\mathbf{x})}_{f_{or}(x)}v_{i}(\mathbf{x})d\mathbf{x}.\end{align*} $$ The first term captures the energy variation due to the smooth variation of the residual in $\Omega\backslash\Gamma_\theta$. It integrates the partial derivative of the residual R with respect to $\theta$ everywhere but at the occluding contours $\Gamma_\theta$, where it is not defined. The October 19,2026 at 14:15:34 UTC from IEEE Xplore. Restrictions apply. --- analytical derivation of $\frac{\partial R}{\partial \theta_i}$ on $\Omega \setminus \Gamma_\theta$ requires application of the chain rule to the functions that define the generative process. The second term in (13) captures the energy variation due to the movement of occluding contours. It integrates the difference between the residual on either side of the occluding contour, multiplied by the normal speed of the boundary when the pose varies. Here, $f_{oc}$ is a vector field whose directions are normal to the curve and whose magnitudes are proportional to the difference of the residuals on either each side of the curve. These occlusion forces account for the displacement of occlusion contours as $\theta$ varies. They are similar to the forces obtained with 2D active regions [^24], which derives from the fact that we kept the image domain $\Omega$ continuous while computing the functional gradient. Because the surface is triangulated, $\Gamma_{\theta}$ is a set of image line segments and we could rewrite (13) in a form that is similar to the gradient flow reported in [^25] for active polygons. Our occlusion forces also bear similarity to the gradient flows of [^26], [^27] for multiview reconstruction, where some terms account for the change of visibility at occlusion boundaries. In their formulation, no shading and texture are attached to the reconstructed surface, which results in substantial differences in the derivation. #### 3.1.3 Computing $\nabla_{\theta}E$ A natural numerical approximation to E is to first sample R at discrete pixel locations and then sum, i.e., $$ E(\theta,T,L)\approx\tilde{E}(\theta,T,L)\equiv\sum_{\mathbf{x}\in\Omega\cap\mathbb{N}^{2}}R(\mathbf{x};\theta,L,T). $$ Similarly, one might discretize the two terms of the gradient $\nabla_{\theta}E$ in (13). The integral over $\Omega\backslash\Gamma_{\theta}$ could be approximated by a finite sum over image pixel in $\Omega\cap\mathbb{N}^{2}$ while the integral along contours $\Gamma_{\theta}$ could be approximated by sampling a fixed number of points along each segment. Let $\nabla_{\theta}E$ be the numerical approximation to the gradient obtained in this way. There are, however, several practical problems with this approach. First, the approximate gradient is not the exact gradient of the approximate objective function, i.e., $\nabla_{\theta}E \neq \nabla_{\theta}E$. Most iterative gradient-based optimization methods require direct numerical evaluation of the objective function to ensure that energy decreases at each iteration. Such methods expect the gradient to be consistent with the objective function. Alternatively, one might try to compute the exact gradient of the approximate objective function, i.e., $\nabla \vec{E}$. Unfortunately, due to the aliasing along occluding edges (i.e., discretization noise in the sampling in (14)), $\vec{E}(\theta, T, L)$ is not a continuous function of $\theta$. Discontinuities occur, for example, when an occlusion boundary crosses the center of a pixel. As a result of these discontinuities, one cannot compute the exact derivative, and numerical difference approximations are likely to be erroneous. ### 3.2 Differentiable Discretization of Energy E To obtain a numerical approximation to E that can be differentiated exactly, we consider an approximation to E with antialiasing. This yields a discretization of the energy, denoted $\bar{E}$, that is continuous in $\theta$ and can be differentiated in closed form using the chain rule, to obtain $\nabla_{\theta}\bar{E}$. The antialiasing technique is inspired by the discontinuity edge overdraw method [^28]. We use it here to form a new residual function, denoted $\bar{R}$. The antialiasing technique progressively blends the residuals on both sides of occlusion boundaries. For a point x that is close to an occluding edge segment, we compute a blending weight that is proportional to the signed distance to the segment: $$ w(\mathbf{x})=\frac{(\mathbf{x}-q(\mathbf{x}))\cdot\hat{n}}{m a x(|\hat{n}_{x}|,|\hat{n}_{y}|)}, $$ where $q(\mathbf{x})$ is a point on the nearby segment and $\hat{n} = (\hat{n}_x, \hat{n}_y)$ is the unit normal to the segment (pointing toward the near-occluded side of the boundary). Dividing the signed distance by $max(|\hat{n}_x|, |\hat{n}_y|)$ is a minor convenience that allows us to interpret terms of the gradient $\nabla_\theta \bar{E}$ in terms of the occlusion forces defined in (13). A detailed derivation is given in Appendix A, which can be found in the Computer Society Digital Library at . Let the image segment $\widetilde{V}_m\widetilde{V}_n$ be defined by end points $\widetilde{V}_m$ and $\widetilde{V}_n$. For $\widetilde{V}_m\widetilde{V}_n$, we define an antialiased region, denoted $\mathcal{A}$, as a set of points on the occluded side near the segment (see Fig. 1 in the Appendix, which can be found in the Computer Society Digital Library at ), i.e., points that project to the segment with weights $w(\mathbf{x})$ in [0,1): $$ \mathcal{A}=\{\mathbf{x}\in\mathbb{R}^{2}|w(\mathbf{x})\in[0,1),~q(\mathbf{x})\in\overline{{\bar{V}_{m}\bar{V}_{n}}}\}. $$ For each point in this region, we define the antialiased residual to be a linear combination of the residuals on the occluded and occluding sides: $$ \begin{aligned}\bar{R}(\mathbf{x};\theta,L,T)&=w(\mathbf{x})R(\mathbf{x};\theta,L,T)\\&\quad+(1-w(\mathbf{x}))R(q(\mathbf{x});\theta,L,T).\end{aligned} $$ The blending is not symmetric with respect to the line segment, but rather it is shifted toward the occluded side. This allows the use of a Z-buffer to handle occlusions (see [^28] for details). For all points outside the antialiased regions, we let $\bar{R}(\mathbf{x};\theta,L,T)=R(\mathbf{x};\theta,L,T)$. Using this antialiased residual image, $\bar{R}$, we define a new approximation to the objective function, denoted $\bar{E}$: $$ \bar{E}(\theta,T,L)=\sum_{\mathbf{x}\in\Omega\cap\mathbb{N}^{2}}\bar{R}(\mathbf{x};\theta,L,T). $$ This antialiasing technique makes $\bar{R}$ and thus $\bar{E}$ continuous with respect to $\theta$ even along occlusion boundaries. There are other antialiasing methods such as oversampling [^29] or A-buffer [^30]. They might reduce the magnitude of the jump in the residual when an occlusion boundary crosses the center of a pixel, as $\theta$ changes, perhaps making the jump visually imperceptible, but the residual remains discontinuous in $\theta$. The edge overdraw method we use in (17) ensures the continuity of $\bar{R}$ and hence $\bar{E}$ with respect to $\theta$. To derive $\nabla_{\theta}\bar{E}$, we differentiate $\bar{R}$ using the chain rule. $$ \nabla_{\theta}\bar{E}=\sum_{\mathbf{x}\in\Omega\cap\mathbb{N}^{2}}\frac{\partial}{\partial\theta}\bar{R}(\mathbf{x};\theta,L,T). $$ Using a backward ordering of derivative multiplications (called adjoint coding in the algorithm differentiation) October 19,2026 at 14:15:34 UTC from IEEE Xplore. Restrictions apply. --- literature [^31]), we obtain an evaluation of the gradient with a computational cost that is comparable to the evaluation of the objective function. The gradient of $\bar{E}$ with respect to $\theta$, i.e., $\nabla_{\theta}\bar{E}$, is one of the possible discretizations of the analytical $\nabla_{\theta}E$. The differentiation of the antialiasing process with respect to $\theta$ yielded terms that sum along the edges and that can been interpreted as a possible discretization of the occlusion forces in (13). This is explained in Appendix A, which can be found in the Computer Society Digital Library at . The advantage of this approach over that discussed in Section 3.1.3 is that $\nabla_{\theta}\bar{E}$ is the exact gradient of $\bar{E}$ and both can be numerically evaluated. This also allows one to check the validity of the gradient $\nabla_{\theta}\bar{E}$ obtained by direct comparison with divided differences on $\bar{E}$. Divided differences are prone to numerical error and are inefficient, but help to detect errors in the gradient computation. ### 3.3 Model Registration #### 3.3.1 Sequential Quadratic Programming During the tracking process, the model of the hand is registered to each new frame by minimizing the approximate objective function $\bar{E}$. Let $P = [\theta, L]^T$ comprise the unknown pose and lighting parameters, and assume that we obtain an initial guess by extrapolating estimates from the previous frames. We then refine the pose and lighting estimates by minimizing $\bar{E}$ with respect to P, subject to a set of linear constraints which enforce joint limits. $$ \tilde{P}=\underset{P}{a r g m i n}\bar{E}(P)s.t.A P\leq b $$ Using the object function gradient, we minimize $\tilde{E}(P)$ using a sequential quadratic programming method [^32] with an adapted Broyden-Fletcher-Goldfarb-Shanno (BFGS) Hessian approximation. This allows us to combine the well-known BFGS quasi-Newton method with the linear joint limit constraints. There are four main steps: 1. A quadratic program is used to determine a descent direction. This uses the energy gradient and an approximation to the Hessian, $\tilde{H}_{t}$, based on a modified BFGS procedure (see Appendix B, which can be found in the Computer Society Digital Library at ): $$ \begin{aligned}\Delta_{P}&=\underset{\Delta_{P}}{argmin}\Biggl(\frac{d\bar{E}}{d P}(P_{t})\Delta_{P}+\frac{1}{2}\Delta_{P}^{t}\tilde{H}_{t}\Delta_{P}\Biggr)\\&s.t.A(P_{t}+\Delta_{P})\leq b\end{aligned} $$ 2. A line search is then performed in that direction: $$ \lambda^{*}=\underset{\lambda}{a r g m i n}\bar{E}(P_{t}+\lambda\Delta_{P})\quad s.t.\lambda\leq1. $$ The inequality $\lambda \leq 1$ ensures that we stay in the linearly constrained subspace. 3. The pose and lighting parameters are updated $$ P_{t+1}=P_{t}+\lambda^{*}\Delta_{P}. $$ 4. The approximate Hessian $\tilde{H}_{t+1}$ is updated using the adapted BFGS formula. These steps are iterated until convergence. To initialize the search with a good initial guess, we first perform a 1D search along the line that linearly extrapolates the two previous pose estimates. Each local minimum obtained during this 1D search is used as a starting point for our SQP method in the full pose space. The number of starting points found is usually just one, but when there are several, the results of the SQP method are compared and the best solution is chosen. Once the model is fitted to a new frame, we then update the texture, which is then used during registration in the next frame. ### 3.4 Texture Update Various methods for mapping images onto a static 3D surface exist. Perhaps the simplest method involves, for each 3D surface point, 1) computing its projected image coordinates, 2) checking its visibility by comparing its depth with the depth buffer, and 3) if the point is visible, interpolating image intensities at the image coordinates and then recovering the albedo by dividing the interpolated intensity by the model irradiance at that point. This approach is not suitable for our formulation for several reasons. First, we want to model texture for hidden areas of the hand, as they may become visible in the next frame. Second, our texture update scheme must avoid progressive contamination by the background color or self-contamination between different parts of the hand. Here, we formulate texture estimation as the minimization of the same basic objective functional as that used for tracking, in combination with a smoothness regularization term (independent of pose and lighting). That is, for texture T we minimize $$ E_{t e x t u r e}(T)=\bar{E}(\theta,L,T)+\beta E_{s m}(T), $$ with the constraint that T satisfies the linear constraints ensuring continuity over the surface, i.e., $\in \mathbb{T}$ (see Section 2.1). For simplicity, the smoothness measure is defined to be the sum of squared differences between adjacent pixels in the texture (texels). That is, $$ E_{s m}(T)=\sum_{i}\sum_{j\in\mathcal{N}_{T}(i)}\|T_{i}-T_{j}\|^{2}, $$ where $\mathcal{N}_T(i)$ represents the neighborhood of the texel i. While updating the texture, we choose $\rho$ (appearing in the definition of R and hence $\bar{R}$ and $\bar{E}$) to be a truncated quadratic instead of the quadratic used for model fitting. This helps to avoid the contamination of the hand texture by background color whenever the hand model fit is not very accurate. The minimization of the function $E_{\text{texture}}(T)$ with $T \in \mathbb{T}$ can be done efficiently using iteratively reweighed least-squares. The smoothness term effectively diffuses the color to texels that do not directly contribute to the image, either because they are associated to a occluded part (Fig. 6) or because of texture aliasing artifacts. To improve robustness, we also remove pixels near occlusion boundaries from the integration domain $\Omega$ when computing $E(\theta, L, T)$, and we bound the difference between the texture in the first frame and in subsequent texture estimates. --- ![Image](images/6ef1009806041ef2bfe835b44278f44445e3e87caa3ab69f1abe60df77c52183.jpg)
Fig. 6. Observed image and extracted texture mapped on the hand under the camera viewpoint and two other viewpoints. The texture is smoothly diffused into regions that are hidden from the camera viewpoint.
![Image](images/2c196d16913755b9a5287f3244e8af8dbc682c0b0edfeea4c155e413ca0f0a73.jpg)
Fig. 7. Estimation of the pose, lighting, and morphological parameters in the first frame. (a) The input image, the synthetic image given rough manual initialization, and its corresponding residual. (b) The result obtained after convergence.
## 4 EXPERIMENTAL RESULTS We apply our hand pose tracking approach to sequences with large amounts of articulation and occlusion and to two sequences used in [^14] for comparison with a state-of-the-art hand tracker. Finally, we also report quantitative results on a synthetic sequence as well as a real sequence for which ground truth data were obtained. While the computational requirements depend on image resolution, for $640 \times 480$ images, our implementation in C++ and Matlab takes approximately 40 seconds per frame on a 2 Ghz workstation. ### 4.1 Initialization A good initial guess for the pose is required for the first frame. One could use a discriminative method to obtain a rough initialization (e.g., [^3]). However, as this is outside the scope of our approach at present, we simply assume prior knowledge. In particular, the hand is assumed to be roughly parallel to the image plane at initialization (Fig. 7). We further assume that the hand color (albedo) is initially constant over the hand surface, so the initial hand appearance is solely a function of pose and shading. The RGB hand color, along with the hand pose, the morphological parameters, and the illuminant are estimated simultaneously using the quasi-Newton method (see row 2 of Fig. 7) (see video 1, which can be found in the Computer Society Digital Library at ). The background image or its histogram is also provided by the user. Once the morphological parameters are estimated in frame 1, they remain fixed for the rest of the sequence. ### 4.2 Finger Bending and Occlusion Forces In the first sequence (Fig. 8 and video 2, which can be found in the Computer Society Digital Library at ), each finger bends in sequential order, eventually occluding other fingers and the palm. The background image is static and is obtained ![Image](images/3d00a677aa3e0d103aa2780d87dfbdf354a5c6225b9f1631c3d3cb7c3f3784df.jpg)
Fig. 8. Improvement due to self-occlusion forces: Rows 2-4 are obtained using our method. Rows 5-7 are obtained by summing the residual on the triangulated surface. Each row shows, from top to bottom: 1) the observed image, 2) the final synthetic image, 3) the final residual image, 4) a synthetic side view at 45 degrees, 5) the final synthetic image with residual summed on surface, 6) the residual for visible points on the surface, and 7) the synthetic side view.
from a frame where the hand is not visible. Each frame has $640 \times 480$ pixels. Note that the edges of the synthetic hand image match the edges in the input image, despite the fact that we do not use any explicit edge-related term in the objective function. Misalignment of the edges in the synthetic and observed images creates a large residual in the area between these edges and produces occlusion forces on the hand pointing in the direction that reduces the misalignment. To illustrate the improvements provided by occlusion forces, we simply remove the occlusion-forces when computing the energy gradient. The resulting algorithm is unable to track the hand for any significant period. It is also interesting to compare the minimization of the energy in (18) with the approach outlined in Section 2.2, which sums errors over the surface of the 3D mesh. These 3D points are uniformly distributed on the surface and their binary visibility is computed at each step of the quasi-Newton process. To account for the change of summation domain (from image to surface), the error associated to each in October 19,2026 at 14:15:34 UTC from IEEE Xplore. Restrictions apply. --- ![Image](images/906920fba9c6869efd0a960a0c1dc4f812e57e3e3552826d67384b9644bee9b4.jpg)
Fig. 9. Recovery from failure of the sum-on-surface method (Section 4.2) by our method with occlusion forces.
point is weighted by the inverse of the ratio of surface areas between the 3D triangular face and its projection. The errors associated with the background are also taken into account to remain consistent with the initial cost function. This also prohibits the hand from receding from the camera so that its projection shrinks. We include occlusion forces between the hand and background to account for variations of the background visibility while computing the energy gradient. The objective functions computed in both approaches would ideally be equal; their difference is bounded by the error induced by the discretization of integrals. For both methods we use a maximum of 100 iterations, which appears usually sufficient for convergence. ![Image](images/d1df52b58bc2639fb97c703715a754f9f405cb2cddec6c6563431128b3d59454.jpg)
Fig. 10. Sequence 2: Each row shows, (a)-(g): (a) the observed image, (b) the final synthetic image with limited pose space, (c) the final residual image, (d) the synthetic side view with an angle of 45 degrees, (e) the final synthetic image with full pose space, (f) the residual image, and (g) the synthetic side view.
![Image](images/f580bd083bee39ec9050068b879800371e4737da8bad921e03932e24989e129c.jpg)
Fig. 11. Sequence 3: Each row shows, from top to bottom: (a) the observed image, (b) the final synthetic image with limited pose space, (c) the final residual image, (d) the synthetic side view with an angle of 45 degrees, (e) the final synthetic image with full pose space, (f) the residual image, and (g) the synthetic side view.
The alternative approach produces reasonable results (Fig. 8, rows 5-7) but fails to recover the precise location of the finger extremities when they bend and occlude the palm. This is evident in column 3 of Fig. 8, where a large portion of the little finger is missing. Our approach (rows 2-4) yields accurate pose estimates through the entire sequence. The alternative approach fails because the hand/background silhouette is not particularly informative about the position of fingertips when fingers are bending. The residual error is localized near the outside extremity of the synthesized finger. Self-occlusion forces are necessary to pull the finger toward this region. We further validate our approach by choosing the hand pose estimated by the alternative method (Fig. 8, column 3, rows 5-7) as an initial state for our tracker with occlusion forces (Fig. 9 and video 3, which can be found in the Computer Society Digital Library at ). The initial pose and the corresponding residual are shown in column 1. The poses after 25 and 50 iterations are shown in columns 2 and 3, at which point the pose is properly recovered. ### 4.3 Stenger Sequences [^14] The second and third sequences (Figs. 10 and 11, videos 4 and 5, which can be found in the Computer Society Digital Library at ) were provided by Stenger for direct comparison with his state-of-the-art hand tracker [^14]. Both ---
(a)
![Image](images/a1597f489fe222697484cd6f1089d11341bd8094b2fcff90b982a86350cc109b.jpg)
(c)
Fig. 12. Sequence 4: Two hands. The rows depict four frames of the sequence, corresponding to (a) the observed images, (b) images synthesized using the estimated parameters, and (c) the residual images.
sequences are $320 \times 240$ pixels. In Sequence 2, the hand opens and closes while rotating. In Sequence 3, the index finger is pointing while the hand rotates in a rigid manner. Both sequences exhibit large interframe displacements, which makes it difficult to predict good initial guesses for minimization. The fingers also touch one another often. This is challenging because our optimization framework does not prohibit the interpenetration of hand parts. Finally, the background in Sequence 3 is dynamic, so the energy function has been expressed using (4). For computational reasons, Stenger et al. [^14] used a form of state-space dimension reduction, adapted to each sequence individually (8D for Sequence 2—2 for articulation and 6 for global motion—and 6D rigid movement for Sequence 3). We test our algorithm both with and without pose-space reductions (respectively, rows 2-4 and 5-7). For pose-space reduction, linear inequalities are defined between pairs or triplets of joint angles. This allows us to limit the pose space while keeping enough freedom of pose variation to make fine registration possible. We do not update the texture for those sequences after the first frame. With pose-space reduction, our pose estimates are qualitatively similar to Stenger's ones, although smoother through time. Without pose-space reduction, Stenger's technique becomes computationally impractical while our approach continues to produce good results. The lack of need for offline training and sequence-specific pose-space reduction are major advantages of our approach. ### 4.4 Occlusion Sequence 4 (Fig. 12 and video 6, which can be found in the Computer Society Digital Library at ) demonstrates the tracking of two hands, with significant occlusions as the left hand progressively occludes the right one. Texture and pose parameters from both hands are simultaneously estimated, and self-occlusions of one hand by another are modeled just as was done with a single hand. Fig. 12 shows that the synthetic image looks similar to the observed image due to the shading and texture models. In row 2, the ring finger and the little finger of the right hand are still properly registered despite the large occlusion. Despite the translational movement of the hands, we kept all 28 DOFs on each hand while performing pose estimation. ### 4.5 Quantitative Performance on Natural Images While existing papers on hand pose tracking have relied mainly on qualitative demonstrations for evaluation, here we ![Image](images/19926d315874d1288d854222613f3b3e2365253bd446bfdb4cde8d0beee4a8f0.jpg)
(a)
![Image](images/c576106681cba837b5a39bb63a8bea338f6a6d5fd25e7a904e3e871197496170.jpg)
(b)
Fig. 13. Distance between reconstructed and ground truth 3D locations: (a) the distances for each of the finger tips and (b) the distances for each of the metacarpophalangeal point and the palm basis.
introduce quantitative measures on a binocular sequence (see video 7, which can be found in the Computer Society Digital Library at ) from which we can estimate 3D ground truth at a number of fiducial points. The hand is captured with two synchronized, calibrated cameras (see rows 1 and 4 in Fig. 14). To obtain ground truth 3D measurements on the hand, we manually identify, at each frame, the locations of 11 fiducial points on the hand, i.e., each finger tip, each metacarpophalangeal (MP) joint, and the base of the palm. These locations are shown in the top-left image of Fig. 14. From these 2D locations in two views, we obtain 3D ground truth by triangulation [^33]. The hand is tracked using just the right image sequence. For each of the 11 fiducial points, we then compute the distance between the 3D ground truth and the estimated position on the reconstructed hand surface. Because the global size of our model does not necessarily match the true size of the hand, in each frame we allow a simple scaling transform of the estimated hand to minimize the sum of squared distances between the 3D ground truth points and the recovered locations. The residual euclidean errors for each point after this scaling are shown in Fig. 13. Errors for the thumb tip are large for frames 340-360 and errors for the middle finger are large for frames 240-350. During those periods, the optimization loses track of the positions of the middle finger (see Fig. 14, column 3) and the thumb (see Fig. 14, column 4). Thumb errors are also large between frames 50 and 150, despite the fact that the estimated pose appears consistent with the right view used for tracking (see residuals in Fig. 14 row 3, columns 1-3). This reflects the intrinsic problem of monocularly estimating depth. The last row in Fig. 14 shows the hand model synthesized from the left camera viewpoint, for which the depth errors are evident. ### 4.6 Quantitative Performance on Synthetic Images We measure quantitative performance using synthetic images. The image sequence is created with the Poser© software (see video 8, which can be found in the Computer Society Digital Library at ). The hand model used in Poser differs somewhat from our model both in the triangulated surface and the joint parameterization. To obtain realistic sequences, we enable ambient occlusion and cast shadows in the rendering process, neither of which --- ![Image](images/9a52fa8305e946f5ced8196ba6b56180c14d7d1dfa65ec4b76eab89f7183cf0c.jpg)
Fig. 14. Binocular Video, which can be found in the Computer Society Digital Library at : The columns show frames 2, 220, 254, and 350. Each row, (a)-(e), depicts: (a) the image from the right camera, (b) the corresponding synthetic image, (c) the residual for the right view, (d) the observed left image (not used for tracking), and (e) the synthetic image seen from the left camera.
is explicitly modeled in our tracker. Row 1 of Fig. 15 shows several frames from the sequence. A quantitative evaluation of estimated pose is obtained by measuring a distance between the reconstructed 3D surface and the ground truth 3D surface. For visualization of depth errors over the visible portion of the hand surface, we define a distance image $D_{s}$: $$ D_{s}(p)=\inf_{q\in S_{g}}\left(\Pi_{S_{s}}^{-1}(p),q\right)\text{if}p\in\Omega_{s},\quad\infty\text{otherwise}, $$ where $S_g$ is the ground truth hand surface, $S_s$ is the reconstructed hand surface, and $\Omega_s$ is the silhouette of the surface $S_s$ in the image plane. In addition, $\Pi_{S_s}^{-1}$ is the back projection function onto the surface $S_s$, i.e., the function that maps each point p in $\Omega_s$ to the corresponding point on the 3D surface $S_s$. To cope with scaling uncertainty, as discussed above, we apply a scaling transform on the reconstructed surface whose fixed point is the optical center of the camera, such that the mean depth of the visible points in $S_s$ is equal to the mean depth of the visible points in $S_g$, i.e., $$ \begin{array}{r}{\frac{\int_{p\in\Omega_{s}}\Pi_{S_{s}}^{-1}(p)\cdot\hat{z}d p}{\int_{p\in\Omega_{s}}1d p}=\frac{\int_{p\in\Omega_{s}}\Pi_{S_{g}}^{-1}(p)\cdot\hat{z}d p}{\int_{p\in\Omega_{g}}1d p},}\end{array} $$
(a)
where $\hat{z}$ is aligned with the optical axis, $\Omega_g$ is the silhouette in the ground truth image, and $\Pi_{S_g}^{-1}$ is the back projection function onto the surface $S_g$. Fig. 15 depicts the performance of our tracker and the hand tracker described in [^10]. The current method is
(b)
![Image](images/a1458045db97b9ea8f6b9932bb1c0cffd16c1891b1f1e2eedd1d4f1f43b17962.jpg)
Fig. 15. Synthetic sequence: (a) Depicts individual frames. (b)-(e) Depict tracking results, i.e., (b) estimated pose, (c) residual image, (d) side view superimposing ground truth and the estimated surfaces, and (e) distance image $D_{s}$. (f)-(i) Depict results obtained with the method in [^10], i.e., (f) generative image of [^10] at convergence, (g) log-likelihood per pixel according to [^10], (h) a side view superimposing ground truth and estimated surfaces, and (i) distance image $D_{s}$.
shown to be significantly more accurate. Despite the fact that the model silhouette matches the ground truth in the row 3 of Fig. 15, the errors over the hand surface are significant, e.g., see the middle and ring fingers. The method in [^10] relies on a matching cost that is solely a function of the hand silhouette and is therefore subject to ambiguities. Many of these ambiguities are resolved with the use of shading and texture. Finally, Fig. 16 plots the median distance $D_{s}(p)$ over points in $\Omega_{s}$ shown as a function of time for both methods. Again, note that our method outperforms that in [^10] for most frames. ## 5 Discussion We describe a novel approach to the recovery of 3D hand pose from monocular images. In particular, we build a detailed generative model that incorporates a polygonal in October 19,2026 at 14:15:34 UTC from IEEE Xplore. Restrictions apply. --- ![Image](images/cc5d91e2486088ed878be22f7fe09fa2026b090afc85159b8e9ddcd248b935a5.jpg)
Fig. 16. Comparison, through all frames of the video, of the median of the distances $D_{s}(p)$ over the point in $\Omega_{s}$ with our method and the one in [^10].
mesh to accurately model shape and a shading. We estimate the model parameters (i.e., shape, pose, texture, and lighting), which are determined through a variational formulation. A rigorous mathematical formulation of the objective function is provided in order to properly deal with occlusions. Estimation with a quasi-Newton technique is demonstrated with various image sequences, some with ground truth, for a hand model with 28 joint degrees of freedom. The model provides state-of-the-art pose estimates, but future work remains in order to extend this work to an entirely automatic technique that can be applied to long image sequences. In particular, there remain several situations in which the technique described here is prone to converge to a poor local minima and therefore lose track of the hand pose: 1. There are occasional reflection ambiguities which are not easily resolved by the image information at a single time instant. 2. Fast motion of the fingers sometimes reduces the effectiveness of our initial guesses based on a simple form of extrapolation from previous frames. 3. Entire fingers are sometimes occluded. In this case, there will be no image support for a pose estimate of that region of the hand. 4. Cast shadows are only modeled as texture, so dynamic cast shadows can be difficult for the shading model to handle well (as can be seen in some of the experiments above). Modeling cast shadows is tempting, but it would mean a discontinuous residual function of $\theta$ along synthetic shadow borders, which are likely very complicated to handle analytically. 5. Collisions or interpenetration of the model fingers creates aliasing along the intersection boundaries in the discretized synthetic image. The change of visibility along self-collisions could be treated using forces along these boundaries, but this is not currently done and therefore the optimization may not converge in these cases. A solution might be to add non-linear nonconvex constraints that would prevent collisions. When our optimization loses track of the hand pose, we have no mechanism to reinitialize the pose, or explore multiple pose hypotheses, e.g., with discriminative methods. Dynamical models of typical hand motions, and/or latent models of typical hand poses may also help resolve some of these problems due to ambiguities and local minima. Last, but not least, the use of discrete optimization methods involving higher order interactions between handparts [^34] with explicit modeling of visibilities [^35] may be a promising approach for improving computational efficiency and for finding better energy minima. ## ACKNOWLEDGMENTS This work was financially supported by grants to David J. Fleet from NSERC Canada and the Canadian Institute for Advanced Research (CIFAR). Martin de La Gorce was with the Ecole Centrale de Paris during the time in which this work was performed. ## REFERENCES [^1]: S. Lu, D. Metaxas, D. Samaras, and J. Oliensis, "Using Multiple Cues for Hand Tracking and Model Refinement," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 443-450, 2003. [^2]: A.O. Bălan, M. Black, H. Haussecker, and L. Sigal, "Shining a Light on Human Pose: On Shadows, Shading and the Estimation of Pose and Shape," Proc. IEEE Conf. Computer Vision, pp. 1-8, 2007. [^3]: R. Rosales, V. Athitsos, L. Sigal, and S. Scarloff, "3D Hand Pose Reconstruction Using Specialized Mappings," Proc. IEEE Conf. Computer Vision, vol. 1, pp. 378-385, 2001. [^4]: N. Shimada, "Real-Time 3-D Hand Posture Estimation Based on 2-D Appearance Retrieval Using Monocular Camera," Proc. IEEE ICCV Workshop Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems, pp. 23-30, 2001. [^5]: V. Athitsos and S. Sclaroff, "Estimating 3D Hand Pose from a Cluttered Image," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 432-442, 2003. [^6]: T.E. de Campos and D.W. Murray, "Regression-Based Hand Pose Estimation from Multiple Cameras," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 782-789, 2006. [^7]: T. Heap and D. Hogg, "Towards 3D Hand Tracking Using a Deformable Model," Proc. Int'l Conf. Automatic Face and Gesture Recognition, pp. 140-145, 1996. [^8]: J. Rehg and T. Kanade, "Model-Based Tracking of Self-Occluding Articulated Objects," Proc. IEEE Int'l Conf. Computer Vision, pp. 612-617, 1995. [^9]: B. Stenger, P.R.S. Mendonça, and R. Cipolla, "Model-Based Hand Tracking Using an Unscented Kalman Filter," Proc. British Machine Vision Conf., vol. 1, pp. 63-72, 2001. [^10]: M. de La Gorce and N. Paragios, "A Variational Approach to Monocular Hand-Pose Estimation," Computer Vision and Image Understanding, vol. 114, no. 3, pp. 363-372, 2010. [^11]: E. Sudderth, M. Mandel, W. Freeman, and A. Willsky, "Visual Hand Tracking Using Nonparametric Belief Propagation," Proc. IEEE Conf. Computer Vision and Pattern Recognition Workshop, pp. 189-196, 2004. [^12]: H. Ouhaddí and P. Horain, "3D Hand Gesture Tracking by Model Registration," Proc. Int'l Workshop Synthetic-Natural Hybrid Coding and 3D Imaging, pp. 70-73, 1999. [^13]: Y. Wu, J.Y. Lin, and T.S. Huang, "Capturing Natural Hand Articulation," Proc. IEEE Int'l Conf. Computer Vision, pp. 426-432, 2001. [^14]: B. Stenger, “Model-Based Hand Tracking Using a Hierarchical Bayesian Filter,” PhD dissertation, Univ. of Cambridge, Mar. 2004. [^15]: M. Brubaker, L. Sigal, and D. Fleet, “Video-Based People Tracking,” Handbook of Ambient Intelligence and Smart Environments, Springer, 2009. [^16]: A.O. Bălan, L. Sigal, M.J. Black, J.E. Davis, and H.W. Haussecker, "Detailed Human Shape and Pose from Images," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-8, 2007. [^17]: M. Bray, E. Koller-Meier, L. Van Gool, and N.N. Schraudolph, "3D Hand Tracking by Rapid Stochastic Gradient Descent Using a Skinning Model," Proc. European Conf. Visual Media Production, pp. 59-68, 2004. [^18]: N. Magnenat-Thalmann, R. Laperrière, and D. Thalmann, "Joint-Dependent Local Deformations for Hand Animation and Object Grasping," Proc. Graphics Interface '88, 1988. [^19]: J.P. Lewis, M. Cordner, and N. Fong, "Pose Space Deformation: A Unified Approach to Shape Interpolation and Skeleton-Driven Deformation," Proc. ACM SIGGRAPH, pp. 165-172, 2000. [^20]: V. Blanz and T. Vetter, “A Morphable Model for the Synthesis of 3D Faces,” Proc. ACM SIGGRAPH, pp. 187-194, 1999. [^21]: M. Soucy, G. Godin, and M. Rioux, "A Texture-Mapping Approach for the Compression of Colored 3D Triangulations," The Visual Computer, vol. 12, no. 10, pp. 503-514, 1996. Authorized licensed use limited to: ShanghaiTech University. Downloaded on October 19,2026 at 14:15:34 UTC from IEEE Xplore. Restrictions apply. --- [^22]: C. Hernández, “Stereo and Silhouette Fusion for 3D Object Modeling from Uncalibrated Images under Circular Motion,” PhD dissertation, Ecole Nationale Supérieure des Télécomm., May 2004. [^23]: H. Sidenbladh, M.J. Black, and D.J. Fleet, "Stochastic Tracking of 3D Human Figures Using 2D Image Motion," Proc. European Conf. Computer Vision, vol. 2, pp. 702-718, 2000. [^24]: N. Paragios and R. Deriche, "Geodesic Active Regions for Supervised Texture Segmentation," Proc. IEEE Int'l Conf. Computer Vision, vol. 2, pp. 926-932, 1999. [^25]: G. Unal, A. Yezzi, and H. Krim, "Information-Theoretic Active Polygons for Unsupervised Texture Segmentation," Int'l J. Computer Vision, vol. 62, no. 3, pp. 199-220, 2005. [^26]: P. Gargallo, E. Prados, and P. Sturm, "Minimizing the Reprojection Error in Surface Reconstruction from Images," Proc. IEEE Int'l Conf. Computer Vision, pp. 1-8, 2007. [^27]: A. Delaunoy, E. Prados, P. Gargallo, J.-P. Pons, and P. Sturm, "Minimizing the Multi-View Stereo Reprojection Error for Triangular Surface Meshes," Proc. British Machine Vision Conf., 2008. [^28]: P.V. Sander, H. Hoppe, J. Snyder, and S.J. Gortler, “Discontinuity Edge Overdraw,” Proc. Symp. Interactive 3D Graphics, pp. 167-174, 2001. [^29]: F.C. Crow, "A Comparison of Antialiasing Techniques," IEEE Computer Graphics and Applications, vol. 1, no. 1, pp. 40-48, Jan. 1981. [^30]: L. Carpenter, "The A-Buffer, an Antialiased Hidden Surface Method," ACM SIGGRAPH Computer Graphics, vol. 18, no. 3, pp. 103-108, 1984. [^31]: A. Griewank, Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation. SIAM, 2000. [^32]: A. Conn, N. Gould, and P. Toint, Trust-Region Methods. SIAM, 2000. [^33]: R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision. Cambridge Univ. Press, 2000. [^34]: N. Komodakis and N. Paragios, “Beyond Pairwise Energies: Efficient Optimization for Higher-Order mrls,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 2985-2992, 2009. [^35]: C. Wang, M. de La Gorce, and N. Paragios, "Segmentation, Ordering and Multi-Object Tracking Using Graphical Models," Proc. IEEE Int'l Conf. Computer Vision, pp. 747-754, 2009. ![Image](images/1594bd49d6f01dfbc1b71371137d7d7f3282cbe386af6f8966ced4bb16dae656.jpg) Martin de La Gorce received the engineer diploma from the Institut Superieur d'Electonique de Paris in 2004, the master diploma in applied mathematics, computer vision, and statistical learning from the Ecole Superieure de Cachan in 2005, and the PhD degree in applied mathematics from the Ecole Centrale de Paris in 2009. He is now employed as a research engineer at Image Metrics, a leading company in 3D facial animation for the entertainment industry. His work has been presented at various international conferences (CVPR, BMVC) and has been published in the Computer Vision and Image Understanding Journal (CVIU). His research interests include image processing, computer vision, and human-computer interaction. He is a student member of the IEEE. ![Image](images/565aa8f8e115ca50e80c6e6e96cd26a9a4c3f35b2539fc5525c8067186ae9033.jpg) David J. Fleet received the PhD degree in computer science from the University of Toronto in 1991. He was on the faculty at Queen's University in Kingston from 1991 to 1998, and then was an area manager and research scientist at the Palo Alto Research Center (PARC) from 1999 to 2003. In 2004, he joined the University of Toronto as a professor of computer science. His research interests include computer vision, image processing, visual perception, and visual neuroscience. He has published research articles, book chapters, and one book on various topics including the estimation of optical flow and stereoscopic disparity, probabilistic methods in motion analysis, modeling appearance in image sequences, motion perception and human stereopsis, hand tracking, human pose tracking, latent variable models, and physics-based models for human motion analysis. In 1996, he was awarded an Alfred P. Sloan Research Fellowship. He has won paper awards at ICCV '99, CVPR '01, UIST '03, and BMVC '09. In 2010, he was awarded the Koenderink Prize for his work with Michael Black and Hedvig Sidenbladh on human pose tracking. He has served as an area chair for numerous computer vision and machine learning conferences. He was program cochair for the 2003 IEEE Conference on Computer Vision and Pattern Recognition, and will be cochair for the 2014 European Conference on Computer Vision. He has been an associate editor, and associate editor-in-chief for the IEEE Transactions on Pattern Analysis and Machine Intelligence, and currently serves on the RPAMI Advisory Board. He is a senior member of the IEEE. Nikos Paragios is a professor at the Ecole Centrale de Paris leading the Medical Imaging and Computer Vision Group in the Applied Mathematics Department. He is also affiliated with INRIA Saclay Ile-de-France, heading the GALEN group, a joint research team between ECP/INRIA. Prior to that he was a professor (2004-2005) at the Ecole Nationale de Ponts et Chaussees and with Siemens Corporate Research (Princeton, New Jersey, 1999-2004) as a project manager, senior research scientist, and research scientist. He has held numerous adjunct and visiting appointments at institutions such as New York University (2004) and Yale University (2007). He has coedited four books, published more than 150 papers in the most prestigious journals and conferences of medical imaging and computer vision, has 17 US issued patents and more than 20 pending. He is a fellow of the IEEE, an associate editor for the IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), an area editor for the Computer Vision and Image Understanding Journal (CVIU), and a member of the editorial board of the International Journal of Computer Vision (IJCV), the Medical Image Analysis Journal (MedIA), the Journal of Mathematical Imaging and Vision (JMIV), and the Imaging and Vision Computing Journal (IVC). His research interests include image processing, computer vision, medical image analysis, and human computer interaction. () ![Image](images/2fa3eb20625b3d3536661e62878032743bb05c5abb4b3f45c790a0de6d47b07a.jpg) For more information on this or any other computing topic, please visit our Digital Library at .