Главная > Документ

IEEE TRANSACTIONS ON Visualization and COMPUTER GRAPHICS, manuscript ID first page9

Interactive marker-less tracking of human limbs

Srinivasa G. Rao, and Larry F. Hodges, Member, IEEE

Abstract— One of the hard problems that combines research in Computer Graphics and Computer Vision is tracking human limbs in real time without using any markers. This problem is known to be notoriously difficult and ill posed. In this paper we present a technique to track human limbs at near interactive rates without markers. This work builds on, modifies, and adds to the state-of-the-art tracking techniques developed by previous researchers. We use a particle filtering tracking algorithm that uses 3d data derived from a modified visual hull algorithm. The particle filtering algorithm runs on a GPU and hence we are able to track human limbs at 4Hz with reasonable accuracy. This method has the potential to be applied to tracking the whole human body without markers.

Index Terms— I.3.7.a Animation, I.4.8 Scene Analysis, I.4.8.n Tracking, I.3.7 Three Dimensional Graphics and Realism.


  • Srinivasa G. Rao is a PhD student in the Department of Computer Science, University of North Carolina, Charlotte, NC 28213. E-mail: srao3@uncc.edu.

  • Professor Larry F. Hodges is the chair of the Department of Computer Science, University of North Carolina, Charlotte, NC 28213. E-mail: lfhodges@uncc.edu.

Manuscript received (insert date of submission if desired). Please note that all acknowledgments should be placed at the end of the paper, before the bibliography.


1 Introduction and Motivation

THE applications of Marker-less Limb Tracking in Real Time (abbreviated as MLTRT from now on) are many. Firstly, if we can track a person’s limb poses over time, we can construct a free view-point video by rendering his 3d model from novel camera positions. The viewer watching this video can alter his view-point in the video at will. Previously, [1], [2], and [3] have achieved this and have synthesized video from novel view-points, but only by offline processing. Secondly, we can render the tracked position of user’s limbs in VR environments without use of cumbersome maker based systems. [21], [22] have shown that if a user in a VR environment can see his limbs, the feeling of immersion is increased. Third, MLTRT will be very useful for new types of user interfaces. We can use human actions such as pointing, waving hands etc. instead of pointing devices such as mice or joysticks to interact with our computer. Fourth, MLTRT can be used for achieving telepresence and telecollaboration in combined virtual and real environments. Gross et al. [14], achieve this by constructing 3d video consisting of video fragments (3d point samples derived from a visual hull) and streaming it across the network. MLTRT would enable us to just stream the joint angle values over the network, thereby reducing the amount of bandwidth required to drive such systems.

Section 2 describes our ideas and further sections show how we implemented these ideas. In section 3 we describe the previous work done both in Computer Vision and Computer Graphics literature on human motion tracking. Section 4 gives an overview of our work. Section 5 goes into details of particle filtering. Section 6 describes implementation of the particle filtering algorithm on a GPU and its tracking speed and accuracy. In section 7 we describe how we obtain the necessary 3d data from a visual hull. Section 8 describes real time implementation of the system. Section 9 describes our attempt to do marker less limb tracking in PCA (Eigen) space. Finally, we conclude and describe our future work.

2 Contributions

Our ideas and technical contributions can be summarized as follows –

(a) Even though vision and computer graphics researchers have used camera images as input for tracking algorithms, only the silhouette information is used for estimating motion parameters [1]. Some of the researchers have used 3d data [6], [7] but have not used a robust tracking framework such as particle filtering. Our major contribution is to integrate current particle filtering techniques primarily found in computer vision literature and three-dimensional visual hull techniques from computer graphics.

(b) Previous techniques for MLT have required offline processing and hence cannot track in real time as data is acquired. We implement a parallelizable particle filtering algorithm with 3d data as input on a GPU that enables us to track human limbs at near interactive rates.

3 Previous Work

This section starts by summarizing the work done on MLT in the computer vision literature. Then we summarize similar research done in the computer graphics literature. MLTRT is challenging due to the high dimensionality of degrees of freedom (DOFs) of the articulated human body, the lack of explicit depth information in 2D images, occlusion, and complexity of human dynamics and kinematics. There has been extensive research in the field of computer vision on MLT, but not at real-time rates (MLTRT). Deutscher et al. [9], [10] have used particle filters to do MLT. Particle filtering provides a robust Bayesian framework for human mo­tion capture. A kinematic body model is used to “search for” the optimal body pose (a vector of angles representing joint angles of the human body) such that, it minimizes the error between the kinematic model’s projection on an image plane and body silhouettes obtained from doing background subtraction on camera images. We refer the interested readers to [9], [10] for a more detailed description. This is the basic framework that we have modified and we describe it in detail in section 5. Sigal et al. [11] pose the problem of 3D human limb tracking as one of inference in a graphical model. Their model of the body is a collection of loosely-connected limbs. Conditional probabilities relating the 3D pose of connected limbs are learned from motion-captured training data. Human pose and motion estimation is then solved with non-parametric belief propagation using a variation of particle filtering that can be applied over a general loopy graph.Gavrila and Davis [12] use a decomposition approach and a best-first technique to search through the high dimensional pose parameter space, to search for the pose of the articulated model that best matches image data. Both ellipsoids [7] and super quadrics [12] have been used in kinematic models. Bregler and Malik [13] have used the product of exponential maps and twist motions to track a walking person in a video captured from his side view. Cheung et al. [8] use an algorithm that iteratively segments points on the silhouettes to each articulated part of the kinematic model and estimates the motion of each individual part using the segmented silhouette. Luck and Small [6] create a force field exerted by voxels in a volumetric data set to fit a kinematic skeleton at near interactive frame rates.

Computer Graphics researches have gone a step further and generated free view-point and virtual videos from novel views after estimating motion parameters. [15], [16] compute a visual hull, which is a coarse representation of human shape from a set of input images acquired from cameras at known positions. The visual hull can then be rendered from novel view-points. Vedula et al. [17] have created models of moving human actors using Multi-view video, voxel-based reconstruction, and space-time interpolation along the 3D scene flow. Carranza et al. [1] have used sophisticated human models for offline parameter estimation and then rendered free view-point video from novel view-points. Gross et al.[14] have used 3d point samples derived from a visual hull for telepresence in a CAVE-like environments. But, a visual hull and its derivatives are a very coarse approximation for a high quality video. Recently [4], [5] have demonstrated that, with the use of many cameras and high bandwidth and/or offline computation, it is possible to generate virtual videos and autostereoscopic 3d display even without estimation of any motion parameters. But, in such systems, not only is the computational and/or bandwidth cost very high, but also novel views generated are very close to and in-between the views captured by the cameras.

4 Overview of our work

Our work aims to generate a high quality LIVE action virtual video at near interactive rates. To accomplish this, we track the joint angles of human motion using a simple kinematic model. Using the joint angle data estimated by the tracking algorithm we can animate a high resolution, high quality, pre-processed, 3d human model from novel view-points. We achieve this by implementing a particle filtering tracking algorithm that runs on a Graphics Processing Unit (GPU) and uses 3d data derived from a visual hull.

MLT, let alone MLTRT, is known to be a notoriously difficult and ill posed problem. In this work, we have decided to solve a much simpler problem first – track only human limbs without any markers on the limbs. Fig. 1 shows architecture of our system.

Fig. 1. The system architecture.

5 Particle filtering algorithm (PFA)

In this section, we describe the particle filtering techniques we use for tracking. Our algorithm is a modification of the techniques originally presented in [9], [10].

The term “particle” should not be confused with particles used in physical simulation of fire/smoke/water etc. Here, “particle” is a vector which has dimension equal to the number of degrees of freedom (say d) of the limbs of an articulated model. It can be thought of as a point in d-dimensional space.

We first simulated our entire work and studied the nature of the MLTRT problem using the CMU Mocap data and viewing software [18]. To determine the accuracy of our tracking approach, we compared the synthetic 3d data points rendered by MOCAP software using the motion captured data (Fig. 2b, yellow points) to the pose determined by our tracking algorithm. Our kinematic model (Fig. 2a) consisted of ellipsoidal limbs of the same bone structure with 10 DOF - three rotation values for each thigh, one rotation value (along X axis (red line)) for each leg, and one rotation value (along X axis) for each foot. Hence in our case each particle is a vector of floating point values with 10 components, for legs, and is a 4 component vector (3DOF for shoulder and 1DOF for arm) for tracking arms.

Fig. 2. Clockwise starting from bottom left (a) kinematic model, (b) synthetic 3d data (c), (d) particles having imperfect match, (e) particle having perfect match with 3d data and (f) typical MLTRT weight function plot, green line indicating ground truth, particle where the weight function has the highest peak.

5.1 Multimodal property of the PFA weight function

Given a particle P (vector with dimension d), it describes a pose of the kinematic model. A measure of how good this particle matches the actual 3d data is the fraction of 3d points that are inside the ellipsoids of the model for that particular pose. This is called the weigh, w, of the particle. Weights are between [0, 1]. For example, the particle in Fig. 2c has a lesser weight than the particle in Fig. 2d. The particle in Fig. 2e has a weight of 1, since it matches the 3d data perfectly – this is the particle/pose that we want to search for. But, even a non perfect match like the one shown in Fig. 2d can have weight close to 1. This is because; ellipsoids representing the right foot and the lower part of the right leg of the kinematic model contain some 3d points in the ground truth that are actually a part of middle part of the right leg! Due to this the weight function in MLTRT is multimodal, which means it has many peaks, which further means we cannot use gradient descent methods or its variants to search for the peak of such a function! Say we have only 4DOF (rotation along X axis for thighs and limbs). Also suppose we know the maximum and minimum angles of rotation for each DOF. We want to find where the highest peak lies. If we take 10 values in between the maximum and minimum values for each DOF, we end up with 104 particles whose weights we have to evaluate to find the peak of the function! Hence, the computation is exponential. Fig. 2f shows a plot (particle number along X axis and weights along Y axis) of weights of 104 particles; only the middle 6000 particles are shown for brevity. This illustrates the multimodal nature of the MLTRT weight function. The green line represents the particle in Fig. 2(e), which has the pose that is closest to ground truth.

Fig. 3. The annealing stack of PFA for arbitrary frame, Frame K– green line gives the ground truth. See sections 5.1, 5.2 for description of the process.

5.2 Details of the PFA

A Particle Filtering Algorithm (PFA for brevity) may be used to search for a peak on such a multimodal function. Fig. 3 demonstrates how the PFA works.

We have to track the 3d data each frame, which means we have to run the PFA each frame. Each run of the PFA has an annealing stack that has several layers. The main idea is based on simulated annealing [23]. At the topmost layer (say M-1), the weight function described in the previous section is smoothed so that the search does not get “distracted” by smaller peaks. As the PFA progress from the topmost layer to the bottommost layer, the degree of smoothing of the weight function is decreased such that at layer 0 we use the original unsmoothed weight function. At the beginning of the PFA, a particular number (say N) of random particles (poses or vectors, each of d dimensions) are generated. Their weights are computed depending on how close their match is with the actual 3d data as described above (for a more detailed description see Section 6). Then, a fraction of particles (say half of them), that have higher weights are selected. Gaussian noise (also a vector) with a mean zero vector and particular variance (diagonal matrix, correlations between noise in different dimensions is assumed to be 0) is generated and added to the selected weighted particles to generate unweighted particles for the next layer. This process of computing weights of particles and “propagating” only a heavier fraction of those to the next layer is repeated M times. Thus, the particles/poses that match the given 3d data more closely “live” and the rest “die out” as the PFA progresses. The PFA is very robust for tracking because some of the particles with less initial weight “live long enough” and sometimes turn out to be a very close match at the end.

We learn the variance for each DOF from the motion captured data and use those values while running the PFA. A histogram of the absolute difference between consecutive frame values for each DOF is generated. A maximum change in that DOF is taken from each histogram after discounting for the noise. The standard deviation of the noise for the particular DOF is set to be one third of this maximum value at the top most layer of the PFA.

As the PFA progresses from the topmost to the bottommost layer, noise of smaller and smaller variance is added to the heavier fraction of weighted particles that are selected for generating un-weighted particles for the next layer and the weight function becomes closer and closer to the actual unsmoothed weight function. This results in many particles that are near the desired peak of the multimodal MLTRT weight function as shown in Fig. 3. After we reach the bottom layer (layer 0), an average of the selected particles (weighted by their weights) is computed to determine the particlethat is very close to the actual 3d data – the particle where the highest peak of multimodal weight function occurs – the particle/pose which we want to search for! Fig. 3 shows the PFA in action with number of particles (N) = 4, selection fraction of 0.5 (i.e. 2 heavier particles getting selected) and, number of layers (M) = 3.

5.3 Technical Contributions

We have modified the PFA algorithm described in [9] and [10] in the following ways.

Fig. 4. Clockwise from top left. 2d image from (a) camera in front (b) camera on the left (c) a camera behind (d) camera on the right (e) view from camera at 45deg (f) result of using 3d data.

(a) Deutscher et al. [9], [10] use 2d images to compute the weight function. The contours of an actor are obtained from doing background subtraction on a set of camera images. They then project the kinematic model onto the camera images and compute the weight of a particle depending on how close the projection matches the contours. We have found that the use of 2d images causes many problems during PFA. An example is shown in Fig. 4 – green is the kinematic model, red is ground truth - (a), (b), (c) and (d) show four images from four surrounding synthetic cameras, placed at 90 degrees separation (front, left, back and right) around the synthetic kinematic model. Notice that, even though the reprojection error (number of green pixels not overlapping with red foreground contour) is not very high, the position picked by the PFA is far from ground truth! The left leg of the kinematic model is matched up with the right leg of the ground truth and vice versa as seen in Fig. 4(e)! But when we use 3d data for tracking, as shown in Fig. 4(f), this problem does not happen. In addition, using 3d data for weight computation is more efficient for GPU implementation as we show in Section 6.

Fig. 5. Clockwise from top left (a) Distribution is only approximately uniform. Hence (b) particles with lesser weights are sometimes Selected more number of times for Generating new particles (# SG) than ones with higher weights, which is avoided by having deterministic algorithm (c). (d) Use of Crossover operator results in particles having more variance in some DOFs.

(b) Deutscher et al. [9], [10] select the particles with a probability proportional to their weights, with replacement, for generating new particles. Initially, we implemented this approach by dividing the unit interval [0, 1] into N (number of particles) intervals whose lengths were equal to normalized particle weights. Then, we generated a uniform random number. Based on which interval this random number belonged to, we chose the corresponding particle. But, pseudo random number generators generate random numbers whose distribution is only approximately uniform (or approximately Gaussian, for that matter). See Fig. 5(a). Due to this sometimes particles with lesser weights were chosen more times than ones with heavier weights for generating new particles for the next layer – contrary to that required by the PFA (Fig. 5(b)). Hence, to fix this problem, we now use a deterministic rather than a probabilistic algorithm to do selection (Fig. 5(c)).

(c) Also, we found that use of the cross-over operator found in Genetic Algorithms [10] actually resulted in particles having more variance during PFA compared to those simulations where we did not use it. The smaller the variance is for the DOFs, the better it is for the PFA. Hence, as compared to [10] we do not use cross-over operator in our implementation. This comparison is shown in Fig. 5(d).

6 Implementation of the PFA on the GPU

The PFA executes partly on the CPU and partly on a GPU. To speed up the execution of the PFA, the weight computation, which is parallelizable, is implemented on a GPU. The remainder of the PFA is implemented on the CPU. The part of the PFA that executes on the CPU supplies appropriate texture data to the GPU during multi-pass rendering, which then executes its part of the PFA. Fig. 6 shows an articulated model (on the left) with a thigh and a leg being represented by one ellipsoid each (solid blue) with their local coordinate systems. On the right, we have synthetic ground truth 3d data color coded depending on which limb (ellipsoid) of the articulated model the 3d point belongs to. The 3d data is in the global coordinate system. Yellow means it belong to the leg, cyan means it belongs to the thigh and blue means it does not belong to any part of the kinematic articulated model. Green colored 3d points are not of interest. The articulated kinematic model and synthetic 3d data have the same “root” position, i.e., the green sphere of the articulated model on the left contains all red 3d points – but here they are shown apart just for clarity. Also shown is the global coordinate system.

6.1 3d weight calculation

The weight of a particle (pose) like the one shown in Fig. 6 can be calculated by this simple algorithm.

(1) Mark all the 3d data points as Belongs to No Ellipsoid (BNE) i.e., set all BNE flags of all 3d points to 1.

Fig. 6. 3d Weight Calculation.

(2) Do for all ellipsoids (having radii xradius, yradius and zradius in corresponding directions) in articulated model -

  • Transform ellipsoid’s local coordinate system to the global coordinate system. We get O, local coordinate system’s origin’s position in global coordinates; and X, Y, Z, unit vectors in the local coordinate system transformed to the global coordinate system.

  • Do for all 3d points, P = (x, y, z), that have BNE set to 1

Let D = (P – O); Dx = (X-O).D; Dy = (Y-O).D; Dz = (Z-O).D; If a 3d point is inside the ellipsoid, i.e., (Dx <= xradius) and (Dy <= yradius) and (Dz <= zradius), then set the BNE flag to 0, indicating that 3d point belongs to one of the ellipsoids.

(3) Weight of the particle w = (total number of 3d points – number of 3d points with BNE flag set to 1)/ (total number of 3d points).

6.2 Parallelization of 3D weight calculation

Computational complexity in the PFA comes from calculating the weight of each particle, in each layer, and for each frame. The process can be easily parallelized. Notice that the BNE flags for different particles are independent of each other. Also notice that, given a single particle we have to process BNE flags for the 1st ellipsoid first; and then for the 2nd ellipsoid, using only those 3d points whose BNE flag was NOT set to 0 by the 1st ellipsoid and so on. So, if we can feed the same 3d data for every particle, the BNE flags for a particular ellipsoid can be processed for all the particles independently in parallel – this corresponds to one rendering. Notice that a particular ellipsoid will have different orientations for different particles. So we process BNE flags for the 1st ellipsoid of all particles (independent of each other) and use the resulting data for the 2nd ellipsoid of all particles and so on. Hence we have to update the frame buffer a number of times equal to the number of ellipsoids in the articulated model. In this way we have parallelized the weight computation.

We use Open GL Shading Language (GLSL) to program the GPU. We use the fragment shader (FS for brevity) on the GPU to do this parallel computation. The vertex shader (VS) is just a pass-through shader. We input 3d data into FS as a 512X512 RGB texture. We do multi-pass rendering because we need to use the data from the previous render as an input for the next render. Each render corresponds to an ellipsoid in the articulated kinematic model. In each rendering, we render N textured squares – each square corresponds to one particle. We use a uniform variable 4X4 matrix (GLSL) to feed in the ellipsoid’s orientation matrix for the corresponding particle to the FS. Each point in the square triggers the FS which checks if the corresponding 3d point (read from texture) has its BNE flag set to 1. If it is, then the FS checks whether the point is inside the current ellipsoid. If so, its BNE flag is set to 0 and that fragment’s color is set to black during rendering. If not, the fragment is rendered with color read originally from the texture. At end of all these renders we are left with only those 3d points which don’t belong to ANY ellipsoid, for each particle, i.e. their BNE flags will still be set to 1. Fig. 7 shows two frame buffer read backs – at start (left) and near the end of multi-pass rendering (right). Notice that some squares have all black pixels (corresponding particles fit 3d data very well) and some don’t. We use two Pbuffers for multi-pass rendering. We first render to the buffer1 and in the next rendering, read from buffer1 and render to buffer2; and in the next rendering read from buffer2 render to buffer1 and so on (usually known as the “ping pong” action). We have 6 ellipsoids (2 thighs, 2 limbs, 2feet) in our articulated model for tracking legs and 2 ellipsoids for arm. We have used N = 81 particles. So we have to do multi-pass rendering 6(or2) times for each anneal layer of PFA, depending on whether we are tracking legs(or arms) followed by a read back of the frame buffer. We have used M=3 anneal layers. Hence, to track each frame, we need 18(or 6) renderings and 3 read backs of frame buffer.

Fig. 7. Two sample frame buffer read backs.

6.3 Tracking Accuracy and Speed

If we implement the PFA on a Pentium 3.06 GHz CPU only it takes 5 seconds to calculate the pose of the articulated model in each frame. When we use both the CPU and a GeForce 5900FX GPU, the time to calculate the pose is reduced to 0.25 seconds – a 20X speedup! We also calculated the accuracy of the PFA by calculating Euclidean distance between the ground truth particle (vector of d dimensions of angles) and the particle found by the PFA for each frame and averaging it over many ground truth sequences. The average absolute joint angle error was 7.7302 degrees. Maximum, minimum and standard deviation of the error were 19.3103, 0.0 and 4.5398 degrees respectively.

6.4 Comparison with 2D weight calculation

We compare our approach with the approach taken in [9], [10] in this section. Let us say that instead of deriving 3d data from camera images, we wanted to use 2d re-projection error on the image plane for weight computation. If we use only 6 renderings per anneal layer to match the number of renderings per anneal layer in our 3d data method, then, in each rendering we must process N (= 81)/6 = 14 particles. If we use 4 cameras as done in [9] and render a 512X512 quadrilateral each time during the PFA, then each camera image would occupy (512X512)/(14*4) = 4681 pixels. Assuming we are capturing 320X240 images, this would mean we have to resize the image to less than 1/16th the size of original image! Apart from having fundamental pose calculation problems like that shown in Fig. 4, this would severely alias the image data which would adversely affect the performance of the PFA.

7 Deriving 3D data from a visual hull

We use a modified Visual Hull algorithm to derive 3d points inside the 3d object directly. We refer interested readers to [15] for more details. We use this 3d point data for the PFA and hence achieve MLT of human limbs at near interactive rates. Since we only need the 3d points inside the 3d object, the modified visual hull algorithm has O(n) complexity instead of O(n2), where n is the number of cameras. Hence, less computation time is spent on computing the 3d data.

Fig. 8. Image contours from 3cameras – one each along X,Y and Z axis and Modified Visual Hull computation.

Consider three calibrated cameras, one each along X, Y and Z axes. The algorithm can be described as follows –

(1) Choose grid points on foreground contour obtained from Camera1 (Fig. 8, red points on black contour).

(2) For each of these points, do

(a) Compute the 3d line (red line) passing through the camera center and the grid point (see Fig. 8).

(b) Compute the corresponding epipolar lines on image planes of Camera2 and Camera3 (green and blue lines on respective image planes). Intersect these with the corresponding contours. Using these points of intersection for Camera2 compute that portion on the red 3d line which, when projected onto the image plane of Camera2, belongs to the foreground contour of Camera2 – say Cam2_3DLine. Similarly compute Cam3_3DLine.

(c) Compute the part of the red 3d line that is the intersection of Cam2_3DLine and Cam3_3DLine – black part of red 3d line in Fig. 8.

(d) Choose 3d points on this line. Number of 3d points chosen is directly proportional to length of the black 3d line (see Fig. 8).

Fig. 9 shows the result of this algorithm at an instant in time. The 3d points inside the 3d object derived from the contours in the camera images are shown on the left. A close up view of the 3d points is shown on the right.

Fig. 9. (Left) 3d point data derived from modified visual hull computation and (Right) Zoomed in version.

8 Problems with doing mltrt in pca space

During this work, we tried another idea. Safonova et al. [19] and Grochow et al. [20] showed that human motion can be simulated and solved for in a lesser dimensional space rather than in the original high DOF space. We thought that this would help in particle filtering too. If we could do PFA in a lesser dimensional Eigen (PCA) space, we could use a smaller number of particles and hence reduce computation time. We computed the covariance matrix and base vectors for the DOFs of our articulated model in the PCA space from the Mocap database [18]. We postulated that the error due to tracking in PCA space would be acceptable. But, it turns out that error in PCA space is not acceptable. Fig. 10 shows two instances where the PFA failed in PCA space. We think this is because Safonova et al. [19] actually solved for motion parameters on a whole sequence by specifying positions for the articulated model at some key frames. This results in the solution of motion parameters “getting pulled to the right path”. Since we do not do any kind of optimization/key frame position specification, doing PFA in PCA space actually decreases its robustness. Also since the position computed for Frame K is used to generate particles for Frame K+1, tracking errors accumulate while doing PFA in PCA space.

Fig. 10. Two instances showing failure of PFA in PCA space. For both, tracking in actual space (left) and tracking in PCA space (right).

9 Real time implementation of our system

We implemented the whole system on a Pentium 4 3.06 GHz processor and a GeForce 5900FX video card. We used 3 IBOT fire-wire web cameras for hand tracking and leg tracking. GLSL and GLEW library were used for GPU programming. Calibration toolbox by [24] was used to calibrate our cameras. DirectX 9.0 library was used to capture camera images and synchronize them in software. The whole program has 7 threads. A main thread creates and controls all other threads. There is one thread for doing only the GPU part of the PFA. There are separate threads for capturing data from each camera. The main thread requests data from all the camera threads and suspends itself until it received updates from all of the camera threads. Then a separate thread computes the visual hull and the 3d data which is then fed to the PFA. A separate debug thread displayed all the debugging information. We were able to track human limbs with reasonable accuracy at 4Hz. See Fig. 11, Fig. 12, Fig. 13 and Fig. 14 for few tracking results.

10 Conclusion and Future work

We have demonstrated how human limbs can be tracked at near interactive rates. We also have demonstrated that using 3d data derived from a visual hull greatly improves the robust particle filtering framework. Implementation of the particle filtering framework on GPU has enabled us to do MLTRT at 4Hz and with mean error of 7.7302 degrees. We are currently working on optimizing the algorithm further and increase the update rate using a faster GPU. We also want to try out full body MLTRT. We will investigate signal processing issues associated with smoothing and sampling of the MLTRT weight function. We will integrate human dynamics and collision detection between limbs into MLTRT. We then will proceed to using this technique for free view-point video, VR gaming, HCI, telepresence in combined VR and real environments.


The first author would like to thank his friends Gajendra Singh, Arvind Somya and Madhusudhan Reddy for their help and moral support during this project.

Fig. 11. First row shows images of a hand at a particular position captured by three cameras. Second row shows corresponding foreground segments extracted by doing background subtraction. Last row show three virtual views of the articulated model that is used to track user’s hands – tracking and rendering of virtual views happens in real time at near interactive rates.

Fig. 12. Same as Fig. 11, but for a different pose of the arm.

Fig. 13. First row shows images of legs at a particular position captured by three cameras. Second row shows corresponding foreground segments extracted by doing background subtraction. Last row show three virtual views of the articulated model that is used to track user’s legs – tracking and rendering of virtual views happens in real time at near interactive rates.


  1. Carranza, J., Theobalt, C., Magnor, M. A., Seidel, H., “Free-View point Video of Human Actors”. In Proceedings of SIGGRAPH 2003, ACM Press / ACM SIGGRAPH, 2003.

  2. Wuermlin, S., Lamboray, E., Staadt, O., Gross, M., “3d video recorder”. In Proceedings of Pacific Graphics 2002, IEEE Computer Society Press, 325–334. 2002.

  3. Matsuyama, T., Akai, T., “Generation, visualization, and editing of 3D video”. In Proc. of 1st International Symposium on 3D Data Processing Visualization and Transmission (3DPVT’02), 234ff, 2002.

Fig. 14. Same as Fig. 13, but for a different pose of the legs.

  1. Matusik, W., and Pfister, H., “3DTV: A Scalable System for Real-Time Acquisition, Transmission, and Auto stereoscopic Display of Dynamic Scenes”. In Proceedings of SIGGRAPH 2004, ACM Press / ACM SIGGRAPH, 2004.

  2. Zitnick, C.L., Kang, S.B., Uyttendaele, M., Winder, S., Szeliski, R., “High quality Video View Interpolation using a Layered Representation”. In Proceedings of SIGGRAPH 2004, ACM Press / ACM SIGGRAPH, 2004.

  3. Luck, J., Small, D., “Real-time markerless motion tracking using linked kinematic chains”. In Proc. of CVPRIP02, 2002.

  4. Cheung, K., Kanade, T., Bouguet, J.Y., Holler, M., “A real time system for robust 3D voxel reconstruction of human motions”. In Proc. of Computer Vision and Pattern Recognition, vol. 2, 714 – 720, 2000.

  5. Cheung, G., Baker, S., Kanade, T., “Shape-From-Silhouette of Articulated Objects and its Use for Human Body Kinematics Estimation and Motion Capture”. Proc. Conf. Computer Vision and Pattern Recognition, 2003.

  6. Deutscher, J., Blake, A., Reid, I., “Articulated body motion capture by annealed particle filtering”. In Proc. Conf. Computer Vision and Pattern Recognition, volume 2, pages 1144-- e 1149, 2000.

  7. Deutscher, J., Davison, A.J., Reid, I., ”Automatic Partitioning of High Dimensional Search Spaces associated with Articulated Body Motion Capture”. Proc. IEEE Conference on Computer Vision and Pattern Recognition, Kauai, 2001.

  8. Sigal, L., Bhatia, S., Roth, S., Black, M.J., Isard, M., “Tracking Loose-limbed People”. Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2004.

  9. Gavrila, D., and Davis, L., “3d model-based tracking of humans in action: a multi-view approach”. Proc. Conf. Computer Vision and Pattern Recognition (1996), 73–80, 1996.

  10. Bregler, C., Malik, J., “Tracking People with Twists and Exponential Maps”. Proc. IEEE Computer Vision and Pattern Recognition 1998.

  11. Gross, M., Würmlin, S., Naef, M., Lamboray, E., Spagno, C., Kunz, A., Koller-Meier, E., Svoboda T., Van Gool, L., Lang S., Strehlke, K., Moere, V.A., StaadT,O., “blue-c: A Spatially Immersive Display and 3D Video Portal for Telepresence”. Proceedings of ACMSIGGRAPH 2003, pp. 819-827, 2003.

  12. Matusik, W., Buehler, C., and McMillan, L., “Polyhedral Visual Hulls for Real-Time Rendering”. In Proceedings of Eurographics Workshop on Rendering, 2001.

  13. Laurentini, A., “The Visual Hull Concept for Silhouette Based Image Understanding”. IEEE PAMI, 16, 2 (1994), pp. 150-162., 1994.

  14. Vedula, S., Baker, S., Kanade, T., “Spatio-temporal view interpolation.” In Proceedings of the 13th ACM Eurographics Workshop on Rendering, 65–75, 2002.

  15. http://mocap.cs.cmu.edu/

  16. Safonova, A., Hodgins, J., Pollard, N., “Synthesizing Physically Realistic Human Motion in Low-Dimensional, Behavior-Specific Spaces”. In Proceedings of SIGGRAPH 2004, ACM Press / ACM SIGGRAPH, 2004.

  17. Grochow K., Martin S.L., Hertzmann A., Popović Z., “Style-based Inverse Kinematics”. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2004), 2004.

  18. M. Slater and M. Usoh, “The Influence of a Virtual Body on Presence in Immersive Virtual Environments.” VR 93, VirtualReality International, Proceedings of the Third Annual Conferenceon Virtual Reality, London, Meckler, 1993, pp 34-42.

  19. M. Slater and M. Usoh, “Body Centred Interaction inImmersive Virtual Environments.” In Artificial Life and Virtual Reality, (N. Magnenat, D. Thalmann, eds.), John Wiley and Sons, 1994, pp. 125-148.

  20. Kirkpatrick, S., C.D. Gelatt Jr, and M.P. Vecchi, “Optimization by Simulated Annealing”, Science, V. 220, No. 4598, pp. 671 – 680, 1983.

  21. Tomas Svoboda, Daniel Martinec, and Tomas Pajdla. “A convenient multi-camera self-calibration for virtual environments”. PRESENCE: Teleoperators and Virtual Environments, 14(4), August 2005.

Srinivasa G. Rao is a PhD student in the Department of Computer Science in the College of Information Technology at the University of North Carolina - Charlotte. He earned his B.E. (Bachelor of Engineering) degree in Computer Science in 1999 from University of Mysore, India. He worked with IBM as a software engineer (1999-2000). He holds a M.S. in Computer Science from University of Maryland at College Park (2000-2002). He also worked at MERL labs as a research intern (2002-2003). His main research interests are Virtual Reality, Augmented Reality, Real time Computer Graphics and Computer Vision.

Dr. Larry F. Hodges received the PhD degree from North Carolina State University in Computer Engineering in 1988. He is a professor and chair of the Department of Computer Science in the College of Information Technology at University of North Carolina at Charlotte. Prior to moving to Charlotte, he spent 14 years as a faculty member in the College of Computing at the Georgia Institute of Technology, where he was a founding member of the Graphics, Visualization and Usability (GVU) Center. He is also cofounder of Virtually Better, a company that specializes in creating virtual environments for clinical applications in psychiatry, psychology, and addiction. His research interests include virtual reality, interactive computer graphics, visualization and 3D HCI.

Скачать документ

Похожие документы:

  1. Elektronisk dansk 104 okt 06

    ... founders Sergey Brin andLarry Page, who ... of computer simulation. A number of major tracksof ... ofhuman reasoning and information processing. In fact, fuzzy logic resembles human ... of Bacteria-Antibiotic Interactions in Batch Culture James T. Murphy, Ray ...
  2. Proposal submittal

    ... A., and G. Sapiro, “Trackingof moving objects under severe and total ... analyzers and spectroscopic instrumentation. REFERENCES: 1. M. K. Royand J. Richter ... and Visualization of their Hole-Mediated Interactions,” Nature, 442, 436 (2006). 2. T. Hynninen ...

Другие похожие документы..