Mean Shift Tracking\\\\[2cm]
IntroductionIn the article we will look at the application of Mean Shift Tracking for color based tracking.
Mean shiftThe object model used in mean shift tracking is color probability distribution.
Now we have a object model,given an image we can compute the likelihood image Each pixel in likelihood image represents the likelihood that pixel belongs to the object model/histogram. \[ L_u(y_i) = p_u*\delta(b(y_i)-u) \] This likelihood image assigns to each pixel a similarity measure wrt object model.
It is reasonable to assume that the region in which the highest similarity measure or highest density is observed is a good estimate of object location.
Thus if we consider a small window and move towards the mean value ie along the mean shift vector we should eventually reach the region of maximum similarity.
The likelihood surface is not smooth,we can give it properties of smoothness using kernel density estimation.
Now we can find the modes of the similarity surface using standard mean shift algorithm.
Let us assume that current estimate of mode of function is at $y$. Thus we consider a small rectangular window about $y$,compute the mean shift vector and take a small step along the mean shift vector.
In principle this should enable to find the local maximum. Similarity surface is discontinuous and to do this we need to perform KDE over entire image consisting of dense grid of points ,which is a expensive operation.(We need to perform convolution at each point with Gaussian with suitably large aperture)
Using Similarity for trackingThe concept of similarity surface can be made useful in tracking application.
Let us consider a small region of interest about a present location y, we can compute the similarity score about this region,perform KDE on this small region,obtain a similarity surface and compute the mean shift vector.
If object is not present in region,similarity surface will be flat and mean shift vector will be zero.
If there is object present in some part of region,it will correspond to modes of similarity surface .The mean shift vector will give us direction to move along.
Now instead of trying to estimate the mode,say we translate the region of interest along direction provided by the mean shift vector. This would typically lead to large portion of object being visible and would expose the region of global similarity surface where a large maximum would lie.
This is the basis of mean shift tracking,keen on translating the region of interest ,till we reach local maximum of similarity surface.
For tracking applications ,since fast computation is required, we can consider a rectangular window with bandwidth equal to that of the region of interest.The present location of point point is the center of the rectangular region.
ImplementationA video of mean shift tracking is shown below.A naive object model based on color probability in HS color space using first frame of the video
Another issue is that if the object is moving too fast and significant part of the object moves out of ROI in successive frames,the object will not be tracked. This can be seen in the following videos
If we encounter a larger object or object that exhibits higher density ,tracking will be lost.In the below case the tracking is lost when object passes over a large blue background which is similar to object color.
There are many other cases where mean shift tracking will fail
As will all tracking approaches ,the performance heavily depends on the object model. The better we are able to model the object and obtain a likelyhood/similarity which does not show high probability for background or other objects in the scene,the more accurate will be the tracking
CodeFor further image processing application a library consisting of high level interface to opencv will be used.The library is called OpenVisionLibrary. https://github.com/pi19404/OpenVision
The project cmake file is included in the repository. the build will create the library and test files in the bin directory To run demo program for mean shift run the binary meanShiftTest
The files for mean shift algorithm are meanshift.cpp and meanshift.hpp https://github.com/pi19404/OpenVision/tree/master/ImgProc/ repository.
To run the test program : Select the region of interest and click the build model button to start tracking.
For video file initially only the first frame is show ,select the ROI in the first frame and then click on build model button to start the tracking.
The button is shown upon clicking on Display properties button on the window.