Tracking by detection is the topic of recent research that has received considerable attention in computer vision community. Mainly off-line classification methods have been used, however, they perform weakly in the case of appearance changes. Training the classifier incrementally and in an online manner solves this problem, but nevertheless, raises drifting due to soft or hard labeling in the online adaptation. In this paper a novel semi-supervised online tracking algorithm based on manifold assumption is proposed. Target object and background patches lie near low-dimensional manifolds. This motivates us to make use of the intrinsic structure of data in classification, and benefit from the smooth variation of the labeling function with respect to the underlying manifold. Unlabeled data make connections between different object poses to overcome difficulties due to appearance changes and partial occlusion. Moreover, the proposed method doesn’t rely on self-training, therefore, it is more robust to drifting. Experimental results substantiate the superiority of the proposed method over the ones that does not consider the geometry of data.