Abstract: To solve the problem of single color feature and the lack of necessary template updating method in mean shift algorithm, an adaptive tracking algorithm of dual template adaptive updating based on multiple features is proposed. The difference of gray mean in the neighborhood of pixel point and the layered space information are introduced to enhance feature differentiation. By analyzing the similarity coefficient of dual template in target and background region, the origin of disturbance factor is estimated truly, and the target template in target region of current frame is updated by similarity coefficient. The speed of template updating is adapted to the transformation of target feature, and at the same time excessive template updating is restrained, which solves the problem of the occasion and speed of template updating. The robustness of the proposed algorithm is demonstrated by simulations in different tracking scenarios.