Abstract: Since Camshift algorithm is easy to lose tracked object in complex background, this paper puts forward a tracking method based on Camshift algorithm, before which a color transfer technique is used between color reference image and fused image in YCBCR space. During color transfer, the proposed method used information of dual-band image effectively, and the target not only had the excellence of high contrast, but also enhanced the drawing of color probability distribution, the proposed method improved the efficiency of Camshift. The comparison of the visual image, color transferred infrared image and the traditional color transferred image was made using the same tracking method, a group of experimental results demonstrate that the distance between the center of tracking window and target centroid is just 3 pixels in the pictures based on proposed method, tracking precision outperforms the results of contrast experiment pictures. At the same time, the tracking time of the proposed method is 20.6 ms, which meets the real time requirement.