Abstract: A fast algorithm of vehicle infrared video colorization was proposed. Contour feature points tracking was utilized to get the contours and areas of different object classes in each frame. Characteristic colors of different classes were transferred to the corresponding areas in each frame. Firstly, we constructed the characteristic color sets of different classes of scenes. The characteristic colors of different scenes in the natural color images were extracted as the corresponding scenes-colors in the infrared images. Secondly, we clustered and segment the scenes in the key infrared video frame using the improved efficient KMeans clustering method, and the contour feature points were extracted. Thirdly, we tracked the contour feature points using the Kanade-LucasTomasi (KLT) algorithm to get and correct their positions in the next frame. The B-spline interpolation method was used to reconstruct the contours. The contours and areas of different object classes in each frame were obtained. Finally, characteristic colors of different classes were transferred to the corresponding areas in each frame. Infrared video colorization was realized and the scenes were with reasonable colors. Experimental results show that the processing speed of the proposed method is nearly 5 times as fast as that of the algorithm based on motion estimation and it can give a natural color look to the infrared video.