Abstract: The stable tracking of small infrared targets in the complex backgrounds such as ground, sea surface and sky is an urgent problem to be solved. Taking into account both the robustness and real-time performances, based on the discriminative scale space tracking algorithm, the generalized structure tensor algorithm was applied that could effectively characterize the gray-scale mutation characteristics of the target area and the target shape information as a feature extraction method. The improved algorithm was more suitable for the rapid infrared image processing, which could improve the tracking robustness, and had a small amount of calculation, high efficiency, and low target feature dimension. In order to improve the stability of the model, it was decided to track the model update according to the confidence level, so as to avoid the model being disturbed by the wrong samples. Compared with the discriminative scale space tracking algorithm, the proposed algorithm has significant advantages in accuracy, real time and robustness, and achieves a tracking speed of 300 fps @256×256 pixels of image.