Abstract: Aiming at the false alarm or missing problems due to the insignificant and obscured targets in sequence images, an indirect localization method for building targets based on automatic selection of auxiliary targets was proposed to improve the stability of target tracking. In this method, firstly, the region with stable shape and significant gray level in the infrared image was automatically selected as the auxiliary target. After locating the building target, its relative positional relationship with the target was extracted, and then, the relative position was used to indirectly locate the target. Finally, the target was located by fusing the result of direct target recognition and the result of indirect target location, which could update the auxiliary target timely to ensure that it is always in the field of view. The experimental result shows that the false alarm or missing problems can be solved by indirect location, and it can improve the accuracy and robustness of the algorithm during the process of target location.