Abstract: A design of video tracking and processing system based on ADSP-BF561 was proposed. This system took full advantage of the excellent performance of dual-core processing and the profile interfaces of encoding and decoding. The paper used a correlation tracking algorithm based on correlation coefficient. And many measures were put forward to improve the speed of the algorithm, in order to satisfy the requirement of object real-time tracking. The experiment result based on the target video in phase alternating line (PAL) shows that the method can achieve object tracking in a complicated background in real time.