Abstract: A segmentation method to separate the moving objects from their backgrounds in a video sequence is proposed. Firstly, the global motion estimation and compensation is performed. Secondly, the continuous twice difference for the adjacent frames in the video sequence is implemented and the noise is filtered out with the adaptive filtering, and then the morphologic treatment is carried out. Thirdly, the effect of uncovered and covered background is eliminated with the difference intersection technology. At last, the mask matching and updating enables the algorithm to handle both quickly changing objects and temporary stop moving objects. Experimental results indicate that the moving objects can be automaticly extracted from the video sequence by the method and it has the strong robustness.