Abstract: To compare the properties of random mask in the optical image encryption, optical image encryption was stimulated and image encryption effects were analyzed using random phase mask and random amplitude mask in the several typical optical image encryption systems. The stimulation result shows that the correlation coefficient between original image and encryption image is bigger than 0.5 when random amplitude mask is used in double random mask optical encryption system based on optical Fourier transform and the original image can not be encrypted effectively. But the correlation coefficient between original image and encryption image approaches 0 when random amplitude mask is used in optical encryption system based on optical Fourier transform holography and the encryption effect is similar to one when random phase mask is used.