Abstract: To improve the security of optical image encryption system, a nonlinear optical image encryption system based on doubleopticalwedge joint Fresnel transform correlator and vector decomposition was proposed. The original image was decomposed into two phase masks. One of them, f1(x), was located at the reference window to realize the image encryption; the wave containing the information of the other phase mask, f2(x), interferenced with the decrypted output wave containing the information of the mask f1(x) to obtain the decrypted image. The encryption and decryption processes were numerically simulated. For both grayscale and binary images, the correlation coefficient between the decrypted and original image is 0.812 7 and 0.810 9, respectively, when the wedge angle is 1.8° and the distance between the centers of the mask f1(x) and the key code k(x) is 18 mm. The feasibility of the encryption method was verified by the simulation of the decryption result using incorrect key code with phase distribution error. Analysis results show that the position of the key code k(x) and the wedge angel are proved to be the additional key parameters and can effectively expand the key space. The encryption system is able to defend ciphertextonly attack, knownplaintext attack and chosenplaintext attack.