Abstract: The optical wave transmission was studied with plane wave passing through the rough surface of layered medium. The scattering coefficient formulae of the transmitted light wave under different polarization states was derived with the perturbation method. The natural rough surface of layered medium was simulated with Gaussian rough surface. The calculation formulae for the transmission coefficient of the plane wave incidence to Gaussian rough surface of the layered medium was derived in combination with the power spectrum of Gaussian rough surface. The variation curves of HH polarization transmission coefficient with the scattering angle of transmission wave were obtained by numerical calculation. The influence of the dielectric constant of substrate medium, the dielectric constant and thickness of intermediate medium, rough surface parameters, and wavelength of incident optical wave on the transmission coefficient was discussed. The numerical results show that such influence is very complex.