Abstract: To calculate the refractive index of every layer accurately in coating process, two methods are introduced to determine the refractive index of layers in real time monitoring. The first method calculates the refractive index of the film inversely with the measured transmittance spectrum. The other one fits the refractive index of the film with the estimation algorithm in the least square method (LSM). The results of the test indicate the online inverse calculation method is more suitable for single point monitoring and the error of calculated refractive index is less than 2%. However, in real coating process, the first method is not preferred because the layers’ parameters differ a lot from each other, which makes the error greater than 25%. Meanwhile, the latter one can collect the information from whole wideband spectrum and make the error smaller, which is normally less than 10%. Therefore, it can improve the precision of the film thickness monitoring significantly.