Patch-match binocular 3D reconstruction based on deep learning

SONG Lizheng; LIN Dongyun; PENG Xiafu; LIU Tengfei

doi:10.5768/JAO202243.0302003

Journal of Applied Optics > 2022 > 43(3): 436-443. > DOI: 10.5768/JAO202243.0302003

SONG Lizheng, LIN Dongyun, PENG Xiafu, LIU Tengfei. Patch-match binocular 3D reconstruction based on deep learning[J]. Journal of Applied Optics, 2022, 43(3): 436-443. DOI: 10.5768/JAO202243.0302003

Citation:

PDF (3393 KB)

Patch-match binocular 3D reconstruction based on deep learning

School of Aeronautics and Astronautics, Xiamen University, Xiamen 361005, China

More Information

Received Date: November 18, 2021
Revised Date: February 06, 2022
Available Online: April 18, 2022

Graphical Abstract

Abstract

Abstract

The patch-match algorithm has been widely used in binocular stereo reconstruction due to its low memory consumption and high reconstruction accuracy. However, the traditional patch-match algorithm needs to iteratively calculate the optimal disparity d for each pixel of image in an orderly manner, which resulting in a high running time. In order to solve this problem, a learning-based model on the basis of traditional patch-match algorithm as a guide to reduce the running time and improve the accuracy of stereo reconstruction was introduced. First, the deep learning model was used to output the initial disparity map of each pixel with heteroscedastic uncertainty, which was used to measure the accuracy of the disparity predicted by the network model. Then, the heteroscedastic uncertainty and initial disparity were taken as the prior information of patch-match algorithm. Finally, in the plane refinement step, the heteroscedastic uncertainty of each pixel was used to dynamically adjust its search interval to achieve the goal of reducing the running time. On the Middlebury dataset, compared with the original algorithm, the running time of the improved algorithm is reduced by 20%, and the reconstruction accuracy of the discontinuous region is slightly improved.
- 3D reconstruction,
- deep learning,
- heteroscedastic uncertainty,
- precision

FullText(HTML)

References (20)

References

[1]	BLEYER M, RHEMANN C, ROTHER C. PatchMatch stereo-stereo matching with slanted support windows[C]//British Machine Vision Conference 2011. Scotland: University of Dundee, 2011.
[2]	XU H, CHEN X, LIANG H, et al. CrossPatch-based rolling label expansion for dense stereo matching[J]. IEEE Access,2020,8:63470-63481. doi: 10.1109/ACCESS.2020.2985106
[3]	BARNES C, SHECHTMAN E, FINKELSTEIN A, et al. PatchMatch: a randomized correspondence algorithm for structural image editing[J]. ACM Transactions on Graphics,2009,28(3):1-24.
[4]	BARNES C, SHECHTMAN E, DAN B G, et al. The generalized PatchMatch correspondence algorithm[C]//European Conference on Computer Vision, Heraklion, Greece, September 5-11, 2010. Berlin: Springer, 2010: 29-43.
[5]	马少斌, 张成文. 基于双路多尺度金字塔池化模型的显著目标检测算法[J]. 应用光学,2021,42(6):1056-1061. doi: 10.5768/JAO202142.0602007 MA Shaobin, ZHANG Chenwen. Salient target detection algorithm based on dual-channel multi-scale pyramid pooling model[J]. Journal of Applied Optics,2021,42(6):1056-1061. doi: 10.5768/JAO202142.0602007
[6]	GIRSHICK R, DONAHUE J, et al. Rich feature hierarchies for accurate object detection and semantic segmentation[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, June 23-28, 2014. USA: IEEE, 2014: 580-587.
[7]	曹富强, 王明泉, 张俊生, 等. 基于深度学习的铸件X射线图像分割研究[J]. 应用光学,2021,42(6):1025-1033. doi: 10.5768/JAO202142.0602003 CAO Fuqiang, WANG Mingquan, ZHANG Junsheng, et al. Research on X-ray image segmentation of castings based on deep learning[J]. Journal of Applied Optics,2021,42(6):1025-1033. doi: 10.5768/JAO202142.0602003
[8]	RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]// 18th International Conference on Medical Image Computing and Computer-Assisted Intervention.Berlin:Springer,2015: 234-241.
[9]	SEKI A, POLLEFEYS M. SGM-Nets: semi-global matching with neural networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition , Honolulu, HI, July 21-26, 2017. USA: IEEE, 2017: 6640-6649.
[10]	WU S, RUPPRECHT C, VEDALDI A. Unsupervised learning of probably symmetric deformable 3D objects from images in the wild[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition , Seattle, WA, June 13-19, 2020. USA. : IEEE, 2020: 1-10.
[11]	KENDALL A, MARTIROSYAN H, et al. End-to-end learning of geometry and context for deep stereo regression[C]//2017 IEEE International Conference on Computer Vision , Venice, Italy, October 22-29, 2017. USA: IEEE, 2017: 66-75.
[12]	BONTAR J, LECUN Y. Computing the stereo matching cost with a convolutional neural network[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition , Boston, MA, June 7-12, 2015. USA. IEEE, 2015: 1592-1599.
[13]	ŽBONTAR J, LECUN Y. Stereo matching by training a convolutional neural network to compare image patches[J]. The Journal of Machine Learning Research,2016,17(1):2287-2318.
[14]	CHANG J R, CHEN Y S. Pyramid stereo matching network[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , Salt Lake City, UT, June 18-23, 2018. USA. IEEE, 2018: 5410-5418.
[15]	KENDALL A G . Geometry and uncertainty in deep learning for computer vision[D]. London: University of Cambridge, 2019.
[16]	HU Yaoyu, ZHEN Weikun, et al. Deep-learning assisted high-resolution binocular stereo depth reconstruction[C]//2020 IEEE International Conference on Robotics and Automation , Paris, France, May 31-August 31, 2020. USA: IEEE, 2020: 8637-8643.
[17]	HIRSCHMULLER H. Stereo processing by semiglobal matching and mutual information[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 30(2): 328 – 341.
[18]	MAYER N, ILG E, et al. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition , Las Vegas, NV, June 27-30, 2016. USA: IEEE, 2016: 4040-4048.
[19]	SCHARSTEIN D, HIRSCHMÜLLER H, KITAJIMA Y, et al. High-resolution stereo datasets with subpixel-accurate ground truth[C]// 36th German Conference on Pattern Recognition, 2014, 8753: 31-42.
[20]	ZHANG Kang, FANG Yuqiang, et al. Cross-scale cost aggregation for stereo matching[C]// 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, June 23-28, 2014. USA, IEEE, 2014: 1590-1597

Cited By

Cited by

Periodical cited type(6)

1.	杨力. 基于双目视觉的三维场景图像表面重建算法. 现代电子技术. 2024(04): 71-75 .
2.	白维维，李俊杰，陈烽. 基于双目图像深度学习的农作物择优采摘仿真. 计算机仿真. 2024(02): 187-191 .
3.	孙同明，任俊，张峰，廖春云. 基于逆向工程的航空发动机叶片三维重建模型构建. 计算机测量与控制. 2024(09): 249-255 .
4.	韦玮，陈芬，张华波，罗英国，张鹏，彭宗举. 基于亚像素和梯度引导的光场图像超分辨率. 应用光学. 2024(05): 956-965 . 本站查看
5.	张子康，殷松峰，曹良才，刘成. 基于身份-年龄共享特征的跨年龄人脸识别方法. 应用光学. 2023(03): 565-570 . 本站查看
6.	宁孟丽，樊亚栋. 仿真技术在我国旅游领域应用现状的研究. 计算机仿真. 2023(08): 264-269 .

Other cited types(2)

Get Citation

PDF

XML

Article views (610) PDF downloads (72) Cited by(8)

Turn off MathJax

Article Contents

Abstract

References

Patch-match binocular 3D reconstruction based on deep learning