3D Mesh Reconstruction from Single 2D Image Using DBSCAN and CNN Architecture.
Keywords:
3D-Reconstruction, Computer Vision, 3D Mesh, 3D from 2D, Point CloudAbstract
One of the most intriguing challenges in this domain is the reconstruction of 3-D objects or scenes from a single 2D image. 3D Reconstruction from a single-view image aims to reconstruct the 3D object from a single-colored image. Previously, the 3D Reconstruction was done using multiple images taken from different angles. But it has limitations in accessibility. Generating a 3D model from a 2D image has broad applications in various industries such as Augmented reality (AR), Virtual reality (VR), Robotics, Video games, and Medical Imaging. We have broken the process into two parts, in the first part we generate a 3D point cloud from a 2D image. For the next part, we then generate a 3D mesh from the point cloud map. A CNN is later used to further optimize the mesh. In the result and conclusion, we have generated an optimized mesh but the end output has not reached the desired accuracy. This accuracy can be increased by creating a Depth-map and view-point angle prior to calculating a point cloud.
Downloads
References
Yang, Xianghui, Guosheng Lin, and Luping Zhou. "Single-view 3D Mesh Reconstruction for Seen and Unseen Categories." IEEE Transactions on Image Processing (2023).
Choy, Christopher B., Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese. "3d-r2n2: A unified approach for single and multi-view 3d object reconstruction." In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14, pp. 628-644. Springer International Publishing, 2016.
Mescheder, Lars, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. "Occupancy networks: Learning 3d reconstruction in function space." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4460-4470. 2019.
Richter, Stephan R., and Stefan Roth. "Matryoshka networks: Predicting 3d geometry via nested shape layers." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1936-1944. 2018.
Popov, Stefan, Pablo Bauszat, and Vittorio Ferrari. "Corenet: Coherent 3d scene reconstruction from a single rgb image." In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pp. 366-383. Springer International Publishing, 2020.
Wu, Jiajun, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. "Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling." Advances in neural information processing systems 29 (2016).
Durou, Jean-Denis, Maurizio Falcone, and Manuela Sagona. "Numerical methods for shape-from- shading: A new survey with benchmarks." Computer Vision and Image Understanding 109, no. 1 (2008): 22-43.
Horn, Berthold KP. "Shape from shading: A method for obtaining the shape of a smooth opaque object from one view." (1970).
Zhang, Ruo, Ping-Sing Tsai, James Edwin Cryer, and Mubarak Shah. "Shape-from-shading: a survey." IEEE transactions on pattern analysis and machine intelligence 21, no. 8 (1999): 690-706.
M. Tatarchenko, S. R. Richter, R. Ranftl, Z. Li, V. Koltun and T. Brox, "What Do Single-View 3D Reconstruction Networks Learn?," 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019, pp. 3400-3409, doi: 10.1109/CVPR.2019.00352.
Gkioxari, Georgia, Jitendra Malik, and Justin Johnson. "Mesh r-cnn." In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9785-9795. 2019.
Liu, Shichen, Tianye Li, Weikai Chen, and Hao Li. "Soft rasterizer: A differentiable renderer for image-based 3d reasoning." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7708-7717. 2019.
Li, Xueting, Sifei Liu, Kihwan Kim, Shalini De Mello, Varun Jampani, Ming-Hsuan Yang, and Jan Kautz. "Self-supervised single-view 3d reconstruction via semantic consistency." In Computer Vision– ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16, pp. 677-693. Springer International Publishing, 2020.
Kanazawa, Angjoo, Shubham Tulsiani, Alexei A. Efros, and Jitendra Malik. "Learning category- specific mesh reconstruction from image collections." In Proceedings of the European Conference on Computer Vision (ECCV), pp. 371-386. 2018.
Kato, Hiroharu, Yoshitaka Ushiku, and Tatsuya Harada. "Neural 3d mesh renderer." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3907-3916. 2018.
Chen, Wenzheng, Huan Ling, Jun Gao, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. "Learning to predict 3d objects with an interpolation-based differentiable renderer." Advances in neural information processing systems 32 (2019).
Pontes, Jhony K., Chen Kong, Sridha Sridharan, Simon Lucey, Anders Eriksson, and Clinton Fookes. "Image2mesh: A learning framework for single image 3d reconstruction." In Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part I 14, pp. 365-381. Springer International Publishing, 2019.
Ye, Yufei, Shubham Tulsiani, and Abhinav Gupta. "Shelf-supervised mesh prediction in the wild." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8843- 8852. 2021.
Nie, Yinyu, Xiaoguang Han, Shihui Guo, Yujian Zheng, Jian Chang, and Jian Jun Zhang. "Total3dunderstanding: Joint layout, object pose and mesh reconstruction for indoor scenes from a single image." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 55-64. 2020.
Chen, Zhiqin, and Hao Zhang. "Learning implicit fields for generative shape modeling." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5939- 5948. 2019.
Saito, Shunsuke, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. "Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization." In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2304-2314. 2019.
Saito, Shunsuke, Tomas Simon, Jason Saragih, and Hanbyul Joo. "Pifuhd: Multi-level pixel- aligned implicit function for high-resolution 3d human digitization." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 84-93. 2020.
Fan, Haoqiang, Hao Su, and Leonidas J. Guibas. "A point set generation network for 3d object reconstruction from a single image." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 605-613. 2017.
Han, Zhizhong, Chao Chen, Yu-Shen Liu, and Matthias Zwicker. "DRWR: A differentiable renderer without rendering for unsupervised 3D structure learning from silhouette images." arXiv preprint arXiv:2007.06127 (2020).
Chao, Chen, Zhizhong Han, Yu-Shen Liu, and Matthias Zwicker. "Unsupervised learning of fine structure generation for 3d point clouds by 2d projection matching." arXiv preprint arXiv:2108.03746 (2021).
Gwak, JunYoung, Christopher B. Choy, Manmohan Chandraker, Animesh Garg, and Silvio Savarese. "Weakly supervised 3d reconstruction with adversarial constraint." In 2017 International Conference on 3D Vision (3DV), pp. 263-272. IEEE, 2017.
Afifi, Ahmed J., Jannes Magnusson, Toufique A. Soomro, and Olaf Hellwich. "Pixel2Point: 3D object reconstruction from a single image using CNN and initial sphere." IEEE Access 9 (2020): 110- 121.
Ugrinovic, Nicolas, Albert Pumarola, Alberto Sanfeliu, and Francesc Moreno-Noguer. "Single- view 3D Body and Cloth Reconstruction under Complex Poses." arXiv preprint arXiv:2205.04087 (2022).
Siddique, Ashraf, and Seungkyu Lee. "Sym3DNet: Symmetric 3D prior network for single-view 3D reconstruction." Sensors 22, no. 2 (2022): 518.
Zhao, Meihua, Gang Xiong, MengChu Zhou, Zhen Shen, and Fei-Yue Wang. "3D-RVP: A method for 3D object reconstruction from a single depth view using voxel and point." Neurocomputing 430 (2021): 94-103.
Tatarchenko, Maxim, Stephan R. Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox. "What do single-view 3d reconstruction networks learn?." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3405-3414. 2019.
Moons, Theo, Luc Van Gool, and Maarten Vergauwen. "3D reconstruction from multiple images part 1: Principles." Foundations and Trends® in Computer Graphics and Vision 4, no. 4 (2010): 287-404.
A. Singh, V. Jaiswal, G. Joshi, A. Sanjeeve, S. Gite and K. Kotecha, "Neural Style Transfer: A Critical Review," in IEEE Access, vol. 9, pp. 131583-131613, 2021, doi: 10.1109/ACCESS.2021.3112996.
Mishra, A.; Dharahas, G.; Gite, S.; Kotecha, K.; Koundal, D.; Zaguia, A.; Kaur, M.; Lee, H.-N. ECG Data Analysis with Denoising Approach and Customized CNNs. Sensors 2022, 22, 1928. https://doi.org/10.3390/s22051928
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
