Hostname: page-component-6bb9c88b65-kzqxb Total loading time: 0 Render date: 2025-07-21T20:13:41.281Z Has data issue: false hasContentIssue false
  • English
  • Français

Robust Semantic Mapping in Challenging Environments

Published online by Cambridge University Press:  21 May 2019

Jiyu Cheng ,
Yuxiang Sun  and
Max Q.-H. Meng
Get access Rights & Permissions [Opens in a new window]

Summary

Visual simultaneous localization and mapping (visual SLAM) has been well developed in recent decades. To facilitate tasks such as path planning and exploration, traditional visual SLAM systems usually provide mobile robots with the geometric map, which overlooks the semantic information. To address this problem, inspired by the recent success of the deep neural network, we combine it with the visual SLAM system to conduct semantic mapping. Both the geometric and semantic information will be projected into the 3D space for generating a 3D semantic map. We also use an optical-flow-based method to deal with the moving objects such that our method is capable of working robustly in dynamic environments. We have performed our experiments in the public TUM dataset and our recorded office dataset. Experimental results demonstrate the feasibility and impressive performance of the proposed method.

Keywords

Semantic MappingDynamic EnvironmentsCRF-RNN

Information

Type
Articles
Information
Robotica , Volume 38 , Issue 2 , February 2020 , pp. 256 - 270
Copyright
© Cambridge University Press 2019 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Klein, G. and Murray, D., “Parallel Tracking and Mapping for Small AR Workspaces,” ISMAR 2007 Proceedings of the 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, Nara, Japan (2007) pp. 225–234.Google Scholar
Engel, J., Schöps, T. and Cremers, D., “LSD-SLAM: Large-Scale Direct Monocular Slam,” European Conference on Computer Vision, Zurich, Switzerland (2014) pp. 834–849.Google Scholar
Mur-Artal, R. and Tardós, J. D., “ORB-SLAM2: An open-source slam system for monocular, stereo, and RGB-D cameras,” IEEE Trans. Robot. 33 (5), 1255–1262 (2017).CrossRefGoogle Scholar
Whelan, T., Leutenegger, S., Salas-Moreno, R., Glocker, B. and Davison, A., “Elasticfusion: Dense Slam Without a Pose Graph.” Robotics: Science and Systems: A Robotics Conferences, Rome, Italy (2015).Google Scholar
Kerl, C., Sturm, J. and Cremers, D., “Dense Visual Slam for RGB-D Cameras,” 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Tokyo, Japan (2013) pp. 2100–2106.CrossRefGoogle Scholar
Wang, C., Meng, L., She, S., Mitchell, I. M., Li, T., Tung, F., Wan, W., Meng, M., de Silva, C. W. and Clarence, W., “Autonomous Mobile Robot Navigation in Uneven and Unstructured Indoor Environments,” 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, Canada (2017) pp. 109–116.CrossRefGoogle Scholar
Zhu, D., Li, T., Ho, D., Wang, C. and Meng, M. Q.-H., “Deep Reinforcement Learning Supervised Autonomous Exploration in Office Environments,” 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia (2018) pp. 7548–7555.CrossRefGoogle Scholar
Sun, Y., Zuo, W. and Liu, M., “Rtfnet: RGB-thermal fusion network for semantic segmentation of urban scenes,” IEEE Robot. Autom. Lett., 2576–2583 (2019).CrossRefGoogle Scholar
Wang, C., Cheng, J., Wang, J., Li, X., and Meng, M. Q.-H., “Efficient object search with belief road map using mobile robot,” IEEE Robot. Autom. Lett. 3 (4), 3081–3088 (2018).CrossRefGoogle Scholar
Cheng, J., Cheng, H., Meng, M. Q.-H. and Zhang, H., “Autonomous Navigation by Mobile Robots in Human Environments: A Survey,” 2018 IEEE International Conference on Robotics and Biomimetics (ROBIO), Kuala Lumpur, Malaysia (2018) pp. 1981–1986.CrossRefGoogle Scholar
Raguram, R., Chum, O., Pollefeys, M., Matas, J. and Frahm, J.-M., “USAC: a universal framework for random sample consensus.” IEEE Trans. Pattern Anal. Mach. Intell. 35 (8), 2022–2038 (2013).CrossRefGoogle ScholarPubMed
Redmon, J., Divvala, S., Girshick, R. and Farhadi, A., “You Only Look Once: Unified, Real-Time Object Detection,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA (2016) pp. 779–788.Google Scholar
Long, J., Shelhamer, E. and Darrell, T., “Fully Convolutional Networks for Semantic Segmentation,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA (2015) pp. 3431–3440.Google Scholar
Mur-Artal, R. and Tardos, J. D., “ORB-SLAM2: an open-source slam system for monocular, stereo and RGB-D cameras,” IEEE Transactions on Robotics 33(5), 1255–1262 (2017).CrossRefGoogle Scholar
Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C. and Torr, P. H., “Conditional Random Fields as Recurrent Neural Networks,” Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile (2015) 1529–1537.Google Scholar
Cheng, J., Sun, Y. and Meng, M. Q.-H., “A Dense Semantic Mapping System Based on CRF-RNN Network,” 2017 18th International Conference on Advanced Robotics (ICAR), Hong Kong, China (2017) pp. 589–594.CrossRefGoogle Scholar
Bloesch, M., Omari, S., Hutter, M. and Siegwart, R., “Robust Visual Inertial Odometry using a Direct EKFBased Approach,” 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany (2015) pp. 298–304.CrossRefGoogle Scholar
Usenko, V., Engel, J., Stückler, J. and Cremers, D., “Direct Visual-Inertial Odometry with Stereo Cameras,” 2016 IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden (2016) pp. 1885–1892.CrossRefGoogle Scholar
Kim, D.-H., Han, S.-B. and Kim, J.-H., “Visual odometry algorithm using an RGB-D sensor and IMU in a highly dynamic environment,” In: Robot Intelligence Technology and Applications 3, (Springer, Cham, 2015) pp. 11–26.CrossRefGoogle Scholar
Sun, Y., Liu, M. and Meng, M. Q.-H., “Active perception for foreground segmentation: An RGB-D databased background modeling method,” IEEE Trans. Autom. Sci. Eng. (2019). Early Access.CrossRefGoogle Scholar
Kim, D.-H. and Kim, J.-H., “Effective background model-based RGB-D dense visual odometry in a dynamic environment,” IEEE Trans. Robot. 32(6), 1565–1573 (2016).CrossRefGoogle Scholar
Sun, Y., Liu, M. and Meng, M. Q.-H., “Improving RGB-D slam in dynamic environments: A motion removal approach,” Robot. Autonom. Syst. 89, 110–122 (2017).CrossRefGoogle Scholar
Li, S. and Lee, D., ‘RGB-D slam in dynamic environments using static point weighting,” IEEE Robot. Autom. Lett. 2(4), 2263–2270 (2017).CrossRefGoogle Scholar
Sun, Y., Liu, M. and Meng, M. Q.-H., “Motion removal for reliable RGB-D slam in dynamic environments,” Robot. Autonom. Syst. 108, 115–128 (2018).CrossRefGoogle Scholar
Zou, D. and Tan, P., “Coslam: Collaborative visual slam in dynamic environments,” IEEE Trans. Pattern Anal. Machine Intell. 35 (2), 354–366 (2013).CrossRefGoogle ScholarPubMed
Wang, Y. and Huang, S., “Motion Segmentation Based Robust RGB-D Slam,” 2014 11th World Congress on Intelligent Control and Automation (WCICA), Shenyang, China (2014) pp. 3122–3127.Google Scholar
Terashima, T. and Hasegawa, O., “A Visual-Slam for First Person Vision andMobile Robots,” 2017 Fifteenth IAPR International Conference on Machine Vision Applications (MVA), Nagoya, Japan (2017) pp. 73–76.CrossRefGoogle Scholar
Cheng, J., Sun, Y., Chi, W., Wang, C., Cheng, H. and Meng, M. Q.-H., “An Accurate Localization Scheme for Mobile Robots Using Optical Flow in Dynamic Environments,” 2018 IEEE International Conference on Robotics and Biomimetics (ROBIO), Kuala Lumpur, Malaysia (2018) pp. 723–728.CrossRefGoogle Scholar
Panchpor, A. A., Shue, S. and Conrad, J. M., “A Survey of Methods for Mobile Robot Localization and Mapping in Dynamic Indoor Environments,” 2018 Conference on Signal Processing and Communication Engineering Systems (SPACES), Vijayawada, India (2018) pp. 138–144.CrossRefGoogle Scholar
Hermans, A., Floros, G. and Leibe, B., “Dense 3d Semantic Mapping of Indoor Scenes from RGB-D Images,” 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China (2014) pp. 2631–2638.CrossRefGoogle Scholar
Salas-Moreno, R. F., Newcombe, R. A., Strasdat, H., Kelly, P. H. and Davison, A. J., “Slam++: Simultaneous Localisation and Mapping at the Level of Objects,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, Oregon, USA (2013) 1352–1359.Google Scholar
Sünderhauf, N., Pham, T. T., Latif, Y., Milford, M. and Reid, I., “Meaningful Maps – Object-Oriented Semantic Mapping,” 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, Canada (2017) pp. 5079–5085.CrossRefGoogle Scholar
Bowman, S. L., Atanasov, N., Daniilidis, K. and Pappas, G. J., “Probabilistic Data Association for Semantic Slam,” 2017 IEEE International Conference on Robotics and Automation (ICRA), Marina Bay Sands, Singapore (2017) pp. 1722–1729.CrossRefGoogle Scholar
Gan, L., Jadidi, M. G., Parkison, S. A. and Eustice, R. M., “Sparse Bayesian inference for dense semantic mapping,” arXiv preprint arXiv:1709.07973, (2017).Google Scholar
Triggs, B., McLauchlan, P. F., Hartley, R. I. and Fitzgibbon, A. W., “Bundle Adjustment – A Modern Synthesis,” International Workshop on Vision Algorithms, Corfu, Greece (1999) pp. 298–372.Google Scholar
Nistér, D., “An efficient solution to the five-point relative pose problem,” IEEE Trans. Pattern Anal. Machine Intell. 26(6), 756–770 (2004).CrossRefGoogle ScholarPubMed
Baker, S. and Matthews, I., “Lucas-kanade 20 years on: A Unifying Framework,” Int. J. Comp. Vision 56(3), 221–255 (2004).CrossRefGoogle Scholar
Sturm, J., Engelhard, N., Endres, F., Burgard, W. and Cremers, D., “A Benchmark for the Evaluation of RGBD Slam Systems,” 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Algarve, Portugal (2012) pp. 573–580.CrossRefGoogle Scholar
Song, S., Lichtenberg, S. P. and Xiao, J., “Sun RGB-D: A RGB-D Scene Understanding Benchmark Suite,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA (2015) pp. 567–576.Google Scholar
Silberman, N. and Fergus, R., “Indoor Scene Segmentation Using a Structured Light Sensor,” 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain (2011) pp. 601–608.CrossRefGoogle Scholar
Xiao, J., Owens, A., and Torralba, A., “Sun3d: A Database of Big Spaces Reconstructed using SFM and Object Labels,” Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia (2013) pp. 1625–1632.Google Scholar
Song, S. and Xiao, J., “Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA (2016) pp. 808–816.Google Scholar
Hua, B.-S., Pham, Q.-H., Nguyen, D. T., Tran, M.-K., Yu, L.-F. and Yeung, S.-K., “Scenenn: A Scene Meshes Dataset with Annotations,” 2016 Fourth International Conference on 3D Vision (3DV), Stanford, California, USA (2016) pp. 92–101.CrossRefGoogle Scholar
Geiger, A., Lenz, P. and Urtasun, R., “AreWe Ready for Autonomous Driving? The KITTI Vision Benchmark Suite,” 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, Rhode Island, USA (2012) pp. 3354–3361.CrossRefGoogle Scholar