Hostname: page-component-745bb68f8f-l4dxg Total loading time: 0 Render date: 2025-01-12T14:05:13.683Z Has data issue: false hasContentIssue false
  • English
  • Français

Lightweight deep network-enabled real-time low-visibility enhancement for promoting vessel detection in maritime video surveillance

Published online by Cambridge University Press:  29 October 2021

Yu Guo ,
Yuxu Lu  and
Ryan Wen Liu Open the ORCID record for Ryan Wen Liu [Opens in a new window]
Yu Guo
Affiliation:
Hubei Key Laboratory of Inland Shipping Technology, School of Navigation, Wuhan University of Technology, Wuhan, China
Yuxu Lu
Affiliation:
Hubei Key Laboratory of Inland Shipping Technology, School of Navigation, Wuhan University of Technology, Wuhan, China
Ryan Wen Liu*
Affiliation:
Hubei Key Laboratory of Inland Shipping Technology, School of Navigation, Wuhan University of Technology, Wuhan, China National Engineering Research Center for Water Transportation Safety, Wuhan, China
*
*Corresponding author. E-mail: wenliu@whut.edu.cn
Get access Rights & Permissions [Opens in a new window]

Abstract

Maritime video surveillance has become an essential part of the vessel traffic services system, intended to guarantee vessel traffic safety and security in maritime applications. To make maritime surveillance more feasible and practicable, many intelligent vision-empowered technologies have been developed to automatically detect moving vessels from maritime visual sensing data (i.e., maritime surveillance videos). However, when visual data is collected in a low-visibility environment, the essential optical information is often hidden in the dark, potentially resulting in decreased accuracy of vessel detection. To guarantee reliable vessel detection under low-visibility conditions, the paper proposes a low-visibility enhancement network (termed LVENet) based on Retinex theory to enhance imaging quality in maritime video surveillance. LVENet is a lightweight deep neural network incorporating a depthwise separable convolution. The synthetically-degraded image generation and hybrid loss function are further presented to enhance the robustness and generalisation capacities of LVENet. Both full-reference and no-reference evaluation experiments demonstrate that LVENet could yield comparable or even better visual qualities than other state-of-the-art methods. In addition, it takes LVENet just 0⋅0045 s to restore degraded images with size 1920 × 1080 pixels on an NVIDIA 2080Ti GPU, which can adequately meet real-time requirements. Using LVENet, vessel detection performance can be greatly improved with enhanced visibility under low-light imaging conditions.

Keywords

maritime video surveillanceintelligent visionvessel detectionlow-visibility enhancementmaritime safety
Type
Research Article
Information
The Journal of Navigation , Volume 75 , Issue 1 , January 2022 , pp. 230 - 250
Check for updates
Copyright
Copyright © The Author(s), 2021. Published by Cambridge University Press on behalf of The Royal Institute of Navigation

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

†

(Yu Guo and Yuxu Lu are co-first authors)

References

Bloisi, D. D., Previtali, F., Pennisi, A., Nardi, D. and Fiorini, W. (2017). Enhancing automatic maritime surveillance systems with visual information. IEEE Transactions on Intelligent Transportation Systems, 18(4), 824–833.CrossRefGoogle Scholar
Bochkovskiy, A., Wang, C. Y. and Liao, H. Y. M. (2020). YOLOv4: Optimal speed and accuracy of object detection. arXiv: 2004.10934.Google Scholar
Bychkovsky, V., Paris, S., Chan, E. and Durand, F. (2011). Learning Photographic Global Tonal Adjustment with Database of Input/Output Image Pairs. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, USA.Google Scholar
Cai, B., Xu, X., Guo, K., Jia, K., Hu, B. and Tao, D. (2017). A Joint Intrinsic-Extrinsic Prior Model for Retinex. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.CrossRefGoogle Scholar
Chaturvedi, S. K. (2019). Study of synthetic aperture radar and automatic identification system for ship target detection. Journal of Ocean Engineering and Science, 4(2), 173–182.CrossRefGoogle Scholar
Chen, S. D. and Ramli, A. R. (2003). Minimum mean brightness error bi-histogram equalization in contrast enhancement. IEEE Transactions on Consumer Electronics, 49(4), 1310–1319.CrossRefGoogle Scholar
Chen, X., Wang, S., Shi, C., Wu, H., Zhao, J. and Fu, J. (2019). Robust ship tracking via multi-view learning and sparse representation. The Journal of Navigation, 72(1), 176–192.CrossRefGoogle Scholar
Chen, X., Yang, Y., Wang, S., Wu, H., Tang, J., Zhao, J. and Wang, Z. (2020). Ship type recognition via a coarse-to-fine cascaded convolution neural network. The Journal of Navigation, 73(4), 813–832.CrossRefGoogle Scholar
Dong, C., Chen, C. L., He, K. and Tang, X. (2016). Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2), 295–307.CrossRefGoogle ScholarPubMed
Everingham, M., Gool, L. V., Williams, C. K. I., Winn, J. and Zisserman, A. (2010). The PASCAL visual object classes (VOC) challenge. International Journal of Computer Vision, 88, 303–338.CrossRefGoogle Scholar
Fu, X., Zeng, D., Huang, Y., Ding, X. and Zhang, X. P. (2013). A Variational Framework for Single Low Light Image Enhancement Using Bright Channel Prior. Proceedings of the IEEE Global Conference on Signal and Information Processing, Austin, TX, USA.CrossRefGoogle Scholar
Fu, X., Zeng, D., Huang, Y., Zhang, X. P. and Ding, X. (2016). A Weighted Variational Model for Simultaneous Reflectance and Illumination Estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.CrossRefGoogle Scholar
Gu, K., Wang, S., Zhai, G., Ma, S., Yang, X., Lin, W., Zhang, W. and Gao, W. (2016). Blind quality assessment of tone-mapped images via analysis of information, naturalness, and structure. IEEE Transactions on Multimedia, 18(3), 432–443.CrossRefGoogle Scholar
Guo, X., Li, Y. and Ling, H. (2017). LIME: Low-light image enhancement via illumination map estimation. IEEE Transactions on Image Processing, 26(2), 982–993.CrossRefGoogle Scholar
Guo, Y., Lu, Y., Liu, R. W., Yang, M. and Chui, K. T. (2020). Low-light image enhancement with regularized illumination optimization and deep noise suppression. IEEE Access, 8, 145297–145315.CrossRefGoogle Scholar
He, K., Sun, J. and Tang, X. (2010). Single image haze removal using dark channel prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(12), 2341–2353.Google ScholarPubMed
He, K., Sun, J. and Tang, X. (2013). Guided image filtering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(6), 1397–1409.CrossRefGoogle ScholarPubMed
He, K., Zhang, X., Ren, S. and Sun, J. (2016). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.CrossRefGoogle Scholar
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M. and Adam, H. (2017). MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv: 1704.04861.Google Scholar
Hu, W. C., Yang, C. Y. and Huang, D. Y. (2011). Robust real-time ship detection and tracking for visual surveillance of cage aquaculture. Journal of Visual Communication and Image Representation, 22(6), 543–556.CrossRefGoogle Scholar
Jiang, B., Woodell, G. A. and Jobson, D. J. (2015). Novel multi-scale retinex with color restoration on graphics processing unit. Journal of Real-Time Image Processing, 10, 239–253.CrossRefGoogle Scholar
Jobson, D. J., Rahman, Z. and Woodell, G. A. (1997a). Properties and performance of a center/surround retinex. IEEE Transactions on Image Processing, 6(3), 451–462.CrossRefGoogle Scholar
Jobson, D. J., Rahman, Z. and Woodell, G. A. (1997b). A multiscale retinex for bridging the gap between color images and the human observation of scenes. IEEE Transactions on Image Processing, 6(7), 965–976.CrossRefGoogle Scholar
Kim, Y. T. (1997). Contrast enhancement using brightness preserving bi-histogram equalization. IEEE Transactions on Consumer Electronics, 43(1), 1–8.Google Scholar
Kim, K., Hong, S., Choi, B. and Kim, E. (2018). Probabilistic ship detection and classification using deep learning. Applied Sciences, 8(6), 936–953.CrossRefGoogle Scholar
Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv: 1412.6980.Google Scholar
Land, E. H. (1977). The Retinex theory of color vision. Scientific American, 237(6), 108–129.CrossRefGoogle ScholarPubMed
Li, B., Peng, X., Wang, Z., Xu, J. and Feng, D. (2017). AOD-Net: All-in-One Dehazing Network. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.CrossRefGoogle Scholar
Li, C., Guo, J., Poriki, F. and Pang, Y. (2018). Lightennet: A convolutional neural network for weakly illuminated image enhancement. Pattern Recognition Letters, 104(1), 15–22.CrossRefGoogle Scholar
Liu, R. W., Nie, J., Garg, S., Xiong, Z., Zhang, Y. and Hossain, M. S. (2021a). Data-driven trajectory quality improvement for promoting intelligent vessel traffic services in 6G-enabled maritime IoT systems. IEEE Internet of Things Journal, 8(7), 5374–5385.CrossRefGoogle Scholar
Liu, R. W., Yuan, W., Chen, X. and Lu, Y. (2021b). An enhanced CNN-enabled learning method for promoting ship detection in maritime surveillance system. Ocean Engineering, 235, 109435.CrossRefGoogle Scholar
Lore, K. G., Akintayo, A. and Sarkar, S. (2017). LLNet: A deep autoencoder approach to natural low-light image enhancement. Pattern Recognition, 61, 650–662.CrossRefGoogle Scholar
Lu, Y., Yang, M. and Liu, R. W. (2021). DSPNet: Deep Learning-Enabled Blind Reduction of Speckle Noise. Proceedings of the International Conference on Pattern Recognition, Milan, Italy.CrossRefGoogle Scholar
Lv, F., Lu, F., Wu, J. and Lim, C. (2018). MBLLEN: Low-Light Image/Video Enhancement Using CNNS. Proceedings of the British Machine Vision Conference, Newcastle, UK.Google Scholar
Mittal, A., Soundararajan, R. and Bovik, A. C. (2013). Making a “completely blind” image quality analyzer. IEEE Signal Processing Letters, 20(3), 209–212.CrossRefGoogle Scholar
Nie, X., Yang, M. and Liu, R. W. (2019). Deep Neural Network-Based Robust Ship Detection under Different Weather Conditions. Proceedings of the IEEE International Conference on Intelligent Transportation Systems, Auckland, NZ.CrossRefGoogle Scholar
Pisano, E. D., Zong, S., Hemminger, B. M., DeLuca, M., Johnston, R. E., Muller, K., Braeuning, M. P. and Pizer, S. M. (1998). Contrast limited adaptive histogram equalization image processing to improve the detection of simulated spiculations in dense mammograms. Journal of Digital Imaging, 11(4), 193–200.CrossRefGoogle ScholarPubMed
Radford, A., Metz, L. and Chintala, S. (2015). Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. Proceedings of the International Conference on Learning Representations, San Juan, PR.Google Scholar
Shao, Z., Wu, W., Wang, Z., Du, W. and Li, C. (2018). Seaships: A large-scale precisely annotated dataset for ship detection. IEEE Transactions on Multimedia, 20(10), 2593–2604.CrossRefGoogle Scholar
Shao, Z., Wang, L., Wang, Z., Du, W. and Wu, W. (2020). Saliency-aware convolution neural network for ship detection in surveillance video. IEEE Transactions on Circuits and Systems for Video Technology, 30(3), 781–794.CrossRefGoogle Scholar
Shen, L., Yue, Z., Fen, F., Chen, Q., Liu, S. and Ma, J. (2017). MSR-net: Low-light image enhancement using deep convolutional network. arXiv: 1711.02488.Google Scholar
Tan, L. T., Sim, K. S. and Tso, C. P. (2012). Image enhancement using background brightness preserving histogram equalization. Electronics Letters, 48(3), 155–157.CrossRefGoogle Scholar
Wang, Z. and Bovik, A. C. (2009). Mean squared error: Love it or leave it? IEEE Signal Processing Magazine, 26(1), 98–117.CrossRefGoogle Scholar
Wang, Z., Bovik, A. C., Sheikh, H. R. and Simoncelli, E. P. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600–612.CrossRefGoogle ScholarPubMed
Wang, S., Zheng, J., Hu, H. and Li, B. (2013). Naturalness preserved enhancement algorithm for non-uniform illumination images. IEEE Transactions on Image Processing, 22(9), 3538–3578.CrossRefGoogle ScholarPubMed
Wei, C., Wang, W., Yang, W. and Liu, J. (2018). Deep Retinex Decomposition for Low-Light Enhancement. Proceedings of the British Machine Vision Conference, Newcastle, UK.Google Scholar
Wu, F., Zhou, Z., Wang, B. and Ma, J. (2018). Inshore ship detection based on convolutional neural network in optical satellite images. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11(11), 4005–4015.CrossRefGoogle Scholar
Yang, C. S., Park, J. H. and Rashid, A. H. A. (2018). An improved method of land masking for synthetic aperture radar-based ship detection. The Journal of Navigation, 71(4), 788–804.CrossRefGoogle Scholar
Yang, M., Nie, X. and Liu, R. W. (2019a). Coarse-to-Fine Luminance Estimation for low-Light Image Enhancement in Maritime Video Surveillance. Proceedings of the IEEE International Conference on Intelligent Transportation Systems, Auckland, NZ.CrossRefGoogle Scholar
Yang, W., Liu, J., Yang, S. and Guo, Z. (2019b). Scale-free single image deraining via visibility enhanced recurrent wavelet learning. IEEE Transactions on Image Processing, 28(6), 2948–2961.CrossRefGoogle Scholar
Zhang, L., Zhang, L., Mou, X. and Zhang, D. (2011). FSIM: A feature similarity index for image quality assessment. IEEE Transactions on Image Processing, 20(8), 2378–2386.CrossRefGoogle ScholarPubMed
Zhang, L., Shen, Y. and Li, H. (2014). VSI: A visual saliency-induced index for perceptual image quality assessment. IEEE Transactions on Image Processing, 23(10), 4270–4281.CrossRefGoogle ScholarPubMed
Zhang, W., Goerlandt, F., Kujala, P. and Wang, Y. (2016). An advanced method for detecting possible near miss ship collisions from AIS data. Ocean Engineering, 124, 1–16.CrossRefGoogle Scholar
Zhang, W. B., Kopca, C., Tang, J., Ma, D. and Wang, Y. (2017). A systematic approach for collision risk analysis based on AIS data. The Journal of Navigation, 70(5), 1117–1132.CrossRefGoogle Scholar
Zhang, Y., Zhang, J. and Guo, X. (2019). Kindling the Darkness: A Practical Low-Light Image Enhancer. Proceedings of the ACM International Conference on Multimedia, Nice, France.CrossRefGoogle Scholar
Zhu, C., Zhou, H., Wang, R. and Guo, J. (2010). A novel hierarchical method of ship detection from spaceborne optical image based on shape and texture features. IEEE Transactions on Geoscience and Remote Sensing, 48(9), 3446–3456.CrossRefGoogle Scholar