Image inpainting based on sparse representation using self-similar joint sparse coding (original) (raw)

Abstract

In order to improve the sparse coding ability of over-complete dictionary and take advantage of the similarity between damaged pixels and their neighbors, we propose an inpainting method based on sparse representation using self-similar joint sparse coding. First, we perform singular value decomposition on the gradient vector of the image patches, and then divide the image patches into three categories: smooth patches, edge patches and texture patches according to the relationship between the primary direction and the secondary direction. Second, we use the KSVD method to train these three types of image patches respectively, and obtain three over-complete dictionaries that adapt to different local features. Third, we define a non-local self-similar matching function and use it to search for the most similar image patch to the current patch in the target region, and then use the similar patch and the current patch for joint sparse coding. Finally, we use the calculated sparse coding and the corresponding over-complete dictionary to reconstruct the current patch. A series of experimental results show that the self-similar joint sparse coding we proposed can not only improve the restoration effect of sparse representation methods to a certain extent, but also has good adaptability and can be combined with other sparse representation methods to improve their restoration effect.

Access this article

Log in via an institution

Subscribe and save

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Aharon M, Elad M, Bruckstein A (2006) K-SVD: an algorithm for designing over-complete dictionaries for sparse representation [J]. IEEE Trans Signal Process 54(11):4311–4322
    Article MATH Google Scholar
  2. Bai H, Hong C, Li S, Zhang YD, Li X (2022) Unit-norm tight frame-based sparse representation with application to speech inpainting [J]. Digi Sign Proc 123:103426
    Article Google Scholar
  3. Barba-J L, Vargas-Quintero L, Calderón-Agudelo JA (2022) Bone SPECT/CT image fusion based on the discrete Hermite transform and sparse representation [J]. Biomed Signal Proc Contr 71:103096
    Article Google Scholar
  4. Bertalmio M, Sapiro G, Caselles V, et al. (2000) Image inpainting [C]. Proceedings of the 27th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co.: 417–424.
  5. Chan TF, Shen J (2001) Nontexture inpainting by curvature-driven diffusions [J]. J Vis Commun Image Represent 12(4):436–449
    Article Google Scholar
  6. Chan TF, Shen J (2002) Mathematical models for local nontexture inpaintings [J]. SIAM J Appl Math 62(3):1019–1043
    Article MathSciNet MATH Google Scholar
  7. Chen Y, Zhang H, Liu L et al (2021) Research on image inpainting algorithm of improved total variation minimization method [J]. J Ambient Intell Humaniz Comput 8:1–10
    Google Scholar
  8. Cheng K, Shichao LI, Rong L et al (2021) Video stabilization via prediction with time-series network and image inpainting with pyramid fusion [J]. Chin J Electron 30(6):1103–1110
    Article Google Scholar
  9. Criminisi A, Pérez P, Toyama K (2004) Region filling and object removal by exemplar-based image inpainting [J]. IEEE Trans Image Process 13(9):1200–1212
    Article Google Scholar
  10. Dong X, Dong J, Sun G, Duan Y, Qi L, Yu H (2019) Learning-based texture synthesis and automatic inpainting using support vector machines [J]. IEEE Trans Ind Electron 66(6):4777–4787
    Article Google Scholar
  11. Elad M, Starck JL, Querre P et al (2005) Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA) [J]. Appl Comput Harmon Anal 19(3):340–358
    Article MathSciNet MATH Google Scholar
  12. Feng XG, Milanfar P (2002) Multiscale principal components analysis for image local orientation estimation [J]. Proceedings of the 36th Asilomar conference on signals. Syst Comput 1:478–482
    Google Scholar
  13. Gapon N, Voronin V, Semenishchev E, et al. (2021) Multimodal image inpainting for an autonomous robot navigation application [C], Multimodal Sensing and Artificial Intelligence: Technologies and Applications II: 117850Y.
  14. Iizuka S, Simo-Serra E, Ishikawa H (2017) Globally and locally consistent image completion [J]. ACM Trans Graph 36(4):107–114
    Article Google Scholar
  15. Jam J, Kendrick C, Walker K et al (2020) A comprehensive review of past and present image inpainting methods [J]. Comput Vis Image Underst 203:103147–103181
    Article Google Scholar
  16. Jin X, Su Y, Zou L, Zhang C, Jing P, Song X (2018) Video logo removal detection based on sparse representation [J]. Multimed Tools Appl 77(22):29303–29322
    Article Google Scholar
  17. Kaur A, Raj A, Jayanthi N, Indu S (2020) Inpainting of irregular holes in a manuscript using UNet and partial convolution [C]. Second Int Conf Inventive Res Comput App (ICIRCA) 2020:778–784
    Google Scholar
  18. Kaur RP, Jindal MK, Kumar M (2021) Text and graphics segmentation of newspapers printed in Gurmukhi script: a hybrid approach [J]. Vis Comput 37:1637–1659
    Article Google Scholar
  19. Kumar M, Chhabra P, Garg NK (2018) An efficient content based image retrieval system using BayesNet and K-NN [J]. Multimed Tools Appl 77(16):1–14
    Article Google Scholar
  20. Li R, Tang L, Bai Y, Wang Q, Zhang X, Liu M (2020) Group-based sparse representation based on lp-norm minimization for image Inpainting [J]. IEEE Access 8:60515–60525
    Article Google Scholar
  21. Li Y, Gao Y, Yan J et al (2021) Image inpainting methods based on deep neural networks: a review [J]. Chinese J Comput 44(11):2295–2316
    Google Scholar
  22. Liu H, Cao S, Ling Y et al (2021) Inpainting for saturation artifacts in optical coherence tomography using dictionary-based sparse representation [J]. IEEE Photonics J 13(2):1–10
    Article Google Scholar
  23. Mirza M, Osindero S (2014) Conditional generative adversarial nets [J]. Comput Therm Sci:2672–2680
  24. Mullah HU, Deka B, Prasad A (2020) Fast multi-spectral image super-resolution via sparse representation [J]. IET Image Process 14(12):2833–2844
    Article Google Scholar
  25. Newson A, Almansa A, Gousseau Y, Pérez P (2017) Non-local patch-based image inpainting [J]. Image Proc Line 7:373–385
    Article MathSciNet Google Scholar
  26. Pathak D, Krahenbuhl P, Donahue J, et al. (2016) Context encoders: feature learning by inpainting [C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA: 2536–2544.
  27. Rathish Kumar BV, Halim A (2019) A linear fourth-order PDE-based gray-scale image inpainting model [J]. Comput Appl Math 38(6):1–21
    MathSciNet MATH Google Scholar
  28. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutionalnetworks for biomedical image segmentation [C]. Proceedings of the International Conference on Medical Image Computing and Computer-assisted Intervention, Munich, Germany: 234–241.
  29. Sreelakshmy IJ, Binsu CK (2021) Towards source-based classification of image inpainting techniques: a survey [J]. J Inf Knowl Manag 20(3):2150039
    Article Google Scholar
  30. Sun M, Meng Q, Wang T et al (2020) Removal of manually induced artifacts in ultrasound images of thyroid nodules based on edge-connection and Criminisi image restoration algorithm[J]. Comput Methods Prog Biomed 200(2):105868–105877
    Google Scholar
  31. Torrado-Carvajal A, Albrecht DS, Lee J, Andronesi OC, Ratai EM, Napadow V, Loggia ML (2021) Inpainting as a technique for estimation of missing voxels in brain imaging [J]. Ann Biomed Eng 49:345–353
    Article Google Scholar
  32. Yan Z, Li X, Li M, et al. (2018) Shift-net: image inpainting via deep feature rearrangement [C]. Proceedings of the European Conference on Computer Vision. Munich, Germany: 1–17.
  33. Yao F (2019) Damaged region filling by improved criminisi image inpainting algorithm for thangka[J]. Clust Comput 22(6):13683–13691
    Article Google Scholar
  34. Zhang L, Chang M (2020) Image inpainting for object removal based on adaptive two-round search strategy [J]. IEEE Access 8:94357–94372
    Article Google Scholar
  35. Zkaya U, Makineci HB, Ztürk A et al (2021) Obtaining missing areas with exemplar inpainting in mosaic unmanned aerial vehicle images [J]. Geomatik 6(1):61–68
    Google Scholar

Download references

Acknowledgements

The research is supported in part by National Natural Science Foundation of China (Grant: 61703363), in part by Scientific and Technological Innovation Programs of Higher Education Institutions in Shanxi Province (Grant: 2020 L0572), in part by Scientific Research Project of Yuncheng University (Grant: XK-2018034, CY-2019025, YQ-2020021), in part by the Industrial Science and Technology Research Project of Henan Province (Grant: 202102210387, 212102210418), in part by the Natural Science Foundation Project of Henan Province (Grant: 222300420582).

Availability of data and material

Please contact authors for data requests.

Code availability

Not applicable.

Author information

Authors and Affiliations

  1. School of Mathematics and Information Technology, Yuncheng University, Yuncheng, 044000, China
    Lei Zhang & Minhui Chang
  2. Software Engineering College, Zhengzhou University of Light Industry, Zhengzhou, 450001, China
    Rui Chen

Authors

  1. Lei Zhang
  2. Minhui Chang
  3. Rui Chen

Corresponding author

Correspondence toLei Zhang.

Ethics declarations

Conflict of interest

The authors declare that they have no competing interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Cite this article

Zhang, L., Chang, M. & Chen, R. Image inpainting based on sparse representation using self-similar joint sparse coding.Multimed Tools Appl 82, 20215–20231 (2023). https://doi.org/10.1007/s11042-023-14337-w

Download citation

Keywords