SS-MVMETRO: Semi-supervised multi-view human mesh recovery transformer (original) (raw)

Abstract

Parametric methods are widely utilized in RGB-based human mesh recovery, relying on precise statistical human body model parameters that are challenging to obtain. In contrast, non-parametric transformer-based approaches excel but are applied only to monocular RGB tasks. To address these limitations, this paper presents Semi-Supervised Multi-View Human Mesh Recovery Transformer (SS-MVMETRO), which combines multi-view information with non-parametric methods for the first time. Our model encodes different images according to their respective view directions, fusing local features around key points of joints and vertices. Then, a residual-like structure is proposed to integrate the fused features in the mesh recovery transformer, which subsequently predicts the 3D coordinates of the human mesh vertices. Additionally, we divide different views into the main view and auxiliary views and propose a semi-supervised training approach that requires fewer matching labels. The efficacy of our work is validated on two datasets, Human3.6M and Mpi_inf_3dph, through quantitative and qualitative experiments. The results indicate that SS-MVMETRO improves the reconstruction accuracy, surpassing previous image-based methods by at least 8.9% in terms of Procrustes Analysis Mean-Per-Joint-Position-Error (PA-MPJPE).

Graphical abstract

Access this article

Log in via an institution

Subscribe and save

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Data availability and access

The “Human3.6M” data that support the findings of this work are available in Human3.6M, the ”Mpi_inf_3dph” data are available in Mpi_inf_3dph, the ”Mpii” data are available in Mpii, the ”Muco” data are available in Muco, the ”Up3d” data are available in Up3d, the ”Coco” data are available in Coco. These datasets are publicly accessible.

References

  1. Loper M, Mahmood N, Romero J et al (2015) Smpl: A skinned multi-person linear model. ACM Transactions on Graphics 34(6):1–16. https://doi.org/10.1145/2816795.2818013
    Article Google Scholar
  2. Ran H, Ning X, Li W et al (2023) 3d human pose and shape estimation via de-occlusion multi-task learning. Neurocomputing 126284. https://doi.org/10.1016/j.neucom.2023.126284
  3. Wei G, Lan C, Zeng W et al (2020) View invariant 3d human pose estimation. IEEE Trans Circuits Syst Video Technol 30(12):4601–4610. https://doi.org/10.1109/TCSVT.2019.2928813
    Article Google Scholar
  4. Gu R, Wang G, Jiang Z et al (2020) Multi-person hierarchical 3d pose estimation in natural videos. IEEE Trans Circuits Syst Video Technol 30(11):4245–4257. https://doi.org/10.1109/TCSVT.2019.2953678
    Article Google Scholar
  5. Kolotouros N, Pavlakos G, Black MJ et al (2019) Learning to reconstruct 3d human pose and shape via model-fitting in the loop. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 2252–2261
  6. Zhang H, Tian Y, Zhou X et al (2021) Pymaf: 3d human pose and shape regression with pyramidal mesh alignment feedback loop. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 11446–11456
  7. Liang J, Lin MC (2019) Shape-aware human pose and shape reconstruction using multi-view images. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 4352–4362
  8. Lin K, Wang L, Liu Z (2021) Mesh graphormer. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 12939–12948
  9. Cho J, Youwang K, Oh TH (2022) Cross-attention of disentangled modalities for 3d human mesh recovery with transformers. In: Proceedings of the European conference on computer vision, Springer, pp 342–359
  10. Dong Y, Yuan Q, Peng R et al (2024) An iterative 3d human body reconstruction method driven by personalized dimensional prior knowledge. Appl Intell 54(1):738–748. https://doi.org/10.1007/s10489-023-05214-y
    Article Google Scholar
  11. Kim J, Gwon MG, Park H et al (2023) Sampling is matter: Point-guided 3d human mesh reconstruction. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 12880–12889
  12. Dai Y, Wen C, Wu H et al (2022) Indoor 3d human trajectory reconstruction using surveillance camera videos and point clouds. IEEE Trans Circuits Syst Video Technol 32(4):2482–2495. https://doi.org/10.1109/TCSVT.2021.3081591
    Article Google Scholar
  13. Zhang B, Ma K, Wu S et al (2023) Two-stage co-segmentation network based on discriminative representation for recovering human mesh from videos. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 5662–5670
  14. Zheng Z, Yu T, Liu Y et al (2022) Pamir: Parametric model-conditioned implicit representation for image-based human reconstruction. IEEE Trans Pattern Anal Mach Intell 44(6):3170–3184. https://doi.org/10.1109/TPAMI.2021.3050505
    Article Google Scholar
  15. Harvey FG, Yurick M, Nowrouzezahrai D et al (2020) Robust motion in-betweening. ACM Trans Graphics (TOG) 39(4):60–1. https://doi.org/10.1145/3386569.3392480
    Article Google Scholar
  16. Henter GE, Alexanderson S, Beskow J (2020) Moglow: Probabilistic and controllable motion synthesis using normalising flows. ACM Trans Graphics (TOG) 39(6):1–14. https://doi.org/10.1145/3414685.3417836
    Article Google Scholar
  17. Tian Y, Zhang H, Liu Y et al (2023) Recovering 3d human mesh from monocular images: A survey. IEEE Trans Pattern Anal Mach Intell 45(12):15406–15425. https://doi.org/10.1109/TPAMI.2023.3298850
    Article Google Scholar
  18. Bogo F, Kanazawa A, Lassner C et al (2016) Keep it smpl: Automatic estimation of 3d human pose and shape from a single image. In: Proceedings of the European conference on computer vision, Springer, pp 561–578
  19. Mahendran S, Ali H, Vidal R (2018) A mixed classification-regression framework for 3d pose estimation from 2d images. In: Proceedings of the British machine vision conference. BMVA Press, pp 72–84
  20. Lin K, Wang L, Liu Z (2021) End-to-end human pose and mesh reconstruction with transformers. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 1954–1963
  21. Shin S, Halilaj E (2020) Multi-view human pose and shape estimation using learnable volumetric aggregation
  22. Li Z, Oskarsson M, Heyden A (2021) 3d human pose and shape estimation through collaborative learning and multi-view model-fitting. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp 1888–1897
  23. Zhang S, Liu Y, Liu J et al (2022) Multi-view high precise 3d human body reconstruction method for virtual fitting. Int J Pattern Recognition Artif Intell 36(15):2256023. https://doi.org/10.1142/S0218001422560237
    Article Google Scholar
  24. Vaswani A, Shazeer N, Parmar N et al (2017) Attention is all you need. In: Proceedings of the International Conference on neural information processing systems, pp 6000–6010
  25. Zhang J, Cai Y, Yan S et al (2021) Direct multi-view multi-person 3d pose estimation. In: Proceedings of the International Conference on neural information processing systems, pp 13153–13164
  26. Hao C, Kong D, Li J et al (2023) Hypergraph based human mesh hierarchical representation and reconstruction from a single image. Comput & Graphics 115:339–347. https://doi.org/10.1016/j.cag.2023.07.011
    Article Google Scholar
  27. Zhou K, Han X, Jiang N et al (2022) Hemlets posh: Learning part-centric heatmap triplets for 3d human pose and shape estimation. IEEE Trans Pattern Anal Machine Intell 44(6):3000–3014. https://doi.org/10.1109/TPAMI.2021.3051173
    Article Google Scholar
  28. Chen D, Song Y, Liang F et al (2023) 3d human body reconstruction based on smpl model. Visual Comput 39(5):1893–1906. https://doi.org/10.1007/s00371-022-02453-x
    Article Google Scholar
  29. Lu Y, Yu H, Ni W et al (2023) 3d real-time human reconstruction with a single rgbd camera. Appl Intell 53(8):8735–8745. https://doi.org/10.1007/s10489-022-03969-4
    Article Google Scholar
  30. Khirodkar R, Tripathi S, Kitani K (2022) Occluded human mesh recovery. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 1715–1725
  31. He K, Zhang X, Ren S et al (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 770–778
  32. Sun K, Xiao B, Liu D et al (2019) Deep high-resolution representation learning for human pose estimation. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 5693–5703
  33. Li Z, Oskarsson M, Heyden A (2022) Detailed 3d human body reconstruction from multi-view images combining voxel super-resolution and learned implicit representation. Appl Intell 52(6):6739–6759. https://doi.org/10.1007/s10489-021-02783-8
    Article Google Scholar
  34. Xu W, Xiang D, Wang G et al (2022) Multiview video-based 3-d pose estimation of patients in computer-assisted rehabilitation environment (caren). IEEE Trans Human-Mach Syst 52(2):196–206. https://doi.org/10.1109/THMS.2022.3142108
    Article Google Scholar
  35. Gerats BG, Wolterink JM, Broeders IA (2023) 3d human pose estimation in multi-view operating room videos using differentiable camera projections. Comput Methods Biomech Biomed Eng: Imaging & Visualization 11(4):1197–1205. https://doi.org/10.1080/21681163.2022.2155580
    Article Google Scholar
  36. Shuai H, Wu L, Liu Q (2023) Adaptive multi-view and temporal fusing transformer for 3d human pose estimation. IEEE Trans Pattern Anal Machine Intell 45(4):4122–4135. https://doi.org/10.1109/TPAMI.2022.3188716
    Article Google Scholar
  37. Zhou ZH (2018) A brief introduction to weakly supervised learning. National Sci Rev 5(1):44–53. https://doi.org/10.1093/NSR/NWX106
    Article Google Scholar
  38. Zhou ZH, Li M (2010) Semi-supervised learning by disagreement. Knowl Inform Syst 24:415–439. https://doi.org/10.1007/s10115-009-0209-z
    Article Google Scholar
  39. Eren ME, Bhattarai M, Joyce RJ et al (2023) Semi-supervised classification of malware families under extreme class imbalance via hierarchical non-negative matrix factorization with automatic model selection. ACM Trans Privacy Secur 26(4):1–27. https://doi.org/10.1145/3624567
    Article Google Scholar
  40. Wu L, Fang L, He X et al (2023) Querying labeled for unlabeled: Cross-image semantic consistency guided semi-supervised semantic segmentation. IEEE Trans Pattern Anal Mach Intell 45(7):8827–8844. https://doi.org/10.1109/TPAMI.2022.3233584
    Article Google Scholar
  41. Yang X, Song Z, King I et al (2023) A survey on deep semi-supervised learning. IEEE Trans Knowl Data Eng 35(9):8934–8954. https://doi.org/10.1109/TKDE.2022.3220219
    Article Google Scholar
  42. Zhao H, Jia J, Koltun V (2020) Exploring self-attention for image recognition. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 10076–10085
  43. Wenxuan Z, Yaqin Z, Zhaoxiang Z et al (2023) Lite transformer network with long-short range attention for real-time fire detection. Fire Technol 59(6):3231–3253. https://doi.org/10.1007/s10694-023-01465-w
    Article Google Scholar
  44. Ranjan A, Bolkart T, Sanyal S et al (2018) Generating 3d faces using convolutional mesh autoencoders. In: Proceedings of the European conference on computer vision, Springer, pp 704–720
  45. Pang S, Peng R, Dong Y et al (2023) Jointmetro: a 3d reconstruction model for human figures in works of art based on transformer. Neural Comput Appl pp 1–15. https://doi.org/10.1007/s00521-023-08844-y
  46. Kocabas M, Huang CHP, Hilliges O et al (2021) Pare: Part attention regressor for 3d human body estimation. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 11127–11137
  47. Ionescu C, Papava D, Olaru V et al (2013) Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Trans Pattern Anal Machine Intell 36(7):1325–1339. https://doi.org/10.1109/TPAMI.2013.248
  48. Andriluka M, Pishchulin L, Gehler P et al (2014) 2d human pose estimation: New benchmark and state of the art analysis. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 3686–3693
  49. Mehta D, Sotnychenko O, Mueller F et al (2018) Single-shot multi-person 3d pose estimation from monocular rgb. In: Proceedings of the IEEE International Conference on 3D vision, pp 120–130
  50. Lassner C, Romero J, Kiefel M et al (2017) Unite the people: Closing the loop between 3d and 2d human representations. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 6050–6059
  51. Lin TY, Maire M, Belongie S et al (2014) Microsoft coco: Common objects in context. In: Proceedings of the European conference on computer vision, Springer, pp 740–755
  52. Mehta D, Rhodin H, Casas D et al (2017) Monocular 3d human pose estimation in the wild using improved cnn supervision. In: Proceedings of the IEEE International Conference on 3D vision, pp 506–516
  53. Deng J, Dong W, Socher R et al (2009) Imagenet: A large-scale hierarchical image database. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 248–255
  54. Loshchilov I, Hutter F (2018) Decoupled weight decay regularization. In: Proceedings of the International Conference on Learning Representations, pp 1–18
  55. Wang L, Liu X, Ma X et al (2022) A progressive quadric graph convolutional network for 3d human mesh recovery. IEEE Trans Circuits Syst Video Technol 33(1):104–117. https://doi.org/10.1109/TCSVT.2022.3199201
    Article Google Scholar
  56. Kolotouros N, Pavlakos G, Jayaraman D et al (2021) Probabilistic modeling for human mesh recovery. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 11605–11614
  57. Yu Z, Zhang L, Xu Y et al (2022) Multiview human body reconstruction from uncalibrated cameras. In: Proceedings of the International Conference on neural information processing systems, pp 7879–7891

Download references

Funding

This work was supported in part by the Youth Innovation Promotion Association of Chinese Academy of Sciences (Y202072) and in part by the Natural Science Foundation of Shandong Province (ZR2021QE205).

Author information

Authors and Affiliations

  1. School of Biomedical Engineering (Suzhou), Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, Anhui, 230026, People’s Republic of China
    Silong Sheng, Zhijie Ren & Weiwei Fu
  2. Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Science, Suzhou, Jiangsu, 215163, People’s Republic of China
    Silong Sheng, Tianyou Zheng, Zhijie Ren, Yang Zhang & Weiwei Fu
  3. Jinan Guoke Medical Technology Development Co., Ltd, Jinan, Shandong, 250000, People’s Republic of China
    Yang Zhang

Authors

  1. Silong Sheng
  2. Tianyou Zheng
  3. Zhijie Ren
  4. Yang Zhang
  5. Weiwei Fu

Contributions

Conceptualization: Silong Sheng; Methodology: Silong Sheng, Tianyou Zheng, Zhijie Ren; Formal analysis and investigation: Silong Sheng, Tianyou Zheng; Writing - original draft preparation: Silong Sheng; Writing - review and editing: Tianyou Zheng, Weiwei Fu, Yang Zhang, Zhijie Ren; Funding acquisition: Weiwei Fu, Yang Zhang; Resources: Weiwei Fu, Yang Zhang; Supervision: Weiwei Fu.

Corresponding authors

Correspondence toTianyou Zheng or Weiwei Fu.

Ethics declarations

Competing interests

The authors have no competing interests to declare that are relevant to the content of this article.

Ethics approval

Not applicable.

Not applicable.

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Cite this article

Sheng, S., Zheng, T., Ren, Z. et al. SS-MVMETRO: Semi-supervised multi-view human mesh recovery transformer.Appl Intell 54, 5027–5043 (2024). https://doi.org/10.1007/s10489-024-05435-9

Download citation

Keywords