FLEX: Parameter-free Multi-view 3D Human Movement Reconstruction

FLEX: Parameter-free Multi-view 3D Human Movement Reconstruction

Present human movement reconstruction strategies utilizing movement seize sensors require a tedious and costly process. The widespread availability of video recordings from RGB cameras could make this activity simpler.

Nonetheless, multi-cameras settings that are used to keep away from occlusion and depth ambiguity are nonetheless an issue. A latest paper on arXiv.org suggests a parameter-free multi-view movement reconstruction algorithm.

Physique movement seize. Picture credit score: Raíssa Ruschel through Flickr, CC BY 2.0

It depends on the perception that the 3D angle between the skeletal components is invariant to the digital camera place. A neural community learns to foretell joint angles and bone lengths with out utilizing any of the digital camera parameters. A novel fusion layer is used to extend the arrogance of every joint detection and mitigate occlusions. Qualitative and quantitative evaluations present that the advised mannequin outperforms state-of-the-art strategies in movement and pose reconstruction by a big margin.

The rising availability of video recordings made by a number of cameras has supplied new means for mitigating occlusion and depth ambiguities in pose and movement reconstruction strategies. But, multi-view algorithms strongly rely upon digital camera parameters, particularly, the relative positions among the many cameras. Such dependency turns into a hurdle as soon as shifting to dynamic seize in uncontrolled settings. We introduce FLEX (Free muLti-view rEconstruXion), an end-to-end parameter-free multi-view mannequin. FLEX is parameter-free within the sense that it doesn’t require any digital camera parameters, neither intrinsic nor extrinsic. Our key concept is that the 3D angles between skeletal components, in addition to bone lengths, are invariant to the digital camera place. Therefore, studying 3D rotations and bone lengths slightly than places permits predicting frequent values for all digital camera views. Our community takes a number of video streams, learns fused deep options by way of a novel multi-view fusion layer, and reconstructs a single constant skeleton with temporally coherent joint rotations. We display quantitative and qualitative outcomes on the Human3.6M and KTH Multi-view Soccer II datasets. We examine our mannequin to state-of-the-art strategies that aren’t parameter-free and present that within the absence of digital camera parameters, we outperform them by a big margin whereas acquiring comparable outcomes when digital camera parameters can be found. Code, skilled fashions, video demonstration, and extra supplies might be accessible on our venture web page.

Analysis paper: Gordon, B., Raab, S., Azov, G., Giryes, R., and Cohen-Or, D., “FLEX: Parameter-free Multi-view 3D Human Movement Reconstruction”, 2021. Hyperlink: https://arxiv.org/abs/2105.01937

Source link