Proposed: Retarget VRM Motion Blender Workflow

Context and Problem Statement

We need to retarget motion for VRM type models.

This is a lengthy offline process to retarget other motions to VRM type models.

Describe the proposed option and how it helps to overcome the problem or limitation

Use Blender to retarget t-posed characters.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

  • Inputs: bvh + t-pose avatar
  • Output: glb w/retargetted animation

Requirements:

  • Blender v2.91 or v2.92 (may work with other versions)
  • Blender Add-On: skeletal-motion-retarget-blender
    • Download .zip
    • Use Blender Edit => Preferences, Add-ons, Install that zip
    • Check the box for the new install an add-on to activate
  • (optional) if using a VRM avatar Blender Add-On: saturday06/VRM_Addon_for_Blender
    • Download .zip
    • Use Blender Edit => Preferences, Add-ons, Install that zip
    • Check the box for the newly install an add-on to activate

Conversion Process

  • Have a rigged, T-Pose humanoid avatar
  • Start a new scene?
    • Under the Scene Settings property panel, set Frame Rate to 30 fps
    • (or you may want to match your .mp4 frame rate used during capture)
  • Import your avatar model
    • if using a VRM avatar, be sure to use Import VRM (otherwise, retargeting may not work!)
  • Import the generated BVH. In the Animation section, enable the following options
  • Use the Pose Transfer tab (vertical tab part of the “N” options panel?)
    • Source: the imported BVH armature
    • Target: the imported avatar armature
    • Click the Rebuild Bone List
      • manually map over any bones not automatically mapped
      • eg: LeftLowerArm => “Left elbow”
      • note: it is possible to save your mapping to .json and reuse
    • Click Retarget Animation
      • retargeting may take several minutes
  • Verify retargeting by playing the animation via the Blender play button
  • Export the scene as a .glb (using default settings)

Common Problems

Q: Retargeted animation seems to have legs crossed / backwards

A: This can happen if you import a VRM avatar as a .gltf (re-import using the VRM importer add-on)

Q: Legs/arms bend unnaturally

A: Check the armature bone name mappings

  • at most, each target bone name should appear once
  • leave BVH joints not present in your avatar blank

Acknowledgments

Thanks to humbletim for being the scribe

Thanks to lyuma for being test pilot flying these steps

Positive Consequences

  • Have a generally robust process of retargeting motions to VRM-type models.

Negative Consequences

  • Only VRM
  • MMD does not work.

Option graveyard: Adobe Mixamo

List the reasons for the rejection: (the Bad traits)

  • Proprietary Online Service
  • Blackbox technology

If this enhancement will not be used often, can it be worked around with a few lines of script?

Motion retargeting is not trivial offline or online.

Is there a reason why this should be core and done by us?

We need motions for our character controller without using an off-the-shelf black box solution.

References

  • https://www.blender.org/
  • https://hackmd.io/EmOAt375S8KFqaNdPDJCXQ
  • https://github.com/V-Sekai/V-Sekai-Blender-tools/tree/main/addons/blender-skeletal-motion-animate

Derivative License

Copyright (c) 2020-2021 V-Sekai contributors.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.