XR_ERROR_RUNTIME_FAILURE: "Simultaneous hands and controllers cannot be enabled while Body API is active"

[BattleAxeVR]

Hello, when I call xrResumeSimultaneousHandsAndControllersTrackingMETA in my IGL-based PC VR streaming app, in order to enable multi-modal, I get an error:

Specific Line of code:

https://github.com/BattleAxeVR/igl/blob/02011a8d3c311d4e842b70376eb8aa1e758b3951/shell/openxr/mobile/XrApp.cpp#L2234

I originally logged the issue here, but was asked to recreate it here in IGL to get some visibility inside Facebook and hopefully a solution can be found.

Repro steps:

1. Enable Meta Body Tracking API via OpenXR
2. Enable Hand Tracking EXT
3. Enable multi-modal hands/controllers, that depends on 2 but is mutually exclusive with 1.

The order of 1-2 above can be swapped but leads to body tracking failing with the same XR_ERROR_RUNTIME_FAILURE, first Hand tracking EXT, then Multi-modal, then Body tracking fails.

Why do I need this?

Well, full body tracking is useful generally for immersion, presence, and fun, but specifically for my app I need the waist pose to implement waist-oriented locomotion, which reduces if not eliminates VR simulation sickness caused by rotations induced by head-oriented during smooth movement, which couples the head and body into one. Waist-loco decouples head and body so you keep walking straight = no rotation, no sickness (I highly encourage people to try it to confirm it for themselves, it works in any SteamVR game using my OK Cloud Streamer app)

The multi-modal support is also needed for my game, to use 2-handed weapons, or 1 main hand weapon and a free offhand, to do things like open doors, turn keys, pick up / manipulate objects, pull levers, cast spells using complex gestures involving fingers, etc. The possibilities are endless. I need at least 1 VR controller with a thumbstick + waist-oriented locomotion in order to navigate in-game. I cannot ship one without the other and permit these combinations of activities.

I don't see any technical reason why FBT and HT EXT can't be made to work at the same time, after all, the hand is a part of the body, and HT EXT returns a subset of the joints that FBT does. FBT + multi-modal should therefore be possible.

I hope Facebook will fix this (if you aren't already doing so) so I can use all these features at the same time to achieve my vision. Thanks.