Many industries are adopting AR and VR technologies inside their processes. For most use cases, traditional handheld controllers are more than enough in product-design and production processes. But for some precise use cases, adding VR finger tracking and haptic feedbacks is a real game-changer.
For instance, you are immersed inside the CAD model of your car, and you are conducting ergonomic studies. It’s one thing to visualize your 3D data in real-time. But it’s another to be able to interact with your model with your bare hands and feel where every element is. In this article, we explore how the level of realism and interaction is a must-have for some professionals.
What is VR finger tracking?
Finger-tracking is a type of tracking that aims to replace VR controllers by the user’s hand in some precise use cases. Generally speaking, finger tracking can be considered as a specialized type of hand tracking system. In this case, the fingers position can be found in real-time when interacting with CAD models in VR, without being restricted by the speed or the hand movements.
Finger-tracking-based interactions coupled with haptics and force-feedback compatible devices create a completely immersive experience, both visually and practically. While interacting with your 1:1 CAD model, you will be able to feel vibrations, textures, and stiffness of the different parts of your 3D model. This way you can block or create resistance for the fingers of the users when you are grabbing a virtual object to increase the realism of the interactions.
They are different ways to accurately track finger movements with interactive virtual reality technology. The one we use with TechViz is VR gloves, as our software is compatible with many tracking sensor devices such as ART Finger Tracking, Vibro Tactile Finger Tracking, Cyberglove or UltraLeap.
What is the difference between VR finger tracking and VR hand tracking?
Hand tracking happens when the VR headset camera or the external sensor station tracks your hands’ position, depth, pace, and orientation. The tracking data is collected by your VR system and translated into a three-dimensional representation of your hands and their movements in the VR environment. Even if hand-tracking scenarios can include sensory and force-feedback devices, they are generally less precise than individual finger tracking. The choice between the two tracking methods highly depends on your use case and your virtual reality system.
What is the difference between VR finger tracking and hand gesture recognition?
By using machine learning, deep neural networks and model-based tracking, several companies like Oculus and Google are now able to recognize hand gestures. However, this technology aims to recognize specific hand movements and poses. If you have specific use cases that require fine finger-tracking, for example to simulate digital instrument clusters in virtual reality and run a cockpit ergonomic study, hand gesture recognition will not be adapted as you won’t be able to precisely touch the screen with your fingers.
What is the difference between finger touch detection and finger tracking?
Full finger tracking is different from finger touch detection. With finger touch detection, you are still using VR controllers. The controller can detect your fingers without any pressing in the buttons or analog sticks where you place your thumb, index, or middle fingers. This enables you to make more natural gestures with your hands during the VR experience. It's a simple and cost-effective way to enable basic gestures such as pointing, waving, and making a thumbs-up.
But what finger tracking really means is that the system can track the individual movements of your fingers without using controllers.
5 Use Cases where VR finger tracking is a must-have
The popularity of immersive VR has been increasing for a wide range of industries. Here are some use cases where “traditional” VR and hand-tracking are limited compared to a full finger tracking system in VR.
1. Meaningful ergonomic and reachability studies
Haptic and force-feedback with finger tracking gloves is essential for virtual reality assessment scenario of in-car human interactions. For instance, it’s very important to know whether the driver can access all the controls of the car without taking their eyes off the road. Meaningful reachability studies in a cockpit can only be performed when VR finger tracking with haptics is included
2. Insertion of non-3D elements within 3D models
Haptic feedback and finger tracking in VR are essential for the user experience assessment of in-car human machine interfaces (HMI). Especially for interacting with digital instrument clusters, digital infotainment or navigation systems. The aim is to emulate a touch-sensitive interaction with independent 2D applications integrated in the mock-up
3. Fully immersive design reviews
Being able to participate in project reviews and presentations with finger tracking speeds up product-design processes. It enables every stakeholder to intuitively engage with the CAD model – even if they have little to no experience in VR. Finger tracking in VR product design provides more precise and secure confirmation of each interaction with the virtual objects
4. Specialized virtual reality training scenarios with hands free
Finger tracking in VR enables high-fidelity interactions with VR simulations and adds real value in your training scenarios. It’s very relevant for scenarios where workers that will require to use their fingers, such as maintenance scenarios for technicians or assembly-line training. With VR controllers, you will still have a gap between the VR training scenario and the real life. A hands-free virtual reality simulation helps to synchronize the mind with the hands and provide more accurate results by avoiding confusion and mistakes
5. Remote collaboration with improved communication
The more precisely the hand and finger movements are translated to VR, the easier communicating gets. Adding finger tracking to VR remote collaboration enables your avatar to mirror the gestures you are doing while explaining something to a colleague. Adding non-verbal communication to VR to enable lifelike conversations reduces miscommunications and increases the overall team’s performances.
Why is VR finger tracking important for engineering?
Finger tracking in VR allows you to interact with your CAD content directly with your fingers without needing VR controllers, and with haptics and force-feedback devices to add another layer of realism to your VR simulations. Thus, it has many advantages for engineering:
- Added realism to VR: The improved perceived realism makes it easier for you to make decisions on UX, manufacturability, or ergonomics of a virtual prototype.
- Deeper sense of presence and immersion: The visualization of your hands in VR indicates to your brain that what you are experiencing is real. Full immersion in VR is great for simulations and learning scenarios, and creates a more realistic and engaging testing environment.
- Better user experience: AR/VR for professionals involves very different users, from designers to engineers to technicians. Finger tracking combined with haptics enables interaction that doesn’t require a steep learning curve, as opposed to interactions that rely on controllers where every interaction is programmed differently. Plus, working with untrained VR users can be critical for testing VR prototypes on user behavior.
- More accurate tools for specialized use cases: With 27 degrees of freedom and the ability to feel small objects, your fingers are great engineering tools. Grabbing, pinching and wiggling fingers makes engineering way more intuitive.
- Easier collaboration with coworkers: People have many ways to interact with their hands. Pointing at something feels way more natural than pressing a button, which helps a lot for collaboration in VR
Which VR system supports finger tracking?
Can you use VR finger tracking work with a projection-based VR system (CAVE, Powerwall…)?
With a projection-based VR system, one or several stereoscopic projectors display shows two sets of slightly different images to create a 3D effect only visible using LCD shutter glasses. When there is one projector, it’s called a Powerwall. When there are at least two, it’s an immersive room or CAVE (Cave Automatic Virtual Environment).
To track the position of the user, you will need many external infrared cameras. But for very precise interactions like finger-tracking, you will need to add finger tracking gloves to your VR system.
Does finger tracking work with any VR headset?
Some modern head-mounted displays natively support the ability to use VR without controllers such as Meta Quest 2, HTC Vive Pro, Pimax 5K and Pimax 8K (with an added $200 hand tracking module) and Varjo XR-3 and VR-3.
For other virtual reality systems, you will need to add a professional finger tracking sensor device to your virtual reality headset.
Can you have finger tracking in augmented reality?
You will be able to find solutions for interactive AR applications with real-time 3D hand tracking and gesture recognition in the market. They mostly focus on marketing use cases, entertainment and video chat. So, not only there is no real finger tracking software for augmented reality technology available on the market, but also, they are not appropriate for industrial use cases.
That being said, if we are going down the Metaverse road, virtual and augmented reality will need to blend together. This will call for a single universal XR interface. So, we will likely have a fine finger tracking interface in AR in the future, but it might not be for the next few years.
