The main focus of the DirActNI framework is to realize and evaluate contactless 3D interaction techniques.
To do this as flexible and adaptable as possible it uses a lightweight installation and is based on only free and standardized software components. The following two sections
will illustrate the main features by introducing the installation and the realized interaction techniques.
As mentioned before, the installation was chosen to be as lightweight as possible.
It requires no instrumentalization on the user and no complicated calibration or time-consuming setup. On the one hand it does not use
any sensors on the user's body (1) because it captures his motions contactless with a depth camera, like the Microsoft Kinect or the ASUS Xtion (2). On the other hand it uses
conventional hardware like a low performance computer (3) to process the motions and visualize the virtual environment and also a
conventional projector (4) and screen (5) to display the environment. Thus the installation provides a very low instrumentalization
of the user, the possibility of an aseptic interaction and is although very lightweight and inexpensive.
On the software side the DirActNI framework uses the free OpenNI
framework for the motion processing to achieve a flexibility in terms of
different depth sensors or operating systems
. On the other hand the DirActNI framework uses with Unity 3D
powerful authoring tool for the creation of platform independent interactive 3D environments
Furthermore the framework uses an intelligent algorithm to detect the opening state of the hands to allow an even more natural interaction.
The DirActNI framework provides the possibility to handle several individual interaction techniques. Therefore the 3D interaction was
divided in typical interaction tasks like locomotion and orientation. This provides the possibility to implement
several techniques for the same task separately and interchange them during
the execution on the fly to test their pro's and con's. Up to now with the Pointer-based and Whole-body technique two main interaction
techniques was realized. Both will be introduced in the following.
Pointer-based interaction technique
The pointer-based technique is based on the same indirect interaction metaphor as a gamepad or a
combination of a keyboard with a mouse use for the interaction in a 3D environment. The difference is that not a joystick on the gamepad, the keys on the keyboard or a mouse control the locomotion and orientations but the user's hands do. The intention behind this analogy is, to make this technique
intuitive for most experienced users. So the user controls the orientation with one hand and the locomotion with the other hand
by moving them in the desired direction. The speed of the locomotion is controlled relatively to
the hand’s position whereas the orientation is controlled in a certain interval depending on the absolute position of the user’s hand. Outside this interval the rotation speed
is controlled relatively. This combination allows the user a direct control of the orientation in front of him and allows him to rotate completely in the
virtual environment without losing a non-surrounding screen out of his field of view. The manipulation also behaves like the manipulation with a gamepad.
For the triggering event the opening state of the orientation hand is used, which is following a
natural grab gesture. In addition to this triggering event a time-based trigger was also implemented to evaluate the
difference for the user experience.
The whole-body interaction technique offers the user the possibility of a realistic natural
and (more) intuitive interaction
, which is
based on natural movements
to be more immersive
and necessary a lower cognitive load
It provides the possibility of a physically movement
in the virtual environment by using the head tracking technique
Unfortunately these techniques only allow a locally limited locomotion due to the tracking and environmental circumstances.
Therefore, an additional lean-based navigation
was designed by which the user can move freely through the virtual environment while leaning the torso in the desired direction. The
speed of the locomotion is controlled by the leaning angle with predefined thresholds. This kind of locomotion is similar to the usage of a Segway
The orientation is also based on already known natural gestures. Thus the horizontal orientation
is controlled by
natural rotation of the shoulders
and the vertical rotation by looking up and down
gesture through moving the head. Both are
controlled absolute to the particular rotation angles. In addition, the horizontal orientation also behaves like the pointer-based technique beyond a certain interval relatively to the rotation angle.
Thereby, the user can physically move in a limited area and furthermore use his body for an unlimited locomotion in the whole virtual environment.
As the user does not need his hands for the Navigation, he can use them for a natural direct manipulation similar to the
real and everyday grabbing and holding gestures. Therefore, the movement of his hands is translated isomorphically to his virtual
hands in the virtual 3D environment. By touching the desired object with an open hand and closing the hand at the same position he can select the virtually touched object.
While keeping the hand closed, the virtual object is being hold by the virtual hand and follows through the virtual environment until he opens the hand again.
Similar to the one hand interaction a two-handed interaction was especially implemented for carrying bigger virtual objects.