3D User Interface Design — Part 1 - Input/Output

Under the word “interface”, most people imagine a flat medium. UX/UI is understood as the standard for a 2D graphic format that is widely used for websites, mobile applications, or any traditional two-dimensional interaction style on the web.

In the VR industry, any other form than 2D UI doesn’t seem to be used very often. UI in virtual reality or virtual environments (VE) means installing flat screens into the VR space, right? The term still seems to perplex many, which is why I decided to put down a broad overview, common use, available techniques and some practical guidelines for 3D interaction design = 3D UI.

Interfacing in Three Dimensions

UIs for computer applications are becoming increasingly varied, with the traditional WIMP (Windows, Icons, Menus and Pointers/Pull-downs) components of mice, keyboards, windows, menus and icons remaining widespread. New technologies such as trackers, 3D pointing devices and gestural input devices are also growing in popularity. 3D stereoscopic projection displays, head-mounted displays (HMDs), spatial audio systems and haptic devices provide multisensory output.

The introduction of this new technology has revealed a number of challenges – people often struggle to comprehend 3D spaces, as well as with performing tasks in free space. It's not surprising that difficulties arise when transferring the physical world to digital form - the real-world offers additional clues, constraints and affordances which cannot be totally replicated through computer simulation. This means it is essential to carefully consider how UIs and interaction techniques for 3D applications are designed. Adapting traditional WIMP methods does not provide a comprehensive answer – real-world elements or alternative metaphors must be used to create novel 3D user interfaces.

Input & Output Devices

Input and output (I/O) devices play a key role in constructing 3D user interfaces for VE applications. For interaction designers, to be able to match interaction techniques with hardware in an organic and effortless fashion, they must understand the ergonomics, benefits and shortcomings of the equipment used. Though there is a plethora of I/O devices employed, these items can usually be categorized into sets that share similar attributes and interface design criteria.

Output Devices

The term “display” is often used to describe output. Visual displays are the most widely known – however, auditory, haptic, tactile and olfactory displays also play important roles. With regards to 3D user interfaces for VEs, visual displays can be separated into fully-immersive and semi-immersive devices. Fully-immersive displays such as HMDs, arm mounted displays and virtual retinal displays keep users from seeing the physical environment in real time (apart from augmented reality). This means that objects must be displayed graphically in order to be seen – this could make it harder to interact with more complex input devices. 

Semi-immersive systems like stereo monitors, workbenches and surround screen virtual reality systems allow users to observe both the virtual world and their own physical surroundings. A number of interface problems may arise with semi-immersive systems. For example, since people can see their own hands in front of the display they may cover up virtual objects that should appear closer than their hands. This is an issue often encountered during manipulation tasks which can be solved if the virtual element is positioned away from the hand so that it stays visible.

There is another issue with using stereo shutter glasses -- they need emitters, and any obstruction of the line of sight will interfere with the stereo effect. Interface designers thus need to consider ways to make sure that users do not have to move their hands or physical objects in between the view of the glasses and emitter. In addition to visual displays, 3D user interfaces for VEs are increasingly making use of auditory, tactile and haptic displays. For auditory output, there is a focus on sound localization, 3D sound generation, as well as sonification – converting certain types of data into sound. This can be particularly helpful in collaborative settings where users are able to identify where others are located within the virtual space. Additionally, it can act as a substitute for missing tactile or haptic responses – an example would be a sound being used instead of the feeling of pressing a button.

Lastly, haptic and tactile displays are key for VE user interfaces, allowing users to touch and experience things in the virtual world adds great value when it comes to object manipulation and collision detection. This has become an area which researchers are dedicating much effort towards.

Input Devices

A crucial distinction must be drawn between input devices and interaction techniques. Physical tools are needed to realize these, and this is where input devices come in. In most cases, a single input device can support multiple interaction techniques - but how it will perform for each one is determined by factors such as naturalness, efficiency and appropriateness. The number of degrees of freedom that an input device has is another significant consideration — if a device's DOFs (degree of freedom) are limited, additional buttons or modifier keys can often compensate.

Input devices can be divided into three categories according to the kind of events they generate:

Discrete input devices — emit signals as an outcome of user-made single events, usually with a binary result (e.g. up/down). An example for this type of device is the Pinch Glove developed by

Fakespace — which activates when two or more fingers are pinched together. Continuous input devices, on the other hand, transmit a series of events based on position and orientation values. Data Gloves are among these, signaling the bend angle of the finger that is wearing it.

Hybrid devices — bringing together discrete and continuous events in one versatile device. Examples for this type are Ring Mouse - combining ultrasonic tracking with two buttons - and pen-based tablets increasingly widespread in virtual environment applications due to their 2D capabilities.

Speech input is one of a kind as it utilizes the human voice as its device. It adds further possibilities to other modes of input, thus being a great choice for multi-modal activities. 3D user interfaces stand to benefit significantly from speech input especially when users' hands are already occupied. One common misconception lies in expecting that all problems can be solved with the help of good speech recognition technology - this is far from true, however, as there are other issues that need to be taken into account, such as understanding when the computer should pay attention to the user's instructions. This can be addressed through an implicit or invisible push-to-talk system, which lets people notify the application whenever they're about to deliver a command. To make sure it doesn't interfere with the natural flow of conversation, this 'push' should be integrated into existing interaction techniques so users don't have to actively remember it.

The next part of the series will be dedicated to mentioned Interaction Techniques, primarily navigation, selection and manipulation, and system control.

Title credits: henque