Motion Sensing Technology

Motion-sensing capability has become a boon in the gaming industry, with 90 million Nintendo Wii units sold since its 2006 introduction and over 10 million units of Microsoft’s Xbox360 Kinect peripheral sold since its winter 2010 launch, then latter having surpassed the former as the world’s fastest-selling consumer electronic device. Distinguishing Kinect from Wii is its dedication to pure, controller-less motion input, you are the controller, as the marketing slogan goes.
One of the key aspects of Kinect that has generated much excitement is its promotion of the natural user interface (NUI) paradigm, which in many ways aims to succeed the traditional graphical user interface (GUI) and command line interface (CLI) paradigms for human-computer interaction. Indeed, Microsoft’s approach to Kinect clearly focuses on the expansion of NUI and a vision for Kinect that goes far beyond gaming. Craig Mundie, chief research and strategy officer at Microsoft recently stated that the “computing interface is evolving from something we drive to something that’s more like us” but while the world searches for applications that may be primed to accept NUI, Kinect and the activity surrounding it offer clues as to where motion-sensing technology may be headed in the next 5-10 years.
Motion-sensing in gaming has had many premature starts, with a history of products for older generation consoles ranging in reputation from grossly unsuccessful to largely forgotten (SegaPods, Sega Activator, PlayStation EyeToy and Eye). The first true success in introducing motion input into gaming came with Nintendo’s Wii, whose appeal opened the floodgates for family-oriented and casual gaming, but which still relied heavily on traditional button/joystick controller input. The first pure (controller-less) motion gaming experience would arrive with Microsoft’s Kinect peripheral for the Xbox 360 console. Kinect’s explosive success has led many to posit many paths for the growth of motion gaming, and this is the starting point for the technology investigation. Though Kinect, as the technology demonstrator, is a gaming device that search has uncovered the potential for motion-sensing to be much more far-reaching in application.
The entrance of non-conventional input mechanisms in gaming began in 2006 with the release of the Nintendo Wii. Breaking away from the usual controller interface the Wii used a remote plus nunchuk (joystick) combination. The primary controller used built-in accelerometers and infrared detection to gauge its position in space from the light emitted from a sensor bar placed on the television screen. As a secondary peripheral the nunchuk also featured an accelerometer as well as an analog joystick. Both components offered buttons for traditional input as well. The success of the Wii served to break the ground for alternative input methods in gaming, specifically in motion input.
Limited camera and sensor capability of the system is capable of 640×480 pixel resolution and can differentiate and capture objects with a spatial horizontal resolution of 3mm. This only affords precision to a few centimeters. Thus the system can differentiate movements of the wrist and elbow but cannot distinguish movements of the fingers and hands.
In addition, the depth resolution of the Kinect can differentiate objects in 1 cm increments. The image output is 60 fps. However the range of the sensor is the clearly limited, it only works in a range between 2.4ft to 10ft to the user. These parameters are not ideal for use in large family rooms and in games which could include up to 4 players.
Another major but less used component of Kinect is its audio sensing capability. As of now, improvements still need to be made to the algorithm of voice recognition. The challenge is two-fold:
1. Filtering out ambient noises and noises from surround sound speakers.
2. Registering the meaning behind the human speech.
To overcome the first hurdle they have developed acoustic echo cancellation algorithm, which is the first algorithm of its kind to be able to filter out not just single point source, but a full surround sound system. Now they can pick up the soft speech of a child from 4 meters away. They are working on differentiating multiple voices speaking simultaneously allowing the implementation of NUI (natural user interface) in multiplayer games. For the second hurdle, the human speech is very imprecise and heavily based on context. In order for the computer be as competent as human beings more research will have to be done in statistical processing of audio signals
For future implementation Microsoft is focusing on moving towards NUI. The best game controller is the type that disappears once you start playing. Being able to use natural speech to command the Xbox is a step in this direction.
Prior to motion sensing’s entrance into gaming the technology has primarily been used for security applications, as with detecting unauthorized entry, lighting control, and triggering cameras to capture images of a moving subject.
The introduction of Kinect technology has gradually moved motion sensing technology towards the creation of an immersive and interactive user interface applicable in multiple situations. One important application is the integration of motion sensing technology with video conferencing technology, such as the Skype software application. Microsoft recently bought Skype for $8.5 billion, and currently there is much speculation on how they plan to integrate Skype with the Kinect system. One possible application of Kinect with Skype is to provide a more immersive video conferencing experience. The MIT Media Lab has done considerable work by using the Kinect system to integrate spatial depth information, audio, and a live video stream to calibrate an immersive conferencing application. One feature of this system built by the MIT Media Lab is called Spatial Augmented Reality. This feature allows the size and position of objects in the conference room to be defined in three dimensions. Users can click on the objects (which are in reality located somewhere else but whose image is projected on a display screen) and zoom into the object and get its dimensions.
Microsoft Research has also been using motion sensing technology to create a more natural user interface. Their project LightSpace seeks to create interactive surface manipulation on any surface and even the spaces in between by combining elements of surface computing with augmented reality research.
LightSpace is innovative because it uses motion sensing technology to allow the user to interact with a display projected onto various surfaces in a small room specially designed for this purpose. It uses 3D real world coordinates to calibrate its depth cameras and projectors, which allows its graphics to correctly project on any visible surface. A standard table or office desk and the air between them can be transformed into an interactive display using selective depth camera projection. The user can transfer an object from one surface to another by touching both the object and the destination. They can also sweep the object into their hand and walk over to the destination surface.
Another field of application for motion sensing technology is consumer multimedia. Wavi Xtion developed by Asus, is a new approach to PC entertainment that combines the Wavi wireless media streaming device with the Xtion motion sensor. It can stream high definition media wirelessly from a PC located in one room to a TV at another location. The Wavi Xtion uses infrared camera for the motion sensing function. The motion sensing is used to control the gesture-based multimedia playback function, games and other applications. This would be the first motion-sensing PC interface
There are currently two active projects that involve military applications. The first project is sponsored by the Office of Naval Research and the Army Research Office. It involves the military use of Kinect in reconnaissance missions. Developed by students at MIT, a drone equipped with a Kinect system makes a map of a room in real time and is able to navigate through it. The combination of a real-time visual odometry system and the Kinect depth sensor allows for fast and accurate measures of the trajectory of the vehicle. Also, all calculations are performed onboard the unit which aligns camera frames by matching features across images and uses depth information derived from Kinect to determine the camera’s motion. The second is a Request for Information from DARPA for Biologically Driven Navigation. The goal is to search for a more immersive military training experience. More specifically they would like to explore novel approaches to sense and interpret biological signals correlated with human user intent, and to translate these signals into actual waypoints or movement commands in a virtual or physical environment. There has been speculation that Kinect technology can be used in this.
The ability of motion-sensing technology to provide touchfree interaction has many implications for medicine, where both patients and their caretakers would benefit from reduced risk of contamination. Interest centers around two applications for the operating room: the use of gesture control to manipulate and view crucial MRI and CT imagery, and the use of gesture control to aid or perform actual surgical procedures through robots.
With imagery the benefits are immediate and clear. Touchfree control allows surgeons to avoid exposure to non-sterile computer surfaces. Currently scrubbing in and out to access the computer safely can add as much as two hours to an operation leading to precarious operation delays (during which patients can suffer complications), fatigue in the operating staff, and prolonged patient recovery time. Having an assistant to aid with accessing imagery alleviates the problem, but with additional human resource and coordination costs in navigating through thousands of images. A Toronto hospital pioneering this concept has reported the dramatic effectiveness of using Kinect to access imagery.
Implications for robotic surgery are much more uncertain with the effectiveness of robotic surgery itself being questionable and many doctors lamenting its shortfalls. Still putting the debate aside for the moment, where robotic surgery is needed or desired, gesture control could improve surgeons’ performance in several ways. By using Kinect’s powerful ability to capture spatial information in a scene, some pioneers have attempted to address one of the most commonly offered criticisms about robotic surgery, the need for haptic, or tactile (touch), feedback for surgical robot operators. Making use of Kinect’s depth camera, the doctors have produced proof-of-concept for operators to feel when the robot has contacted a surface, in particular with areas they wish to avoid damaging. The system restricts control stick movement in response to the information from the depth camera, thus providing feedback to the operator.
Motion sensing, though important in gaming will ultimately make its biggest impact in non-gaming applications. Though it has been effective in expanding the gaming demographic to include females and older users, it will never completely replace the controller based gaming of today. This is due to the inability of pure motion sensing to completely supplant legacy controllers with regard to games requiring peripherals, such as driving and shooting games. With the upcoming release of the official Kinect SDK, Microsoft plans to encourage the contributions of hackers in developing additional uses for the Kinect, signifying their acceptance of the movement of Kinect to non-gaming applications. In particular research the most likely commercial use of motion sensing will be in video conferencing. This is due to Microsoft’s recent acquisition of Skype, which is significant to their plan to formally introduce Kinect as a peripheral for a better conferencing session on Skype. I expect to see formal videoconferencing products within the next 1-3 years.

References
1. http://en.wikipedia.org/wiki/History_of_video_games
2. http://milandesignweek2011.asus.com/archives/cebit-2011/asus-reveals-the-worlds-first-motion-sensing-experience-for-pc/
3. http://research.microsoft.com/en-us/projects/lightspace/
4. http://en.wikipedia.org/wiki/Kinect
5. http://en.wikipedia.org/wiki/Wii
6. http://www.istartedsomething.com/20110319/microsoft-research-reveals-guts-of-kinect-technology/
7. Andrew Blake et al., “Real-Time Human Pose Recognition in Parts from Single Depth Images”, Microsoft Research Cambridge & Xbox Incubation.
8. http://kinectedconference.media.mit.edu/.
9. Stephen Totilo, Kinect Review. http://kotaku.com/#!5680501/review-kinect
10. http://www.vgchartz.com/weekly.php
11. http://support.xbox.com/en-us/pages/kinect/more-topics/control-your-xbox-360-with-your-voice.aspx
12. http://en.wikipedia.org/wiki/Wii#System_sales