The SixthSense Device: Projecting the digital world into reality

Designed and built by Pranav Mistry

A child develops a deep understanding with its physical world by interacting with everyday objects. We learn to associate objects with gestures, the movements we use to interact with an object, and how we can use an object to interact with other people. The use of gestures generally breaks down at the interface with computers. Data is typically accessed through a mouse, keyboard, or touch screen, and misses the intuitiveness of interacting with a physical object. SixthSense is a mobile device designed to bring alive intuitive physical objects with digital information.

Pranav Mistry wishes to bring the digital world back into reality in a cost effective way that everyone can afford. His experiments began with a series of simple devices that allow the use of real world gestures to interact with a computer. For example, by deconstructing an ordinary mouse, removing the rollers and attaching a basic system of pulleys, he was able to create a device to sense hand movements that could control a virtual pointer.

A common theme in his experiments was to bring a part of the physical world into the digital world, but on realising that what is really desired is information, and that we don't care about the computer, why not reverse the process and extract data from the digital world and paste it into the physical world. This idea lead to the SixthSense device.

The device itself is relatively simple. A potable projector hung

around the neck does the job pasting data onto any real world object, such as a wall. To allow interaction with the projection a small camera tracks the movement of a users fingers (as long as the users wears coloured rings). Your fingers can then be used to perform any number of gestures such as pinching to grab a graphic, pulling to zoom, or making the shape of picture frame to take a snapshot. When combined with a mobile internet connection, the SixthSense device allows you to carry your entire digital world with you.

The device can also recognise objects, such as a book cover. Imagine you are browsing in a bookstore. On looking at book the device can recognise and instantly project a review of the book directly onto the cover, or maybe the best online price, or a rating from your favourite book review web site. Information can also be inputted into the system via the camera. If reading a magazine article why not grab it with a pinch of the fingers and drag it onto your projected display, where you can recognise it as text and begin editing it, emailing it friends, or blogging it.

The SixthSense device is a fantastic piece of kit thats within the budget of the masses, and all the software is freely available. Its a great way to pull yourself back out of the digital world and reconnect with the physical.