Writing In Air Not Pie In The Sky: Student Device Turns Phone Gestures Into Email

Instead of relying on your memory, or finding a place to put all your stuff down to find a pen and paper, wouldn't it be so convenient to simply write "level 4, row H" in the air and be able to retrieve it later?

Engineering students at Duke University have taken advantage of the accelerometers in emerging cell phones to create an application that permits users to write short notes in the air with their phone, and have that message automatically sent to an e-mail address.

Accelerometers are the devices in phones that not only keep track of the phone's movements, but make it possible for the display screens to rotate from landscape to portrait modes depending on how the phone is rotated. These devices are always "on," so there is no additional burden on the phone to use this new application.

"We developed an application that uses the built-in accelerometers in cell phones to recognize human writing," said Sandip Agrawal, electrical and computer engineering senior at Duke's Pratt School of Engineering, who with Duke graduate student Ionut Constandache developed the PhonePoint Pen. "By holding the phone like a pen, you can write short messages or draw simple diagrams in the air.

"The accelerometer converts the gestures to images, which can be sent to any e-mail address for future reference," Constandache said. "Also, say you're in a class and there is an interesting slide on the screen. We foresee being able to take a photo of the slide and write a quick note on it for future reference. The potential uses are practically limitless. That this prototype works validates the feasibility of such a pen."

Agrawal, a Pratt Engineering Undergraduate Fellow, received the inaugural Hoffman + Krippner Award for Excellence in Student Engineering for the development of the PhonePoint Pen application. The award, created by the German technology firm Hoffman + Krippner, was presented Tuesday, June 9 during the 2009 Sensors Expo and Conference in Chicago.

While this first generation application permits the writing of short messages or simple drawings, it is only a matter of time before this prototype system will be able to handle larger and more complex air-writing capabilities, according to Agrawal's mentor, Romit Roy Choudhury, assistant professor of electrical and computer engineering.

"One of the efforts of our group is to take a fresh look at how people get their information into the Internet," Roy Choudhury said. "We're trying to get past the whole idea of typing on a keyboard or using a stylus to enter information into devices. Many people get discouraged with current phones and their small keys. As phones get smaller, this frustration will only grow.

"And today, especially now in the age of Twitter and micro-blogs, the speed in which you send information becomes more important," Roy Choudhury said. "To be able to write quickly using only one hand would be very attractive to many people."

Although challenges still remain to broaden the capabilities of the PhonePoint Pen, the engineers are confident they can be solved. Currently, air-writers must pause briefly between letters, which can slow the process down and rules out the use of cursive writing. Also, each letter must be written large. These improvements would come as a result of improved algorithms and more sophisticated accelerometers, the scientists said.

"It is only a matter of time before we improve the performance of this application," Agrawal said. "We plan to further augment the pen with real-time feedback, character recognition and better support for drawing diagrams."

Roy Choudhury expects that the PhonePoint Pen prototype will be available for download within the next few months.

Another member of the team was computer engineer Shravan Gaonkar at the University of Illinois Urbana-Champaign. Roy Choudhury's research is supported by the National Science Foundation, Nokia and Verizon.