KAIST

John, an insurance planner, took several photos of a competitors’ new brochures. At a meeting, he opened a photo gallery to discuss the documents with his colleagues. He found, however, that the photos of the document had the wrong orientation; they had been rotated in 90 degrees clockwise. He then rotated his phone 90 degrees counterclockwise, but the document photos also rotated at the same time. After trying this several times, he realized that it was impossible to display the document photos correctly on his phone. Instead, he had to set his phone down on a table and move his chair to show the photos in the correct orientation. It was very frustrating for John and his colleagues, because the document photos had different patterns of orientation errors.

Professor Uichin Lee and his team at KAIST have identified the key reasons for such orientation errors and proposed novel techniques to solve this problem efficiently. Interestingly, it was due to a software glitch in screen rotation–tracking algorithms, and all smartphones on the market suffer from this error.

When taking a photo of a document, your smartphone generally becomes parallel to the flat surface, as shown in the figure above (right). Professor Lee said, “Your phone fails to track the orientation if you make any rotation changes at that moment.” This is because software engineers designed the rotation tracking software in conventional smartphones with the following assumption: people hold their phones vertically either in portrait or landscape orientations. Orientation tracking can be done by simply measuring the gravity direction using an acceleration sensor in the phone (for example, whether gravity falls into the portrait or landscape direction).

Professor Lee’s team conducted a controlled experiment to discover how often orientation errors happen in document-capturing tasks. Surprisingly, their results showed that landscape document photos had error rates of 93%. Smartphones’ camera apps display the current orientation using a camera-shaped icon, but users are unaware of this feature, nor do they notice its state when they take document photos. This is why we often encounter rotation errors in our daily lives, with no idea of why the errors are occurring.

 The team developed a technique that can correct a phone’s orientation by tracking the rotation sensor in a phone. When people take document photos their smartphones become parallel to the documents on a flat surface. This intention of photographing documents can be easily recognizable because gravity falls onto the phone’s surface. The current orientation can be tracked by monitoring the occurrence of significant rotation.

In addition, the research team discovered that when taking a document photo, the user tends to tilt the phone, just slightly, towards the user (called a “micro-tilt phenomenon”). While the tilting degree is very small—almost indistinguishable to the naked eye—these distinct behavioral cues are enough to train machine-learning models that can easily learn the patterns of gravity distributions across the phone.

The team’s experimental results showed that their algorithms can accurately track phone orientation in document-capturing tasks at 93% accuracy. Their approaches can be readily integrated into both Google Android and Apple iPhones. The key benefits of their proposals are that the correction software works only when the intent of photographing documents is detected, and that it can seamlessly work with existing orientation tracking methods without conflict. The research team even suggested a novel user interface for photographing documents. Just like with photocopiers, the capture interface overlays a document shape onto a viewfinder so that the user can easily double-check possible orientation errors.

Professor Lee said, “Photographing documents is part of our daily activities, but orientation errors are so prevalent that many users have difficulties in viewing their documents on their phones without even knowing why such errors happen.” He added, “We can easily detect users’ intentions to photograph a document and automatically correct orientation changes. Our techniques not only eliminate any inconvenience with orientation errors, but also enable a range of novel applications specifically designed for document capturing.” This work, supported by the Korean Government (MSIP), was published online in the International Journal of Human-Computer Studies in March 2017. In addition, their US patent application was granted in March 2017.

 (Photo caption: The team of Professor Lee and his Ph.D.student Jeungmin Oh  developed a technique that can correct a phone’s orientation by tracking the rotation sensor in a phone.)