Freehand Circling for Object Selection Using a Head-mounted Camera

Posted on Aug 5, 2014

Baidu Eye is prototype head-mounted device with front facing camera. As a research intern in Baidu’s Institute of Deep Learning, I was responsible for developing a light algorithm that allows wearer to circle around things in his view using his pointing finger in order to select a specific object from a large scene. The selected item was then sent to Baidu’s image search API for object recognition.

This can be used for searching image of products, banners, logos, and even clipping a portion of an advertisement.

Main challanges

  • Avoiding registration of close points to prevent hand shaking errors
  • Ending selection by approaching a loop
  • Getting a clean shot of object that is not covered by hand
  • Counteracting background movements to fix the selection using:
    • Lucas-Kanade optical flow algorithm
    • Classifying camera versus other movements (hand, background noise)
Other contributors: Jiawei Gu , Jiabin He