Home

Learning Stable Event-based Keypoints Using a Deep Siamese Architecture

Keypoint detection and matching is a fundamental problem in Computer Vision, with applications in autonomous navigation, augmented reality, etc. Event-based cameras are an ideal sensor for these applications thanks to their properties of low power consumption, high temporal resolution and high dynamic range. However, detection and matching of keypoints from an event-based camera is an open and challenging problem.

The goal of this project is to learn to detect stable keypoints for event-based cameras directly from raw data. Similarly to what is done in [1] for regular camera computer vision, we propose to train a deep siamese network to learn a feature representation which is stable across multiple views of a scene.

The project will include the following steps:

  1. Generate an event-based dataset for keypoint detection (Ground truth information can be directly extracted form gray-level images or from an OptiTrack);
  2. Train a siamese neural network to learn a representation that is as close as possible for corresponding keypoints and different for non-corresponding ones;
  3. Evaluate the results.

Prerequisites: Good programming skills (Python/C++); Mathematical and/or Computer Science background; Experience with frame-based Computer Vision and Deep Learning is a plus.

[1] Yi, K. M., Trulls, E., Lepetit, V., Fua, P. LIFT: Learned Invariant Feature Transform. European Conference on Computer Vision (ECCV), 2016.

For more information, please contact Vincent Lepetit.