Constrained PDE optimization methods for motion segmentation and layer extraction

Abstract

A layered representation of images allows us to capture motion, shape, appearance and occlusion structure without going into the complexity of a full 3D representation of the scene. A unified computational framework to integrate much of the current and prior work on layered models in the framework of PDEs and calculus of variations is presented. A novelty of this modeling technique is that it relaxes the \textit{brightness constancy constraint} for moving objects making the model a better fit for tracking objects in most real life scenarios. The technique links to the simplification of an earlier approach to layering (Jackson 2008) which significantly reduces the computational complexity of the model yet still provides enough modeling richness for pertinent applications. More importantly a subtle yet fundamental modeling flaw in this earlier work is discovered which severely degraded the performance of the technique. The cause of this is traced to a bias in the original formulation that improperly extended the classical Mumford-Shah segmentation model to layering which unintentionally penalized layer occlusion thereby producing the effect. The problem is resolved by replacing the prior joint shape and appearance optimization strategy with an alternative Lagrangian style constrained shape optimization subject to PDE based appearance constraints. MOVE (Moving Object Video Encoding), a novel technique for representing video that utilizes the framework is introduced.