Composite Objective Optimization and Learning for Massive Datasets

Show full item record

Please use this identifier to cite or link to this item:

Title: Composite Objective Optimization and Learning for Massive Datasets
Author: Singer, Yoram
Abstract: Composite objective optimization is concerned with the problem of minimizing a two-term objective function which consists of an empirical loss function and a regularization function. Application with massive datasets often employ a regularization term which is non-differentiable or structured, such as L1 or mixed-norm regularization. Such regularizers promote sparse solutions and special structure of the parameters of the problem, which is a desirable goal for datasets of extremely high-dimensions. In this talk, we discuss several recently developed methods for performing composite objective minimization in the online learning and stochastic optimization settings. We start with a description of extensions of the well-known forward-backward splitting method to stochastic objectives. We then generalize this paradigm to the family of mirrordescent algorithms. Our work builds on recent work which connects proximal minimization to online and stochastic optimization. We focus in the algorithmic part on a new approach, called AdaGrad, in which the proximal function is adapted throughout the course of the algorithm in a data-dependent manner. This temporal adaptation metaphorically allows us to find needles in haystacks as the algorithm is able to single out very predictive yet rarely observed features. We conclude with several experiments on large-scale datasets that demonstrate the merits of composite objective optimization and underscore superior performance of various instantiations of AdaGrad.
Description: Yoram Singer, Senior Research Scientist of Google Research presented a lecture on September 3, 2010 at 2:00 pm in room 1447 of the Klaus Advanced Computing Building on the Georgia Tech campus. Yoram Singer is a senior research scientist at Google. From 1999 through 2007 he was an associate professor at the Hebrew University of Jerusalem, Israel. He was member of the technical staff at AT&T Research from 1995 through 1999. He served as an associate editor of Machine Learning Journal and is now on the editorial board of the Journal of Machine Learning Research and IEEE Signal Processing Magazine. He was the co-chair of COLT'04 and NIPS'07. He is a AAAI Fellow and won for several awards for his research papers, most recently the 10 years retrospect award for the most influential paper of ICML 2000. Runtime: 56:18 minutes
Type: Lecture
Date: 2010-09-03
Contributor: Google Research
Georgia Institute of Technology. School of Computational Science and Engineering
Publisher: Georgia Institute of Technology
Subject: Machine learning

All materials in SMARTech are protected under U.S. Copyright Law and all rights are reserved, unless otherwise specifically indicated on or in the materials.

Files in this item

Files Size Format View Description
singer.mp4 148.3Mb MPEG-4 video View/ Open Download Video
singer_streaming.html 918bytes HTML View/ Open Streaming Video

This item appears in the following Collection(s)

Show full item record