GSoC_2016_project_large_gps

Large-Scale Gaussian Processes

Note: This project description is yet incomplete

Polish, update, and extend Shogun's framework for Gaussian Processes, with a focus on large-scale problems and sparse variational inference.

Mentors

• Heiko (github: karlnapf, IRC: HeikoS)
• Wu (github: yorkerlin, IRC: yorkerlin)
• Emtiyaz Khan
• Vincent Adam

Difficulty & Requirements

Medium to Difficult You need know

• Gaussian Process basics (You should understand the GP Notebook)
• Variational approximation basics (You should understand at least the full GP part of the Notebook for Variational Inference)
• Linear Algebra (math & in C++)

Description

Following previous successful project on variational learning for Big Data, we attempt to bring Shogun's Gaussian Processes (GP) to Big Data land. From a high level perspective, this means that the goal is to implement established methodology on how to scale up GPs to be able to process hundreds of thousands of points.

Details

We attempt to put attention to the following sub-tasks:

• Variational inference for (full) GP
• Variational inference for sparse GP
• Stochastic variational inference for sparse GP
• Applications

The project will start from the existing code base, which already contains a huge amount of work. We aim to fill the gaps with respect to the above methods.

• Implement all standard methods that are not yet in the framework
• Benchmark existing methods against competing implementations (such as GPStuff, GPflow, GPML, etc)
• Improve efficiency if necessary. Make explicit use of multi-core computation.
• Unify gradient computations within Shogun.

Waypoints and initial work

• Exact inference for (full) GP regression using Tensorflow (entrance task) (ExactInferenceMethod)
• Variational inference for (full) GP binary classification using Tensorflow (KLCovarianceInferenceMethod, KLCholeskyInferenceMethod, and KLApproxDiagonalInferenceMethod)
• Variational inference for sparse GP (regression & classification) using Tensorflow (FITCInferenceMethod and SparseVGInferenceMethod)
• Make beautiful demos and benchmarks
• Stochastic variational inference for sparse GP using Tensorflow (optional)

Refactoring existing framework

Variational Gaussian inference (Suggested Roadmap)

• base class for computing gradient of Evidence Lower BOund (ELBO) wrt variaitonal variables
• base class for computing gradient of ELBO wrt hyper-parameters in likelihoods, mean functions, and co-variance/kernel functions
• (base) class for using external or build-in minimizers (LBFGSMinimizer and NLOPTMinimizer)
• (for full GP) classes for computing gradient wrt variaitonal variables using Tensorflow and existing hand-implemented codes
• (for full GP) classes for computing gradient wrt hyper-parameters using Tensorflow and existing hand-implemented codes (tricky)
• Benchmarks and notebooks for demos
• base class for MC samplers
• classes for using existing MC samplers
• (for sparse GP) classes for computing gradient wrt variaitonal variables using Tensorflow and existing hand-implemented codes
• (for sparse GP) classes for computing gradient wrt hyper-parameters using Tensorflow and existing hand-implemented codes (tricky)
• classes for HMC samplers from Stan (optional)
• base class for model selection (eg, Bayesian OPT) (optional)

Optional

MCMC inference (optional)

Other:

• Deep GP

Why this is cool

Our primary goal is to scale up GPs to make it possible to apply GPs to many such applications useful for big data. GPs are becoming more and more popular for big data since not only they provide accurate predictions but they also tell us how confident we should be about our prediction (aka uncertainty quantification) and that whether we have selected the right model (aka model selection). These issues are even more relevant in the era of big data since the amount of noise also increases with the amount of data. Recent work extends the use of GPs beyond regression and classification, to a wide range of appliations from numerical optimization to recommendation system and even to deep networks, making GPs a popular choice. The main bottleneck in these applications is scalability and we want to make easy-to-use scalable code which will help the use of GPs for the machine learning community even more.

Useful ressources

• GPflow using Tensorflow
• Shogun's variational Gaussian inference for full GP
• Shogun's variational Gaussian inference for sparse GP
• Notebook about Variational inference for GP
• Approximations for Binary Gaussian Process Classification
• Concave Gaussian Variational Approximations for Inference in Large-Scale Bayesian Linear Models
• Variational Learning of Inducing Variables in Sparse Gaussian Processes
• Gaussian Processes for Big data
• Scalable Variational Gaussian Process Classification
• Distributed Gaussian Processes
• Deep Gaussian Processes