Emukit: Decision making under uncertainty

Andrei Paleyes

SciPy 2023

Part 1: Intuition

f(x) = y

Image source: https://bookdown.org/johnnydoorn/bayesbookdown/

Bayesian optimisation:

Bayesian quadrature:

Experiment design:

Revisiting "Decision making under uncertainty"

"Decision making"?

"Under uncertainty"?

Part 2: Scope

Models

Gaussian process
Bayesian neural network
Random forest
Multi-fidelity

Model is the belief we talked about. It is a data-driven representation of whatever data we have about the objective function, and our uncertainty about it. It is often referred to as "statistical emulator". Usually a special type of models, known as Gaussian process, is used, but sometimes you can also see random forests, BNNs etc. Emukit is not a modeling framework, as there are so many great probabilistic programming tools around we didn't feel the need to build one more. So we expect users to define the model in a framework of their choice, and plug it into Emukit. With one exception being multi-fidelity modeling with GPs. We had a lot of interest in the topic, and there were no packages for it at the time, so we've built it into Emukit. But generally spreaking, we expect users to come with their own model definitions.

Methods

Bayesian optimisation
Bayesian quadrature
Experimental design

Tasks

Hyperparameter optimisation
Data efficient learning
Benchmarking

Part 3: Using the library


		model_gpy = GPy.models.GPRegression(X,Y)
		model_emukit = GPyModelWrapper(model_gpy)
		parameter_space = ParameterSpace([
			ContinuousParameter('x1', -5, 10),
			ContinuousParameter('x2', 0, 15)
		])

		model_variance_acquisition = ModelVariance(model = model_emukit)
		experimental_design_loop = ExperimentalDesignLoop(
			model = model_emukit,
			space = parameter_space,
			acquisition = model_variance_acquisition
		)
		experimental_design_loop.run(f, 10)


        while not stopping_condition.should_stop(self.loop_state):
            self._update_models()
            new_x = self.candidate_point_calculator.compute_next_points(self.loop_state, context)

            results = user_function.evaluate(new_x)
  
            self.loop_state.update(results)


	class IPriorHyperparameters:
		def generate_hyperparameters_samples(...
		
		def fix_model_hyperparameters(...


	class IntegratedHyperParameterAcquisition(Acquisition):
		def __init__(
			self,
			model: Union[IModel, IPriorHyperparameters],
		...

Image source: https://blog.knoldus.com/why-we-need-solid-principles-and-its-types/

Wait!

Numpy arrays ...

Agnostic of modeling framework ...

Performance!

Part 4: But why use Emukit then?

Future plans

Friendly to beginners
Feature completeness for UQ
Integrations
Ideas?

So this is where the package is now. What does the future hold? There are multiple possible directions. First, Emukit could be more friendly to absolute beginners. I'd love to be able to say "optimise your models with Emukit in 3 lines of code", but I simply cannot, because the package wasn't built that way. Second, there are functionalities, like sensitivity analysis and Bayesian calibration, that were always planned but never really implemented. These would make the package more complete as far as uncertainty quantification methods go. Third, integration with other tools. For example create custom optimisers with Emukit for Scipy, Ray Tune or Optuna. Finally, one of the reasons I am here (well, kind of here) today is to gather feedback from this amazing community. And maybe through some conversations and comments a new interesting direction for Emukit emerges.

Credit to the Emukit team (current team in bold green)

Andrei Paleyes

Maren Mahsereci

Neil Lawrence

Mark Pullin

Cliff McCollum

Javier Gonzalez

Thanks!

Website: https://emukit.github.io/

Code: https://github.com/EmuKit/emukit

Speaker (and this talk's slides): https://paleyes.info/

Questions?

Mic in the room! Slack!

✉ ap2169@cam.ac.uk