Add Brownian motion based primitive kernels

[SamuelBrand1]

Hi there,

I think one gap in AutoGP is that it doesn't offer non-stationary kernel primitives, e.g. Brownian motion or integrated brownian motion (IBM).

In particular, cubic spline regression for 1d data (i.e. time series) can be recovered by the kernel compostion: CONST + LINEAR + IBM. I think thats quite powerful to expose as something that can be learnt using involutive MCMC moves on the structure.

I do note however, that adding a new kernel primitive would require replumbing the GPConfig struct, so its a bit of a lift.

[SamuelBrand1]

Looking at this... it looks very hard to achieve with the current SMC proposal system which seems to hard-code the precise primitive/leaf kernels.

@fsaad would you be interested in a PR on this, aimed at making the SMC proposals more extensible along with adding a couple more primitive kernels?

[fsaad]

I think one gap in AutoGP is that it doesn't offer non-stationary kernel primitives

The LIN kernel, and compositions thereof, define nonstationary Gaussian processes, see here.

integrated brownian motion (IBM)

By IBM, do you mean Brownian Integral Kernel? If so, then it appears incompatible with AutoGP. Our covariance kernels take the form $$k(t,t')$$ for two real time points $t,t'$, whereas the BIK is defined over time intervals $k((s,e), (s',e'))$ where $(s,e) \in \mathbb{R}^2$ is a time a interval for the start and end time.

[SamuelBrand1]

Hi @fsaad,

No, I meant the simpler 1-fold Wiener process as per (6.30) in https://gaussianprocess.org/gpml/chapters/RW.pdf , that I think of as integrated brownian motion, but 1-fold Wiener process is probably more specific. https://juliagaussianprocesses.github.io/KernelFunctions.jl/stable/kernels/#KernelFunctions.WienerKernel Has l = -1,..., 3 implemented.

I don't think we want to limit to compositions of LIN for non-stationary kernels, although if AutoGP gets more user extensible e.g. #46 or something like it lands then I'm happy with doing the extra kernels from my application side rather than adding to AutoGP code base.

[SamuelBrand1]

I don't think we want to limit to compositions of LIN for non-stationary kernels

To expand on this, some times you might want polynomial regression (e.g. LIN x LIN for quadratic) but a lot of the time you'd prefer cubic spline LIN + IBM (+ CONST)

[fsaad]

The GP Kernel Library in AutoGP is somewhat limited, we should consider migrating our base GP library to use the Kernels.jl package that you referenced: https://juliagaussianprocesses.github.io/KernelFunctions.jl/stable/

In the meantime, are you able to use #46 locally on your installation for time being, until it is merged?

[SamuelBrand1]

The GP Kernel Library in AutoGP is somewhat limited, we should consider migrating our base GP library to use the Kernels.jl package that you referenced: https://juliagaussianprocesses.github.io/KernelFunctions.jl/stable/

I think this would be nice, and I think (dangerous word) pretty easy, you'd wrap the cov calculations and other methods for AutoGP in KernelFunction 1 dim specialisation and utils like rescale rather than doing your own and extend the LeafNode concept to include any Kernel from KernelFunction.

In the meantime, are you able to use #46 locally on your installation for time being, until it is merged?

Yes, I have a fork with this simple change. The change lets me 1) add a custom LeafNode 2) add that to the list of primitives that can be sampled. Everything else just works with standard AutoGP SMC/MCMC approach.