Gaussian processes are models that are equivalent to neural networks with infinitely many hidden units, and have many desirable properties, such as tractable Bayesian inference and sensible uncertainty estimates. In recent years, there has been much progress on approximate inference for large datasets and non-conjugate likelihoods. However, the model structure of Gaussian processes has remained simple, especially compared to deep models. In this talk, we show how convolutional structure can be embedded in a Gaussian process and how to construct an tailored variational inference scheme for practical and accurate inference. We show that this structure significantly improves performance on classification tasks, as was seen in neural networks. We hope that this work will inspire work on more interesting Gaussian process models, where we obtain the benefits of both accurate inference and complex model structure.