In this talk, I provide my perspective on the machine learning community’s efforts to develop inference procedures with Bayesian characteristics that go beyond Bayes' Rule as an epistemological principle. I will explain why these efforts are needed, as well as the forms which they take. Focusing on some of my own contributions to the field, I will trace out the community’s most important milestones, as well as the challenges that lie ahead. Throughout, I will provide success stories of the field, and emphasise the new opportunities that open themselves up to us once we dare to go beyond orthodox Bayesian procedures.