This course treats two advanced topics in machine learning: causal inference (the study of cause-effect relations), and reinforcement learning (learning to interact with an environment).

Learning goals: After completing the course, the student

• understands the description of causal relations in terms of directed graphs and structural equation models;
• can explain the implications of causal assumptions;
• is able to test these implications using observational and interventional data;
• can implement machine learning methods in such a way that causal information is taken into account;
• knows the main concepts and algorithms of reinforcement learning;
• can describe the role played by function approximation methods (such as neural networks) in reinforcement learning;
• is able to explain the additional challenges that arise with off-policy methods.

The following knowledge will be assumed in this course:

• solid proficiency in mathematics, in particular probability theory (e.g. ability to understand and manipulate formulas involving conditional probabilities and expectations), linear algebra, basic calculus
• programming skill in Python
• understanding of basic machine learning theory and methods, for example from the bachelor course Machine Learning (KI3V15001)

• Judea Pearl, Madelyn Glymour, Nicholas P. Jewell. Causal Inference in Statistics: A Primer. Wiley, 2016.
• Richard S. Sutton, Andrew G. Barto. Reinforcement Learning: An Introduction (second edition). MIT Press, 2018. (pdf available from authors' website: http://incompleteideas.net/book/the-book-2nd.html)
• additional material that will be made available online

Modern machine learning methods have achieved spectacular results on various tasks. Yet there are pitfalls and limitations that can't be overcome simply by increasing the amounts of data and computing power. For example, standard methods assume that the data are drawn from a single, unchanging probability distribution. The two main topics that we cover in this course both deal with situations where that is not the case.

The first topic, causal inference, is the subfield of machine learning that studies causes and effects: if we make a change to one random variable in a system, for which other variables does the distribution change? An understanding of these cause-and-effect relations allows us to predict the results of a change in the environment. We will also look at the problem of learning these relations from data.

Second, reinforcement learning is about the design of agents that can learn to interact with an unknown environment. Recent advances in supervised learning (such as deep learning) can be built on by reinforcement learning methods. This brings with it a unique set of challenges that we will cover in this course.