LV 340.300 Principles of Interaction:
Part: Selected Topics of interactive Machine Learning – Interaction with Agents
Summer Term 2017

Active-Machine-Learning-OracleWelcome Students!

In this part of the LV 340.300 “Principles of interaction” you will learn the difference between automatic Machine Learning (aML) and interactive Machine Learning (iML).

The goal of Machine Learning (ML) is to design and develop algorithms which can learn from data and improve over time automatically without any human interaction. Such automated Machine Learning (aML) approaches (e.g. “Google car“) work well in application domains where big data sets are available – automated approaches need many training samples. However, in some application domains, e.g. the health domain, we are confronted with a small number of data sets or rare events, where aML approaches suffor of insufficient training samples. In such situations interactive Machine Learning (iML) approaches may be of help, because humans can make sense from a very few training samples, thus a “human-in-the-loop” can be beneficial in solving computationally hard problems, e.g. subspace clustering, protein folding, or k-anonymization, where human expertise and intuition can help to reduce an exponential search space through heuristic selection of samples. Therefore, what would otherwise remain an NP-hard problem, reduces greatly in complexity through the input and the assistance of a human agent involved in the machine learning phase.

Definition:

We define iML-approaches as algorithms that can interact with both computational and human agents *) and can optimize its learning behaviour trough this interaction.

*) Such agents are called in Active Learning “oracles” (see e.g.: Settles, B. 2011. From theories to queries: Active learning in practice. In: Guyon, I., Cawley, G., Dror, G., Lemaire, V. & Statnikov, A. (eds.) Active Learning and Experimental Design Workshop 2010. Sardinia: JMLR Proceedings. 1-18. (the sketch at the right represents the oracle of delphi, drawn by Katharina Holzinger, 2016)

Goal:

This graduate course follows a research-based teaching (RBT) approach and provides a broad overview of models and discusses methods for combining human intelligence with machine intelligence to solve computational problems. The application focus is on the health domain.

Syllabus:

This course contains three parts: In part 1 the integrative ML approach will be explained, an example for complexity given, which leads to uncertainty and probabilistic information and provides a short explaination of statistical learning. After showing some examples and disadvantages of automatic approaches, some future challenges are presented, reaching from mutli-task and transfer learning to federated machine learning. In part 2 Multi-Agent (Hybrid) Systems are discussion and some recent research in the gamification of iML approaches shown. Finally, part 3 is for deepening the understanding of human abilities in presenting the resemblance of computational reinforcement learning and human reinforcement learning.

Pre-Reading and Starting material:

1) Read this first: Holzinger, A. 2016. Interactive Machine Learning for Health Informatics: When do we need the human-in-the-loop? Springer Brain Informatics (BRIN), 3, (2), 119-131, doi:10.1007/s40708-016-0042-6.

For german speakers there is a light version available here: Holzinger, A. 2016. Interactive Machine Learning (iML). Informatik Spektrum, 39, (1), 64-68, doi:10.1007/s00287-015-0941-6.

2) Watch this 20-minutes video: https://goo.gl/0FFrNR

For impatient students there is a light 3-minutes version availabe: https://goo.gl/0hcPOY

3 ) Mathematical Notations can be found here as pdf (221 KB)

4) Python Tutorial: M. D. Bloice and A. Holzinger, “A Tutorial on Machine Learning and Data Science Tools with Python“, in Machine Learning for Health Informatics, Lecture Notes in Artificial Intelligence LNAI 9605, Springer, 2016, pp. 437-483.

5) Enjoy these Games: https://human-centered.ai/gamification-interactive-machine-learning
and please provide feedback to me

Selected Topics of interactive Machine Learning: Interaction with Agents
Part 1: Top-Level Overview

In the first part we get a top-level overview on the differences between automatic machine learnig and interactive machine learning and we discuss some future challenges as a teaser.

Topic 01: The HCI-KDD approach: Towards an integrative MAKE-pipeline
Topic 02: Understanding Intelligence
Topic 03: Example for complexity: the application area health
Topic 04: Probabilistic Information & Gaussian Processes
Topic 05: Automatic Machine Learning (aML)
Topic 06: Interactive Machine Learning (iML)
Topic 07: Active Representation Learning
Topic 08: Multi-Task Learning
Topic 09: Generalization and Transfer Learning
Topic 10: Federated Machine Learning

Lecture slides 2×2 (7,490 kB): 01-340300-HOLZINGER-interactive-Machine-Learning-slides-2×2

Additional study material:

Selected Topics of interactive Machine Learning: Interaction with Agents
Part 2: Multi-Agents and the human-in-the-loop

In second part we concentrate on the interaction of agents and on testing iML approaches with Gamification.

Topic 00: Reflection: human information processing
Topic 01: Intelligent Agents
Topic 02: Multi-Agent (Hybrid) Systems
Topic 03: Applications in Health
Topic 04: Decision Making as a Search Problem
Topic 05: iML Gamification
Topic 07: Collective Intelligence

Lecture slides 2×2 (5,330 kB): 02-340300-HOLZINGER-Multi-Agent-Interaction-slides-2×2

Addtional study material:

Selected Topics of interactive Machine Learning: Interaction with Agents
Part 3: Reinforcement Learning and Multi-Armed Bandits

In the third part we deepen our knowledge and compare human reinforcement learing with computational reinforcement learning.

Topic 00: Reflection: Limits of the Human-in-the-Loop
Topic 01: What is Reinforcement Learning, why is it interesting?
Topic 02: Decision making under uncertainty
Topic 03: Roots of Reinforcement Learning
Topic 04: Cognitive Science of Reinforcement Learning
Topic 05: The Anatomy of an RL-Agent
Topic 06: Example: Multi-Armed Bandits
Topic 07: Multi-Task Learning
Topic 08: RL-Application in Health

Lecture slides 2×2 (5,781 kB): 03-3403000-HOLZINGER-Reinforcement-Learning-2017-slides-2×2

Additional study material:

Some Quick Explanations:

Active Learning (AL) := to select training samples to enable a minimization of loss in future cases; a learner must take actions to gain information, and has to decide which actions will provide the information that will optimally minimize future loss. The basic idea goes back to Fedorov, V. (1972). Theory of optimal experiments. New York: Academic Press. According to Sanjoy Dasgupta the frontier of active learning is mostly unexplored, and except for a few specic cases, we do not have a clear sense of how much active learning can reduce label complexity: whether by just a constant factor, or polynomially, or exponentially. The fundamental statistical and algorithmic challenges involved along with huge practical application possibility make AL a very important area for future research.

Agent Based Model (ABM):= class of models for simulating interactions of autonomous intelligent agents (individual and in multi-agents these are collective entities) routed in evolutionary computing going back to John Holland (1929-2015),  Holland, J. H. & Miller, J. H. 1991. Artificial adaptive agents in economic theory. The American economic review, 81, (2), 365-370.

Interactive Machine Learning (iML) := machine learning algorithms which can interact with – partly human – agents and can optimize its learning behaviour trough this interaction. Holzinger, A. 2016. Interactive Machine Learning for Health Informatics: When do we need the human-in-the-loop? Brain Informatics (BRIN), 3, (2), 119-131.

Preference learning (PL) := concerns problems in learning to rank, i.e. learning a predictive preference model from observed preference information, e.g. with label ranking, instance ranking, or object ranking.  Fürnkranz, J., Hüllermeier, E., Cheng, W. & Park, S.-H. 2012. Preference-based reinforcement learning: a formal framework and a policy iteration algorithm. Machine Learning, 89, (1-2), 123-156.

Reinforcement Learning (RL) := examination on how an agent may learn from a series of reinforcements (sucess/rewards or failure/punishments). A must read is Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996). Reinforcement learning: A survey. Journal of artificial intelligence research, 237-285.

Multi-Agent Systems (MAS) := include collections of several independent agents, could also be a mixture of computer agents and human agents. An exellent pointer of the later one is: Jennings, N. R., Moreau, L., Nicholson, D., Ramchurn, S. D., Roberts, S., Rodden, T. & Rogers, A. 2014. On human-agent collectives. Communications of the ACM, 80-88.

Transfer Learning (TL) := The ability of an algorithm to recognize and apply knowledge and skills learned in previous tasks to
novel tasks or new domains, which share some commonality. Central question: Given a target task, how do we identify the
commonality between the task and previous tasks, and transfer the knowledge from the previous tasks to the target one?
Pan, S. J. & Yang, Q. 2010. A Survey on Transfer Learning. IEEE Transactions on Knowledge and Data Engineering, 22, (10), 1345-1359, doi:10.1109/tkde.2009.191.