Machine Learning for Health Informatics class 2021

Current as of: June, 08, 2021 – 17:00 CEST (UTC+2), 08:00 PDT (UTC-7), 09:00 MDT (UTC-6), 11:00 EDT (UTC-4)

Machine Learning for Health Informatics

“It is remarkable that a science which began with the consideration of games of chance
should have become the most important object of human knowledge”
Pierre Simon de Laplace, 1812.

2021S, 2.0 h 3.0 ECTS, Type: VU Lecture with Python Excercises, Language: English [TISS]
066 936 Master program Medical Informatics [TISS]
066 646 Master program Computational Science & Engineering SB Computational Informatics [TISS]

According to the COVID-regulations the class of 2021 will be held on-line;
to enroll to this course please enroll via TISS and additonally send an e-Mail to: andreas.holzinger AT tuwien.ac.at
please put “LV 185.A83 Class of 2021 AY21W” into the header to bypass the spamfilter
and please describe in a few lines your background and your expectations to the course.

Lecture starts Tuesday, March, 23, 2021, 17:00 CET) via WebEx (enrolled via e-Mail)
Course parts will be blocked througout the semester – so please look up the Course Homepage for the correct dates.

Lecturers:
Andreas HOLZINGER, Anna SARANTI, Marcus BLOICE, Florian ENDEL, Human-Centered AI Lab (Holzinger Group)
Rudi FREUND, Theory & Logic Group

This year we are very gratedul to be supported by the following international guest professors:
Marco Tulio RIBEIRO, Microsoft Research and University of Washington, Seattle, WA, USA
Craig MERMEL and Po-Hsuan Cameron CHEN, Google Health, Mountain View, CA, USA
Igor JURISICA, Data Science Discovery Centre for Chronic Diseases and University of Toronto, Canada
Michael SNYDER, Synder Lab, Department of Genetics, School of Medicine, Stanford University, Palo Alto, CA, USA

(see abstracts below)

Course Description:

The health domain is evolving into a data-driven science. Health AI is working to effectively and efficiently use machine learning methods to solve problems in the comprehensive field of health and life sciences. This needs a synergistic approach to put the human-in-control of AI and align it with human values, privacy, security, and safety. This master’s course takes a research-centered teaching approach. Topics covered include methods for combining human intelligence and machine intelligence to support medical decision making. Since 2018, the European General Data Protection Regulation explicitly provides for a legal “right to explanation”, and the EU Parliament recently adopted a resolution on “explainable AI” as part of the European Digitization Initiative. This calls for solutions that must enable medical experts to understand, replicate and comprehend machine results. The central focus of Class 2021 is even more on making machine decisions transparent, comprehensible and interpretable for medical experts. A critical requirement for successful AI applications in the future will be that human experts must be able to at least understand the context and be able to explore the underlying explanatory factors, with the goal of answering the question WHY a particular machine decision was made. This is desirable in many domains, but mandatory in the medical domain. In addition, explainable AI should enable a healthcare professional to ask counterfactual questions, such as “what if?” questions, to also gain new insights. Ultimately, such approaches foster confidence for future solutions from artificial intelligence – which will inevitably enter everyday medical practice.

Grading:

Machine learning is a highly practical field, consequently this class is a VU: there will be a written exam at the end of the course, and during the course the students will solve related assignments. ECTS Breakdown: 75 hours in 15 hours lecture, 15 hours preparation for the lecture and practicals, 30 hours assignments, 15 hours preparation for the 1 hour written exam.

Guest Lecture, April, 13, 2021, 18:00 CEST (UTC+2) = 09:00 am PDT (UTC-7)

Prof. Dr. Marco Tulio RIBEIRO, Microsoft Research and University of Washington, Seattle, WA, USA

Title: Explaining explainable AI

Abstract: In this talk, I present my view of how one should think of explainable AI research and techniques. I discuss: (1) what explanations are for, (2) an overview of different explanation types and techniques, (3) a framework for thinking about which explanation/technique is appropriate for specific contexts.

Bio: Marco Tulio RIBEIRO is a Senior Researcher at Microsoft Research and Affiliate Professor at the University of Washington, where he received his PhD advised by Carlos GUESTRIN and Sameer SINGH. Marco’s work is on facilitating the communication between humans and machine learning models, helping humans to interact with machine learning models meaningfully, which includes interpretability, trust, debugging, feedback, robustness, testing, etc., Marco is the pioneer of the Local Interpretable Model Agnostic Explanation (LIME) approach [link]

Guest Lecture, Tuesday, April 20, 2021, 18.00 CEST = 09:00 PST

Dr. Craig MERMEL and Dr. Po-Hsuan Cameron CHEN

Title: From Diagnosis to Prognosis: How Deep Learning is Changing Healthcare

Abstract: Advances in machine learning and the availability of digitized healthcare data are revolutionizing the healthcare industry. In this talk, I will discuss our recent works on how machine learning leads to the advancement in disease detection, grading, and outcome prediction. These works have demonstrated the potential of ML in improving the accuracy and efficiency of patient care.

Bio 1: Craig MERMEL, MD, PhD is a Senior Staff Research Scientist at Google and the Research Lead for Pathology at Google Health. Dr. Mermel graduated with a BA in Mathematics and Biochemistry from Washington University in St. Louis. He obtained an MD from Harvard Medical School and a PhD in Genetics from Harvard University. His thesis dissertation, titled “The Analysis of Somatic Copy Number Alteration in Human Cancers,” was completed in the laboratory of Dr. Matthew Meyerson at the Dana-Farber Cancer Institute and the Broad Institute of Harvard and MIT. Dr. Mermel completed residency training in Clinical Pathology at the Massachusetts General Hospital and is board-certified in Clinical Pathology by the American Board of Pathology.

Bio 2: Cameron CHEN is a Staff Software Engineer and a Tech Lead Manager of Machine Learning at Google Health. Cameron’s primary research interests lie at the intersection of machine learning and healthcare. His research has been published in leading scientific, clinical and machine learning venues, including Nature, JAMA, NeurIPS, etc. Those research has also been covered by various media outlets, including the New York Times, Forbes, Engadget, etc. He received his PhD in Electrical Engineering and Neuroscience from Princeton University and his BS from National Taiwan University. Cameron was also a recipient of the Google PhD Fellowship.

Guest Lecture, April, 27, 2021, 18:00 CEST (UTC+2) = 12:00 EDT (UTC-4)

Prof. Dr. Igor JURISICA

Title: AI is Not Enough: Explainable Biology for Improved Therapies

Abstract: Integrative computational biology and AI help improving treatment of complex diseases by building explainable models. From systematic data analysis to improved biomarkers, drug mechanism of action, and patient selection, such analyses influence multiple steps of drug discovery pipeline. Data mining, machine learning, graph theory and advanced visualization help characterize interactome and drug orphans with accurate predictions, making disease modeling more comprehensive. Intertwining computational prediction and modeling with biological experiments will lead to more useful findings faster and more economically. Learning Objectives: Participants will learn about the main challenges and opportunities in precision medicine using artificial intelligence, big data analytics and integrative computational biology workflows.

Bio: Igor JURISICA, PhD, DrSc is a Senior Scientist at Osteoarthritis Research Program, Division of Orthopedic Surgery, Schroeder Arthritis Institute and Krembil Research Institute, Professor at the University of Toronto and Visiting Scientist at IBM CAS. He is also an Adjunct Professor at the Department of Pathology and Molecular Medicine at Queen’s University, and an adjunct scientist at the Institute of Neuroimmunology, Slovak Academy of Sciences in Bratislava. Since 2015, he has also served as Chief Scientist at the Creative Destruction Lab, Rotman School of Management. His research focuses on integrative informatics and the representation, analysis and visualization of high-dimensional data to identify prognostic/predictive signatures, determine clinically relevant combination therapies, and develop accurate models of drug mechanism of action and disease-altered signaling cascades. He has published extensively on data mining, visualization and integrative computational biology, including multiple papers in Science, Nature, Nature Medicine, Nature Methods, J Clinical Oncology, J Clinical Investigations. He has been included in Thomson Reuters 2014, 2015 & 2016 lists of Highly Cited Researchers (http://highlycited.com), and The World’s Most Influential Scientific Minds: 2015 & 2014 Reports. In 2019, he has been included in the Top 100 AI Leaders in Drug Discovery and Advanced Healthcare list (Deep Knowledge Analytics, http://analytics.dkv.global).

Guest Lecture, May, 18, 2021, 18:00 CET (UTC+2) – 09:00 am PDT (UTC-7)

Prof. Dr. Michael SNYDER, School of Medicine, Stanford University, Palo Alto, CA, USA

Title: Monitoring health and predicting disease using big data

Abstract: tba

Bio: Michael SNYDER is the Stanford Ascherman Professor and Chair of Genetics and the Director of the Center of Genomics and Personalized Medicine. Dr. Snyder received his Ph.D. training at the California Institute of Technology and carried out postdoctoral training at Stanford University. He is a leader in the field of functional genomics and multiomics, and one of the major participants of the ENCODE project. His laboratory study was the first to perform a large-scale functional genomics project in any organism, and has developed many technologies in genomics and proteomics. These including the development of proteome chips, high resolution tiling arrays for the entire human genome, methods for global mapping of transcription factor (TF) binding sites (ChIP-chip now replaced by ChIP-seq), paired end sequencing for mapping of structural variation in eukaryotes, de novo genome sequencing of genomes using high throughput technologies and RNA-Seq. These technologies have been used for characterizing genomes, proteomes and regulatory networks. Seminal findings from the Snyder laboratory include the discovery that much more of the human genome is transcribed and contains regulatory information than was previously appreciated (e.g. lncRNAs and TF binding sites), and a high diversity of transcription factor binding occurs both between and within species. He launched the field of personalized medicine by combining different state-of–the-art “omics” technologies to perform the first longitudinal detailed integrative personal omics profile (iPOP) of a person, and his laboratory pioneered the use of wearables technologies (smart watches and continuous glucose monitoring) for precision health. He is a cofounder of many biotechnology companies, including Personalis, SensOmics, Qbio, January, Protos, Oralome, Mirvie and Filtricine.

Additional Information:

Students can watch this TEDx talk: https://www.youtube.com/watch?v=UuiV0icAlRs

Deutschsprachige Literatur siehe [1], [2], [3], for English speakers see [4], [5], [6], [7], [8] and for Python newbies see [9].

[1] Andreas Holzinger (2018): Explainable AI (ex-AI). Informatik-Spektrum, 41, (2), 138-143, doi:10.1007/s00287-018-1102-5.

[2] Andreas Holzinger (2018): Interpretierbare KI: Neue Methoden zeigen Entscheidungswege künstlicher Intelligenz auf. c’t Magazin für Computertechnik, 22, 136-141 [pdf]

[3] Andreas Holzinger & Heimo Müller (2020). Verbinden von Natürlicher und Künstlicher Intelligenz: eine experimentelle Testumgebung für Explainable AI (xAI). Springer HMD Praxis der Wirtschaftsinformatik, 57, (1), 33–45, doi:10.1365/s40702-020-00586-y

[4] Andreas Holzinger (2018): From Machine Learning to Explainable AI. 2018 World Symposium on Digital Intelligence for Systems and Machines (DISA), 23-25 Aug. 2018 2018. 55-66, doi:10.1109/DISA.2018.8490530.

[5] Andreas Holzinger (2016): Interactive Machine Learning for Health Informatics: When do we need the human-in-the-loop? Brain Informatics, 3, (2), 119-131, doi:10.1007/s40708-016-0042-6

[6] Andreas Holzinger, Georg Langs, Helmut Denk, Kurt Zatloukal & Heimo Müller (2019). Causability and Explainability of Artificial Intelligence in Medicine. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 9, (4), 1-13, https://doi.org/10.1002/widm.1312

[7] Andreas Holzinger, Andre Carrington & Heimo Müller (2020). Measuring the Quality of Explanations: The System Causability Scale (SCS). Comparing Human and Machine Explanations. KI – Künstliche Intelligenz (German Journal of Artificial intelligence), Special Issue on Interactive Machine Learning, Edited by Kristian Kersting, TU Darmstadt, 34, (2), 193-198, doi:10.1007/s13218-020-00636-z

[8] Andreas Holzinger, Bernd Malle, Anna Saranti & Bastian Pfeifer (2021). Towards Multi-Modal Causability with Graph Neural Networks enabling Information Fusion for explainable AI. Information Fusion, 71, (7), 28-37, doi:10.1016/j.inffus.2021.01.008

[9] For practical applications we focus on Python – which is to date the worldwide most used ML-language. Tutorial: Python-Tutorial-for-Students-Machine-Learning-course pdf, 2,279 kB – reference as: Marcus D. Bloice & Andreas Holzinger 2016. A Tutorial on Machine Learning and Data Science Tools with Python. In: Lecture Notes in Artificial Intelligence LNAI 9605. Springer, pp. 437-483, https://doi.org/10.1007/978-3-319-50478-0_22

Week 11 (2021) Lecture 00 – Primer on Probability, Information and Learning from Data

Lecture Outline: This pre-lecture is for those of you who need a refresher of some fundamentals in statistics for dealing with uncertainty.

Lecture Keywords: integrative AI/ML, complexity, automatic ML, interactive ML, explainable AI, explainability, causability

Topic 00: Mathematical Notations
Topic 01: Probability, particularly Probability Distribution and Probability Density
Topic 02: Expectation and Expected Utility Theory
Topic 03: Joint Probability and Conditional Probability
Topic 04: Independent and Identically Distributed Data (IIDD)
Topic 05: Bayes (Price), and Laplace
Topic 06: Measuring Information: Kullback-Leibler Divergence and Entropy

Course slides full size (pdf – 7,332 kB) –

Learning Goals: At the end of the this primer the students …
+ should have refreshed their basic understanding of dealing with uncertainty
+ get some pointers for basic information
+ have a common understanding of the underlying theories of statistical learning
+ have refreshed their knowledge on information measures

Reading for Students: (some prereading/postreading and video recommendations):

David J.C. Mackay 2003. Information theory, inference and learning algorithms, Boston (MA), Cambridge University Press.
Online available: https://www.inference.org.uk/itprnn/book.html
Slides online available: https://www.inference.org.uk/itprnn/Slides.shtml
An excellent resource is: David Poole, Alan Mackworth & Randy Goebel 1998. Computational intelligence: a logical approach, New York, Oxford University Press, where there is a new edition available along with excellent student resources available online:
https://artint.info/2e/html/ArtInt2e.html
Please watch about the “war” between “Frequentists” and “Bayesians” in this talk “All About that Bayes: Probability, Statistics, and the Quest to Quantify Uncertainty”, provided by Kristin LENNOX on July, 28, 2016 (last accessed on April, 2, 2021) https://www.youtube.com/watch?v=eDMGDhyDxuY
Very interesting is MIT 6.S191 which shows that most modern technologies rely on the very fundamentals of statistics: Evidential Deep Learning and Uncertainty by Alexander AMINI provided on January, 26, 2021 (last accessd on April, 2, 2021): https://www.youtube.com/watch?v=toTcf7tZK8c

Week 12 (2021) Lecture 01 – Introduction: From health informatics to ethical responsible medical AI

Lecture Outline: In the first lecture you get a quick introduction to the application area health informatics, why this application area is complex and why and how probabilistic learning can help. We start first with a few defintions to get a mutual understanding and then get an overview on the differences between automatic machine learning and interactive machine learning and discuss a few future challenges of the HCAI approach to ensure ethical responsible AI/ML. This shall emphasize the integrative ML approach, where at first we learn from prior data, then extract knowledge in order to generalize and to detect certain patterns in the data and use these to make predictions and help to make decisons under uncertainty. The grand future goal for medical AI in the future is in re-traceability, interpretability and sense-making.

Lecture Keywords: integrative AI/ML, complexity, automatic ML, interactive ML, explainable AI, explainability, causability

Topic 00: A few definitions – for mutual understanding
Topic 01: Machine Learning health examples
Topic 02: Application Area Health: On the complexity of health informatics
Topic 03: Probabilistic learning on the example of Gaussian processes
Topic 04: Automatic Machine Learning (aML)
Topic 05: Interactive Machine Learning (iML) and why we need a human-in-the-loop
Topic 06: “Explainable AI” and Methods of Explainability
Topic 07: Causability – Measuring the Quality of Explanations

Lecture slides full size (pdf – 7,332 kB) – 01-185A83-Health-AI-class-of-2021-Introduction-PRINT
Lecture slides 2 x 2 (pdf – 8,427 kB) – 01-185A83-Health-AI-class-of-2021-Introduction-PRINT-2×2

To get a preview you can have a look at the slides of the last course years: 2020, 2019, 2018, 2017, 2016
however, please note that for the 2021 exam the 2021 slides are of course relevant

Learning Goals: At the end of the first lecture the students …
+ become aware of some problems of the application domain medicine and health
+ have an overview on current trends, challenges, hot topics and future aspects of AI/ML for health informatics
+ know the differences, advantages and disadvantages of automatic ML and interactive ML
+ get an understanding of the importance of re-traceability, transparency, explainability and causality
+ gain awareness for the importance of ethical, legal, and social responsibility in health AI

Reading for Students: (some prereading/postreading and video recommendations):

Holzinger, A. 2016. Interactive Machine Learning for Health Informatics: When do we need the human-in-the-loop? Springer Brain Informatics, 1-13. doi: doi: 10.1007/s40708-016-0042-6
Dossier: HOLZINGER (2016) Dossier interactive Machine Learning Health Informatics
Watch the video of Andreas Holzinger: https://youtu.be/lc2hvuh0FwQ
Watch the video of Google DeepMindHealth: https://youtu.be/teZ2m5oTKwM
“Medicine is so complex, the challenges are so great … we need everything that we can bring to make our diagnostics more precise, more accurate and our therapeutics more focused on that patient.” Sir Malcolm GRANT, NHS England, in: Machine learning : ROYAL SOCIETY Conference report, Part of the conference series Breakthrough science and technologies Transforming our future with machine learning), https://royalsociety.org/topics-policy/projects/machine-learning
Watch the videos: https://www.youtube.com/playlist?list=PLg7f-TkW11iX3JlGjgbM2s8E1jKSXUTsG

Easter break (Keep calm, collect your Ester eggs and pepare for your Python programming assignment)

Week 15 (2021) Lecture 02 – Tutorial Self-Supervised Learning and Assignment 01

Semi-supervised and self-supervised learning in computer vision and image analysis

Semi-supervised learning is a sub-field of machine learning that has been popular in the field of natural language processing (NLP) for several years and is the basis for some of the most powerful language models currently in use, such as BERT [1]. It is used in situations where both labelled and unlabelled data are available. It is especially used in cases where the number of unlabelled samples far exceeds the number of labelled samples. Very recently, the technique is also beginning to see adoption in deep learning for computer vision and image analysis. Interestingly, we are now approaching the point, where unsupervised and semi-supervised image classification algorithms are outperforming purely supervised methods on the same datasets, using only a fraction of the number of labelled samples. In the given assignment, you will apply semi and self-supervised learning by implementing the SimCLR algorithm described in [2], which is very well described in [3].

Dowload the one-sheet description here: TU-Wien-LV-185A83-Assignment-01-Class-of-2021 (pdf, 101 kB)

Discord Server: https://discord.gg/bzXtr6h4K9

Download the slides with links to the Discord server here: Semi-Supervised-Machine-Learning-Intro-for-assignment-2021 (pdf, 400 kB)

[1] Jacob Devlin, Ming-Wei Chang, Kenton Lee & Kristina Toutanova (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. https://arxiv.org/abs/1810.04805
[2] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi & Geoffrey Hinton (2020). Big self-supervised models are strong semi-supervised learners. https://arxiv.org/abs/2006.10029
[3] Desription of [2] by Yannic KILCHER, ETH Zurich (last accessed: April, 2, 2021) https://www.youtube.com/watch?v=2lkUNDZld-4

Week 15 (2021) Lecture 03 – Guest Lecture Prof. Dr. Marco Tulio RIBEIRO

April, 13, 2021, 18:00 CEST (UTC+2) = 09:00 am PDT (UTC-7)

Marco Tulio RIBEIRO, Microsoft Research and University of Washington, Seattle, WA, USA

Title: Explaining explainable AI

Reading:

[1] Ribeiro, Marco Tulio, Singh, Sameer & Guestrin, Carlos 2016. Why should i trust you?: Explaining the predictions of any classifier. 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2016). San Francisco (CA): ACM. 1135-1144, doi:10.1145/2939672.2939778. (this paper has 5,062 citations as of 30.03.2021)

[2] Marco Tulio Ribeiro, Sameer Singh & Carlos Guestrin (2016). Model-Agnostic Interpretability of Machine Learning. arXiv:1606.05386.

Lecture Slides: Marco-Tulio-Ribeiro-for-Holzinger-class (pdf, 7,203 kB)

Week 15 (2021) Lecture 04 – From data for machine learning to probabilistic information, entropy and knowledge: On data quality, data integration, data augmentation and information theory

Lecture Outline: The importance of the quality of the overall machine learning ecosystem is often underestimated. In order to carry out successful machine learning, we need not only appropriate algorithms, but above all top quality – and relevant – data, and appropriate domain knowledge! You will always get a result, the crucial question is whether and to what extent the results are relevant to support medical decision making from uncertainty. In the second lecture we get an overview of three essential topics: Data, Information and Knowledge. We will see that the big challenges in AI/machine learning lie in these areas. Data quality is extremely important. Data integration is the grand challenge in medical AI. Context understanding is the far-off goal of future artificial intelligence. We follow in our course the definition of the American Association of Medical Informatics (AMIA): Biomedical informatics (BMI) is the interdisciplinary field that studies and pursues the effective use of biomedical data, information, and knowledge for scientific problem solving, and decision making, motivated by efforts to improve human health. Medicine is ongoing decision making under uncertainty and our quest is to provide relevant information for making better decisions.

Lecture Keywords: data, information, probability, entropy, cross-entropy, Kullback-Leibler divergence, knowledge, ontology, classification, terminology

Topic 00 Reflection (quiz about the last lecture)
Topic 01 Data – The underlying physics of data
Topic 02 Data – Biomedical data sources – taxonomy of data
Topic 03 Data – Integration, Mapping and Fusion of data, digression on medical communication and data augmentation
Topic 04 Information – Theory and Entropy
Topic 05 Knowledge Representation – Ontologies – Medical Classifications

Course slides: 02-185A83-Health-AI-class-of-2021-DATA (pdf, 10,480 kB)
Course slides 2 x 2 02-185A83-Health-AI-class-of-2021-DATA-2×2 (pdf, 14,330 kB

To get a preview you can have a look at the slides of the last course years: 2020, 2019, 2018, 2017, 2016
however, please note that for the 2021 exam of course the 2021 slides are relevant

Learning Goals: At the end of this lecture the students …

+ become aware that results are always dependent on data quality and that in the domain of medicine we are always confronted with non-iid data
+ are aware of the underlying problems of health data and understand the importance of data integration and fusion in the life sciences and the relevance of multi-modal information fusion
+ have a “feel” for biomedical data sources, i.e. where the data comes from and how to handle it
+ recognize the usefulness of relative entropy, called Kullback–Leibler divergence which is very important, particularly for sparse variational methods between stochastic processes
+ have insight into the problematic of knowledge representation, an overview on the usefulness and limitations of ontologies, terminologies and medical classifications.

Reading for Students: (some prereading/postreading recommendations):

Banerjee, O., El Ghaoui, L. & D’aspremont, A. 2008. Model selection through sparse maximum likelihood estimation for multivariate gaussian or binary data. The Journal of Machine Learning Research, 9, 485-516, https://www.jmlr.org/papers/v9/banerjee08a.html
De Boer, P.-T., Kroese, D. P., Mannor, S. & Rubinstein, R. Y. 2005. A tutorial on the cross-entropy method. Annals of operations research, 134, (1), 19-67. doi:10.1007/s10479-005-5724-z
Galas, D. J., Dewey, T. G., Kunert-Graf, J. & Sakhanenko, N. A. 2017. Expansion of the Kullback-Leibler Divergence, and a new class of information metrics. arXiv:1702.00033.
Holzinger, A., Dehmer, M. & Jurisica, I. (2014). Knowledge Discovery and interactive Data Mining in Bioinformatics – State-of-the-Art, future challenges and research directions. BMC Bioinformatics, 15, (S6), I1. doi:10.1186/1471-2105-15-S6-I1
Holzinger, A., Hörtenhuber, M., Mayer, C., Bachler, M., Wassertheurer, S., Pinho, A. & Koslicki, D. (2014). On Entropy-Based Data Mining. In: Holzinger, A. & Jurisica, I. (eds.) Interactive Knowledge Discovery and Data Mining in Biomedical Informatics, Lecture Notes in Computer Science, LNCS 8401. Berlin Heidelberg: Springer, pp. 209-226. doi:10.1007/978-3-662-43968-5_12
Online available: https://pure.tugraz.at/portal/files/3108669/HOLZINGER_Entropy_based_data_mining.pdf
Loshchilov, Ilya, Schoenauer, Marc & Sebag, Michele (2013). KL-based Control of the Learning Schedule for Surrogate Black-Box Optimization. arXiv:1308.2655.
Matthews, A., Hensman, J., Turner, R. E. & Ghahramani, Z. On sparse variational methods and the Kullback-Leibler divergence between stochastic processes. Proceedings of the Nineteenth International Conference on Artificial Intelligence and Statistics (AISTATS), 2016. JMLR, 231-239 https://www.jmlr.org/proceedings/papers/v51/matthews16.html

Additional Reading: (to foster a deeper understanding of information theory related to the life sciences):

Manca, Vincenzo (2013). Infobiotics: Information in Biotic Systems. Heidelberg: Springer. (This book is a fascinating journey through the world of discrete biomathematics and a continuation of the 1944 Paper by Erwin Schrödinger: What Is Life? The Physical Aspect of the Living Cell, Dublin, Dublin Institute for Advanced Studies at Trinity College)

Week 16 (2021) Lecture 05 – From Decision Making under Uncertainty to Probabilistic Graphical Models

Lecture Outline: In order to get well prepared for the second tutorial on probabilistic programming, this module provides some basics on graphical models and goes towards methods for Monte Carlo sampling from probability distributions based on Markov Chains (MCMC). This is not only very important, it is awesome, as it is similar as our brain may work. It allows for computing hierachical models having a large number of unknown parameters and also works well for rare event sampling wich is often the case in the health informatics domain. So, we start with reasoning under uncertainty, provide some basics on graphical models and go towards graph model learning. One particular MCMC method is the so-called Metropolis-Hastings algorithm which obtains a sequence of random samples from high-dimensional probability distributions -which we are often challenged in the health domain. The algorithm is among the top 10 most important algorithms and is named after Nicholas METROPOLIS (1915-1999) and Wilfred K. HASTINGS (1930-2016); the former found it in 1953 and the latter generalized it in 1970 (remember: Generalization is a grand goal in science).

Lecture Keywords: Reasoning under uncertainty, graph extraction, network medicine, metrics and measures, point-cloud data sets, graphical model learning, MCMC, Metropolis-Hastings Algorithm

Topic 00 Reflection from last lecture
Topic 01 Decision Making under uncertainty
Topic 02 Some basics of Graphs/Networks
Topic 03 Bayesian Networks (BN) – digression Markov Processes in machine learning
Topic 04 Markov Chain Monte Carlo (MCMC) – digression graphical models and decision making
Topic 05 Metropolis Hastings Algorithm (MH)
Topic 07 Probabilistic Programming (PP) – digression on concept learning

03-185A83-Health-AI-class-of-2021-GRAPHS (pdf, 7,824 kB)
03-185A83-Health-AI-class-of-2021-GRAPHS-2×2 (pdf, 12,535 kB)

To get a preview you can have a look at the slides of the last course years: 2020, 2019, 2018, 2017, 2016
however, please note that for the 2020 exam of course the 2020 slides are relevant

Learning Goals: At the end of this lecture the students
+ are aware of reasoining and decision making
+ have an idea of graphical models
+ understand the advantages of probabilistic programming

Reading for Students:

Bishop, Christopher M (2007) Pattern Recognition and Machine Learning. Heidelberg: Springer [Chapter 8: Graphical Models]
Chenney, S. & Forsyth, D. A. 2000. Sampling plausible solutions to multi-body constraint problems. Proceedings of the 27th annual conference on Computer graphics and interactive techniques. ACM. 219-228, doi:10.1145/344779.344882.
Ghahramani, Z. 2015. Probabilistic machine learning and artificial intelligence. Nature, 521, (7553), 452-459, doi:10.1038/nature14541
Gordon, A. D., Henzinger, T. A., Nori, A. V. & Rajamani, S. K. Probabilistic programming. Proceedings of the on Future of Software Engineering, 2014. ACM, 167-181, doi:10.1145/2593882.2593900
KOLLER, Daphne & FRIEDMAN, Nir (2009) Probabilistic graphical models: principles and techniques. Cambridge (MA): MIT press.
Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H. & Teller, E. 1953. Equation of State Calculations by Fast Computing Machines. The Journal of Chemical Physics, 21, (6), 1087-1092, doi:10.1063/1.1699114. (34,123 citations as of 21.03.2017)
Wainwright, Martin J. & Jordan, Michael I. (2008) Graphical Models, Exponential Families, and Variational Inference. Foundations and Trends in Machine Learning, Vol.1, 1-2, 1-305, doi: 10.1561/2200000001 [Link to pdf]
Wood, F., Van De Meent, J.-W. & Mansinghka, V. A New Approach to Probabilistic Programming Inference. AISTATS, 2014. 1024-1032.

A hot topic in ML are graph bandits:

Villar, S. S., Bowden, J. & Wason, J. 2015. Multi-armed Bandit Models for the Optimal Design of Clinical Trials: Benefits and Challenges. Statistical Science, 199-215, doi:10.1214/14-STS504, accesible via: https://arxiv.org/abs/1507.08025

Week 16 (2021) Lecture 06 – Guest Lecture by Dr. Craig MERMEL and Dr. Po-Hsuan Cameron CHEN

Title: From Diagnosis to Prognosis: How Deep Learning is Changing Healthcare

Week 17 (2021) Lecture 07 – Guest Lecture April, 27, 2021, 17:00 CEST (UTC+2) = 11:00 EDT (UTC-4)

Prof. Dr. Igor JURISICA

Title: AI is Not Enough: Explainable Biology for Improved Therapies

Slides: 21-APR-27-TUWien-Jurisica-for-Holzinger-course (pdf, 13,747 kB)

More papers of Igor Jurisica see here:
https://www.cs.toronto.edu/~juris/home.html

Week 17 (2021) Lecture 08 – A practical introduction to Graph Neural Networks (GNNs)

Lecture Outline: Along with the more theoretical introduction in Lecture 09 here you get a concise more practical introduction to Graph Neural Networks. Graph Neural Networks (GNN) are a special form of Artificial Neural Networks that do not use pixels as input but the data type graph. Therefore it is important to know the basics of such graphs: A graph consists of several points (nodes or vertices) which are connected to each other (by edges) and thus form pairs. A well-known subform of a graph is the tree: There, the nodes are connected in such a way that there is always only one path (even across several nodes) between point A and point B. The edges can have either one direction or no direction at all. The edges can have either a direction or no direction. In a graph, the connections are as important as the data itself. Both each edge and each node can be given attributes. A graph is very good at representing real-world conditions which is a challenge in Deep Learning. In a GNN, nodes gather information from their neighbors as nodes exchange messages with each other on a regular basis. In this way, the Graph Neural Network can learn: Information is passed on and incorporated into the properties of the respective node.

Lecture Keywords: graphs, networks, graph neural networks, knowlege graphs

Course slides: Presentation_Saranti_20210427_TUWien (pdf, 4,170 kB)

Week 18 (2021), Lecture 09 – Guest Lecture May, 04, 2021, 17:00 CEST (UTC+2)

Prof. Dr. Ute SCHMID and Bettina FINZEL, MSc, University of Bamberg, Germany

Title: Learning from Mutual Explanations for Cooperative Decision Making in Medicine

Abstract: Medical decision making is one of the most relevant real world domains where intelligent support is necessary to help human experts master the ever growing complexity. At the same time, standard approaches of data driven black box machine learning are not recommendable since medicine is a highly sensitive domain where errors may have fatal consequences. In the talk, we will advocate interactive machine learning from mutual explanations to overcome typical problems of purely data driven approaches to machine learning. Mutual explanations, realised with the help of an interpretable machine learning approach, allow to incorporate expert knowledge in the learning process and support the correction of erroneous labels as well as dealing with noise. Mutual explanations therefore constitute a framework for explainable, comprehensible and correctable classification. Specifically, we present an extension of the inductive logic programming system Aleph which allows for interactive learning. We introduce our application LearnWithME which is based on this extension. LearnWithME gets input from a classifier such as a Convolutional Neural Net‘s prediction on medical images. Medical experts can ask for verbal explanations in order to evaluate the prediction. Through interaction with the verbal statements they can correct classification decisions and in addition can also correct the explanations. Thereby, expert knowledge is taken into account in form of constraints for model adaptation.

Short Bios:

Ute SCHMID is professor for Cognitive Systems at the University of Bamberg, Germany. She holds a diploma in psychology and a diploma in computer science, both from Technical University Berlin (TUB), Germany. She received both her doctoral degree (Dr. rer.nat.) and her habilitation from the Department of Computer Science of TUB, and researched previously at the Carnegie Mellon University machine learning department. Ute is researching in the area of Artificial Intelligence, machine learning and cognitive modelling for more than 15 years. Her research focus is on interpretable and human-like machine learning, inductive programming, and multimodal explanations. Current research projects are on interpretable and explainable machine learning for medical image diagnosis (BMBF – TraMeExCo), for facial expression analysis (DFG – PainFaceReader), and for detecting irrelevant digital objects (DFG – Dare2Del in the priority program Intentional Forgetting). Since 2019 Ute Schmid is a fortiss resesarch fellow for Inductive Programming, where she is engaged in the flagship project Robust AI, as well as in the IBM fortiss Center for AI. Ute is engaged to bring AI education to school and holds many outreach talks to give a realistic picture of the benefits and risks of AI applications.

Bettina FINZEL is a research assistant in the BMBF funded project Transparent Medical Expert Companion (TraMeExCo). She has a master as well as a bachelor of science both in Applied Computer Science from the University of Bamberg. She is mainly interested in comprehensible and interactive machine learning approaches for the medical domain. Bettina Finzel is active in measures to engage female high school students in computer science.

Project Page: https://www.uni-bamberg.de/en/cogsys/research/projects/bmbf-project-trameexco/

Lecture Slides: SCHIMD-FINZEL-Guest-Lecture-for-Holzinger-course (pdf, 4,324 kB)

Week 18 (2021) Lecture 10 – Tutorial T2 – Probabilistic Programming with Python (Tutor: Florian ENDEL) and Assignment 02

In this tutorial, we will explore probabilistic programming with the Python framework PyMC3. “Probabilistic programming allows for automatic Bayesian inference on user-defined probabilistic models.” [1]

We will start with a brief repetition of the previous lecture by discussing the Bayes’ theorem, Bayesian models and Bayesian parameter estimation using Markov Chain Monte Carlo (MCMC) sampling. Next on, we will dive deeper into the capabilities, workflow and specific utilization of PyMC3. Language primitives, stochastic variables and the intuitive syntax to define complex models and networks will be explored. Increasingly complex examples including, e.g., a simple statistical test, linear (LM) and generalized linear (GLM) models as well as multilevel modelling will highlight the applicability of Bayes’ methodology as well as the potential and simplicity of probabilistic programming with PyMC3. An exercise based on real world research [2] will demonstrate the advantage of multilevel modelling and probabilistic programming.

The presentation is created with a Jupyter plugin, the results shown were calculated “live”. Here are the slides as interactive html output (with precalculated results):

PyMC3 Introduction

https://dl.endel.at/PyMC3/PyMC3%20Introduction.slides.html#/

Multilevel Modelling with PyMC3

https://dl.endel.at/PyMC3/multilevel_modeling.slides.html#/

Note: You can navigate with the space bar or the arrow keys. Pressing “o” takes you to an overview.

Assignment Instruction 2021: Exercise-Therapeutic-Touch-2021 (pdf, 119 kB)

—-

1) The Tutorial of the class of 2020 can be seen here

2) The Exercise instruction Exercise Therapeutic Touch

3) The Exercise data TherapeuticTouchData

——–

[1] John Salvatier, Thomas V. Wiecki & Christopher Fonnesbeck 2016. Probabilistic programming in Python using PyMC3. PeerJ Computer Science, 2, e55, doi:10.7717/peerj-cs.55

[2] Linda Rosa, Emily Rosa, Larry Sarner & Stephen Barrett 1998. A Close Look at Therapeutic Touch. JAMA, 279, (13), 1005-1010, doi:10.1001/jama.279.13.1005

Additional resources:

Introduction to PyMC3: https://florian.endel.at/Presentation/PyMC3Intro/

Assignment Instruction: Exercise-Therapeutic-Touch-LV185A83-2018

The 2019 class will again cover Multilevel Modelling (adapted from Chris Fonnesbeck):
https://florian.endel.at/Presentation/PyMC3Intro/multilevel_modeling#/

Please refer to our Github pages: https://github.com/human-centered-ai-lab/cla-185A83-machine-learning-health-class-2019

Lecture slides 2017: full size (815 kB) 2017-04-04 Probabilistic Programming – Endel
Examples 2017: https://github.com/FlorianEndel/Probabilistic-Programming-Tutorial

MCMC: https://chi-feng.github.io/mcmc-demo/app.html

[3] Avi Pfeffer (2016). Practical probabilistic programming, Shelter Island (NY), Manning.

[4] C. Davidson-Pilon, Bayesian methods for hackers: probabilistic programming and Bayesian inference. New York: Addison-Wesley, 2016.

[5] J. K. Kruschke, Doing Bayesian data analysis: a tutorial with R, JAGS, and Stan, Edition 2. Boston: Academic Press, 2015.

[6] A very good tutorial by Christopher FONNESBECK from the Department of Biostatistics, at Vanderbilt Unviersity Medical Center in Probabilistic Programming and Bayesian Modeling with PyMC 3 https://www.youtube.com/watch?v=M-kBB2I4QlE

Week 18 (2021) Lecture 11 – From probabilistic graphical models to Graph Machine Learning

Lecture Outline: Graphs are ubiquitous in health, from molecular interaction maps, to dependencies between diseases in an individual, to populations spanning social and health interactions – very timely right now regarding Covid 19. We want to get another overview here – although we can only scratch the surface – and look primarily at graph machine learning, with multi-modality being a key aspect of this. We also want to touch on knowledge graphs very briefly.

Lecture Keywords: graphs, networks, graph neural networks, knowlege graphs

Topic 00: Reflection from last lecture
Topic 01: Machine Learning on Graphs
Topic 02: Graph Neural Networks (GNN)
Topic 03: Knowledge Graph Embeddings
Topic 04: Application Examples
Topic 05: Graph metrics and Graph mesures (emergence, robustness, modularity)
Topic 06: How do you get point cloud data from natural images?
Topic 07: Browser-based Graph extraction from pixel images
Topic 08: Application (Web based) Tumor Growth Visualization

Course slides: 11-185A83-Health-AI-GRAPH-MACHINE-LEARNING-2021 (pdf, 8,264 kB)

Please read our paper on in-silico cancer research: https://bmccancer.biomedcentral.com/articles/10.1186/s12885-018-4302-0

Learning Goals: At the end of the this lecture the students …
+ should have a further overview on graphs in medicine
+ get a little insight into graph machine learning
+ have at least touched the idea of knowledge graphs

Reading for Students: (some prereading/postreading and video recommendations):

William L. Hamilton (2021) Graph Representation Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, online avialable: https://www.cs.mcgill.ca/~wlh/grl_book/
Albert-Laszlo Barabasi (2015) Network Science, Online available: http://networksciencebook.com/
Michelle M. Li, Kexin Huang & Marinka Zitnik (2021). Representation Learning for Networks in Biology and Medicine: Advancements, Challenges, and Opportunities. https://arxiv.org/abs/2104.04883
Andreas Holzinger, Bernd Malle, Anna Saranti & Bastian Pfeifer (2021). Towards Multi-Modal Causability with Graph Neural Networks enabling Information Fusion for explainable AI. Information Fusion, 71, (7), 28-37, doi:10.1016/j.inffus.2021.01.008

Week 19 (2021) Lecture 12 – Tutorial on LRP (Tutor: Anna SARANTI) and Assignment 03

In this tutorial you get an introduction to a very popular and useful method of explainable AI – the Layer Wise Relevance Propagation (LRP).
For a machine learning model to generalize well it must be ensured that its decisions are supported by meaningful patterns in the input data. A prerequisite for the model is to be able “to explain itself”, e.g. by highlighting which input features it uses to support its given prediction. LRP is a technique that brings such explainability and scales to complex deep neural networks, concrete, it operates by propagating the prediction backward in the neural network, using a set of designed propagation rules [1], [2].

Slides: Presentation_Saranti_20210511_TUWien_B (pdf, 7,812 kB)

[1] http://www.heatmapping.org

[2] Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Wojciech Samek & Klaus-Robert Müller (2019). Layer-wise relevance propagation: an overview. Explainable AI: interpreting, explaining and visualizing deep learning. pp. 193-209, doi:https://doi.org/10.1007/978-3-030-28954-6_10.

[3] Sebastian Lapuschkin, Alexander Binder, Gregoire Montavon, Klaus-Robert Müller & Wojciech Samek (2016). The LRP toolbox for artificial neural networks. The Journal of Machine Learning Research (JMLR), 17, (1), 3938-3942.

All material can be found on our GitHub page:

https://github.com/human-centered-ai-lab/MLHI-2020

If you have any technical questions please open an issue on the repository itself, via:

https://github.com/human-centered-ai-lab/MLHI-2020/issues

Introduction to Python can be found here:
Python-Tutorial-for-Students-Machine-Learning-course (pdf, 2,279 kB)

Week 20, Lecture 13 – Guest Lecture May, 18, 2021, 17:00 CEST (UTC+2)

Prof. Dr. Mike SNYDER, Stanford, School of Medicine, Palo Alto, CA, USA

Title: Monitoring health and predicting disease using big data

https://med.stanford.edu/snyderlab.html

Michael SNYDER is part of our HEAP (Human Exposome Assessment Platform) Project

Week 22, Lecture 14 – Guest Lecture June, 01, 2021, 17:00 CEST (UTC+2)

Prof. Dr. Jussi TOHKA, AI Virtanen Institute for Molecular Sciences, University of Eastern Finland

Title: Predictive Brain Image Analysis

Abstract: Brain imaging can reveal minute changes in brain structure and function that precede clinical symptoms of brain diseases. As a result identifying biomarkers of brain disorders from neuroimaging data has become a rapidly growing research area at the intersection of
biomedical engineering, machine learning, and neuroscience. My group develops ML methods to overcome methodological challenges in this area, including
1) combining imaging data across different centers and cohorts,
2) poor interpretability of the derived biomarkers, and
3) combining data from multiple imaging techniques and behavioral, lifestyle, genetic and demographic information.
My research program aims to develop an interpretable ML framework capable of integrating multiple types of imaging and other data into prediction models while effectively controlling for multiple nuisance factors.

Bio: Jussi TOHKA is a Full Professor of Biomedical Image Analysis and head of Biomedical Image Analysis Group at A.I. Virtanen Institute for Molecular Sciences of University of Eastern Finland. His research focuses on developing new methods to analyze imaging data and developing machine learning approaches for predicting the course of brain diseases at individual level. He received his PhD degree (with commendation) in Signal Processing from the Tampere University of Technology, Finland, in 2003. He was a post-doctoral fellow in the Laboratory of Neuro Imaging, University of California, Los Angeles, USA, and thereafter held various research positions – including a highly regarded Academy of Finland research fellow position – at the Department of Signal Processing, Tampere University of Technology, Finland and was affiliated with Universidad Carlos III de Madrid as CONEX professor in 2015 – 2016 until joining UEF in 2017. He has published over 110 full-length research articles in refereed international journals or conferences and supervised 8 PhD theses to completion. Several software tools developed by him and his team are in wide use in medical imaging laboratories around the world.

https://www.jussitohka.net/

Week 24, Lecture 15 – Guest Lecture June, 15, 2021, 17:00 CEST (UTC+2)

Prof. Dr. Przemyslaw BIECEK, Warsaw University of Technology, PL

Title: Interactive Explanatory Model Analysis for Covid mortality prediction

Abstract: During the talk I will introduce the set of XAI techniques that constitute Explanatory Model Analysis (http://ema.drwhy.ai/). I will show how they can be applied to better understand how predictive models work. I will then show an example of a predictive model for covid related mortality, and based on this model I will show how XAI combined with machine learning can bring us closer to personalised medicine.

Bio: What have I learned from building predictive models in business and academia over the last 18 years? A simple truth – there is no free lunch. Machine learning and artificial intelligence are atomic energy. They can support radiologists, improve credit risk models or streamline business operations. But if not implemented responsibly then discrimination, model drift hidden artefacts can kill any initiative. In 2016 I joined the Warsaw University of Technology as an associate professor in machine learning. In 2017 I set up MI2-DataLab a group that works on new methods, tools and initiatives to deliver AI/ML solutions responsibly. Since then we develop methods and tools for ResponsibleAI.

More information: https://pbiecek.github.io/

Week 25, Lecture 16 – Guest Lecture June, 22, 2021, 17:00 CEST (UTC+2)

Prof. Dr. Natalia DIAZ-RODRIGUEZ, INSTA Paris, FR

Title: Never-Ending Learning through dreaming, EXplainable Reinforcement Learning (XRL) & experts in the loop

Abstract:
In this presentation I will talk about different dimensions leading to Never-Ending Learning (NEL). First, I will present the DREAM (Deferred Restructuring of Experience in Autonomous Machines) architecture, a result of a four years EU project, where we tackled Open Ended Learning and proposed a Developmental Approach to Open-Ended Learning in Robotics. This is a yet more challenging setting that considers more areas than the ones traditionally considered in Continual Learning (CL). Later I will talk about the paradigm of eXplainable AI (XAI) as a dimension to lead to NEL and Responsible AI, which is widely acknowledged as a crucial feature for the practical deployment of AI models. The overviews presented establish a novel definition of explainable Machine Learning that covers prior conceptual propositions with a major focus on the audience for which the explainability is sought. We propose taxonomies aimed at classifying methods for explaining Deep Learning and Reinforcement Learning models. Finally we show some multi-agent RL examples and an explainability set up in two use cases, learning both symbolic and deep representations with domain expert knowledge graphs, and learning models where the human in the loop can offer help while training.
Reference papers:

Alexandre Heuillet, Fabien Couthouis & Natalia Díaz-Rodríguez (2021). Explainability in deep reinforcement learning. Knowledge-Based Systems, 214, doi:10.1016/j.knosys.2020.106685

https://www.sciencedirect.com/science/article/pii/S1566253519308103

https://arxiv.org/abs/2104.11914

https://arxiv.org/abs/1910.10045

https://arxiv.org/abs/2006.00882

https://arxiv.org/abs/2008.06693

Bio: Natalia Díaz-Rodríguez got her double PhD from Abo Akademi (Finland) and University of Granada in 2015 on symbolic Artificial Intelligence. She is currently Assistant Professor of Artificial Intelligence at the Autonomous Systems and Robotics Lab at ENSTA, Institut Polytechnique of Paris (INRIA Flowers team on developmental robotics). Her current interests include deep unsupervised learning, open-ended learning, continual/lifelong learning, representation learning, neural-symbolic computation, explainable and responsible AI. She worked at CERN, Philips Research, University of California Santa Cruz, SETI-NASA and in Silicon Valley (Stitch Fix). She has been Google Anita Borg Scholar, Heidelberg Laureate Forum, Nokia Foundation fellow, Google Research Scholar, and is co-founder of non-profit ContinualAI.org (http://continualai.org/).

Google Scholar: https://scholar.google.com/citations?user=aia6ZgYAAAAJ&hl=en

Personal Page: https://nataliadiaz.github.io/

Week 26, FINAL LECTURE Guest Lecture June, 29, 2021, 17:00 CEST (UTC+2)

Dr. Thomas KIPF, Google Brain, Amsterdam

Graph Neural Networks and Relational Structure Discovery

Abstract: Graphs are a powerful abstraction: they allow us to efficiently describe the entities and their pairwise relationships which are relevant for a particular task or application. The past four years have seen an incredible proliferation of graph neural networks (GNNs): neural network architectures that are effective at learning with data provided in the form of a graph. But we rarely ask the question where and how the entities and relations are obtained from in the first place on which we deploy our models. In this talk, I will give an introduction to GNNs and on how to apply them to problems such as node classification or link prediction. I will further focus on the question of how we can build effective relational machine learning models even in the absence of annotated links or relations, using techniques such as attention mechanisms and neural relational inference.

Bio: Thomas KIPF is a Research Scientist at Google Research in the Brain Team in Amsterdam. He obtained his PhD on “Deep Learning with Graph-Structured Representations” at the University of Amsterdam under the supervision of Prof. Max Welling. His research focuses on relational models in machine learning with applications in network analysis, computer vision, modeling of physical systems, and object-centric learning.

Reading: Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling & Richard Zemel. Neural relational inference for interacting systems. International Conference on Machine Learning, (2018). PMLR. http://proceedings.mlr.press/v80/kipf18a.html

Week tba (2021) Lecture xx – Selected Methods of explainable AI (xAI)

Lecture Outline: Medical action is permanent decsion making under uncertainty within limited time (“5 -Minutes”). The problem of the most successful AI/ML methods (e.g. deep learning; see the differences between AI-ML-DL here) is that they are often considered to be “black-boxes” which is not quite true. However, even if we understand the underlying mathematical and theoretical principles, it is difficult to re-enact and to answer the question of why a certain machine decision has been reached. A general serious drawback is that such models have no explicit declarative knowledge representation, hence have difficulty in generating the required explanatory structures – the context – which considerably limits the achievement of their full potential. Interestingly the “old symbolic and logic based AI-approaches” did have such explanatory structures, at least for a very narrow domain space. One future goal is in implicit knowledge elicitation through efficient human-AI interfaces.

Lecture Keywords: medical decsion making, transparency, re-traceability, re-enaction, re-producibility, explainability, interpretability

Topic 01 Explainability, Interpretability, Causability, Students read [1, 2]
Topic 02 is xAI new? History of DSS = History of AI – explainable AI is actually the oldest field of Artificial Intelligence
Topic 03 Examples for ante-hoc models (explainable models, glass-boxes, interpretable machine learning)
Topic 04 Examples for post-hod models (making the “black-box” model interpretable)
Topic 04a LIME, 04b BETA, 04c LRP, 04d Taylor, 04e Prediction difference analysis, 04f TCAV

Stay healthy!

Lecture slides full size (7.023 kB): 06-185A83-HOLZINGER-health-AI-explainability-class-2020
Lecture slides 2 x 2 (9.413 kB): 06-185A83-HOLZINGER-health-AI-explainability-class-2020-2×2
Youtube Video recording mp4 (303,104 kB, 01:54,44): https://www.youtube.com/watch?v=UayEs-JFoQ4

To get a preview you can have a look at the slides of the last course years: 2019, 2018, 2017, 2016
however, please note that for the 2020 exam of course the 2020 slides are relevant

Learning Goals: At the end of this lecture the students …
+ know the roots of explainable AI, causality and causability and how to measure the quality of explanations
+ see the importance of future human-AI interfaces for medical experts
+ have an overview of post-hoc and ante-hoc methods of explainable AI
+ see how important ground truth in the medical domain is and how explainablity and causability must be mapped for a mutual understanding
+ know a selection of some of the most relevant methods of explainable AI

for more details please go to the course page (taking place each semester at Graz University of Technology):
https://human-centered.ai/explainable-ai-causability-2019

Student read:

[1] Andreas Holzinger, Georg Langs, Helmut Denk, Kurt Zatloukal & Heimo Mueller 2019. Causability and Explainability of Artificial Intelligence in Medicine. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 9, (4), doi:10.1002/widm.1312.
Online available: https://onlinelibrary.wiley.com/doi/full/10.1002/widm.1312

[2] Andreas Holzinger, Andre Carrington & Heimo Müller 2020. Measuring the Quality of Explanations: The System Causability Scale (SCS). Comparing Human and Machine Explanations. KI – Künstliche Intelligenz (German Journal of Artificial intelligence), Special Issue on Interactive Machine Learning, Edited by Kristian Kersting, TU Darmstadt, 34, (2), doi:10.1007/s13218-020-00636-z.
Online available: https://link.springer.com/article/10.1007/s13218-020-00636-z

Why do we need research in the field of explainability?
Due to raising social, ethical and leagal aspects (e.g. due to GDPR, IVDR) in the biomedical domain explainability is becoming important for several reasons:

a) Transparcency (for a discussion on the need of transparecny see the paper of Adrian Weller 2019: https://arxiv.org/abs/1708.01870

b) Causality: the question of why is to explain and understand the underlying explanatory factors; and the question of what-if (counterfactual): What if I change parameter x, what will happen? What if I remove feature y, what will happen?

c) Bias: for all data-driven machine learning the quality of data is crucial, but bias can not only be in the learned representations, bias can also brougth in from a human-in-the-loop, so a central question for us is “How to ensure that the AI system has not learned bias – or at least that we have a change to recognize and to understand the bias.

d) Fairness: What does fairness mean and what is fair to whom?

e) Safety and Trust: In the medical domain it is important that a medical expert can be confident in the reliability of the results and without any explanation and addtionally to increased trust with explanations – but are the explanations correct? This is closely related to the fundamental problem of generalization in statistical learning theory.

Tutorial on Data Augmentaion

All material can be found on our GitHub page:

https://github.com/human-centered-ai-lab/MLHI-2020

If you have any technical questions please open an issue on the repository itself, via:

https://github.com/human-centered-ai-lab/MLHI-2020/issues

Introduction to Python can be found here:
Python-Tutorial-for-Students-Machine-Learning-course (pdf, 2,279 kB)

For more information on data augmentation please read:

Marcus D. Bloice, Peter M. Roth & Andreas Holzinger 2019. Biomedical image augmentation using Augmentor. Bioinformatics, 35, (1), Oxford Academic Press, 4522-4524, doi:10.1093/bioinformatics/btz259.
Online available: https://academic.oup.com/bioinformatics/article/35/21/4522/5466454

In this paper we present the Augmentor software package for image augmentation. It provides a stochastic, pipeline-based approach to image augmentation with a number of features that are relevant to biomedical imaging, such as z-stack augmentation and randomized elastic distortions. The software has been designed to be highly extensible meaning an operation that might be specific to a highly specialized task can easily be added to the library, even at runtime. There are two versions available, one in Python and one in Julia.

N.B.: In this course we stay with Python, although Julia is a really good alternative for machine learning experiments. We appreciate that Stefan Karpinski, Viral Shah und Jeff Bezanson received the JH. Wilkinson Prize for Numerical Software in 2019.

Final Lecture

Here you find the exam quiz sheet of 2020 – please fill it with your data and send it back to Andreas Holzinger

LV-185A83-Machine-Learning-for-Health-Informatics-Exam-class-of-2020 (pdf, 165 kB)

The grading consists of three independent parts:

I) Final Exam (written test quiz, 30%) – see sample exam here
STUDENT-LV-185A83-Machine-Learning-for-Health-Informatics-Exam-class-of-2019

II) Presentations of the assigments (orally, 10 %) – will be held online

III) Grading of the assignments (coding, 20 % each, 60 % total)

Short Bio of Lecturer:

Andreas HOLZINGER <expertise> teaches and researches on data driven Artificial Intelligence (AI) and machine learning (ML) to support and improve human health. Andreas pioneered in interactive ML with the human-in-the-loop, paving the way towards multi-modal causability. Andreas promotes a synergistic approach towards Human-Centered AI (HCAI) to put the human-in-control of AI, and to align AI with human values, privacy, security and safety, ultimately fostering ethical responsible machine learning, trust and acceptance for AI.

Andreas obtained a Ph.D. in Cognitive Science from Graz University in 1998 and his Habilitation (second Ph.D.) in Computer Science from Graz University of Technology in 2003. Andreas was Visiting Professor for Machine Learning & Knowledge Extraction in Verona, RWTH Aachen, University College London and Middlesex University London. Since 2016 Andreas is Visiting Professor for Machine Learning in Health Informatics at the Faculty of Informatics at Vienna University of Technology. Currently, Andreas is Visiting Professor for explainable AI, Alberta Machine Intelligence Institute, University of Alberta, Canada.

Personal Homepage: https://www.aholzinger.at

Short Bio of Tutors:

Marcus BLOICE is finishing his PhD this year with the application of deep learning on medical images. Currently, he is working on the Augmentor project and the Digital Pathology project, and is involved in the featureCloud EU project. He has a background in computer science from the University of Sunderland (UK). He is a programmer in Python and has experience with most of the popular machine learning pipelines. Marcus has also experience in machie learning on large medical images.

Florian ENDEL started working as a database developer in the general field of healthcare research in 2007 – after gathering first experiences as high school teacher for two years and working as freelance Web designer, A specific highlight is the development and supervision of “GAP-DRG”, a database holding massive amounts of reimbursement data from the Austrian social insurance system, since 2008. Since then, he was part of several national and international research projects handling, among others, data management, data governance, statistical analytics and secure computing infrastructure. He is currently participating in the EU FP7 project CEPHOS-LINK, the FFG K-Projekt DEXHELPP and still finishing his master’s thesis.

Anna SARANTI is now pursuing her PhD in the field of explainable AI, as doctoral student financed by the FWF project P-32554 “A reference model of explainable Artificial Intelligence for the Medical Domain”. She completed her Master’s studies (Dipl.-Ing.) with a work on Applying Probabilistic Graphical Models and Deep Reinforcement Learning in a Learning-Aware Application, supervised by Andreas Holzinger and Martin Ebner at Graz University of Technology. Anna was previously in the Privatwirtschaft, working as professional machine learning engieer in Vienna.

Additional pointers and reading suggestions can be found a the
Learning Machine Learning page

Excellent Ressources for excercises
Github repository by Alberto Blanco Garcés https://github.com/alberduris

Recommended general books in Machine Learning:

MITCHELL, Tom M., 1997. Machine learning, New York: McGraw Hill. (Book Webpages)
Undoubtedly, this is the classic source from the pioneer of ML for getting a perfect first contact with the fascinating field of ML, for undergraduate and graduate students, and for developers and researchers. No previous background in artificial intelligence or statistics is required.
FLACH, Peter, 2012. Machine Learning: The Art and Science of Algorithms that Make Sense of Data. Cambridge: Cambridge University Press. (Book Webpages)
Introductory for advanced undergraduate or graduate students, at the same time aiming at interested academics and professionals with a background in neighbouring disciplines. It includes necessary mathematical details, but emphasizes on how-to.
MURPHY, Kevin, 2012. Machine learning: a probabilistic perspective. Cambridge (MA): MIT Press. (Book Webpages)
This books focuses on probability, which can be applied to any problem involving uncertainty – which is highly the case in medical informatics! This book is suitable for advanced undergraduate or graduate students and needs some mathematical background.
BISHOP, Christopher M., 2006. Pattern Recognition and Machine Learning. New York: Springer-Verlag. (Book Webpages)
This is a classic work and is aimed at advanced students and PhD students, researchers and practitioners, not asuming much mathematical knowledge.
HASTIE, Trevor, TIBSHIRANI, Robert, FRIEDMAN, Jerome, 2009. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. New York: Springer-Verlag (Book Webpages)
This is the classic groundwork from supervised to unsupervised learning, with many applications in medicine, biology, finance, and marketing. For advanced undergraduates and graduates with some mathematical interest.

To get an understanding of the complexity of the health informatics domain:

Andreas HOLZINGER, 2014. Biomedical Informatics: Discovering Knowledge in Big Data.
New York: Springer. (Book Webpage)
This is a students textbook for undergraduates, and graduate students in health informatics, biomedical engineering, telematics or software engineering with an interest in knowledge discovery. This book fosters an integrated approach, i.e. in the health sciences, a comprehensive and overarching overview of the data science ecosystem and knowledge discovery pipeline is essential.
Gregory A PETSKO & Dagmar RINGE, 2009. Protein Structure and Function (Primers in Biology). Oxford: Oxford University Press (Book Webpage)
This is a comprehensive introduction into the building blocks of life, a beautiful book without ballast. It starts with the consideration of the link between protein sequence and structure, and continues to explore the structural basis of protein functions and how this functions are controlled.
Ingvar EIDHAMMER, Inge JONASSEN, William R TAYLOR, 2004. Protein Bioinformatics: An Algorithmic Approach to Sequence and Structure Analysis. Chicheser: Wiley.
Bioinformatics is the study of biological information and biological systems – such as of the relationships between the sequence, structure and function of genes and proteins. The subject has seen tremendous development in recent years, and there are ever-increasing needs for good understanding of quantitative methods in the study of proteins. This book takes the novel approach of covering both the sequence and structure analysis of proteins and from an algorithmic perspective.

Amongst the many tools (we will concentrate on Python), some useful and popular ones include:

WEKA. Since 1993, the Waikato Environment for Knowledge Analysis is a very popular open source tool. In 2005 Weka received the SIGKDD Data Mining and Knowledge Discovery Service Award: it is easy to learn and easy to use [WEKA]
Mathematica. Since 1988 a commercial symbolic mathematical computation system, easy to use [Mathematica]
MATLAB. Short for MATrix LABoratory, it is a commercial numerical computing environment since 1984, coming with a proprietary programming language by MathWorks, very popular at Universities where it is licensed, awkward for daily practice [Matlab]
R. Coming from the statistics community it is a very powerful tool implementing the S programming language, used by data scientists and analysts. [The R-Project]
Python. Currently maybe the most popular scientific language for ML [Python Software Foundation]
An excellent source for learning numerics and science with Python is: https://www.scipy-lectures.org/
Julia. Since 2012, raising scientific language for technical computing with better performance than Python. IJulia, a collaboration between the Jupyter and Julia, provides a powerful browser-based graphical notebook interface to Julia. [julialang.org]

Please have a look at: What tools do people generally use to solve problems?

Recommendable reading on tools include:

Wes McKINNEY (2012) Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Beijing et al.: O’Reilly.
This is a practical introduction from the author of the Pandas library. [Google-Books]
Ivo BALBAERT (2015) Getting Started with Julia Programming. Birmingham: Packt Publishing.
A good start for the Julia language and more focused on scientific computing projects, it is assumed that you already know about a high-level dynamic language such as Python. [Google-Books]

International Courses on Machine Learning:

Carnegie Mellon University > Machine Learning Course 10-701 2015
by Eric XING (expertise) and Ziv-Bar JOSEPH (expertise)
https://www.cs.cmu.edu/~epxing/Class/10701/lecture.html
Carnegie Mellon University > Machine Learning Course 10-701/15-781 2011
by Tom MITCHELL (expertise)
https://www.cs.cmu.edu/~tom/10701_sp11/
Carnegie Mellon University > Machine Learning Course 10-601 2015
by Maria-Florina BALCAN (expertise) and Tom MITCHELL (expertise)
https://www.cs.cmu.edu/~ninamf/courses/601sp15/
Carnegie Mellon University > Machine Learning Course 10-701 2013
by Alex SMOLA (expertise)
https://alex.smola.org/teaching/cmu2013-10-701/index.html
Carnegie Mellon University > Machine Learnigng Course 10601b 2015
by Seyoung KIMhttps://www.cs.cmu.edu/~10601b/
Cornell University > Machine Learning CS 4780/5780 2014
by Thorsten JOACHIMS (expertise)
https://www.cs.cornell.edu/courses/cs4780/2014fa/
Cornell University > General Machine Learning, Knowledge Extraction
and Data Science courses
https://machinelearning.cis.cornell.edu/pages/courses.php
Oxford > Department of Computer Science > Machine Learning: 2014-2015
by Nando de FREITAS (expertise)
https://www.cs.ox.ac.uk/teaching/courses/2014-2015/ml/index.html

Conferences on Machine Learning with a special focus on health application

CD-MAKE – Cross Domain Conference for MAchine Learning and Knowledge Extraction
https://cd-make.net
NIPS (now called NeurIPS) – has always workshops on machine learning for health
https://neurips.cc
ICML – has also always workshops/sessions on and for health
https://icml.cc/

Pointers:

A) Students with a GENERAL interest in machine learning should definitely browse these sources:

TALKING MACHINES – Human conversation about machine learning by Katherine GORMAN and Ryan P. ADAMS <expertise>
excellent audio material – 24 episodes in 2015 and three new episodes in season two 2016 (as of 14.02.2016)
This Week in Machine Learning and Artificial Intelligence Podcast
https://twimlai.com
Data Skeptic – Data science, statistics, machine learning, artificial intelligence, and scientific skepticism
https://dataskeptic.com
VIDEOLECTURES.NET Machine learning talks (3,580 items up to 31.01.2017) ML is grouped into subtopics
and displayed as map – highly recommendable
TUTORIALS ON TOPICS IN MACHINE LEARNING by Bob Fisher from the University of Edinburgh, UK

B) Students with a SPECIFIC interest in interactive machine learning should have a look at:
https://human-centered.ai/lv-706-315-interactive-machine-learning/

LV 185.A83 Machine Learning for Health Informatics (Class of 2021)