EXPLAINABILITY OF A REINFORCEMENT LEARNING CONTROL POLICY
Faculty members : Jean-Michel Loubes (IMT) & Edouard Pauwels (IRIT)
Student : M2 internship + PhD
Little is known about how the information is processed in deep learning networks in order to obtain a prediction, which explains why such models are widely considered as black-box models.
The goal of this challenge is to propose new technics to have a better understanding of how a
network takes its decisions.
Machine learning algorithms build predictive models which are nowadays used for a large variety of tasks. They have become extremely popular in various applications such as finance, insurance risk, health-care, recommendation systems as well as industrial applications of all kinds including predictive maintenance, defect detection or industrial liability. Such algorithms are designed to assist human experts by giving access to valuable predictions and even tend to replace human decisions in many fields, achieving an extremely good performance. Over the last decades, the complexity of such algorithms has grown, going from simple and interpretable prediction models based on regression rules to very complex models such as random forest, gradient boosting and models using deep neural networks. Such models are designed to maximize the accuracy of their predictions at the expense of the interpretability of the decision rule. Little is also known about how the information is processed in order to obtain a prediction, which explains why such models are widely considered as black-box models.
Examples of industrial use cases
Increasing confidence on black box models around the distribution of dataset through stress tests.
Black box models will be provided by industrial teams (example: Renault, detection of turn lights).
State of the art and limitations
Scientific approach to solve the challenge
Expected scientific outcomes
This method will enable to construct databases that undergo a stress in a very large variety of ways: such database can be used in two different ways: on the one hand, stressing the test sample enable to study robustness of machine learning procedures and their ability to resist deviation or precise adversarial attacks pointing out their sensitive to interpretable features. So this provides a tool to understand deeply the behavior of AI algorithms with respect to the parameters. This will also enable to build test samples tailored to follow a certain behavior and thus provide a new way of designing benchmark data bases. On the other hand, stressing the learning sample provides a new way to study the landscape of minimizers of the algorithm and to understand the learning procedure in an interpretable way. The outcome will be a better understanding of the optimization step and its stability but also a way to qualify the database by pointing out sensitivity of observations or lack of observations.
Dataset required for the challenge
Data coming from challenges 1 and 2 and the training networks.
Success criteria for the challenge
Challenge will be successful if we have developped:
- Tools for Automatic data augmentation undergoing some modifications on the distribution of the dataset
- Tools for studying robustness and stability of black box models at large scale and deeply understand the behavior of these models in a large variety of cases undergoing some modifications on the distribution of the dataset
 Bachoc et al (2018) Entropic Variable Boosting for Explainability & Interpretability in Machine Learning.
 Mark Craven and Jude W. Shavlik. Extracting tree-structured representations of trained net- works. In Advances in Neural Information Pro- cessing Systems 8, NIPS, Denver, CO, Novem- ber 27-30, 1995, pages 24?30, 1995
 Mary T Dzindolet, Scott A Peterson, Regina A Pomranky, Linda G Pierce, and Hall P Beck. The role of trust in automation reliance. International journal of human-computer studies, 58(6):697?718, 2003
 Jonathan L Herlocker, Joseph A Konstan, and John Riedl. Explaining collaborative filtering recommendations. In Proceedings of the 2000 ACM conference on Computer supported coop- erative work, pages 241?250. ACM, 2000
 Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should I trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Confer- ence on Knowledge Discovery and Data Mining , pages 1135?1144. ACM, 2016
 Ramprasaath R Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that ? arXiv preprint arXiv:1611.07450, 2016
 S. Chatterjee and A.S. Hadi. Influential observations, high leverage points, and outliers in linear regression. Statistical Science, 1(3):379-393, 1986