Academic literature on the topic 'Post-hoc interpretability'
Create a spot-on reference in APA, MLA, Chicago, Harvard, and other styles
Consult the lists of relevant articles, books, theses, conference reports, and other scholarly sources on the topic 'Post-hoc interpretability.'
Next to every source in the list of references, there is an 'Add to bibliography' button. Press on it, and we will generate automatically the bibliographic reference to the chosen work in the citation style you need: APA, MLA, Harvard, Chicago, Vancouver, etc.
You can also download the full text of the academic publication as pdf and read online its abstract whenever available in the metadata.
Journal articles on the topic "Post-hoc interpretability"
1
Feng, Jiangfan, Yukun Liang, and Lin Li. "Anomaly Detection in Videos Using Two-Stream Autoencoder with Post Hoc Interpretability." Computational Intelligence and Neuroscience 2021 (July 26, 2021): 1–15. http://dx.doi.org/10.1155/2021/7367870.
Full textAbstract:
The growing interest in deep learning approaches to video surveillance raises concerns about the accuracy and efficiency of neural networks. However, fast and reliable detection of abnormal events is still a challenging work. Here, we introduce a two-stream approach that offers an autoencoder-based structure for fast and efficient detection to facilitate anomaly detection from surveillance video without labeled abnormal events. Furthermore, we present post hoc interpretability of feature map visualization to show the process of feature learning, revealing uncertain and ambiguous decision boundaries in the video sequence. Experimental results on Avenue, UCSD Ped2, and Subway datasets show that our method can detect abnormal events well and explain the internal logic of the model at the object level.
2
Zhang, Zaixi, Qi Liu, Hao Wang, Chengqiang Lu, and Cheekong Lee. "ProtGNN: Towards Self-Explaining Graph Neural Networks." Proceedings of the AAAI Conference on Artificial Intelligence 36, no. 8 (2022): 9127–35. http://dx.doi.org/10.1609/aaai.v36i8.20898.
Full textAbstract:
Despite the recent progress in Graph Neural Networks (GNNs), it remains challenging to explain the predictions made by GNNs. Existing explanation methods mainly focus on post-hoc explanations where another explanatory model is employed to provide explanations for a trained GNN. The fact that post-hoc methods fail to reveal the original reasoning process of GNNs raises the need of building GNNs with built-in interpretability. In this work, we propose Prototype Graph Neural Network (ProtGNN), which combines prototype learning with GNNs and provides a new perspective on the explanations of GNNs. In ProtGNN, the explanations are naturally derived from the case-based reasoning process and are actually used during classification. The prediction of ProtGNN is obtained by comparing the inputs to a few learned prototypes in the latent space. Furthermore, for better interpretability and higher efficiency, a novel conditional subgraph sampling module is incorporated to indicate which part of the input graph is most similar to each prototype in ProtGNN+. Finally, we evaluate our method on a wide range of datasets and perform concrete case studies. Extensive results show that ProtGNN and ProtGNN+ can provide inherent interpretability while achieving accuracy on par with the non-interpretable counterparts.
3
Xu, Qian, Wenzhao Xie, Bolin Liao, et al. "Interpretability of Clinical Decision Support Systems Based on Artificial Intelligence from Technological and Medical Perspective: A Systematic Review." Journal of Healthcare Engineering 2023 (February 3, 2023): 1–13. http://dx.doi.org/10.1155/2023/9919269.
Full textAbstract:
Background. Artificial intelligence (AI) has developed rapidly, and its application extends to clinical decision support system (CDSS) for improving healthcare quality. However, the interpretability of AI-driven CDSS poses significant challenges to widespread application. Objective. This study is a review of the knowledge-based and data-based CDSS literature regarding interpretability in health care. It highlights the relevance of interpretability for CDSS and the area for improvement from technological and medical perspectives. Methods. A systematic search was conducted on the interpretability-related literature published from 2011 to 2020 and indexed in the five databases: Web of Science, PubMed, ScienceDirect, Cochrane, and Scopus. Journal articles that focus on the interpretability of CDSS were included for analysis. Experienced researchers also participated in manually reviewing the selected articles for inclusion/exclusion and categorization. Results. Based on the inclusion and exclusion criteria, 20 articles from 16 journals were finally selected for this review. Interpretability, which means a transparent structure of the model, a clear relationship between input and output, and explainability of artificial intelligence algorithms, is essential for CDSS application in the healthcare setting. Methods for improving the interpretability of CDSS include ante-hoc methods such as fuzzy logic, decision rules, logistic regression, decision trees for knowledge-based AI, and white box models, post hoc methods such as feature importance, sensitivity analysis, visualization, and activation maximization for black box models. A number of factors, such as data type, biomarkers, human-AI interaction, needs of clinicians, and patients, can affect the interpretability of CDSS. Conclusions. The review explores the meaning of the interpretability of CDSS and summarizes the current methods for improving interpretability from technological and medical perspectives. The results contribute to the understanding of the interpretability of CDSS based on AI in health care. Future studies should focus on establishing formalism for defining interpretability, identifying the properties of interpretability, and developing an appropriate and objective metric for interpretability; in addition, the user's demand for interpretability and how to express and provide explanations are also the directions for future research.
4
Gill, Navdeep, Patrick Hall, Kim Montgomery, and Nicholas Schmidt. "A Responsible Machine Learning Workflow with Focus on Interpretable Models, Post-hoc Explanation, and Discrimination Testing." Information 11, no. 3 (2020): 137. http://dx.doi.org/10.3390/info11030137.
Full textAbstract:
This manuscript outlines a viable approach for training and evaluating machine learning systems for high-stakes, human-centered, or regulated applications using common Python programming tools. The accuracy and intrinsic interpretability of two types of constrained models, monotonic gradient boosting machines and explainable neural networks, a deep learning architecture well-suited for structured data, are assessed on simulated data and publicly available mortgage data. For maximum transparency and the potential generation of personalized adverse action notices, the constrained models are analyzed using post-hoc explanation techniques including plots of partial dependence and individual conditional expectation and with global and local Shapley feature importance. The constrained model predictions are also tested for disparate impact and other types of discrimination using measures with long-standing legal precedents, adverse impact ratio, marginal effect, and standardized mean difference, along with straightforward group fairness measures. By combining interpretable models, post-hoc explanations, and discrimination testing with accessible software tools, this text aims to provide a template workflow for machine learning applications that require high accuracy and interpretability and that mitigate risks of discrimination.
5
Marconato, Emanuele, Andrea Passerini, and Stefano Teso. "Interpretability Is in the Mind of the Beholder: A Causal Framework for Human-Interpretable Representation Learning." Entropy 25, no. 12 (2023): 1574. http://dx.doi.org/10.3390/e25121574.
Full textAbstract:
Research on Explainable Artificial Intelligence has recently started exploring the idea of producing explanations that, rather than being expressed in terms of low-level features, are encoded in terms of interpretable concepts learned from data. How to reliably acquire such concepts is, however, still fundamentally unclear. An agreed-upon notion of concept interpretability is missing, with the result that concepts used by both post hoc explainers and concept-based neural networks are acquired through a variety of mutually incompatible strategies. Critically, most of these neglect the human side of the problem: a representation is understandable only insofar as it can be understood by the human at the receiving end. The key challenge in human-interpretable representation learning (hrl) is how to model and operationalize this human element. In this work, we propose a mathematical framework for acquiring interpretable representations suitable for both post hoc explainers and concept-based neural networks. Our formalization of hrl builds on recent advances in causal representation learning and explicitly models a human stakeholder as an external observer. This allows us derive a principled notion of alignment between the machine’s representation and the vocabulary of concepts understood by the human. In doing so, we link alignment and interpretability through a simple and intuitive name transfer game, and clarify the relationship between alignment and a well-known property of representations, namely disentanglement. We also show that alignment is linked to the issue of undesirable correlations among concepts, also known as concept leakage, and to content-style separation, all through a general information-theoretic reformulation of these properties. Our conceptualization aims to bridge the gap between the human and algorithmic sides of interpretability and establish a stepping stone for new research on human-interpretable representations.
6
Degtiarova, Ganna, Fran Mikulicic, Jan Vontobel, et al. "Post-hoc motion correction for coronary computed tomography angiography without additional radiation dose - Improved image quality and interpretability for “free”." Imaging 14, no. 2 (2022): 82–88. http://dx.doi.org/10.1556/1647.2022.00060.
Full textAbstract:
AbstractObjectiveTo evaluate the impact of a motion-correction (MC) algorithm, applicable post-hoc and not dependent on extended padding, on the image quality and interpretability of coronary computed tomography angiography (CCTA).MethodsNinety consecutive patients undergoing CCTA on a latest-generation 256-slice CT device were prospectively included. CCTA was performed with prospective electrocardiogram-triggering and the shortest possible acquisition window (without padding) at 75% of the R-R-interval. All datasets were reconstructed without and with MC of the coronaries. The latter exploits the minimal padding inherent in cardiac CT scans with this device due to data acquisition also during the short time interval needed for the tube to reach target currents and voltage (“free” multiphase). Two blinded readers independently assessed image quality on a 4-point Likert scale for all segments.ResultsA total of 1,030 coronary segments were evaluated. Application of MC both with automatic and manual coronary centerline tracking resulted in a significant improvement in image quality as compared to the standard reconstruction without MC (mean Likert score 3.67 [3.50;3.81] vs 3.58 [3.40;3.73], P = 0.005, and 3.7 [3.55;3.82] vs 3.58 [3.40;3.73], P < 0.001, respectively). Furthermore, MC significantly reduced the proportion of non-evaluable segments and patients with at least one non-evaluable coronary segment from 2% to as low as 0.3%, and from 14% to as low as 3%. Reduction of motion artifacts was predominantly observed in the right coronary artery.ConclusionsA post-hoc device-specific MC algorithm improves image quality and interpretability of prospectively electrocardiogram-triggered CCTA and reduces the proportion of non-evaluable scans without any additional radiation dose exposure.
7
Lao, Danning, Qi Liu, Jiazi Bu, Junchi Yan, and Wei Shen. "ViTree: Single-Path Neural Tree for Step-Wise Interpretable Fine-Grained Visual Categorization." Proceedings of the AAAI Conference on Artificial Intelligence 38, no. 3 (2024): 2866–73. http://dx.doi.org/10.1609/aaai.v38i3.28067.
Full textAbstract:
As computer vision continues to advance and finds widespread applications across various domains, the need for interpretability in deep learning models becomes paramount. Existing methods often resort to post-hoc techniques or prototypes to explain the decision-making process, which can be indirect and lack intrinsic illustration. In this research, we introduce ViTree, a novel approach for fine-grained visual categorization that combines the popular vision transformer as a feature extraction backbone with neural decision trees. By traversing the tree paths, ViTree effectively selects patches from transformer-processed features to highlight informative local regions, thereby refining representations in a step-wise manner. Unlike previous tree-based models that rely on soft distributions or ensembles of paths, ViTree selects a single tree path, offering a clearer and simpler decision-making process. This patch and path selectivity enhances model interpretability of ViTree, enabling better insights into the model's inner workings. Remarkably, extensive experimentation validates that this streamlined approach surpasses various strong competitors and achieves state-of-the-art performance while maintaining exceptional interpretability which is proved by multi-perspective methods. Code can be found at https://github.com/SJTU-DeepVisionLab/ViTree.
8
Jalali, Anahid, Alexander Schindler, Bernhard Haslhofer, and Andreas Rauber. "Machine Learning Interpretability Techniques for Outage Prediction: A Comparative Study." PHM Society European Conference 5, no. 1 (2020): 10. http://dx.doi.org/10.36001/phme.2020.v5i1.1244.
Full textAbstract:
Interpretable machine learning has recently attracted a lot of interest in the community. Currently, it mainly focuses on models trained on non-time series data. LIME and SHAP are well-known examples and provide visual-explanations of feature contributions to model decisions on an instance basis. Other post-hoc approaches, such as attribute-wise interpretations, also focus on tabular data only. Little research has been done so far on the interpretability of predictive models trained on time series data. Therefore, this work focuses on explaining decisions made by black-box models such as Deep Neural Networks trained on sensor data. In this paper, we present the results of a qualitative study, in which we systematically compare the types of explanations and the properties (e.g., method, computational complexity) of existing interpretability approaches for models trained on the PHM08-CMAPSS dataset. We compare shallow models such as regression trees (with limited depth) and black-box models such as Long-Short Term Memories (LSTMs) and Support Vector Regression (SVR). We train models on processed sensor data and explain their output using LIME, SHAP, and attribute-wise methods. Throughout our experiments, we point out the advantages and disadvantages of using these approaches for interpreting models trained on time series data. Our investigation results can serve as a guideline for selecting a suitable explainability method for black-box predictive models trained on time-series data.
9
García-Vicente, Clara, David Chushig-Muzo, Inmaculada Mora-Jiménez, et al. "Evaluation of Synthetic Categorical Data Generation Techniques for Predicting Cardiovascular Diseases and Post-Hoc Interpretability of the Risk Factors." Applied Sciences 13, no. 7 (2023): 4119. http://dx.doi.org/10.3390/app13074119.
Full textAbstract:
Machine Learning (ML) methods have become important for enhancing the performance of decision-support predictive models. However, class imbalance is one of the main challenges for developing ML models, because it may bias the learning process and the model generalization ability. In this paper, we consider oversampling methods for generating synthetic categorical clinical data aiming to improve the predictive performance in ML models, and the identification of risk factors for cardiovascular diseases (CVDs). We performed a comparative study of several categorical synthetic data generation methods, including Synthetic Minority Oversampling Technique Nominal (SMOTEN), Tabular Variational Autoencoder (TVAE) and Conditional Tabular Generative Adversarial Networks (CTGANs). Then, we assessed the impact of combining oversampling strategies and linear and nonlinear supervised ML methods. Lastly, we conducted a post-hoc model interpretability based on the importance of the risk factors. Experimental results show the potential of GAN-based models for generating high-quality categorical synthetic data, yielding probability mass functions that are very close to those provided by real data, maintaining relevant insights, and contributing to increasing the predictive performance. The GAN-based model and a linear classifier outperform other oversampling techniques, improving the area under the curve by 2%. These results demonstrate the capability of synthetic data to help with both determining risk factors and building models for CVD prediction.
10
Wang, Zhengguang. "Validation, Robustness, and Accuracy of Perturbation-Based Sensitivity Analysis Methods for Time-Series Deep Learning Models." Proceedings of the AAAI Conference on Artificial Intelligence 38, no. 21 (2024): 23768–70. http://dx.doi.org/10.1609/aaai.v38i21.30559.
Full textAbstract:
This work undertakes studies to evaluate Interpretability Methods for Time Series Deep Learning. Sensitivity analysis assesses how input changes affect the output, constituting a key component of interpretation. Among the post-hoc interpretation methods such as back-propagation, perturbation, and approximation, my work will investigate perturbation-based sensitivity Analysis methods on modern Transformer models to benchmark their performances. Specifically, my work intends to answer three research questions: 1) Do different sensitivity analysis methods yield comparable outputs and attribute importance rankings? 2) Using the same sensitivity analysis method, do different Deep Learning models impact the output of the sensitivity analysis? 3) How well do the results from sensitivity analysis methods align with the ground truth?
Dissertations / Theses on the topic "Post-hoc interpretability"
1
Jeyasothy, Adulam. "Génération d'explications post-hoc personnalisées." Electronic Thesis or Diss., Sorbonne université, 2024. http://www.theses.fr/2024SORUS027.
Full textAbstract:
La thèse se place dans le domaine de l'IA explicable (XAI, eXplainable AI). Nous nous concentrons sur les méthodes d'interprétabilité post-hoc qui visent à expliquer à un utilisateur la prédiction pour une donnée d'intérêt spécifique effectuée par un modèle de décision entraîné. Pour augmenter l'interprétabilité des explications, cette thèse étudie l'intégration de connaissances utilisateur dans ces méthodes, et vise ainsi à améliorer la compréhensibilité de l'explication en générant des explications personnalisées adaptées à chaque utilisateur. Pour cela, nous proposons un formalisme général qui intègre explicitement la connaissance via un nouveau critère dans les objectifs d'interprétabilité. Ce formalisme est ensuite décliné pour différents types connaissances et différents types d'explications, particulièrement les exemples contre-factuels, conduisant à la proposition de plusieurs algorithmes (KICE, Knowledge Integration in Counterfactual Explanation, rKICE pour sa variante incluant des connaissances exprimées par des règles et KISM, Knowledge Integration in Surrogate Models). La question de l'agrégation des contraintes de qualité classique et de compatibilité avec les connaissances est également étudiée et nous proposons d'utiliser l'intégrale de Gödel comme opérateur d'agrégation. Enfin nous discutons de la difficulté à générer une unique explication adaptée à tous types d'utilisateurs et de la notion de diversité dans les explications<br>This thesis is in the field of eXplainable AI (XAI). We focus on post-hoc interpretability methods that aim to explain to a user the prediction for a specific data made by a trained decision model. To increase the interpretability of explanations, this thesis studies the integration of user knowledge into these methods, and thus aims to improve the understandability of the explanation by generating personalized explanations tailored to each user. To this end, we propose a general formalism that explicitly integrates knowledge via a new criterion in the interpretability objectives. This formalism is then declined for different types of knowledge and different types of explanations, particularly counterfactual examples, leading to the proposal of several algorithms (KICE, Knowledge Integration in Counterfactual Explanation, rKICE for its variant including knowledge expressed by rules and KISM, Knowledge Integration in Surrogate Models). The issue of aggregating classical quality and knowledge compatibility constraints is also studied, and we propose to use Gödel's integral as an aggregation operator. Finally, we discuss the difficulty of generating a single explanation suitable for all types of users and the notion of diversity in explanations
2
SEVESO, ANDREA. "Symbolic Reasoning for Contrastive Explanations." Doctoral thesis, Università degli Studi di Milano-Bicocca, 2023. https://hdl.handle.net/10281/404830.
Full textAbstract:
La necessità di spiegazioni sui sistemi di Machine Learning (ML) sta crescendo man mano che i nuovi modelli superano in performance i loro predecessori, diventando più complessi e meno comprensibili per gli utenti finali.
Un passaggio essenziale nella ricerca in ambito eXplainable Artificial Intelligence (XAI) è la creazione di modelli interpretabili che mirano ad approssimare la funzione decisionale di un algoritmo black box.
Sebbene negli ultimi anni siano stati proposti diversi metodi di XAI, non è stata prestata sufficiente attenzione alla spiegazione di come i modelli modificano il loro comportamento in contrasto con altre versioni (ad esempio, a causa di nuovi addestramenti dei modelli o modifica dei dati sottostanti). In questi casi, un sistema XAI dovrebbe spiegare perché il modello cambia le sue previsioni sui risultati passati.
In diverse situazioni pratiche, i decisori umani si confrontano con più di un modello di apprendimento automatico. Di conseguenza, sta crescendo l'importanza di capire come funzionano due modelli di Machine Learning al di là delle loro performance predittive, per comprendere il loro comportamento, le loro differenze e la loro somiglianza.
Ad oggi, i modelli interpretabili sono sintetizzati per spiegare i cosiddetti modelli black-box e le loro previsioni, e possono essere utili per rappresentare formalmente e misurare le differenze nel comportamento del modello ri-addestrato nel trattare dati nuovi e diversi.
Catturare e comprendere tali differenze è fondamentale, poiché la necessità di fiducia è fondamentale in qualsiasi applicazione a supporto dei processi decisionali umani-IA.
Questa è l'idea di ContrXT, un nuovo approccio che (i) traccia i criteri decisionali di un classificatore black box codificando i cambiamenti nella logica decisionale attraverso Binary Decision Diagrams. Quindi (ii) fornisce spiegazioni globali, agnostici dalla tipologia di modello, Model-Contrastive (M-contrast) in linguaggio naturale, stimando perché -e in quale misura- il modello ha modificato il suo comportamento nel tempo. Abbiamo implementato e valutato questo approccio su diversi modelli ML supervisionati addestrati su set di dati di benchmark e un'applicazione reale, dimostrando che è efficace nel rilevare classi notevolmente modificate e nello spiegare la loro variazione attraverso un user study.
L'approccio è stato implementato ed è disponibile per la comunità sia come pacchetto Python che tramite API REST, fornendo contrastive explanations as a service.<br>The need for explanations of Machine Learning (ML) systems is growing as new models outperform their predecessors while becoming more complex and less comprehensible for their end-users.
An essential step in eXplainable Artificial Intelligence (XAI) research is to create interpretable models that aim at approximating the decision function of a black box algorithm.
Though several XAI methods have been proposed in recent years, not enough attention was paid to explaining how models change their behaviour in contrast with other versions (e.g., due to retraining or data shifts). In such cases, an XAI system should explain why the model changes its predictions concerning past outcomes.
In several practical situations, human decision-makers deal with more than one machine learning model. Consequently, the importance of understanding how two machine learning models work beyond their prediction performances is growing, to understand their behavior, their differences, and their likeness.
To date, interpretable models are synthesised for explaining black boxes and their predictions and can be beneficial for formally representing and measuring the differences in the retrained model's behaviour in dealing with new and different data.
Capturing and understanding such differences is crucial, as the need for trust is key in any application to support human-Artificial Intelligence (AI) decision-making processes.
This is the idea of ContrXT, a novel approach that (i) traces the decision criteria of a black box classifier by encoding the changes in the decision logic through Binary Decision Diagrams. Then (ii) it provides global, model-agnostic, Model-Contrastive (M-contrast) explanations in natural language, estimating why -and to what extent- the model has modified its behaviour over time. We implemented and evaluated this approach over several supervised ML models trained on benchmark datasets and a real-life application, showing it is effective in catching majorly changed classes and in explaining their variation through a user study.
The approach has been implemented, and it is available to the community both as a python package and through REST API, providing contrastive explanations as a service.
3
Laugel, Thibault. "Interprétabilité locale post-hoc des modèles de classification "boites noires"." Electronic Thesis or Diss., Sorbonne université, 2020. http://www.theses.fr/2020SORUS215.
Full textAbstract:
Cette thèse porte sur le domaine du XAI (explicabilité de l'IA), et plus particulièrement sur le paradigme de l'interprétabilité locale post-hoc, c'est-à-dire la génération d'explications pour une prédiction unique d'un classificateur entraîné. En particulier, nous étudions un contexte totalement agnostique, c'est-à-dire que l'explication est générée sans utiliser aucune connaissance sur le modèle de classification (traité comme une boîte noire) ni les données utilisées pour l'entraîner. Dans cette thèse, nous identifions plusieurs problèmes qui peuvent survenir dans ce contexte et qui peuvent être préjudiciables à l'interprétabilité. Nous nous proposons d'étudier chacune de ces questions et proposons des critères et des approches nouvelles pour les détecter et les caractériser. Les trois questions sur lesquelles nous nous concentrons sont : le risque de générer des explications qui sont hors distribution ; le risque de générer des explications qui ne peuvent être associées à aucune instance d'entraînement ; et le risque de générer des explications qui ne sont pas assez locales. Ces risques sont étudiés à travers deux catégories spécifiques d'approches de l'interprétabilité : les explications contrefactuelles et les modèles de substitution locaux<br>This thesis focuses on the field of XAI (eXplainable AI), and more particularly local post-hoc interpretability paradigm, that is to say the generation of explanations for a single prediction of a trained classifier. In particular, we study a fully agnostic context, meaning that the explanation is generated without using any knowledge about the classifier (treated as a black-box) nor the data used to train it. In this thesis, we identify several issues that can arise in this context and that may be harmful for interpretability. We propose to study each of these issues and propose novel criteria and approaches to detect and characterize them. The three issues we focus on are: the risk of generating explanations that are out of distribution; the risk of generating explanations that cannot be associated to any ground-truth instance; and the risk of generating explanations that are not local enough. These risks are studied through two specific categories of interpretability approaches: counterfactual explanations, and local surrogate models
4
Radulovic, Nedeljko. "Post-hoc Explainable AI for Black Box Models on Tabular Data." Electronic Thesis or Diss., Institut polytechnique de Paris, 2023. http://www.theses.fr/2023IPPAT028.
Full textAbstract:
Les modèles d'intelligence artificielle (IA) actuels ont fait leurs preuves dans la résolution de diverses tâches, telles que la classification, la régression, le traitement du langage naturel (NLP) et le traitement d'images. Les ressources dont nous disposons aujourd'hui nous permettent d'entraîner des modèles d'IA très complexes pour résoudre différents problèmes dans presque tous les domaines : médecine, finance, justice, transport, prévisions, etc. Avec la popularité et l'utilisation généralisée des modèles d'IA, la nécessite d'assurer la confiance dans ces modèles s'est également accrue. Aussi complexes soient-ils aujourd'hui, ces modèles d'IA sont impossibles à interpréter et à comprendre par les humains. Dans cette thèse nous nous concentrons sur un domaine de recherche spécifique, à savoir l'intelligence artificielle explicable (xAI), qui vise à fournir des approches permettant d'interpréter les modèles d'IA complexes et d'expliquer leurs décisions. Nous présentons deux approches, STACI et BELLA, qui se concentrent sur les tâches de classification et de régression, respectivement, pour les données tabulaires. Les deux méthodes sont des approches post-hoc agnostiques au modèle déterministe, ce qui signifie qu'elles peuvent être appliquées à n'importe quel modèle boîte noire après sa création. De cette manière, l'interopérabilité présente une valeur ajoutée sans qu'il soit nécessaire de faire des compromis sur les performances du modèle de boîte noire. Nos méthodes fournissent des interprétations précises, simples et générales à la fois de l'ensemble du modèle boîte noire et de ses prédictions individuelles. Nous avons confirmé leur haute performance par des expériences approfondies et étude d'utilisateurs<br>Current state-of-the-art Artificial Intelligence (AI) models have been proven to be verysuccessful in solving various tasks, such as classification, regression, Natural Language Processing(NLP), and image processing. The resources that we have at our hands today allow us to trainvery complex AI models to solve different problems in almost any field: medicine, finance, justice,transportation, forecast, etc. With the popularity and widespread use of the AI models, the need toensure the trust in them also grew. Complex as they come today, these AI models are impossible to be interpreted and understood by humans. In this thesis, we focus on the specific area of research, namely Explainable Artificial Intelligence (xAI), that aims to provide the approaches to interpret the complex AI models and explain their decisions. We present two approaches STACI and BELLA which focus on classification and regression tasks, respectively, for tabular data. Both methods are deterministic model-agnostic post-hoc approaches, which means that they can be applied to any black-box model after its creation. In this way, interpretability presents an added value without the need to compromise on black-box model's performance. Our methods provide accurate, simple and general interpretations of both the whole black-box model and its individual predictions. We confirmed their high performance through extensive experiments and a user study
5
Bhattacharya, Debarpan. "A Learnable Distillation Approach For Model-agnostic Explainability With Multimodal Applications." Thesis, 2023. https://etd.iisc.ac.in/handle/2005/6108.
Full textAbstract:
Deep neural networks are the most widely used examples of sophisticated mapping functions from feature space to class labels. In the recent years, several high impact decisions in domains such as finance, healthcare, law and autonomous driving, are made with deep models. In these tasks, the model decisions lack interpretability, and pose difficulties in making the models accountable. Hence, there is a strong demand for developing explainable approaches which can elicit how the deep neural architecture, despite the astounding performance improvements observed in all fields, including computer vision, natural language processing, generates the output decisions. The current frameworks for explainability of deep models are based on gradients (eg. GradCAM, guided-gradCAM, Integrated gradients etc) or based on locally linear assumptions (eg. LIME). Some of these approaches require the knowledge of the deep model architecture, which may be restrictive in many applications. Further, most of the prior works in the literature highlight the results on a set of small number of examples to illustrate the performance of these XAI methods, often lacking statistical evaluation.
This thesis proposes a new approach for explainability based on mask estimation approaches, called the Distillation Approach for Model-agnostic Explainability (DAME). The DAME is a saliency-based explainability model that is post-hoc, model-agnostic (applicable to any black box architecture), and requires only query access to black box.
The DAME is a student-teacher modeling approach, where the teacher model is the original model for which the explainability is sought, while the student model is the mask estimation model. The input sample is augmented with various data augmentation techniques to produce numerous samples in the immediate vicinity of the input.
Using these samples, the mask estimation model is learnt to generate the saliency map of the input sample for predicting the labels. A distillation loss is used to train the DAME model, and the student model tries to locally approximate the original model. Once the DAME model is trained, the DAME generates a region of the input (either in space or in time domain for images and audio samples, respectively) that best explains the model predictions.
We also propose an evaluation framework, for both image and audio tasks, where the XAI models are evaluated in a statistical framework on a set of held-out of examples with the Intersection-over-Union (IoU) metric. We have validated the DAME model for vision, audio and biomedical tasks. Firstly, we deploy the DAME for explaining a ResNet-50 classifier pre-trained on ImageNet dataset for the object recognition task. Secondly, we explain the predictions made by ResNet-50 classifier fine-tuned on Environmental Sound Classification (ESC-10) dataset for the audio event classification task. Finally, we validate the DAME model on the COVID-19 classification task using cough audio recordings. In these tasks, the DAME model is shown to outperform existing benchmarks for explainable modeling.
The thesis concludes with a discussion on the limitations of the DAME approach along with the potential future directions.
Book chapters on the topic "Post-hoc interpretability"
1
Kamath, Uday, and John Liu. "Post-Hoc Interpretability and Explanations." In Explainable Artificial Intelligence: An Introduction to Interpretable Machine Learning. Springer International Publishing, 2021. http://dx.doi.org/10.1007/978-3-030-83356-5_5.
Full text
2
Greenwell, Brandon M. "Peeking inside the “black box”: post-hoc interpretability." In Tree-Based Methods for Statistical Learning in R. Chapman and Hall/CRC, 2022. http://dx.doi.org/10.1201/9781003089032-6.
Full text
3
Santos, Flávio Arthur Oliveira, Cleber Zanchettin, José Vitor Santos Silva, Leonardo Nogueira Matos, and Paulo Novais. "A Hybrid Post Hoc Interpretability Approach for Deep Neural Networks." In Lecture Notes in Computer Science. Springer International Publishing, 2021. http://dx.doi.org/10.1007/978-3-030-86271-8_50.
Full text
4
Ann Jo, Ashly, and Ebin Deni Raj. "Post hoc Interpretability: Review on New Frontiers of Interpretable AI." In Lecture Notes in Networks and Systems. Springer Nature Singapore, 2023. http://dx.doi.org/10.1007/978-981-99-1203-2_23.
Full text
5
Molnar, Christoph, Giuseppe Casalicchio, and Bernd Bischl. "Quantifying Model Complexity via Functional Decomposition for Better Post-hoc Interpretability." In Machine Learning and Knowledge Discovery in Databases. Springer International Publishing, 2020. http://dx.doi.org/10.1007/978-3-030-43823-4_17.
Full text
6
Stevens, Alexander, Johannes De Smedt, and Jari Peeperkorn. "Quantifying Explainability in Outcome-Oriented Predictive Process Monitoring." In Lecture Notes in Business Information Processing. Springer International Publishing, 2022. http://dx.doi.org/10.1007/978-3-030-98581-3_15.
Full textAbstract:
AbstractThe growing interest in applying machine and deep learning algorithms in an Outcome-Oriented Predictive Process Monitoring (OOPPM) context has recently fuelled a shift to use models from the explainable artificial intelligence (XAI) paradigm, a field of study focused on creating explainability techniques on top of AI models in order to legitimize the predictions made. Nonetheless, most classification models are evaluated primarily on a performance level, where XAI requires striking a balance between either simple models (e.g. linear regression) or models using complex inference structures (e.g. neural networks) with post-processing to calculate feature importance. In this paper, a comprehensive overview of predictive models with varying intrinsic complexity are measured based on explainability with model-agnostic quantitative evaluation metrics. To this end, explainability is designed as a symbiosis between interpretability and faithfulness and thereby allowing to compare inherently created explanations (e.g. decision tree rules) with post-hoc explainability techniques (e.g. Shapley values) on top of AI models. Moreover, two improved versions of the logistic regression model capable of capturing non-linear interactions and both inherently generating their own explanations are proposed in the OOPPM context. These models are benchmarked with two common state-of-the-art models with post-hoc explanation techniques in the explainability-performance space.
7
Turbé, Hugues, Mina Bjelogrlic, Mehdi Namdar, et al. "A Lightweight and Interpretable Model to Classify Bundle Branch Blocks from ECG Signals." In Studies in Health Technology and Informatics. IOS Press, 2022. http://dx.doi.org/10.3233/shti220393.
Full textAbstract:
Automatic classification of ECG signals has been a longtime research area with large progress having been made recently. However these advances have been achieved with increasingly complex models at the expense of model’s interpretability. In this research, a new model based on multivariate autoregressive model (MAR) coefficients combined with a tree-based model to classify bundle branch blocks is proposed. The advantage of the presented approach is to build a lightweight model which combined with post-hoc interpretability can bring new insights into important cross-lead dependencies which are indicative of the diseases of interest.
8
Dumka, Ankur, Vaibhav Chaudhari, Anil Kumar Bisht, Ruchira Rawat, and Arnav Pandey. "Methods, Techniques, and Application of Explainable Artificial Intelligence." In Advances in Environmental Engineering and Green Technologies. IGI Global, 2024. http://dx.doi.org/10.4018/979-8-3693-2351-9.ch017.
Full textAbstract:
With advancement in machine learning, the use of machine learning has been increased, and explainable artificial intelligence (XAI) has emerged as an area of research and development for addressing the opacity and complexity of machine learning models. This chapter has proposed the overview of the current state of explainable artificial intelligence with highlighting its significance, disadvantages, and its potential applications in different fields. This chapter explores several explainable artificial techniques ranging from post-hoc methods like SHAP, LIME to decision tree and rule-based systems. This chapter also focusses on complexity and interpretability of a model.
9
Li, Yaoman, and Irwin King. "Neural Architecture Search for Explainable Networks." In Frontiers in Artificial Intelligence and Applications. IOS Press, 2023. http://dx.doi.org/10.3233/faia230423.
Full textAbstract:
One of the main challenges in machine learning is providing understandable explanations for complex models. Despite outperforming humans in many tasks, machine learning models are often treated as black boxes that are difficult to interpret. Post-hoc explanation methods have been developed to create interpretable surrogate models that explain the behavior of black-box models. However, these methods have been shown to perpetuate bad practices and lack stability. Recently, inherent explainable approaches have been proposed to provide built-in explainability to models. However, most of these methods sacrifice performance. This paper proposes the Neural Architecture Search for Explainable Networks (NASXNet) approach to address the trade-off between performance and interpretability. Our method applies architecture search to generate high-performance and explainable neural networks for image classification tasks. We conduct experiments on four datasets: CUB-200-2011, Stanford Cars, CIFAR 10, and CIFAR 100. The results demonstrate that our models provide a high-level interpretation of prediction results, achieving state-of-the-art performance that is on par with non-explainable models. This paper contributes by solving the trade-off problem between performance and interpretability. It is the first to apply neural architecture search to develop explainable deep learning models, generating state-of-the-art explainable models that outperform existing approaches. Additionally, a new training process is proposed that enables faster convergence during model training.
Conference papers on the topic "Post-hoc interpretability"
1
Laugel, Thibault, Marie-Jeanne Lesot, Christophe Marsala, Xavier Renard, and Marcin Detyniecki. "The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations." In Twenty-Eighth International Joint Conference on Artificial Intelligence {IJCAI-19}. International Joint Conferences on Artificial Intelligence Organization, 2019. http://dx.doi.org/10.24963/ijcai.2019/388.
Full textAbstract:
Post-hoc interpretability approaches have been proven to be powerful tools to generate explanations for the predictions made by a trained black-box model. However, they create the risk of having explanations that are a result of some artifacts learned by the model instead of actual knowledge from the data. This paper focuses on the case of counterfactual explanations and asks whether the generated instances can be justified, i.e. continuously connected to some ground-truth data. We evaluate the risk of generating unjustified counterfactual examples by investigating the local neighborhoods of instances whose predictions are to be explained and show that this risk is quite high for several datasets. Furthermore, we show that most state of the art approaches do not differentiate justified from unjustified counterfactual examples, leading to less useful explanations.
2
Vieira, Carla Piazzon, and Luciano Antonio Digiampietri. "Machine Learning post-hoc interpretability: a systematic mapping study." In SBSI: XVIII Brazilian Symposium on Information Systems. ACM, 2022. http://dx.doi.org/10.1145/3535511.3535512.
Full text
3
Attanasio, Giuseppe, Debora Nozza, Eliana Pastor, and Dirk Hovy. "Benchmarking Post-Hoc Interpretability Approaches for Transformer-based Misogyny Detection." In Proceedings of NLP Power! The First Workshop on Efficient Benchmarking in NLP. Association for Computational Linguistics, 2022. http://dx.doi.org/10.18653/v1/2022.nlppower-1.11.
Full text
4
Sujana, D. Swainson, and D. Peter Augustine. "Explaining Autism Diagnosis Model Through Local Interpretability Techniques – A Post-hoc Approach." In 2023 International Conference on Data Science, Agents & Artificial Intelligence (ICDSAAI). IEEE, 2023. http://dx.doi.org/10.1109/icdsaai59313.2023.10452575.
Full text
5
Gkoumas, Dimitris, Qiuchi Li, Yijun Yu, and Dawei Song. "An Entanglement-driven Fusion Neural Network for Video Sentiment Analysis." In Thirtieth International Joint Conference on Artificial Intelligence {IJCAI-21}. International Joint Conferences on Artificial Intelligence Organization, 2021. http://dx.doi.org/10.24963/ijcai.2021/239.
Full textAbstract:
Video data is multimodal in its nature, where an utterance can involve linguistic, visual and acoustic information. Therefore, a key challenge for video sentiment analysis is how to combine different modalities for sentiment recognition effectively. The latest neural network approaches achieve state-of-the-art performance, but they neglect to a large degree of how humans understand and reason about sentiment states. By contrast, recent advances in quantum probabilistic neural models have achieved comparable performance to the state-of-the-art, yet with better transparency and increased level of interpretability. However, the existing quantum-inspired models treat quantum states as either a classical mixture or as a separable tensor product across modalities, without triggering their interactions in a way that they are correlated or non-separable (i.e., entangled). This means that the current models have not fully exploited the expressive power of quantum probabilities. To fill this gap, we propose a transparent quantum probabilistic neural model. The model induces different modalities to interact in such a way that they may not be separable, encoding crossmodal information in the form of non-classical correlations. Comprehensive evaluation on two benchmarking datasets for video sentiment analysis shows that the model achieves significant performance improvement. We also show that the degree of non-separability between modalities optimizes the post-hoc interpretability.