The Quarrel of Local Post-hoc Explainers for Moral Values Classification in Natural Language Processing

Although popular and effective, large language models (LLM) are characterised by a performance vs. transparency trade-off that hinders their applicability to sensitive scenarios. To address this issue, several approaches have been recently proposed by the XAI research community, mostly focusing on local post-hoc explanations. However, to the best of our knowledge, a thorough comparison among available explainability techniques is currently missing, mainly for the lack of a general metric to measure their benefits. We compare state-of-the-art local post-hoc explanation mechanisms for models trained over moral value classification tasks based on a measure of correlation. By relying on a novel framework for comparing global impact scores, our experiments show how most local post-hoc explainers are loosely correlated, and highlight huge discrepancies in their results—their ``quarrel'' about explanations. Finally, we compare the impact scores distribution obtained from each local post-hoc explainer with human-made dictionaries, showing alarmingly how there exists no correlation between explanation outputs and the concepts considered to be salient by humans.