Recently, the eXplainable AI (XAI) research community has focused on developing methods making Machine Learning (ML) predictors more interpretable or explainable. Unfortunately, researchers are struggling to converge towards an unambiguous definition of notions such as interpretation or explanation—which are often (and mistakenly) used interchangeably. Furthermore, in spite of the sound metaphors that Multi-Agent System (MAS) could easily provide to address such a challenge, an agent-oriented perspective on the topic is still missing. Thus, this paper proposes an abstract and formal framework for XAI-based MAS, reconciling notions and results from the literature.