Symbolic Knowledge-Extraction Evaluation Metrics: The FiRe Score

Symbolic knowledge-extraction (SKE) techniques are becoming of key importance for AI applications since they enable the explanation of opaque black-box predictors, enhancing trust and transparency. Among all the available SKE techniques, the best option for the case at hand should be selected. However, an automatic comparison between different options can be performed only if an adequate metric - such as a scoring function resuming all the interesting features of the extractors - is provided. The definition of evaluation metrics for symbolic knowledge extractors is currently neglected in the literature. Accordingly, in this paper we introduce the FiRe score metric to assess the quality of a symbolic knowledge-extraction procedure, taking into account both its predictive performance and the readability of the extracted knowledge. It is compared to another existing scoring metric and a rigorous mathematical formulation is provided along with several practical examples to highlight its effectiveness to the end of being exploited inside automatic hyper-parameter tuning procedures.