Modern distributed systems require communicating agents to agree on a shared, formal semantics for the data they exchange and operate upon. The Semantic Web offers tools to encode semantics in the form of ontologies, where data is represented in the form knowledge graphs (KG). Applying such tools to intelligent agents equipped with machine learning (ML) capabilities is of particular interest, as it may enable a higher degree of interoperability among heterogeneous agents. Indeed, inputs and outputs of ML models can be formalised through ontologies, while the data they operate upon can be represented as KG.
In this paper we explore the combination of Semantic Web tools with knowledge extraction-that is, a research line aimed at extracting intelligible rules mimicking the behaviour of ML predictors, with the purpose of explaining their behaviour. Along this line, we study whether and to what extent ontologies and KG can be exploited as both the source and the outcome of a rule extraction procedure. In other words, we investigate the extraction of semantic rules out of sub-symbolic predictors trained upon data as KG-possibly adhering to some ontology. In doing so, we extend our PSyKE framework for rule extraction with Semantic Web support. In practice, we make PSyKE able to  train ML predictors out of OWL ontologies and RDF knowledge graphs, and (ii) extract semantic knowledge out of them, in the form of SWRL rules. A discussion among the major benefits and issues of our approach is provided, along with a description of the overall workflow.