Path reasoning is a notable recommendation approach that models high-order user-product relations, based on a Knowledge Graph (KG). This approach can extract reasoning paths between recommended products and already experienced products and, then, turn such paths into textual explanations for the user. A benchmarking of the state-of-the-art approaches, in terms of accuracy and beyond-accuracy perspectives, provides a picture of the progress in this field, highlighting open issues and future directions.

In a study, conducted with Giacomo Balloccu, Christian Cancedda, Gianni Fenu, and Mirko Marras, and published in the proceedings of ECIR 2023, we replicated three state-of-the-art relevant path reasoning recommendation methods proposed in top-tier conferences. Under a common evaluation protocol, based on two public data sets and in comparison with other knowledge-aware methods, we then studied the extent to which they meet recommendation utility and beyond objectives, explanation quality, and consumer and provider fairness.

The source code for our study is available at: https://github.com/giacoballoccu/rep-path-reasoning-recsys. A preprint is also available online.

Methods Replication

To collect existing path reasoning methods, we systematically scanned the recent proceedings of top-tier information retrieval events and journals edited by top-tier publishers. We attempted to replicate the method of each relevant paper, relying as much as possible on the original source code. Three out of the seven relevant papers were replicable with a reasonable effort. As per the non-replicable ones, three did not provide any source code. The other one included unavailable external dependencies.

For each relevant paper, we analyzed the rationale of the proposed method and the characteristics of the experimental setting. In this post, we report the details of the approaches we replicated:

• PGPR (original source code: https://github.com/orcax/PGPR) was based on the idea of training a reinforcement learning (RL) agent for finding paths. During training, the agent starts from a user and learns to reach the correct products, with high rewards. During inference, the agent directly walks to correct products for recommendation, without enumerating all the paths between users and products.
• CAFE (original source code: https://github.com/orcax/CAFE) follows the coarse-to-fine paradigm. Given the KG, a user profile is created to capture user-centric patterns in the coarse stage. To conduct multi-hop path reasoning guided by the user profile, the reasoner is decomposed into an inventory of neural reasoning modules. Then, these modules are combined based on the user profile, to efficiently perform path reasoning.
• UCPR (original source code: https://github.com/johnnyjana730/UCPR/) introduces a multi-view structure leveraging not only local sequence reasoning information, but also a view of the user’s demand portfolio. The user demand portfolio, built in a pre-processing phase and updated via a multi-step refocusing, makes the path selection process adaptive and effective.

Evaluation

We conducted experiments on two data sets: MovieLens (ML1M) and LastFM (LFM1B). To better contextualize our study, we therefore decided to provide comparisons (when interesting) against two knowledge-aware models based on knowledge embeddings, namely CKE and CFKG, and a knowledge-aware model based on propagation, namely KGAT.

Here are the main outcomes emerging from our results:

• Path reasoning methods trade recommendation utility and coverage for explanation power, especially in LFM1M. Conversely, they resulted in higher estimates on other beyond utility objectives and provider fairness than knowledge-aware non-explainable baselines.
• Path reasoning methods show very different patterns in terms of fidelity. CAFE’s fidelity is high and stable across data sets and recommended list sizes. On the other hand, PGPR provides higher but rapidly decaying fidelity in ML1M than LFM1M, viceversa for UCPR.
• Path reasoning methods often yield substantially different paths in terms of recency, popularity, and diversity. Although they exist, no remarkable disparate impacts on explanation quality were found.