Papers
Topics
Authors
Recent
Search
2000 character limit reached

Lightweight Embeddings for Graph Collaborative Filtering

Published 27 Mar 2024 in cs.IR | (2403.18479v2)

Abstract: Graph neural networks (GNNs) are currently one of the most performant collaborative filtering methods. Meanwhile, owing to the use of an embedding table to represent each user/item as a distinct vector, GNN-based recommenders have inherited the long-standing defect of parameter inefficiency. As a common practice for scalable embeddings, parameter sharing enables the use of fewer embedding vectors (i.e., meta-embeddings). When assigning meta-embeddings, most existing methods are a heuristically designed, predefined mapping from each user's/item's ID to the corresponding meta-embedding indexes, thus simplifying the optimization problem into learning only the meta-embeddings. However, in the context of GNN-based collaborative filtering, such a fixed mapping omits the semantic correlations between entities that are evident in the user-item interaction graph, leading to suboptimal recommendation performance. To this end, we propose Lightweight Embeddings for Graph Collaborative Filtering (LEGCF), a parameter-efficient embedding framework dedicated to GNN-based recommenders. LEGCF innovatively introduces an assignment matrix as an extra learnable component on top of meta-embeddings. To jointly optimize these two heavily entangled components, aside from learning the meta-embeddings by minimizing the recommendation loss, LEGCF further performs efficient assignment update by enforcing a novel semantic similarity constraint and finding its closed-form solution based on matrix pseudo-inverse. The meta-embeddings and assignment matrix are alternately updated, where the latter is sparsified on the fly to ensure negligible storage overhead. Extensive experiments on three benchmark datasets have verified LEGCF's smallest trade-off between size and performance, with consistent accuracy gain over state-of-the-art baselines. The codebase of LEGCF is available in https://github.com/xurong-liang/LEGCF.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (68)
  1. Adi Ben-Israel and Thomas NE Greville. 2003. Generalized inverses: theory and applications. Vol. 15. Springer Science & Business Media.
  2. Revisiting graph based collaborative filtering: A linear residual graph convolutional network approach. In AAAI.
  3. Learning elastic embeddings for customizing on-device recommenders. In SIGKDD.
  4. Clustered Embedding Learning for Recommender Systems. In WWW.
  5. Differentiable neural input search for recommender systems. arXiv preprint arXiv:2006.04466 (2020).
  6. Google news personalization: scalable online collaborative filtering. In WWW.
  7. Deeplight: Deep lightweight feature interactions for accelerating ctr predictions in ad serving. In WSDM.
  8. Semantically constrained memory allocation (scma) for embedding in efficient recommendation systems. arXiv preprint arXiv:2103.06124 (2021).
  9. A survey of graph neural networks for recommender systems: Challenges, methods, and directions. ReSys 1, 1 (2023).
  10. Mixed dimension embeddings with application to memory-efficient recommendation systems. In ISIT.
  11. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In AISTATS.
  12. MultiBiSage: A Web-Scale Recommendation System Using Multiple Bipartite Graphs at Pinterest. VLDB 16, 4 (2022).
  13. Lightweight compositional embeddings for incremental streaming recommendation. arXiv preprint arXiv:2202.02427 (2022).
  14. LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. In SIGIR.
  15. Neural Collaborative Filtering. In WWW.
  16. Geoffrey E Hinton et al. 2012. Improving neural networks by preventing co-adaptation of feature detectors. (2012).
  17. xLightFM: Extremely memory-efficient factorization machine. In SIGIR.
  18. Neural Input Search for Large Scale Recommendation Models. In SIGKDD.
  19. Learning to Embed Categorical Features without Embedding Tables for Recommendation. In SIGKDD.
  20. George Karypis and Vipin Kumar. 1997. METIS: A software package for partitioning unstructured graphs, partitioning meshes, and computing fill-reducing orderings of sparse matrices. (1997).
  21. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
  22. AutoSrh: An Embedding Dimensionality Search Framework for Tabular Data Prediction. TKDE (2022).
  23. Matrix Factorization Techniques for Recommender Systems. Computer 42, 8 (2009).
  24. FractalNet: Ultra-Deep Neural Networks without Residuals. In ICLR.
  25. Dual Contrastive Learning for Efficient Static Feature Representation in Sequential Recommendations. TKDE (2023).
  26. Lightweight self-attentive sequential recommendation. In CIKM.
  27. LightRec: A Memory and Search-Efficient Recommender System. In WWW.
  28. Product quantized collaborative filtering. TKDE 33, 9 (2020).
  29. Learning Compact Compositional Embeddings via Regularized Pruning for Recommendation. In ICDM.
  30. Online optimized product quantization. In ICDM.
  31. Automated Embedding Size Search in Deep Recommender Systems. In SIGIR.
  32. Online additive quantization. In KDD.
  33. Learnable Embedding Sizes for Recommender Systems. ICLR (2021).
  34. OptEmbed: Learning Optimal Embedding Table for Click-through Rate Prediction. In CIKM.
  35. Dheevatsa Mudigere et al. 2022. Software-hardware co-design for fast and scalable training of deep learning recommendation models. In Proceedings of the 49th Annual International Symposium on Computer Architecture. 993–1011.
  36. Efficient neural architecture search via parameters sharing. In ICML.
  37. Single-shot embedding dimension search in recommender system. In SIGIR.
  38. Budgeted embedding table for recommender systems. In WSDM.
  39. Continuous Input Embedding Size Search For Recommender Systems. In SIGIR.
  40. Steffen Rendle. 2010. Factorization Machines. In ICDM.
  41. BPR: Bayesian Personalized Ranking from Implicit Feedback. (2009), 10.
  42. Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems. SIGKDD (2020).
  43. Beyond user embedding matrix: Learning to hash for modeling large-scale users in recommendation. In SIGIR.
  44. Gilbert W Stewart. 1993. On the early history of the singular value decomposition. SIAM review 35, 4 (1993).
  45. Hgcf: Hyperbolic graph convolution networks for collaborative filtering. In WWW.
  46. Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction.
  47. Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research (2008).
  48. A location-sentiment-aware recommender system for both home-town and out-of-town users. In KDD.
  49. Next Point-of-Interest Recommendation on Resource-Constrained Mobile Devices. In WWW.
  50. Neural graph collaborative filtering. In SIGIR.
  51. A theoretical analysis of NDCG ranking measures. In COLT.
  52. Feature hashing for large scale multitask learning. In ICML.
  53. Graph Neural Networks in Recommender Systems: A Survey. Comput. Surveys (2023).
  54. On-Device Next-Item Recommendation with Self-Supervised Knowledge Distillation. In SIGIR.
  55. Binary code based hash embedding for web-scale applications. In CIKM.
  56. Learning effective and efficient embedding via an adaptively-masked twins-based layer. In ICKM.
  57. i-Razor: A Neural Input Razor for Feature Selection and Dimension Search in Large-Scale Recommender Systems. arXiv preprint arXiv:2204.00281 (2022).
  58. Tt-rec: Tensor train compression for deep learning recommendation models. Proceedings of Machine Learning and Systems 3 (2021).
  59. Nimble GNN Embedding with Tensor-Train Decomposition. In SIGKDD.
  60. On-Device Recommender Systems: A Comprehensive Survey. arXiv preprint arXiv:2401.11441 (2024).
  61. Graph convolutional neural networks for web-scale recommender systems. In SIGKDD.
  62. Model Size Reduction Using Frequency Based Double Hashing for Recommender Systems. In RecSys.
  63. Double-scale self-supervised hypergraph learning for group recommendation. In CIKM.
  64. Anisotropic additive quantization for fast inner product search. In AAAI.
  65. Deep Learning based Recommender System: A Survey and New Perspectives. Comput. Surveys 52 (2020).
  66. Autoemb: Automated embedding dimensionality search in streaming recommendations. (2021).
  67. Personalized Elastic Embedding Learning for On-Device Recommendation. TKDE (2024).
  68. AutoML for Deep Recommender Systems: A Survey. TOIS (2023).
Citations (1)
View on Semantic Scholar

Summary

  • The paper introduces LEGCF, reducing parameter inefficiency by substituting unique embeddings with a shared pool of meta-embeddings and a learnable assignment matrix.
  • It demonstrates significant improvements in recommendation accuracy and efficiency, evidenced by enhanced NDCG and Recall metrics on benchmark datasets.
  • The methodology simplifies assignment matrix optimization via closed-form solutions, offering a scalable and robust solution for large-scale recommender systems.

Insights into Lightweight Embeddings for Graph Collaborative Filtering

The paper "Lightweight Embeddings for Graph Collaborative Filtering" offers an incremental advancement in the field of recommender systems, particularly those based on Graph Neural Networks (GNNs). This research seeks to address a critical issue tied to the scalability and efficiency of embedding representations in GNN-based recommenders. Traditional embedding methods for such systems are constrained by high parameter costs due to their reliance on large embedding tables where each user/item entity is represented by a distinct vector. This paper introduces an innovative framework called LEGCF, which circumvents this parameter inefficiency by employing a mechanism of meta-embeddings and a learnable assignment matrix.

Problem Formulation and Proposed Solution

The conventional approach in GNN-based collaborative filtering embodies a fundamental defect inherent in classic collaborative filtering methods: parameter inefficiency stemming from extensive embedding tables. To address this, LEGCF leverages a parameter-efficient paradigm where it utilizes a smaller, shared pool of meta-embeddings. Each user/item is represented by an optimal combination of these meta-embeddings rather than being allocated a unique, separate vector. This approach significantly reduces the number of parameters, enhancing scalability.

A noteworthy novelty in LEGCF is the incorporation of an assignment matrix as an additional learnable component. This matrix enables the dynamic assignment of meta-embeddings to user/item entities, optimized via a semantic similarity constraint. Remarkably, the assignment matrix problem—typically a combinatorial challenge—is simplified by leveraging closed-form solutions derived from matrix pseudo-inverses. This facilitates its execution without invoking computationally intensive operations such as those associated with reinforcement learning approaches.

Numerical Results and Evaluation

The paper's experimental validation underscores the advantages of LEGCF. Conducted across three benchmark datasets, the results vividly demonstrate LEGCF's superior performance compared to state-of-the-art baselines concerning both parameter efficiency and recommendation accuracy. For instance, on the Gowalla dataset, the LEGCF framework achieves noteworthy improvements in NDCG and Recall at varying cutoff points, 10 and 20, under significantly reduced memory constraints compared to conventional full embedding tables.

This performance consistency indicates LEGCF's effective trade-off between model complexity and performance, particularly crucial for applications requiring real-time computation under constrained computational resources. The authors also explore hyperparameter tuning, showcasing LEGCF's robustness against variations in meta-embedding bucket size and assignment sparsity levels.

Theoretical and Practical Implications

The implications of this research are manifold. Theoretically, it extends the understanding of embedding optimization under graph structures, integrating the semantic relationships inherent in user-item interaction graphs with the parameter-efficient embedding framework. This affords opportunities to further refine graph-based collaborative filtering paradigms while maintaining a high degree of flexibility and minimal computational burden.

Practically, LEGCF serves as a valuable tool for enhancing the performance and scalability of recommender systems, potentially influencing large-scale applications in e-commerce and social media platforms where the system's ability to efficiently process and recommend at scale is paramount.

Future Directions

While the paper provides robust solutions, several avenues for future exploration exist. More dynamic and context-aware assignment strategies could further enhance embedding expressiveness. Moreover, extending LEGCF to encompass adaptive learning rates for the assignment matrix or exploring hybrid models integrating other neural components such as transformers could inspire new methodologies in collaborative filtering paradigms.

In conclusion, the paper presents a lucid strategy for advancing GNN-based recommender systems through lightweight embedding optimization. By addressing key inefficiencies head-on, LEGCF positions itself as a compelling solution within the evolving landscape of AI and machine learning-driven recommendations.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

GitHub

  1. GitHub - xurong-liang/LEGCF (4 stars)