Papers
Topics
Authors
Recent
Search
2000 character limit reached

Uncertainty in GNN Learning Evaluations: The Importance of a Consistent Benchmark for Community Detection

Published 10 May 2023 in cs.LG and cs.AI | (2305.06026v5)

Abstract: Graph Neural Networks (GNNs) have improved unsupervised community detection of clustered nodes due to their ability to encode the dual dimensionality of the connectivity and feature information spaces of graphs. Identifying the latent communities has many practical applications from social networks to genomics. Current benchmarks of real world performance are confusing due to the variety of decisions influencing the evaluation of GNNs at this task. To address this, we propose a framework to establish a common evaluation protocol. We motivate and justify it by demonstrating the differences with and without the protocol. The W Randomness Coefficient is a metric proposed for assessing the consistency of algorithm rankings to quantify the reliability of results under the presence of randomness. We find that by ensuring the same evaluation criteria is followed, there may be significant differences from the reported performance of methods at this task, but a more complete evaluation and comparison of methods is possible.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. Journal of machine learning research 13(2) (2012)
  2. In: International conference on machine learning, pp. 115–123. PMLR (2013)
  3. In: 2016 Annual Conference on Information Science and Systems (CISS), pp. 584–589. IEEE (2016)
  4. Tech. rep., Carnegie-mellon univ pittsburgh pa school of computer Science (1998)
  5. arXiv preprint arXiv:2003.00982 (2020)
  6. arXiv preprint arXiv:1912.09893 (2019)
  7. Field, A.P.: Kendall’s coefficient of concordance. Encyclopedia of Statistics in Behavioral Science 2, 1010–11 (2005)
  8. In: Proceedings of the third ACM conference on Digital libraries, pp. 89–98 (1998)
  9. In: International Conference on Machine Learning, pp. 4116–4126. PMLR (2020)
  10. In: proceedings of the 25th international conference on world wide web, pp. 507–517 (2016)
  11. arXiv preprint arXiv:2103.09430 (2021)
  12. IEEE Transactions on Knowledge and Data Engineering (2021)
  13. arXiv preprint arXiv:2103.14958 (2021)
  14. arXiv preprint arXiv:1609.02907 (2016)
  15. arXiv preprint arXiv:1611.07308 (2016)
  16. arXiv preprint arXiv:2005.08225 (2020)
  17. arXiv preprint arXiv:2211.12875 (2022)
  18. In: Proceedings of the ACM Web Conference 2022, pp. 1392–1403 (2022)
  19. Information Retrieval 3(2), 127–163 (2000)
  20. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 5145–5152. IEEE (2021)
  21. arXiv preprint arXiv:1902.06673 (2019)
  22. arXiv preprint arXiv:2007.08663 (2020)
  23. In: Proceedings of the 2020 genetic and evolutionary computation conference, pp. 533–541 (2020)
  24. arXiv preprint arXiv:2203.00112 (2022)
  25. The Journal of Machine Learning Research 22(1), 7459–7478 (2021)
  26. In: AAAI: DLG-AAAI’22 (2022)
  27. Salzberg, S.L.: On comparing classifiers: Pitfalls to avoid and a recommended approach. Data mining and knowledge discovery 1, 317–328 (1997)
  28. Schaeffer, S.E.: Graph clustering. Computer science review 1(1), 27–64 (2007)
  29. arXiv preprint arXiv:1811.05868 (2018)
  30. In: Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 990–998 (2008)
  31. In: ICLR 2021 Workshop on Geometrical and Topological Representation Learning (2021)
  32. arXiv preprint arXiv:2006.16904 (2020)
  33. ICLR (Poster) 2(3), 4 (2019)
  34. arXiv preprint arXiv:1906.06532 (2019)
  35. 10.48550/ARXIV.2206.07897. URL https://arxiv.org/abs/2206.07897
  36. Cambridge university press (1994)
  37. nature 393(6684), 440–442 (1998)
  38. In: 2013 IEEE 13th international conference on data mining, pp. 1151–1156. IEEE (2013)
  39. Neurocomputing 415, 295–316 (2020)
  40. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 386–394 (2021)
  41. arXiv preprint arXiv:2006.04131 (2020)
  42. Journal of artificial intelligence research 70, 409–472 (2021)
Citations (4)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 2 likes about this paper.

Sign Up for Free