On the Volatility of Shapley-Based Contribution Metrics in Federated Learning
Abstract: Federated learning (FL) is a collaborative and privacy-preserving Machine Learning paradigm, allowing the development of robust models without the need to centralize sensitive data. A critical challenge in FL lies in fairly and accurately allocating contributions from diverse participants. Inaccurate allocation can undermine trust, lead to unfair compensation, and thus participants may lack the incentive to join or actively contribute to the federation. Various remuneration strategies have been proposed to date, including auction-based approaches and Shapley-value-based methods, the latter offering a means to quantify the contribution of each participant. However, little to no work has studied the stability of these contribution evaluation methods. In this paper, we evaluate participant contributions in federated learning using gradient-based model reconstruction techniques with Shapley values and compare the round-based contributions to a classic data contribution measurement scheme. We provide an extensive analysis of the discrepancies of Shapley values across a set of aggregation strategies and examine them on an overall and a per-client level. We show that, between different aggregation techniques, Shapley values lead to unstable reward allocations among participants. Our analysis spans various data heterogeneity distributions, including independent and identically distributed (IID) and non-IID scenarios.
- Flower: A friendly federated learning research framework. arXiv preprint arXiv:2007.14390, 2020.
- Machine learning with adversaries: Byzantine tolerant gradient descent. Advances in neural information processing systems, 30, 2017.
- On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
- An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
- Data shapley: Equitable valuation of data for machine learning. In International conference on machine learning, pages 2242–2251. PMLR, 2019.
- Measuring the effects of non-identical data distribution for federated visual classification, 2019.
- Fastshap: Real-time shapley value estimation, 2022.
- Towards efficient data valuation based on the shapley value. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1167–1176. PMLR, 2019.
- Secure, privacy-preserving and federated machine learning in medical imaging. Nature Machine Intelligence, 2(6):305–311, 2020.
- Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885–1894. PMLR, 2017.
- Trustworthy ai: From principles to practices, 2022a.
- Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pages 965–978. IEEE, 2022b.
- Advances, challenges and opportunities in creating data for trustworthy ai. Nature Machine Intelligence, 4(8):669–677, 2022.
- Communication-efficient learning of deep networks from decentralized data, 2017.
- Cooperative game theory and its application to natural, environmental, and water resource issues: 3. application to water resources. Application to Water Resources (November 2006). World Bank Policy Research Working Paper, (4074), 2006.
- Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
- Adaptive federated optimization, 2021.
- Rewarding high-quality data via influence functions. arXiv preprint arXiv:1908.11598, 2019.
- The future of digital health with federated learning. NPJ digital medicine, 3(1):119, 2020.
- Motivating workers in federated learning: A stackelberg game perspective. IEEE Networking Letters, 2(1):23–27, 2019.
- Lloyd S Shapley. Notes on the n-person game—ii: The value of an n-person game. 1951.
- Profit allocation for federated learning. In 2019 IEEE International Conference on Big Data (Big Data), pages 2577–2586. IEEE, 2019.
- Incentive mechanisms for federated learning: From economic and game theoretic perspective, 2021.
- Measure contribution of participants in federated learning. In 2019 IEEE international conference on big data (Big Data), pages 2597–2604. IEEE, 2019.
- A principled approach to data valuation for federated learning, 2020.
- Byzantine-robust distributed learning: Towards optimal statistical rates, 2021.
- Fmore: An incentive scheme of multi-dimensional auction for federated learning in mec. In 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS). IEEE, November 2020. doi: 10.1109/icdcs47774.2020.00094.
- Hierarchically fair federated learning. arXiv preprint arXiv:2004.10386, 2020.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.