Papers
Topics
Authors
Recent
Search
2000 character limit reached

A Rate-Distortion Analysis for Composite Sources Under Subsource-Dependent Fidelity Criteria

Published 20 May 2024 in cs.IT and math.IT | (2405.11818v2)

Abstract: A composite source, consisting of multiple subsources and a memoryless switch, outputs one symbol at a time from the subsource selected by the switch. If some data should be encoded more accurately than other data from an information source, the composite source model is suitable because in this model different distortion constraints can be put on the subsources. In this context, we propose subsource-dependent fidelity criteria for composite sources and use them to formulate a rate-distortion problem. We solve the problem and obtain a single-letter expression for the rate-distortion function. Further rate-distortion analysis characterizes the performance of classify-then-compress (CTC) coding, which is frequently used in practice when subsource-dependent fidelity criteria are considered. Our analysis shows that CTC coding generally has performance loss relative to optimal coding, even if the classification is perfect. We also identify the cause of the performance loss, that is, class labels have to be reproduced in CTC coding. Last but not least, we show that the performance loss is negligible for asymptotically small distortion if CTC coding is appropriately designed and some mild conditions are satisfied.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (41)
  1. C. Shannon, “A mathematical theory of communication,” Bell Syst. Tech. J., vol. 27, pp. 379–423 and 623–656, Jul. 1948.
  2. ——, “Coding theorems for a discrete source with a fidelity criterion,” IRE Conv. Rec., 1959.
  3. L. Rabiner and R. Schafer, “Introduction to digital speech processing,” Found. Trends Signal Process., vol. 1, no. 1-2, pp. 1–194, Dec. 2007.
  4. C. Christopoulos, A. Skodras, and T. Ebrahimi, “The JPEG2000 still image coding system: An overview,” IEEE Trans. Consum. Electron., vol. 46, no. 4, pp. 1103–1127, Nov. 2000.
  5. R. Gray, “Conditional rate-distortion theory,” Information System Laboratory, Stanford University, Stanford, California, Tech. Rep. 6502-2, Oct. 1972.
  6. ——, “A new class of lower bounds to information rates of stationary sources via conditional rate-distortion functions,” IEEE Trans. Inf. Theory, vol. 19, no. 4, pp. 480–489, Jul. 1973.
  7. A. Wyner and J. Ziv, “The rate-distortion function for source coding with side information at the decoder,” IEEE Trans. Inf. Theory, vol. 22, no. 1, pp. 1–10, Jan. 1976.
  8. T. Berger, “The source coding game,” IEEE Trans. Inf. Theory, vol. 17, no. 1, pp. 71–76, Jan. 1971.
  9. T. Linder, R. Zamir, and K. Zeger, “On source coding with side-information-dependent distortion measures,” IEEE Trans. Inf. Theory, vol. 46, no. 7, pp. 2697–2704, Nov. 2000.
  10. E. Martinian, G. Wornell, and R. Zamir, “Source coding with distortion side information,” IEEE Trans. Inf. Theory, vol. 54, no. 10, pp. 4638–4665, Oct. 2008.
  11. R. Dobrushin and B. Tsybakov, “Information transmission with additional noise,” IRE Trans. Inf. Theory, vol. 8, no. 5, pp. 293–304, Sep. 1962.
  12. J. Wolf and J. Ziv, “Transmission of noisy information to a noisy receiver with minimum distortion,” IEEE Trans. Inf. Theory, vol. 16, no. 4, pp. 406–411, Jul. 1970.
  13. H. Witsenhausen, “Indirect rate distortion problems,” IEEE Trans. Inf. Theory, vol. 26, no. 5, pp. 518–521, Sep. 1980.
  14. R. Fontana, “Universal codes for a class of composite sources (corresp.),” IEEE Trans. Inf. Theory, vol. 26, no. 4, pp. 480–482, Jul. 1980.
  15. H. Rosenthal and J. Binia, “On the epsilon entropy of mixed random variables,” IEEE Trans. Inf. Theory, vol. 34, no. 5, pp. 1110–1114, Sep. 1988.
  16. A. Gyorgy, T. Linder, and K. Zeger, “On the rate-distortion function of random vectors and stationary sources with mixed distributions,” IEEE Trans. Inf. Theory, vol. 45, no. 6, pp. 2110–2115, Sep. 1999.
  17. C. Weidmann and M. Vetterli, “Rate distortion behavior of sparse sources,” IEEE Trans. Inf. Theory, vol. 58, no. 8, pp. 4969–4992, Aug. 2012.
  18. R. Gray and L. Davisson, “Source coding theorems without the ergodic assumption,” IEEE Trans. Inf. Theory, vol. 20, no. 4, pp. 502–516, Jul. 1974.
  19. H. Kalveram and P. Meissner, “Itakura-Saito clustering and rate distortion functions for a composite source model of speech,” Signal Processing, vol. 18, no. 2, pp. 195–216, Oct. 1989.
  20. J. Gibson and J. Hu, “Rate distortion bounds for voice and video,” Found. Trends Commun. Inf. Theory, vol. 10, no. 4, pp. 379–514, Feb. 2014.
  21. J. Gibson, “Rate distortion functions and rate distortion function lower bounds for real-world sources,” Entropy, vol. 19, no. 11, pp. 1–22, Nov. 2017.
  22. L. Rabiner, “A tutorial on hidden Markov models and selected applications in speech recognition,” Proc. IEEE, vol. 77, no. 2, pp. 257–286, Feb. 1989.
  23. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances Neural Inf. Process. Syst., vol. 27, 2014, pp. 1–9.
  24. D. P. Kingma and M. Welling, “Auto-encoding variational Bayes,” in Proc. Int. Conf. Learn. Representations, 2014, pp. 1–14.
  25. F.-A. Croitoru, V. Hondru, R. T. Ionescu, and M. Shah, “Diffusion models in vision: A survey,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 9, pp. 10 850–10 869, Sep. 2023.
  26. D. Gündüz, Z. Qin, I. E. Aguerri, H. S. Dhillon, Z. Yang, A. Yener, K. K. Wong, and C.-B. Chae, “Beyond transmitting bits: Context, semantics, and task-oriented communications,” IEEE J. Sel. Areas Commun., vol. 41, no. 1, pp. 5–41, Jan. 2023.
  27. A. Kipnis, S. Rini, and A. J. Goldsmith, “The rate-distortion risk in estimation from compressed data,” IEEE Trans. Inf. Theory, vol. 67, no. 5, pp. 2910–2924, May 2021.
  28. J. Liu, W. Zhang, and H. V. Poor, “A rate-distortion framework for characterizing semantic information,” in Proc. IEEE Int. Symp. Inf. Theory (ISIT), 2021, pp. 2894–2899.
  29. J. Liu, S. Shao, W. Zhang, and H. V. Poor, “An indirect rate-distortion characterization for semantic sources: General model and the case of Gaussian observation,” IEEE Trans. Commun., vol. 70, no. 9, pp. 5946–5959, Sep. 2022.
  30. T. Guo, Y. Wang, J. Han, H. Wu, B. Bai, and W. Han, “Semantic compression with side information: A rate-distortion perspective,” Aug. 2022. [Online]. Available: https://arxiv.org/abs/2208.06094
  31. Y. Wang, T. Guo, B. Bai, and W. Han, “The estimation-compression separation in semantic communication systems,” in Proc. IEEE Inf. Theory Workshop (ITW), Nov. 2022, pp. 315–320.
  32. Y. Xiao, X. Zhang, Y. Li, G. Shi, and T. Başar, “Rate-distortion theory for strategic semantic communication,” in Proc. IEEE Inf. Theory Workshop (ITW), Nov. 2022, pp. 279–284.
  33. P. Stavrou and M. Kountouris, “The role of fidelity in goal-oriented semantic communication: A rate distortion approach,” IEEE Trans. Commun., vol. 71, no. 7, pp. 3918–3931, Jul. 2023.
  34. A. Das, A. DeJaco, S. Manjunath, A. Ananthapadmanabhan, J. Huang, and E. Choy, “Multimode variable bit rate speech coding: an efficient paradigm for high-quality low-rate representation of speech signal,” in Proc. IEEE Int. Conf. Acoust., Speech, and Signal Process. (ICASSP), Mar. 1999, pp. 2307–2310.
  35. H. Wang, Q. Li, H. Sun, Z. Chen, Y. Hao, J. Peng, Z. Yuan, J. Fu, and Y. Jiang, “VaBUS: Edge-cloud real-time video analytics via background understanding and subtraction,” IEEE J. Sel. Areas Commun., vol. 41, no. 1, pp. 90–106, Jan. 2023.
  36. R. Gray, “Gauss mixture vector quantization,” in Proc. IEEE Int. Conf. Acoust., Speech and Signal Process. (ICASSP), 2001, pp. 1769–1772.
  37. A. Subramaniam and B. Rao, “PDF optimized parametric vector quantization of speech line spectral frequencies,” IEEE Trans. Speech Audio Process., vol. 11, no. 2, pp. 130–142, Mar. 2003.
  38. D. Zhao, J. Samuelsson, and M. Nilsson, “On entropy-constrained vector quantization using Gaussian mixture models,” IEEE Trans. Commun., vol. 56, no. 12, pp. 2094–2104, Dec. 2008.
  39. S. Arimoto, “An algorithm for computing the capacity of arbitrary discrete memoryless channels,” IEEE Trans. Inf. Theory, vol. 18, no. 1, pp. 14–20, Jan. 1972.
  40. R. Blahut, “Computation of channel capacity and rate-distortion functions,” IEEE Trans. Inf. Theory, vol. 18, no. 4, pp. 460–473, Jul. 1972.
  41. P. Elias, “Universal codeword sets and representations of the integers,” IEEE Trans. Inf. Theory, vol. 21, no. 2, pp. 194–203, Mar. 1975.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

Sign Up for Free