Papers
Topics
Authors
Recent
Search
2000 character limit reached

Statistical inverse learning problems with random observations

Published 23 Dec 2023 in math.ST, stat.ML, and stat.TH | (2312.15341v1)

Abstract: We provide an overview of recent progress in statistical inverse problems with random experimental design, covering both linear and nonlinear inverse problems. Different regularization schemes have been studied to produce robust and stable solutions. We discuss recent results in spectral regularization methods and regularization by projection, exploring both approaches within the context of Hilbert scales and presenting new insights particularly in regularization by projection. Additionally, we overview recent advancements in regularization using convex penalties. Convergence rates are analyzed in terms of the sample size in a probabilistic sense, yielding minimax rates in both expectation and probability. To achieve these results, the structure of reproducing kernel Hilbert spaces is leveraged to establish minimax rates in the statistical learning setting. We detail the assumptions underpinning these key elements of our proofs. Finally, we demonstrate the application of these concepts to nonlinear inverse problems in pharmacokinetic/pharmacodynamic (PK/PD) models, where the task is to predict changes in drug concentrations in patients.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (86)
  1. Designing truncated priors for direct and inverse Bayesian problems. Electronic Journal of Statistics, 16(1):158–200, 2022.
  2. Learning the optimal Tikhonov regularizer for inverse problems. Advances in Neural Information Processing Systems, 34:25205–25216, 2021.
  3. Nachman Aronszajn. Theory of reproducing kernels. Transactions of the American Mathematical Society, 68:337–404, 1950.
  4. On regularization algorithms in learning theory. Journal of Complexity, 23(1):52–72, 2007.
  5. Trust your source: quantifying source condition elements for variational regularisation methods. arXiv preprint arXiv:2303.00696, 2023.
  6. Modern regularization methods for inverse problems. Acta numerica, 27:1–111, 2018.
  7. Rajendra Bhatia. Matrix analysis. In Graduate Texts in Mathematics, volume 169. Springer-Verlag, New York, 1997.
  8. Convergence rates of general regularization methods for statistical inverse problems and applications. SIAM Journal on Numerical Analysis, 45(6):2610–2636, 2007.
  9. Early stopping for statistical inverse problems via truncated SVD estimation. Electronic Journal of Statistics, 12(2), 2018.
  10. Optimal adaptation for early stopping in statistical inverse problems. SIAM/ASA Journal on Uncertainty Quantification, 6(3):1043–1075, 2018.
  11. Discrepancy principle for statistical inverse problems with application to conjugate gradient iteration. Inverse problems, 28(11):115011, 2012.
  12. Lepskii principle in supervised learning. arXiv preprint arXiv:1905.10764, 2019.
  13. Optimal rates for regularization of statistical inverse learning problems. Foundations of Computational Mathematics, 18(4):971–1013, 2018.
  14. Convex regularization in statistical inverse learning problems. Inverse Problems and Imaging, 17(6):1193–1225, 2023.
  15. A variational reconstruction method for undersampled dynamic x-ray tomography based on physical motion models. Inverse Problems, 33(12):124008, 2017.
  16. Large noise in variational regularization. Transactions of Mathematics and its Applications, 2(1):1–45, 2018.
  17. Convergence rates of convex variational regularization. Inverse problems, 20(5):1411, 2004.
  18. Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3):331–368, 2007.
  19. Cross-validation based adaptation for regularization operators in learning theory. Analysis and Applications, 8(2):161–183, 2010.
  20. Laurent Cavalier. Inverse problems in statistics. In Inverse Problems and High-Dimensional Estimation: Stats in the Château Summer School, August 31-September 4, 2009, pages 3–96. Springer, 2011.
  21. Oracle inequalities for inverse problems. The Annals of Statistics, 30(3):843–874, 2002.
  22. Sharp adaptation for inverse problems with random noise. Probability Theory and Related Fields, 123(3):323–354, 2002.
  23. On learning the optimal regularization parameter in inverse problems. arXiv preprint arXiv:2311.15845, 2023.
  24. Statistical approach to some ill-posed problems for linear partial differential equations. Probability theory and related fields, 113(3):421–441, 1999.
  25. A semismooth newton method for nonlinear parameter identification problems with impulsive noise. SIAM journal on imaging sciences, 5(2):505–536, 2012.
  26. Best choices for regularization parameters in learning theory: on the bias-variance problem. Foundations of Computational Mathematics, 2(4):413–428, 2002.
  27. On the mathematical foundations of learning. Bull. Am. Math. Soc., 39(1):1–49, 2002.
  28. Learning theory: An approximation theory viewpoint, volume 24. Cambridge Monographs on Applied and Computational Mathematics, Cambridge University Press, Cambridge, UK, 2007.
  29. Ingrid Daubechies. Ten lectures on wavelets. Philadelphia, PA, USA: SIAM, 1992.
  30. A machine learning approach to optimal Tikhonov regularization I: affine manifolds. Analysis and Applications, 20(02):353–400, 2022.
  31. Adaptive kernel methods using the balancing principle. Foundations of Computational Mathematics, 10:455–479, 2010.
  32. Discretization error analysis for Tikhonov regularization. Analysis and Applications, 4(01):81–99, 2006.
  33. Learning from examples as an inverse problem. Journal of Machine Learning Research, 6(5), 2005.
  34. Learning from examples as an inverse problem. J. Mach. Learn. Res., 6:883–904, 2005.
  35. David L. Donoho. Nonlinear solution of linear inverse problems by wavelet-vaguelette decomposition. Applied and computational harmonic analysis, 2(2):101–126, 1995.
  36. Regularization of inverse problems, volume 375. Mathematics and its applications, Kluwer Academic Publishers Group, Dordrecht, The Netherlands, 1996.
  37. Consistency of Bayesian inference with Gaussian process priors in an elliptic inverse problem. Inverse Problems, 36(8):085001, 2020.
  38. Convergence rates of spectral regularization methods: A comparison between ill-posed inverse problems and statistical kernel learning. SIAM Journal on Numerical Analysis, 58(6):3504–3529, 2020.
  39. Bayesian linear inverse problems in regularity scales. Annales de l’Institut Henri Poincaré Probabilités et Statistiques, 56(3):2081–2107, 2020.
  40. A convergence analysis of the Landweber iteration for nonlinear ill-posed problems. Numerische Mathematik, 72(1):21–37, 1995.
  41. Beyond the Bakushinkii veto: regularising linear inverse problems without knowing the noise distribution. Numerische Mathematik, 145:581–603, 2020.
  42. Nonparametric goodness-of-fit testing for parametric covariate models in pharmacometric analyses. CPT: Pharmacometrics & Systems Pharmacology, 10(6):564–576, 2021.
  43. Tapio Helin. Least squares approximations in linear statistical inverse learning problems. arXiv preprint arXiv:2211.12121, 2022.
  44. Bernd Hofmann. Approximate source conditions in Tikhonov-Phillips regularization and consequences for inverse problems with multiplication operators. Mathematical Methods in the Applied Sciences, 29(3):351–371, 2006.
  45. Analysis of profile functions for general linear regularization methods. SIAM Journal on Numerical Analysis, 45(3):1122–1141, 2007.
  46. Nonlinear Tikhonov regularization in Hilbert scales for inverse boundary value problems with random noise. Inverse Problems and Imaging, 2(271-290):96, 2008.
  47. Inverse problems with poisson data: statistical regularization theory, applications and algorithms. Inverse Problems, 32(9):093001, 2016.
  48. On the regularizing property of stochastic gradient descent. Inverse Problems, 35(1):015004, 2018.
  49. Discretization effects in statistical inverse problems. Journal of complexity, 7(1):1–34, 1991.
  50. Statistical inverse problems: discretization, model reduction and inverse crimes. Journal of Computational and Applied Mathematics, 198(2):493–504, 2007.
  51. Iterative regularization methods for nonlinear ill-posed problems. Walter de Gruyter, 2008.
  52. O. V. Lepskii. On a problem of adaptive estimation in Gaussian white noise. Theory of Probability & Its Applications, 35(3):454–466, 1991.
  53. Spectral algorithms for supervised learning. Neural Computation, 20(7):1873–1897, 2008.
  54. Balancing principle in supervised learning for a general regularization scheme. Applied and Computational Harmonic Analysis, 48(1):123–148, 2020.
  55. Mark A. Lukas. Comparisons of parameter choice methods for regularization with discrete noisy data. Inverse Problems, 14(1):161, 1998.
  56. Statistical inverse estimation in Hilbert scales. SIAM Journal on Applied Mathematics, 56(5):1424–1444, 1996.
  57. Regularization in kernel learning. The Annals of Statistics, 38(1):526 – 565, 2010.
  58. Consistent inversion of noisy non-abelian x-ray transforms. Communications on Pure and Applied Mathematics, 74(5):1045–1099, 2021.
  59. Stochastic gradient descent in Hilbert scales: smoothness, preconditioning and earlier stopping. arXiv preprint arXiv:2006.10840, 2020.
  60. Frank Natterer. Error bounds for Tikhonov regularization in Hilbert scales. Applicable Analysis, 18(1-2):29–37, 1984.
  61. Richard Nickl. Bayesian non-linear statistical inverse problems. Zurich Lectures in Advanced Mathematics. EMS Press, 2023.
  62. Convergence rates for penalized least squares estimators in PDE constrained regression problems. SIAM/ASA Journal on Uncertainty Quantification, 8(1):374–413, 2020.
  63. Convergence rates for regularized solutions of integral equations from discrete noisy data. The Annals of Statistics, 17:556–572, 1989.
  64. Lumping of physiologically-based pharmacokinetic models and a mechanistic derivation of classical compartmental models. Journal of Pharmacokinetics and Pharmacodynamics, 37:365–405, 2010.
  65. Abhishake Rastogi. Tikhonov regularization with oversmoothing penalty for nonlinear statistical inverse problems. Communications on Pure and Applied Analysis, 19(8):4111–4126, 2020.
  66. Abhishake Rastogi. Statistical inverse problems in Hilbert scales. arXiv preprint arXiv:2208.13289, 2022.
  67. Convergence analysis of Tikhonov regularization for non-linear statistical inverse problems. Electronic Journal of Statistics, 14(2):2798–2841, 2020.
  68. Inverse learning in Hilbert scales. Machine Learning, 112:2469–2499, 2023.
  69. Markus Reiß. Asymptotic equivalence for nonparametric regression with multivariate and random design. Annals of Statistics, 36:1957–1982, 2008.
  70. Theory of reproducing kernels and applications. Springer, Singapore, 2016.
  71. Jun Shao. Mathematical statistics. Springer Science & Business Media, 2003.
  72. Shannon sampling II: Connections to learning theory. Applied and Computational Harmonic Analysis, 19(3):285–302, 2005.
  73. Learning theory estimates via integral operators and their approximations. Constructive approximation, 26(2):153–172, 2007.
  74. Bernhard Stankewitz. Smoothed residual stopping for statistical inverse problems via truncated SVD estimation. Electronic Journal of Statistics, 14:3396–3428, 2020.
  75. Support vector machines. Information Science and Statistics, Springer, 2008.
  76. Optimal rates for regularized least squares regression. In S. Dasgupta and A. Klivans, editors, Proceedings of the 22nd Annual Conference on Learning Theory, pages 79–93, 2009.
  77. Andrew M. Stuart. Inverse problems: a Bayesian perspective. Acta numerica, 19:451–559, 2010.
  78. Andrei N. Tikhonov. Regularization of incorrectly posed problems. In Soviet Mathematics Doklady, volume 4, pages 1624–1627, 1963.
  79. Andrei N. Tikhonov. Solution of incorrectly formulated problems and the regularization method. Soviet Mathematics Doklady, 4:1035–1038, 1963.
  80. Tullio Valent. Boundary value problems of finite elasticity – local theorems on existence, uniqueness, and analytic dependence on data, volume 31. Springer Tracts in Natural Philosophy, 1987.
  81. Aad W. van der Vaart and Jon A. Wellner. Weak convergence and empirical processes. Springer Series in Statistics, Springer-Verlag, New York, 1996. With Applications to Statistics.
  82. Vladimir Vapnik. Statistical learning theory, volume 1. Wiley, New York, 1998.
  83. Grace Wahba. Practical approximate solutions to linear operator equations when the data are noisy. SIAM journal on numerical analysis, 14(4):651–667, 1977.
  84. Optimal convergence rates for Tikhonov regularization in besov spaces. SIAM Journal on Numerical Analysis, 58(1):21–47, 2020.
  85. Convergence analysis of (statistical) inverse problems under conditional stability estimates. Inverse Problems, 36(1):015004, 2019.
  86. On early stopping in gradient descent learning. Constructive Approximation, 26:289–315, 2007.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free