Papers
Topics
Authors
Recent
Search
2000 character limit reached

Toward a Spectral Foundation Model: An Attention-Based Approach with Domain-Inspired Fine-Tuning and Wavelength Parameterization

Published 27 Jun 2023 in astro-ph.IM, astro-ph.GA, and astro-ph.SR | (2306.15703v1)

Abstract: Astrophysical explorations are underpinned by large-scale stellar spectroscopy surveys, necessitating a paradigm shift in spectral fitting techniques. Our study proposes three enhancements to transcend the limitations of the current spectral emulation models. We implement an attention-based emulator, adept at unveiling long-range information between wavelength pixels. We leverage a domain-specific fine-tuning strategy where the model is pre-trained on spectra with fixed stellar parameters and variable elemental abundances, followed by fine-tuning on the entire domain. Moreover, by treating wavelength as an autonomous model parameter, akin to neural radiance fields, the model can generate spectra on any wavelength grid. In the case with a training set of O(1000), our approach exceeds current leading methods by a factor of 5-10 across all metrics.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (33)
  1. The GALAH Survey: non-LTE departure coefficients for large spectroscopic surveys. Astronomy and Astrophysics, 642:A62, October 2020. doi: 10.1051/0004-6361/202038650.
  2. Elemental abundance trends in the Galactic thin and thick disks as traced by nearby F and G dwarf stars. Astronomy and Astrophysics, 410:527–551, November 2003. doi: 10.1051/0004-6361:20031213.
  3. Solar oxygen abundance. Monthly Notices of the Royal Astronomical Society, 508(2):2236–2253, jul 2021. doi: 10.1093/mnras/stab2160. URL https://doi.org/10.1093%2Fmnras%2Fstab2160.
  4. The GALAH+ Survey: Third Data Release. arXiv e-prints, art. arXiv:2011.02505, November 2020.
  5. An application of deep learning in the analysis of stellar spectra. Monthly Notices of the Royal Astronomical Society, 475(3):2978–2993, dec 2017. doi: 10.1093/mnras/stx3298. URL https://doi.org/10.1093%2Fmnras%2Fstx3298.
  6. Fuhrmann, K. Nearby stars of the Galactic disk and halo. Astronomy and Astrophysics, 338:161–183, October 1998.
  7. Non-LTE radiative transfer with turbospectrum. Astronomy and Astrophysics, 669:A43, jan 2023. doi: 10.1051/0004-6361/202243673. URL https://doi.org/10.1051%2F0004-6361%2F202243673.
  8. The Gaia-ESO Public Spectroscopic Survey. The Messenger, 147:25–31, March 2012.
  9. Transformer in transformer. Advances in Neural Information Processing Systems, 34:15908–15919, 2021.
  10. On the effectiveness of adapter-based tuning for pretrained language model adaptation. arXiv preprint arXiv:2106.03164, 2021.
  11. LoRA: Low-Rank Adaptation of Large Language Models. arXiv e-prints, art. arXiv:2106.09685, June 2021. doi: 10.48550/arXiv.2106.09685.
  12. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34:1022–1035, 2021.
  13. Kurucz, R. SYNTHE Spectrum Synthesis Programs and Line Data. SYNTHE Spectrum Synthesis Programs and Line Data. Kurucz CD-ROM No. 18. Cambridge, 18, January 1993.
  14. Kurucz, R. L. Model atmospheres for G, F, A, B, and O stars. The Astrophysical Journal Supplement Series, 40:1–340, May 1979. doi: 10.1086/190589.
  15. Kurucz, R. L. ATLAS12, SYNTHE, ATLAS9, WIDTH9, et cetera. Memorie della Societa Astronomica Italiana Supplementi, 8:14, 2005.
  16. Kurucz, R. L. ATLAS12: Opacity sampling model atmosphere program. Astrophysics Source Code Library, March 2013.
  17. satlas: spherical versions of the atlas stellar atmosphere program. Astronomy and Astrophysics, 491(2):633–641, November 2008. doi: 10.1051/0004-6361:200810578.
  18. Deep learning of multi-element abundances from high-resolution spectroscopic data. Monthly Notices of the Royal Astronomical Society, nov 2018. doi: 10.1093/mnras/sty3217. URL https://doi.org/10.1093%2Fmnras%2Fsty3217.
  19. The first data release (DR1) of the LAMOST regular survey. Research in Astronomy and Astrophysics, 15(8):1095, August 2015. doi: 10.1088/1674-4527/15/8/002.
  20. The Apache Point Observatory Galactic Evolution Experiment (APOGEE). The Astronomical Journal, 154:94, September 2017. doi: 10.3847/1538-3881/aa784d.
  21. Nerf: Representing scenes as neural radiance fields for view synthesis, 2020.
  22. The cannon: A data-driven approach to stellar label determination. The Astrophysical Journal, 808(1):16, jul 2015. doi: 10.1088/0004-637x/808/1/16. URL https://doi.org/10.1088%2F0004-637x%2F808%2F1%2F16.
  23. Cycle-StarNet: Bridging the Gap between Theory and Data by Leveraging Large Data Sets. The Astrophysical Journal, 906(2):130, January 2021. doi: 10.3847/1538-4357/abca96.
  24. Image transformer. In International conference on machine learning, pp. 4055–4064. PMLR, 2018.
  25. Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations. arXiv e-prints, art. arXiv:2111.13152, November 2021. doi: 10.48550/arXiv.2111.13152.
  26. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155, 2018.
  27. Implicit neural representations with periodic activation functions, 2020.
  28. Zeta-payne: A fully automated spectrum analysis algorithm for the milky way mapper program of the SDSS-v survey. The Astronomical Journal, 163(5):236, apr 2022. doi: 10.3847/1538-3881/ac5f49. URL https://doi.org/10.3847%2F1538-3881%2Fac5f49.
  29. The payne: Self-consistent ab initio fitting of stellar spectra. The Astrophysical Journal, 879(2):69, jul 2019. doi: 10.3847/1538-4357/ab2331. URL https://doi.org/10.3847%2F1538-4357%2Fab2331.
  30. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  31. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020.
  32. Stellar labels for hot stars from low-resolution spectra. I. The HotPayne method and results for 330 000 stars from LAMOST DR6. Astronomy and Astrophysics, 662:A66, June 2022. doi: 10.1051/0004-6361/202141570.
  33. ResiDual: Transformer with Dual Residual Connections. arXiv e-prints, art. arXiv:2304.14802, April 2023. doi: 10.48550/arXiv.2304.14802.
Citations (3)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up