Papers
Topics
Authors
Recent
Search
2000 character limit reached

Switchable Decision: Dynamic Neural Generation Networks

Published 7 May 2024 in cs.CL, cs.AI, and cs.LG | (2405.04513v1)

Abstract: Auto-regressive generation models achieve competitive performance across many different NLP tasks such as summarization, question answering, and classifications. However, they are also known for being slow in inference, which makes them challenging to deploy in real-time applications. We propose a switchable decision to accelerate inference by dynamically assigning computation resources for each data instance. Automatically making decisions on where to skip and how to balance quality and computation cost with constrained optimization, our dynamic neural generation networks enforce the efficient inference path and determine the optimized trade-off. Experiments across question answering, summarization, and classification benchmarks show that our method benefits from less computation cost during inference while keeping the same accuracy. Extensive experiments and ablation studies demonstrate that our method can be general, effective, and beneficial for many NLP tasks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding. arXiv preprint arXiv:2310.05424, 2023.
  2. Skip rnn: Learning to skip state updates in recurrent neural networks. arXiv preprint arXiv:1708.06834, 2017.
  3. Learning efficient object detection models with knowledge distillation. Advances in neural information processing systems, 30, 2017.
  4. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pp.  177–190. Springer, 2005.
  5. One model, multiple modalities: A sparsely activated approach for text, sound, image, video and code. arXiv preprint arXiv:2205.06126, 2022.
  6. Bert: pre-training of deep bidirectional transformers for language understanding. arxiv. arXiv preprint arXiv:1810.04805, 2018.
  7. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
  8. Unified language model pre-training for natural language understanding and generation. Advances in Neural Information Processing Systems, 32, 2019.
  9. Reducing transformer depth on demand with structured dropout. arXiv preprint arXiv:1909.11556, 2019.
  10. Bayesian attention modules. arXiv preprint arXiv:2010.10604, 2020.
  11. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.
  12. Fantastic rewards and how to tame them: A case study on reward learning for task-oriented dialogue systems. arXiv preprint arXiv:2302.10342, 2023.
  13. Speed reading: Learning to read forbackward via shuttle. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.  4439–4448, 2018.
  14. Lossless acceleration for seq2seq generation with aggressive decoding. arXiv preprint arXiv:2205.10350, 2022.
  15. Compressing bert: Studying the effects of weight pruning on transfer learning. arXiv preprint arXiv:2002.08307, 2020.
  16. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015a.
  17. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015b.
  18. Neural speed reading with structural-jump-lstm. arXiv preprint arXiv:1904.00761, 2019.
  19. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015.
  20. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2(7), 2015.
  21. Token dropping for efficient bert pretraining. arXiv preprint arXiv:2203.13240, 2022.
  22. Distilling knowledge from reader to retriever for question answering. In ICLR 2021, 9th International Conference on Learning Representations, 2021.
  23. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351, 2019.
  24. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  25. Albert: A lite bert for self-supervised learning of language representations. ArXiv, abs/1909.11942, 2020.
  26. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019.
  27. Dq-bart: Efficient sequence-to-sequence model via joint distillation and quantization. arXiv preprint arXiv:2203.11239, 2022.
  28. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 human language technology conference of the North American chapter of the association for computational linguistics, pp.  150–157, 2003.
  29. Fixed point quantization of deep convolutional networks. In International conference on machine learning, pp.  2849–2858. PMLR, 2016.
  30. Runtime neural pruning. Advances in neural information processing systems, 30, 2017.
  31. Fastbert: a self-distilling bert with adaptive inference time. arXiv preprint arXiv:2004.02178, 2020a.
  32. The microsoft toolkit of multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:2002.07972, 2020b.
  33. Fusedream: Training-free text-to-image generation with improved clip+ gan space optimization. arXiv preprint arXiv:2112.01573, 2021.
  34. Text summarization with pretrained encoders. arXiv preprint arXiv:1908.08345, 2019.
  35. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692, 2019a.
  36. Roberta: A robustly optimized bert pretraining approach. arXiv e-prints, pp.  arXiv–1907, 2019b.
  37. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, pp.  55–60, 2014.
  38. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.
  39. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  40. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.
  41. Squad: 100,000+ questions for machine comprehension of text. Empirical Methods in Natural Language Processing (EMNLP), 2016.
  42. Know what you don’t know: Unanswerable questions for squad. Annual Meetings of the Association for Computational Linguistics (ACL), 2018.
  43. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 35:17456–17472, 2022.
  44. The right tool for the job: Matching model and instance complexities. arXiv preprint arXiv:2004.07453, 2020.
  45. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017.
  46. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.  8815–8821, 2020.
  47. Pre-trained summarization distillation. arXiv preprint arXiv:2010.13002, 2020.
  48. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp.  1631–1642, 2013.
  49. Mobilebert: a compact task-agnostic bert for resource-limited devices. arXiv preprint arXiv:2004.02984, 2020.
  50. Pouf: Prompt-oriented unsupervised fine-tuning for large pre-trained models. In International Conference on Machine Learning, pp.  33816–33832. PMLR, 2023.
  51. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  52. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018a.
  53. Skipnet: Learning dynamic routing in convolutional networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp.  409–424, 2018b.
  54. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL-HLT, 2017a.
  55. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426, 2017b.
  56. Blockdrop: Dynamic inference paths in residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  8817–8826, 2018.
  57. Deebert: Dynamic early exiting for accelerating bert inference. arXiv preprint arXiv:2004.12993, 2020.
  58. Berxit: Early exiting for bert with better fine-tuning and extension to regression. In Proceedings of the 16th conference of the European chapter of the association for computational linguistics: Main Volume, pp.  91–104, 2021.
  59. Regularizing a model-based policy stationary distribution to stabilize offline reinforcement learning. In International Conference on Machine Learning, pp.  24980–25006. PMLR, 2022a.
  60. A unified framework for alternating offline model training and policy learning. Advances in Neural Information Processing Systems, 35:17216–17232, 2022b.
  61. Preference-grounded token-level guidance for language model fine-tuning. Advances in Neural Information Processing Systems, 36, 2024.
  62. Fast and accurate text classification: Skimming, rereading and early stopping. 2018.
  63. Are all layers created equal? 2019.
  64. Bayesian attention belief networks. In International Conference on Machine Learning, pp.  12413–12426. PMLR, 2021a.
  65. Alignment attention by matching key and query distributions. Advances in Neural Information Processing Systems, 34:13444–13457, 2021b.
  66. Knowing more about questions can help: Improving calibration in question answering. arXiv preprint arXiv:2106.01494, 2021c.
  67. Learning with different amounts of annotation: From zero to many labels. arXiv preprint arXiv:2109.04408, 2021d.
  68. Passage-mask: A learnable regularization strategy for retriever-reader models. arXiv preprint arXiv:2211.00915, 2022a.
  69. Allsh: Active learning guided by local sensitivity and hardness. arXiv preprint arXiv:2205.04980, 2022b.
  70. Language rectified flow: Advancing diffusion language generation with probabilistic flows. arXiv preprint arXiv:2403.16995, 2024.
  71. Bert loses patience: Fast and robust inference with early exit. Advances in Neural Information Processing Systems, 33:18330–18341, 2020.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

Sign Up for Free