Papers
Topics
Authors
Recent
Search
2000 character limit reached

Toward Zero-Shot Instruction Following

Published 4 Aug 2023 in cs.CL | (2308.03795v2)

Abstract: This work proposes a challenging yet more realistic setting for zero-shot cross-task generalization: zero-shot instruction following, presuming the existence of a paragraph-style task definition while no demonstrations exist. To better learn the task supervision from the definition, we propose two strategies: first, to automatically find out the critical sentences in the definition; second, a ranking objective to force the model to generate the gold outputs with higher probabilities when those critical parts are highlighted in the definition. The joint efforts of the two strategies yield state-of-the-art performance on the Super-NaturalInstructions. Our code is available on GitHub.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. Fatima T. AL-Khawaldeh. 2019. A Study of the Effect of Resolving Negation and Sentiment Analysis in Recognizing Text Entailment for Arabic. CoRR, abs/1907.03871.
  2. Language Models are Few-shot Learners. Advances in neural information processing systems, 33:1877–1901.
  3. Knowprompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction. In Proceedings of the ACM Web Conference 2022, pages 2778–2788.
  4. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  5. Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 915–929.
  6. Improving Empathetic Response Generation by Recognizing Emotion Cause in Conversations. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 807–819.
  7. Robustness of learning from task instructions. In Proceedings of ACL Findings. Association for Computational Linguistics.
  8. Page: A position-aware graph-based model for emotion cause entailment in conversation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE.
  9. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Comput., 9(8):1735–1780.
  10. In-context Learning for Few-shot Dialogue State Tracking. arXiv preprint arXiv:2203.08568.
  11. Reframing Instructional Prompts to GPTk’s Language. In Findings of the Association for Computational Linguistics: ACL 2022, pages 589–612.
  12. Diederik P Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980.
  13. Can language models learn from explanations in context? arXiv preprint arXiv:2204.02329.
  14. Unsupervised Cross-Task Generalization via Retrieval Augmentation. arXiv preprint arXiv:2204.07937.
  15. Chin-Yew Lin. 2004. Rouge: A Package for Automatic Evaluation of Summaries. In Text summarization branches out, pages 74–81.
  16. RoBERTa: A Robustly Optimized Bert Pretraining Approach. arXiv preprint arXiv:1907.11692.
  17. MUFFIN: Curating multi-faceted instructions for improving instruction following. In The Twelfth International Conference on Learning Representations.
  18. Is prompt all you need? no. a comprehensive and broader view of instruction learning. arXiv preprint arXiv:2303.10475.
  19. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-shot Prompt Order Sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086–8098.
  20. Gradts: A gradient-based automatic auxiliary task selection method based on transformer networks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5621–5632.
  21. Contributions of transformer attention heads in multi-and cross-lingual tasks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1956–1966.
  22. The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables. arXiv preprint arXiv:1611.00712.
  23. Noisy Channel Language Model Prompting for Few-shot Text Classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5316–5330.
  24. MetaICL: Learning to Learn In Context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2791–2809.
  25. Rethinking the Role of Demonstrations: What Makes In-context Learning Work? arXiv preprint arXiv:2202.12837.
  26. Cross-Task Generalization via Natural Language Crowdsourcing Instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3470–3487.
  27. OpenAI. 2022. Chatgpt.
  28. OpenAI. 2023. Gpt-4 technical report. ArXiv preprint.
  29. Recognizing emotion cause in conversations. Cognitive Computation, 13:1317–1332.
  30. Language Models are Unsupervised Multitask Learners. OpenAI blog.
  31. Exploring the Limits of Transfer Learning with a Unified Text-To-Text Transformer. J. Mach. Learn. Res., 21(140):1–67.
  32. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392.
  33. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv e-prints, page arXiv:1606.05250.
  34. Learning to Retrieve Prompts for In-context Learning. arXiv preprint arXiv:2112.08633.
  35. Timo Schick and Hinrich Schütze. 2021. Few-shot Text Generation with Natural Language Instructions. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 390–402.
  36. Get To The Point: Summarization with Pointer-Generator Networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073–1083.
  37. Umie: Unified multimodal information extraction with instruction tuning. arXiv preprint arXiv:2401.03082.
  38. Policy Gradient Methods for Reinforcement Learning with Function Approximation. Advances in neural information processing systems, 12.
  39. Improving English-Arabic transliteration with phonemic memories. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3262–3272, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  40. Attention is All You Need. Advances in neural information processing systems, 30.
  41. Pointer Networks. Advances in neural information processing systems, 28.
  42. InstructionNER: A Multi-task Instruction-based Generative Framework for Few-shot NER. arXiv preprint arXiv:2203.03903.
  43. Ranked List Loss for Deep Metric Learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5207–5216.
  44. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085–5109, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  45. More: A Metric Learning Based Framework for Open-Domain Relation Extraction. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7698–7702.
  46. Albert Webson and Ellie Pavlick. 2022. Do Prompt-based Models Really Understand the Meaning of Their Prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2300–2344.
  47. Adaptive chameleon or stubborn sloth: Unraveling the behavior of large language models in knowledge conflicts. arXiv preprint arXiv:2305.13300.
  48. Travelplanner: Toward real-world planning with language agents.
  49. ConTinTin: Continual Learning from Task Instructions. In ACL, pages 3062–3072.
  50. SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient. In Proceedings of the AAAI conference on artificial intelligence, volume 31.
  51. Magicbrush: A manually annotated dataset for instruction-guided image editing. In Advances in Neural Information Processing Systems.
  52. Calibrate Before Use: Improving Few-shot Performance of Language Models. In International Conference on Machine Learning, pages 12697–12706.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Authors (2)

  1. Renze Lou 
  2. Wenpeng Yin 

Collections

Sign up for free to add this paper to one or more collections.

Sign Up