Papers
Topics
Authors
Recent
Search
2000 character limit reached

Constraint Back-translation Improves Complex Instruction Following of Large Language Models

Published 31 Oct 2024 in cs.CL and cs.AI | (2410.24175v2)

Abstract: LLMs struggle to follow instructions with complex constraints in format, length, etc. Following the conventional instruction-tuning practice, previous works conduct post-training on complex instruction-response pairs generated by feeding complex instructions to advanced LLMs. However, even advanced LLMs cannot follow complex instructions well, thus limiting the quality of generated data. In this work, we find that existing datasets inherently contain implicit complex constraints and propose a novel data generation technique, constraint back-translation. Specifically, we take the high-quality instruction-response pairs in existing datasets and only adopt advanced LLMs to add complex constraints already met by the responses to the instructions, which naturally reduces costs and data noise. In the experiments, we adopt Llama3-70B-Instruct to back-translate constraints and create a high-quality complex instruction-response dataset, named CRAB. We present that post-training on CRAB improves multiple backbone LLMs' complex instruction-following ability, evaluated on extensive instruction-following benchmarks. We further find that constraint back-translation also serves as a useful auxiliary training objective in post-training. Our code, data, and models will be released to facilitate future research.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. Longalign: A recipe for long context alignment of large language models. arXiv preprint arXiv:2401.18058.
  2. Benchmarking foundation models with language-model-as-an-examiner. Advances in NeurlPS, 36.
  3. Yake! keyword extraction from single documents using multiple local features. Information Sciences, 509:257–289.
  4. Chateval: Towards better llm-based evaluators through multi-agent debate. In ICLR.
  5. Self-play fine-tuning converts weak language models to strong language models. In ICML.
  6. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6.
  7. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377.
  8. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  9. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475.
  10. Shahul Es. 2023. Orca-chat: A high-quality explanation-style chat dataset.
  11. Víctor Gallego. 2024. Refined direct preference optimization with synthetic data for behavioral alignment of llms. arXiv preprint arXiv:2402.08005.
  12. Reverse training to nurse the reversal curse. arXiv preprint arXiv:2403.13799.
  13. From complex to simple: Enhancing multi-constraint complex instruction following ability of large language models. Preprint, arXiv:2404.15846.
  14. Joey Hejna and Dorsa Sadigh. 2024. Inverse preference learning: Preference-based rl without a reward function. Advances in Neural Information Processing Systems, 36.
  15. Iterative back-translation for neural machine translation. In 2nd Workshop on Neural Machine Translation and Generation, pages 18–24. ACL.
  16. Camels in a changing climate: Enhancing lm adaptation with tulu 2. arXiv preprint arXiv:2311.10702.
  17. Mistral 7b. arXiv preprint arXiv:2310.06825.
  18. FollowBench: A multi-level fine-grained constraints following benchmark for large language models. In Proceedings of ACL, pages 4667–4688.
  19. Aligning large language models through synthetic feedback. In Proceedings of EMNLP, pages 13677–13700.
  20. Longform: Effective instruction tuning with reverse instructions. arXiv preprint arXiv:2304.08460.
  21. Openassistant conversations-democratizing large language model alignment. Advances in NeurlPS, 36.
  22. Self-alignment with instruction backtranslation. arXiv preprint arXiv:2308.06259.
  23. Self-alignment with instruction backtranslation. In ICLR.
  24. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval.
  25. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  26. Aligning teacher with student preferences for tailored training data generation. arXiv preprint arXiv:2406.19227.
  27. Edward Loper and Steven Bird. 2002. Nltk: The natural language toolkit. arXiv preprint cs/0205028.
  28. Paraphrasing revisited with neural machine translation. In Proceedings of ACL, pages 881–893.
  29. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734.
  30. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707.
  31. OpenAI. 2022. Introducing ChatGPT.
  32. OpenAI. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.
  33. Training language models to follow instructions with human feedback. Advances in NeurlPS, 35:27730–27744.
  34. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277.
  35. Suri: Multi-constraint instruction following for long-form text generation. arXiv preprint arXiv:2406.19371.
  36. Style transfer through back-translation. arXiv preprint arXiv:1804.09000.
  37. Adelie: Aligning large language models on information extraction. arXiv preprint arXiv:2405.05008.
  38. Infobench: Evaluating instruction following ability in large language models. arXiv preprint arXiv:2401.03601.
  39. Direct preference optimization: Your language model is secretly a reward model. In Advances in NeurIPs.
  40. Rico Sennrich. 2015. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709.
  41. Conifer: Improving complex constrained instruction-following ability of large language models. arxiv preprint arXiv:2404.02823.
  42. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models., 3(6):7.
  43. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118.
  44. Martina Toshevska and Sonja Gievska. 2021. A review of text style transfer using deep learning. IEEE Transactions on Artificial Intelligence, 3(5):669–684.
  45. Zephyr: Direct distillation of lm alignment. Preprint, arXiv:2310.16944.
  46. Finetuned language models are zero-shot learners. In Proceedings of ICLR.
  47. Learning paraphrastic sentence embeddings from back-translated bitext. arXiv preprint arXiv:1706.01847.
  48. Huggingface’s transformers: State-of-the-art natural language processing. arxiv. arXiv preprint arXiv:1910.03771.
  49. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244.
  50. Qwen2 technical report. arXiv preprint arXiv:2407.10671.
  51. Self-rewarding language models. In ICML.
  52. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792.
  53. A survey of large language models. arXiv preprint arXiv:2303.18223.
  54. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911.
  55. Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization. In Findings of ACL, pages 10586–10613.

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 5 tweets with 0 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free