Unknown LLM Decision Process in Unconstrained Recommendations

Determine the decision-making mechanism by which large language models select a candidate application when responding to unconstrained information-retrieval prompts that do not specify evaluation criteria (for example, requests to "find any text editor"), in order to clarify how such selections are made.

Background

The paper analyzes software-related AI queries to assess when LLMs are necessary versus when lower-cost alternatives would suffice, using a taxonomy that distinguishes factoid, convenience, and complex queries. A key concern is whether some information-retrieval uses rely on deep reasoning or merely replicate simple lookup behavior, which has implications for sustainability via digital sobriety.

To illustrate this, the authors compare an unconstrained prompt ("find any text editor") with a criteria-based prompt ("find the best text editor for software development"). They explicitly note that for the unconstrained case, the internal basis for the model’s selection is unknown, raising a transparency question that affects how such usage should be judged in terms of necessity and sustainability.