Small LLMs with Expert Blocks Are Good Enough for Hyperparamter Tuning

Abstract: Hyper-parameter Tuning (HPT) is a necessary step in ML pipelines but becomes computationally expensive and opaque with larger models. Recently, LLMs have been explored for HPT, yet most rely on models exceeding 100 billion parameters. We propose an Expert Block Framework for HPT using Small LLMs. At its core is the Trajectory Context Summarizer (TCS), a deterministic block that transforms raw training trajectories into structured context, enabling small LLMs to analyze optimization progress with reliability comparable to larger models. Using two locally-run LLMs (phi4:reasoning14B and qwen2.5-coder:32B) and a 10-trial budget, our TCS-enabled HPT pipeline achieves average performance within ~0.9 percentage points of GPT-4 across six diverse tasks.