Defining “Scaling Up” for Foundation Models

Clarify which modifications to foundation models should be classified as mere scaling up (e.g., increases in data and parameters) versus qualitative changes in approach, to enable principled assessments of the scaling hypothesis.

Background

The discussion around Percy Liang’s talk contrasted continued scaling of data and parameters with incorporating multi-modal data, embodiment, or architectural changes. Participants noted ambiguity about what counts as scaling versus a change in kind.

Establishing clear criteria would help evaluate claims about the viability and limits of the scaling paradigm and guide future model development strategies.

References

However, it was not totally clear exactly what kinds of changes to foundation models would count as mere scaling up.

— Embodied, Situated, and Grounded Intelligence: Implications for AI  (2210.13589 - Millhouse et al., 2022) in Discussion — “Are Foundation Models Castles in the Air?”