Limited task interference in over-parameterized multitask supernets

Establish whether, when training an over-parameterized supernet of sufficiently large capacity with shared weights across tasks as in the FBNetV5 multitask neural architecture search, interference among tasks during optimization is small enough to be ignored, and determine the extent to which such task interference impacts the architectures searched for each task.

Background

FBNetV5 searches architectures for multiple computer vision tasks by training an over-parameterized supernet with shared weights and sampling task-specific architectures from learned distributions. A potential concern in multitask learning is interference among tasks during joint optimization, which could harm the quality of the searched architectures. To justify approximating separate task optimization with a single shared supernet, the paper explicitly conjectures that such interference is negligible in large-capacity supernets and has limited impact on search outcomes.

Validating this conjecture would strengthen the theoretical and empirical foundation of FBNetV5’s single-run multitask NAS, clarifying whether and when shared-weight supernets can be safely used to search task-specific architectures without significant cross-task degradation.