Improving Human-Machine Cooperative Visual Search With Soft Highlighting

Abstract: Advances in machine learning have produced systems that attain human-level performance on certain visual tasks, e.g., object identification. Nonetheless, other tasks requiring visual expertise are unlikely to be entrusted to machines for some time, e.g., satellite and medical imagery analysis. We describe a human-machine cooperative approach to visual search, the aim of which is to outperform either human or machine acting alone. The traditional route to augmenting human performance with automatic classifiers is to draw boxes around regions of an image deemed likely to contain a target. Human experts typically reject this type of hard highlighting. We propose instead a soft highlighting technique in which the saliency of regions of the visual field is modulated in a graded fashion based on classifier confidence level. We report on experiments with both synthetic and natural images showing that soft highlighting achieves a performance synergy surpassing that attained by hard highlighting.