Material classification by man and machine: Understanding human perception using psychophysics and convolutional neural networks (360G-Wellcome-218657_Z_19_Z)

The primary focus is to uncover computational mechanisms underlying material perception, with particular emphasis on "perceptual constancy" – the stability of judgements about objects in the world, despite changes in the proximal stimulus. My approach to this problem is to capitalise on recently developed deep-learning paradigms by evaluating and interpreting activation maps of convolutional neural networks (CNNs) that have been trained to replicate human performance in classifying material properties, such as colour, gloss and translucency, and to use these to identify candidate models of human perception. The key stages are to (i) generate computer-rendered training images with diagnostic conditions of lighting, viewpoint and material parameters; (ii) obtain human perceptual labels for 15,000 images using online methods, validated with lab-based measurement; (iii) train CNNs with either ground-truth or human labels; (iv) interpret the CNNs using visualisation tools; and (v) run psychophysical tests of CNN-inspired models of human vision. The goal is to learn about mechanisms of human material perception by decoding the internal structures of these networks and comparing their behavioural responses to new images. Cue-perturbation methods will be used to test the importance of the recovered features for human perception. Perceptually labelled images will be additionally analysed with traditional methods.