Privileged representational axes in biological and artificial neural networks

Abstract

The widespread finding of neural populations apparently tuned to specific, identifiable fea- tures of our external environment (e.g., faces, places, speech) suggests that brains may favor certain representational axes over others. But despite decades of research, we have no formal understanding of whether and why brains use privileged bases for representing the natural world. Here, we develop a formal framework for investigating the extent to which a repre- sentational system has privileged axes. First, we formulate two axis-dependent alignment metrics, which enable us to demonstrate that the axes of neural representations of sensory stimuli are in fact aligned across humans and across macaques. Parallel analyses of computational models of neurobiological systems reveals that Deep Convolutional Neural Networks (DCNNs) also have privileged axes. Strikingly, we further observed that the representational axes of DCNNs trained on natural images and sounds are aligned with the privileged axes found in the visual and auditory cortices respectively. These results further provide a novel unified computational account of certain types of neural tuning (e.g. category selectivity) reported in high-level sensory areas, suggesting that it can, in principle, arise without explicit downstream pressures and simply from symmetry-breaking mechanisms that privilege certain axes over others in neural networks. Additionally, we demonstrate that the favored basis of brains and DCNNs results in higher lifetime sparseness and reduced downstream wiring costs, and promotes better few-shot generalization compared to an arbitrary basis, provid- ing insight into the computational implications of this privileged axis. Finally, our alignment measures can better differentiate the fit of different neural architectures to the brain than standard measures and tools that are blind to axis alignment like RSA, CKA and linear predictivity. Altogether, these findings offer a new computational framework for understanding the systematic neural tuning observed in distinct brain regions.