By Johannes Bill

Motion relations in everyday visual scenes carry an abundance of behaviorally relevant information. We recognize people by the way they walk, track objects through occlusion, and predict hazardous situations from the traffic flow. Thereby, motion percepts are often organized in hierarchies: birds in a flock share a common flock velocity, and when observing the flock while driving your car, your self-motion adds yet an additional velocity component on top of the scene. We wondered if the brain exploits hierarchical motion relations when solving perceptual tasks.

Hierarchical motion structure in natural scenes. The complex velocity patterns of migrating birds can be decomposed by our visual system into a shared flock velocity and smaller relative motion components of individual animals.

We approached this question in a Multiple Object Tracking experiment in which several dots moved randomly about the screen, and a subset of initially marked target dots had to be reidentified at the end of the trial. When we introduced motion structure in the scene — for example, all dots shared a common velocity component, like the bird’s in a flock — human tracking performance greatly improved, compared to scenes of independently moving dots.

Yet, we were puzzled about the origins of these performance gains. Was it really the use of structure information by the brain that led to the improvement? Or was it merely a spurious effect of the stimulus, such as easier tracking due to an altered spatial arrangement of the dots when motion structure was present? To disambiguate these different computational origins, we developed a new class of mathematically tractable stimuli that granted us tight control over all object velocities and locations in the scene, and supported development of a Bayesian observer model of multiple object tracking. The artificial observer replicated human tracking performance for a variety of motion structures — but only, when it exploited the scene’s motion structure during tracking. When disregarding motion relations, in contrast, the model’s performance dropped to the performance for independently moving dots, even when structure was present in the scene. This suggested that structure knowledge was actually employed by our participants.

The use of structure by the brain was further substantiated in a second experiment in which the trajectories of two occluded dots had to be predicted from the motion of other, visible dots. In this task, the response error patterns carry signatures of the motion structure priors employed for making the predictions. We found that our participants reliably used correctly structured motion priors, even for hierarchically nested motion scenes.

Our work predicts which hidden variables the brain needs to track to use structure knowledge for object tracking. Thus, it lays a foundation for studying how the brain identifies motion relations in visual scenes, and how neural circuits in the visual system represent motion structure knowledge.

Johannes Bill is a postdoctoral fellow in the labs of Jan Drugowitsch (Department of Neurobiology at Harvard Medical School) and Sam Gershman (Department of Psychology at Harvard). His research was supported by the Alice and Rodman W. Moorhead III Collaborative Grants Fund for HBI, as well as an Alice and Joseph Brooks Fund Fellowship.

This story will also appear in the HMS Neurobiology Department newsletter, The Action Potential.

Learn more in the original research article:
Hierarchical structure is employed by humans during visual motion perception. Bill J, Pailian H, Gershman SJ, Drugowitsch J. Proc Natl Acad Sci U S A. 2020 Sep 29;117(39):24581-24589. doi: 10.1073/pnas.2008961117.