Scene Semantics and Syntax

Our visual world is complex, but its composition usually follows a set of rather strict rules. Knowledge of these rules - we call them a scene’s “grammar” or a la Biederman et al. (1982) scene semantics and syntax - helps us recognize objects and guide search in scenes. Does this “scene grammar” share common grounds with the grammar studied in linguistics? How do grammatical violations of objects embedded in scenes affect scene viewing? Can we process higher-level object-scene inconsistencies in our visual periphery? And how do these modulate eye movement behavior prior and upon their fixation? I have recently begun testing whether the brain codes scene syntax and semantics in a manner similar to the way that it codes syntax and semantics in language. In language processing, semantic and syntactic processing elicit differential brain responses: An N400 signals semantic violations, while a P600 marks inconsistent syntactic structure. I used event-related brain potentials (ERPs) to look for similar semantic and syntactic signals in scenes. In fact, we have found a clear dissociation between ERP signatures of semantic and syntactic scene processing: As predicted by the sentence processing literature, semantic inconsistencies produced negative deflections in the N300/N400 time window, while syntactic inconsistencies elicited a late positivity resembling the P600 found for syntax manipulations.

Related Publications

Võ, M. L.-H. (2021). The Meaning and Structure of Scenes. Vision Research, 181, 10-20. doi: doi.org/10.1016/j.visres.2020.11.003 pdf

Võ, M. L.-H., & Wolfe, J. M. (2013). Differential ERP Signatures Elicited by Semantic and Syntactic Processing in Scenes. Psychological Science, 24(9), 1816 –1823.

doi: 10.1177/0956797613476955 pdf

Võ, M. L.-H., & Henderson, J. M. (2011). Object-scene Inconsistencies do not Capture Gaze: Evidence from the Flash-Preview Moving-Window Paradigm. Attention, Perception, & Psychophysics, 73, 1742–1753. doi: doi.org/10.3758/s13414-011-0150-6 pdf

Võ, M. L.-H., & Henderson, J. M. (2009). Does Gravity Matter? Effects of Semantic and Syntactic Inconsistencies on the Allocation of Attention during Scene Perception. Journal of Vision, 9(3):24, 1-15. doi: doi.org/10.1167/9.3.24 pdf