Sign language (SL) conveys linguistic information using gestures instead of sounds. Here, we apply a meta-analytic estimation approach to neuroimaging studies (N = 23; subjects = 316) and ask whether SL comprehension in deaf signers relies on the same primarily left-hemispheric cortical network implicated in spoken and written language (SWL) comprehension in hearing speakers. We show that: (a) SL recruits bilateral fronto-temporo-occipital regions with strong left-lateralization in the posterior inferior frontal gyrus known as Broca’s area, mirroring functional asymmetries observed for SWL. (b) Within this SL network, Broca’s area constitutes a hub which attributes abstract linguistic information to gestures. (c) SL-specific voxels in Broca’s area are also crucially involved in SWL, as confirmed by meta-analytic connectivity modeling using an independent large-scale neuroimaging database. This strongly suggests that the human brain evolved a lateralized language network with a supramodal hub in Broca’s area which computes linguistic information independent of speech.