The neurophysiological response during comprehension and production of sign language has been studied using functional magnetic resonance imaging (fMRI) and positron emission tomography (PET) since the advent of neuroimaging. Deaf signers have been shown to recruit similar perisylvian regions for sign language processing as those identified in studies on verbal language. To date, the neuroimaging literature on sign language has only been reviewed qualitatively and the involvement of right-hemispheric homologs of left perisylvian language regions remains subject to debate. We use Activation Likelihood Estimation to observe spatial convergence across studies on sign language comprehension (foci = 391; subjects = 316) and production (foci = 363; subjects = 90) in deaf signers. We further compare our dataset to an independent dataset of fMRI and PET studies of action observation (non-linguistic manual and facial actions) in hearing non-signers (foci = 549; subjects = 431), to identify regions associated with processing of human actions and movements irrespective of the presence of linguistic information. We find that sign language processing recruits a widely distributed bilateral fronto-occipito-temporal network with strong left-lateralization in left inferior frontal gyrus (IFG) and especially Brodmann area (BA) 44. Activity in right IFG during sign language comprehension is not specific to language processing but may be specific to perceiving language in the visuo-spatial modality. Meta-Analytic Connectivity Mapping (MACM) confirms that the observed activation in left IFG primarily relates to language processing. Sign language production recruits a similar yet completely left-lateralized network with additional involvement of cognitive control regions. These results strongly suggest that Broca’s region (left BA 44 and 45) is a core area being recruited for language processing independently of the modality of language use (spoken, written, or signed).