The neurophysiological response during comprehension and production of sign language has been studied using functional magnetic resonance imaging (fMRI) and positron emission tomography (PET) since the advent of neuroimaging. Deaf signers have been shown to recruit similar persivyliyan regions for sign language processing as those identified in studies on verbal language. To date, the neuroimaging literature on sign language has only been reviewed qualitatively and the involvement of right-hemispheric homologs of left persivyliyan language regions remains subject to debate. We use Activation Likelihood Estimation to observe spatial convergence across studies on sign language comprehension and production in deaf signers. To identify regions associated with processing of human actions and movements irrespective of the presence of linguistic information we compare our dataset to an independent dataset of fMRI and PET studies of action observation (non-linguistic manual and bodily actions) in hearing non-signers. We find that sign language processing recruits widely distributed bilateral fronto-occipito-temporal networks, with the largest cluster in left inferior frontal gyrus (IFG) spanning Brodmann areas (BA) 44 and 45. Activity in right IFG, bilateral temporo-occipital cortex, and left insula converges with activation found in hearing non-signers when observing manual actions. We find no spatial convergence in left temporal cortex but observe right superior temporal gyrus (BA 22) activity specific to sign language comprehension. Meta-analytic connectivity modelling confirms that the observed activation in left BA 44 and 45 primarily relates to language processing. Sign language production recruits a similar yet completely left-lateralized network with additional involvement of cognitive control regions. In sum, these results point to left IFG as a modality-independent hub in the language network.