The capacity for language enables humans to flexibly link meaning to sound or sign by means of grammar. Significantly, the computations underlying this ability appear to be fundamentally abstract and therefore potentially independent of the modality of language use (e.g., sign or speech). In this talk, I will present functional and structural data from magnetic resonance imaging (MRI) with deaf native signers of German Sign Language (DGS) to argue that core aspects of language processing are subserved by a modality-independent primarily left-hemispheric fronto-temporal network. This network subserves the integration of lexical items (i.e., words or signs) into phrases and sentences regardless of whether deaf native signers are using a sign language or hearing speakers process their native spoken language (auditorily or in written form). Structural differences in white-matter pathways resulting from deafness and sign language acquisition can primarily be observed in right parietal cortex, which has previously been implicated in modality-specific aspects of sign language processing.