Deafness and sign languages provide a stringent test of whether or to what extent the brain’s language system is tied to speech or constitutes a modality-independent neural architecture. Converging evidence from neuroimaging meta-analyses, task-based fMRI and diffusion MRI supports a primarily left-hemispheric fronto-temporal language network that is implicated in integrating words or signs into sentences across sign language, speech, as well as written language comprehension. The principal white-matter pathways connecting this functional network do not differ detectably between deaf signers and hearing non-signers. These findings indicate that the brain’s core language network develops independently of auditory experience. However, diffusion MRI also reveals macro- and micro-structural alterations primarily in right-hemispheric temporal, frontal, and parietal pathways, potentially reflecting the visuospatial and attentional demands of sign language. In contrast, an analysis of T1-weighted scans from the same participants failed to replicate structural group differences frequently reported in the literature, underscoring the limited power and reproducibility of small-sample comparisons. Preliminary structural and resting-state MRI analyses from a large multisite sample provide a broader test of these effects: Morphometric and connectivity-gradient analyses suggest substantial preservation of macro-scale cortical organisation alongside selective changes in auditory cortex and its functional embedding within attention and control systems. Together, these findings support the notion of a modality-independent core network for language that flexibly interfaces with modality-specific systems depending on the modality of language use.