One of the major insights of modern linguistics has been that the human capacity for language is not bound to speech but may also be externalized and perceived in the visuo-spatial modality of sign language. Linguistic analysis has demonstrated that despite the differences between modalities signed and spoken language exhibit deep similarities. In this lecture, we will start out by reviewing the basics of sign language linguistics, before moving to a discussion of major psycho- and neurolinguistic studies of sign language over the past decades. We will see that the available neuroimaging evidence indicates that signed, spoken and, written language is processed in a partially overlapping primarily left-hemispheric fronto-temporal network. However, sign language processing also recruits brain regions beyond the canonical language network. Against this background, we will argue that the human brain has evolved a so-called “core language network” which processes linguistic information independent of modality and flexibly interacts with non-linguistic and modality-specific networks during language processing, depending on the modality of language use.