The capacity for language constitutes a cornerstone of human cognition and distinguishes our species from other animals. Research in the cognitive sciences has demonstrated that this capacity is not bound to speech but can also be externalized in the form of sign language. Sign languages are the naturally occurring languages of the deaf and rely on movements and configurations of hands, arms, face, and torso in space. This chapter reviews the functional and structural organisation of the neural substrates of sign language as identified by neuroimaging research over the past decades. Most aspects of sign language processing in adult deaf signers markedly mirror the well-known functional left-lateralization of spoken and written language. However, both hemispheres exhibit a certain equipotentiality for processing linguistic information and the right hemisphere seems to specifically support processing of some constructions unique to the signed modality. Crucially, the so-called “core language network” in the left hemisphere constitutes a functional and structural asymmetry in typically developed deaf and hearing populations alike: This network is (i) pivotal for processing complex syntax independent of the modality of language use, (ii) matures in accordance with a genetically determined biological matrix, and (iii) may have constituted an evolutionary prerequisite for the emergence of the human capacity for language.