Over the first weeks and months following birth, infants’ initial, broad-based perceptual sensitivities become honed to the characteristics of their native language. In this article, we review this process of emerging specialization within the context of a cascading “critical period” (CP) framework, in which periods of maximal openness to experience of different aspects of language occur at sequential, overlapping points in development. Importantly, as infants’ experience of speech is not limited to auditory signals, but is informed by—for example—their experience of talking faces and their own oral motor movements, we review the trajectory of perceptual specialization in multisensory language processing. Throughout, we highlight the impact of increasing perceptual specialization on later language outcomes (e.g., word learning, foundations of syntax, literacy), and consider how the outcomes can be compromised if/when the timing of perceptual specialization has been perturbed.