This study investigated how infants deal with cross-talker variability in the perception of native lexical tones, paying specific attention to developmental changes and the role of task demands. Using the habituation-based visual fixation procedures, we tested Cantonese-learning infants of different age groups on their ability to discriminate Cantonese Tone 1 (high level) and Tone 3 (mid level) produced by either multiple talkers or a single talker. Results demonstrated that the 12-month-old and 24-month-old groups showed reliable discrimination across talkers, whereas the 18-month-old group did not (Experiment 1), despite their ability to discriminate the same contrast when the talker was held constant (Experiment 2). In a task that included a novel object as a referent to the sound, the 18-month-olds discriminated the contrast across talkers from Tone 1 to Tone 3 (Experiment 3). These results revealed a U-shaped developmental path and perceptual asymmetry in native lexical tone discrimination across talkers.