It is one of the most important abilities for human to make some predictions based on prior experience and to act accordingly. Previous studies reveal that prior experience stored in long-term memory could affect attention. In the real world, information usually comes from multiple sensory modalities, and attention may shift from the information in one sensory modality to the information in another modality. Previous studies support that there exists both spatial-based and object-based crossmodal attention. The present study aims to explore whether prior experience can guide visual selective attention crossmodally. By adopting prime stimuli from both the same and different modalities, we are also interested in the difference between intra-modal and crossmodal semantic priming on the modulation of visual selective attention. In the present study, a priming paradigm and a dot probe paradigm were combined to examine the effect of prior experience on crossmodal selective attention. In Experiment 1 and 2, a visual or auditory prime stimulus was presented, which was high relevant to only one visual cue in the dot probe task. After the prime stimuli, a dot probe task was adopted, in which two visual cues were presented at two opposite positions on the screen (left or right). After two visual cues disappeared, a visual probe was presented at one of the cue positions. The visual probe were three dots align in one line vertically or horizontally, and participants were required to discriminate the orientation of the probe stimulus. To investigate the time course of crossmodal priming on selective attention, the stimulus onset asynchrony (SOA) between the prime stimulus and the cue was 600 ms in Experiment 1, while 900 ms in Experiment 2. Thus, a 2×2 within-participants design was adopted with the following two variables: Modality of prime stimuli (visual vs. auditory modality) and Cue type (high-relevant vs. low-relevant to the prime stimulus). The results of Experiment 1 show that under the condition of auditory-visual crossmodal priming, the participants respond faster when the probe follows the high-relevant cue than that when the probe follows the low-relevant cue. By contrast, under the condition of visual-visual intramodal priming, no significant difference of reaction times is found for two types of probes. The results of Experiment 2 show that the modulation effect of crossmodal priming on the visual selective attention disappears under the SOA of 900 ms. To sum up, by adopting the natural sounds and pictures of real objects, our study supports that the visual selective attention could be promoted by the crossmodal semantic links based on the experience-level. Moreover, the results in the present study reveal that crossmodal priming effect on visual selective attention could be observed when SOA between the prime stimulus and the cue is 600 ms and disappears when SOA is 900 ms. Our study extends the current research in the field of crossmodal integration and crossmodal attention and could be explained by the “unity assumption”. Moreover, for the natural objects, the existence of crossmodal semantic priming on the visual attention could be explained by the ‘unity assumption’, which could be further studied in the future.