In “Estimating the reproducibility of psychological science” published in 《Science》, the authors report the results of their replication of 100 experimental and correlational studies published in three prestigious psychology journals in 2008 (Open Science Collaboration, 2015). Whereas 97% of original studies had significant results (P<.05), only 36% of replications were so. The authors offer some explanations for these results and emphasise that this is not a defeat for psychology. They conclude that accumulating evidence is the scientific community’s method of self-correction and is the best available option for achieving the ultimate goal: truth. In this process, statistically significant innovations from the original studies would point out paths that are possible, whilst those that subsequently pass the test of replication would point out paths that are likely. Scientific progress relies on both innovations and replications — hence journals should not dismiss replication studies lightly. Whilst we are in broad agreement with the authors’ explanations and conclusions, we note, first, that the problem of irreproducibility is not unique to psychology. It has happened to other sciences such as oncology, haematology and genetics, sometimes to an even greater extent, and has led to better scientific tools such as double-blind procedures to reduce the risk of unconscious biases (Nuzzo, 2015). Second, many of the replications that fail to reach statistically significant P values are in the same trend as the original studies rather than contradictory. This raises the question of the real scientific significance of P values in the research process and its implications. In the second paper reviewed herein (“P values are just the tip of the iceberg” published in《Nature》), Leek and Peng (2015) criticise the decision made by the journal Basic and Applied Social Psychology in 2015 to ban the use of P values and other methods of NHST (null hypothesis significance testing). The authors argue, first, that although P values are widely abused, banning it would open more doors to dubious game statistics for getting a paper accepted for publication. Second, and more importantly, they point out that a successful study has many stages in the making before reaching the P values stage. The latter stage is only the tip of the iceberg, and its impact on the eventual success of a study as a whole is much less critical than careful research design and analysis that lie below it. Hence the more fruitful debate should be on all aspects of the research process rather than on P values primarily. This conclusion has implications not only for the reproducibility study’s over-reliance on P values, but also for the training of psychology research students in contemporary China. Too often, the training has focused on the “tip of the iceberg” of the research process to the neglect of those parts of the process below. In this training culture, students learn the technicalities competently. They also learn to reduce their cognitive uncertainties about doing research by placing their trust in authorities, and can complete super-efficiently the “required” multiple studies in a bundle rather than in a programatic manner to build one study on the results of the one preceding. Then they blame the authorities for failures instead of taking personal responsibility for their own research. The combined take-home message of the two 《Science》 and《Nature》psychology papers is perhaps best summed up by Francis Bacon (1561–1626): “If a man will begin with certainties, he shall end in doubts; but if he will be content to begin with doubts, he shall end in certainties.” (Quoted in Sun, 2004, p. 581)