Recent advances in artificial intelligence (AI), particularly large language models (LLMs), have profoundly reshaped both technological landscapes and scientific inquiry in psychology. This paper systematically examines the bidirectional synergy between psychometrics and AI, arguing that their integration is not merely instrumental but foundational for the emergence of a new measurement paradigm in the era of human-AI coexistence.

On one hand, psychometrics provides essential theoretical frameworks and methodological rigor to evaluate AI systems. By adapting established psychological constructs—such as intelligence, creativity, personality, theory of mind, and moral reasoning—researchers can assess AI’s “psychological” profiles, benchmark its capabilities against human standards, and uncover its developmental trajectories and limitations. Current approaches include the direct application of human psychometric instruments (e.g., WAIS, Big Five inventories) and large-scale AI-specific benchmarks (e.g., MMLU, BIG-bench). Empirical studies show that models like GPT-4 already match or exceed human performance in domains such as verbal reasoning and creativity fluency, yet lag in visual-spatial tasks and emotional depth. However, these methods often suffer from anthropomorphic bias, prompt sensitivity, and a lack of grounding in psychological theory, leading to questionable validity. Moreover, most evaluations rely on classical test theory, yielding ordinal rankings that hinder fine-grained, cross-model comparisons on a common metric. To address this, we advocate for integrating modern psychometric models—such as item response theory and cognitive diagnosis models—to enable equated, interpretable, and diagnostic assessments of AI capabilities. Beyond trait measurement, psychometrics also enables the systematic study of (1) human cognition, emotion, and attitudes toward AI; (2) AI’s impact on human psychological development in education, mental health, and socialization; and (3) interdependent dynamics in human-AI collaboration, including role allocation, trust calibration, and interaction patterns.

On the other hand, AI is revolutionizing psychometrics itself. LLMs facilitate automated item generation, significantly reducing development costs and human bias while enabling dynamic, context-sensitive assessments. AI also supports implicit and multimodal measurement through the analysis of natural language, facial expressions, voice, and behavioral logs, moving beyond traditional self-report questionnaires. Furthermore, deep learning enables the unsupervised extraction of latent psychological dimensions from real-world data (e.g., social media), potentially refining or even redefining psychological constructs. In scoring and interpretation, AI systems can provide reliable, scalable, and diagnostic feedback on open-ended responses, while predictive modeling allows for early risk detection and personalized interventions. For instance, transformer-based models like CLIP and Flamingo enable cross-modal integration of text, image, and audio, while graph neural networks model complex problem-solving trajectories. Nevertheless, these advances raise critical concerns about algorithmic “black boxes,” data bias, cross-cultural fairness, and the ethical use of sensitive behavioral data.

The paper identifies key challenges in both directions. For psychometrics-to-AI, issues include the risk of uncritical anthropomorphism, unstable AI responses due to prompt and parameter sensitivity, cultural bias in benchmarks, and the lack of fine-grained diagnostic feedback in current evaluations. For AI-to-psychometrics, concerns center on transparency, the validity of AI-generated content, data privacy, and the scarcity of interdisciplinary expertise. To address these, we propose five strategies: (1) developing AI-specific psychometric paradigms that account for AI’s “data-algorithm-model” nature and incorporate functional traits (e.g., reasoning stability, cross-context adaptability); (2) creating stability metrics, such as output consistency indices, to quantify AI’s trait volatility; (3) embedding core psychometric principles—reliability, validity, fairness—into AI systems from the design stage; (4) training domain-specific “AI psychometricians” that integrate psychological theory with computational methods; and (5) establishing ethical guidelines for data collection, use, and synthetic data generation.

Ultimately, we envision a co-evolutionary future in which psychometrics and AI mutually inform each other: psychometrics offers interpretability, standardization, and ethical grounding, while AI contributes scalability, adaptivity, and multimodal integration. Their deep integration may give rise to intelligent psychometrics—a new discipline that merges theory-driven and data-driven approaches to understand both human and artificial minds in an increasingly intertwined world. This synergy not only advances scientific understanding but also ensures that AI development remains human-centered, scientifically sound, and ethically responsible.