Generative Artificial Intelligence (AI) holds transformative potential for addressing persistent limitations in traditional psychological health education systems, particularly constraints related to accessibility, uneven distribution of resources, and the lack of personalized support. However, critical concerns persist regarding their reliability, interpretability, and fairness, particularly in high-stakes scenarios such as psychological guidance.

This study employed a word embedding-based Comprehensive Semantic Behavioral Analysis Framework (CSBAF) to systematically evaluate the semantic consistency, response strategies, and systemic bias of LLMs in psychological health education contexts. Grounded in the theory of verbal behavior, the framework conceptualizes AI-generated language as both informational content and social action. By integrating semantic structure analysis with contextual strategy evaluation over iterative interactions, the framework offered advantages over traditional evaluation criteria such as content accuracy, providing a deeper behavioral perspective on AI performance in psychologically sensitive domains. To operationalize this framework, we utilized DeepSeek as the primary model and conducted comparative testing with ChatGPT and Doubao to assess cross-model generalizability. The evaluation was based on 21 structured prompt templates adapted from established psychological education handbooks, encompassing key themes including depression, anxiety, general health, substance use, meaning and existence, lifestyle, and interpersonal relationship. Each model was evaluated under three sampling configurations, by adjusting the sampling parameters of temperature and top_p. For the semantic consistency assessment, responses were transformed into vector representations using Chinese word embeddings. Semantic similarity across 30 repeated dialogue iterations was quantified using the Frobenius norm and visualized using dimensionality reduction techniques (PCA and t-SNE). Clustering analysis was employed to identify and characterize distinct response strategies exhibited by each model. In addition, expert-based evaluation methods were employed to systematically assess the primary model across six dimensions: accuracy, clarity, relevance, empathy, engagement, and ethical considerations, with all assessments situated within the contextual frameworks of gender and ethnicity.

This study yielded three principal findings regarding the performance of LLMs in multi-turn psychological dialogue scenarios. First, in terms of semantic structural similarity, the primary model demonstrated a strong correlation between response patterns and question types. Although semantic distribution exhibited structural changes with adjustments in sampling parameters, the impact of question type on semantic stability surpassed that of parameter variations. Cross-model comparisons showed parameter settings play a major role in generative patterns. Nonetheless, for certain question types, the prompts remained the dominant factor influencing semantic behavior. Second, in terms of response strategies, each model showed relatively stable and distinguishable strategic preferences for specific question types, and these tendencies were closely related to model architecture and parameter settings. Third, in the bias analysis, male-context prompts were more likely to elicit information-focused responses, while female-context prompts triggered more emotionally expressive outputs. These results suggest the presence of implicit social role tendencies in LLMs.

In summary, these findings validate the practical potential of LLMs for augmenting psychological health education. Future research should further investigate how generative AI could be integrated into human-AI collaborative systems to better support educational practice.