中国人民公安大学学报（自然科学版）

伴随大语言模型(Large Language Models, LLMs)在国家安全领域展现出新的赋能潜力，通过构建该领域的系统数据集与评估体系并开展测评，探索其潜在价值。鉴于大语言模型评估体系难以有效捕捉“战略模糊性”等高阶政治概念，提出主观/客观-真实/生成一致性验证方法，整合国家安全学、情报学与计算机科学知识，构建主客观测评框架及高质量基准数据集。实验选取10个开源大语言模型展开测评，同时引入生成测评数据进行对比分析。此外，结合“4·15国家安全教育日”等实际场景，对重点领域进行针对性测评。研究表明，不同大语言模型答题能力差异显著，Doubao-pro-32k/240615模型以85.4分(满分100分)领先，合成数据题的表现优于真实题，但主观题作答存在内容规范性不足等问题。大语言模型具有显著的效率提升和决策辅助价值，在实际场景中展现出巨大的应用潜力。

With large language models(LLMs) exhibiting emerging potential in the national security domain, a systematic dataset and evaluation framework is constructed to explore their latent value. Given the difficulty of existing LLMs evaluation systems in effectively capturing high-level political concepts such as “strategic ambiguity”, a subjective/objective-real/generated consistency verification method is proposed. Knowledge from national security studies, intelligence science and computer science are integrated to establish a comprehensive evaluation framework along with a high-quality benchmark dataset. Experiments evaluate 10 open-source LLMs, with comparative analysis incorporating synthetically generated test data. Furthermore, targeted assessments in key areas are conducted, incorporating real-world scenarios such as China's “April 15 National Security Education Day”. Findings reveal significant disparities in performance across models, with Doubao-pro-32k/240615 achieving the highest score(85.4/100). Synthetic data questions yield better results than real-data questions, though subjective responses exhibit issues such as insufficient content standardization. LLMs offers notable efficiency gains and decision-support value, showing great potential in practical applications.