AI聊天机器人评估量表和评分标准

数据集获取（Figshare）

• 源站标题：AI chatbot evaluation scale and rubric.
• DOI：10.1371/journal.pone.0300487.t001
• 发布日期：2025-03-19
• 文件数：1 个
• 文件总体积：5.63 KB
• 源站页面：https://plos.figshare.com/articles/dataset/AI_chatbot_evaluation_scale_and_rubric_/28627141

命令行下载

curl -L -o "Table 1.xls" "https://ndownloader.figshare.com/files/53113435"

文件清单

• Table 1.xls（5.63 KB）

文件总体积 5.63 KB（源站 API 实测）。

数据加载示例（图像类）

from PIL import Image
import glob, os

files = (glob.glob(os.path.join(path, "**", "*.png"), recursive=True)
       + glob.glob(os.path.join(path, "**", "*.jpg"), recursive=True)
       + glob.glob(os.path.join(path, "**", "*.tif"), recursive=True))
print("图像文件数:", len(files))
img = Image.open(files[0]); print("尺寸/模式:", img.size, img.mode)

# torchvision Dataset 方式:
# from torchvision import datasets
# ds = datasets.ImageFolder(path)  # 要求 子目录=类别

目录组织与标注格式以源站说明和下载后实际文件为准。

数据说明

Background
Generative artificial intelligence (GenAI) has the potential to revolutionise healthcare delivery. The nuances of real-life clinical practice and complex clinical environments demand a rigorous, evidence-based approach to ensure safe and effective deployment of AI.
Methods
We present a protocol for the systematic evaluation of large language models (LLMs) as GenAI chatbots within the context of clinical microbiology and infectious diseases clinical consultations. We aim to critically assess recommendations produced by four leading GenAI models, including Claude 2, Gemini Pro, GPT-4.0, and a GPT-4.0-based custom AI chatbot.
Discussion
A standardised, healthcare-specific, universal prompt template is developed to elicit clinically impactful AI responses. Generated responses will be graded by two panels of practicing clinicians, encompassing a wide spectrum of domain expertise in clinical microbiology and virology, as well as infectious diseases. Evaluations will be performed using a 5-point Likert scale across four clinical domains: factual consistency, comprehensiveness, coherence, and medical harmfulness. Our study will offer insights into the feasibility, limitations, and boundaries of GenAI in clinical consultations, providing guidance for future research and clinical implementation. Ethical guidelines and safety guardrails should be developed to uphold patient safety and clinical standards.

许可

CC BY 4.0

引用

学术使用请引用 DOI 10.1371/journal.pone.0300487.t001（source: https://plos.figshare.com/articles/dataset/AI_chatbot_evaluation_scale_and_rubric_/28627141）。