ASTRID临床问答评估数据集
包含370条临床问题-答案-上下文三元组，用于评估RAG临床问答系统，涵盖白内障术后随访等多场景。
基本信息
资源简介
ASTRID临床问答评估数据集
【数据集背景】
本数据集源站标题为《arXiv Query: search_query=&id_list=2501.08208v1&start=0&max_results=1》。源站首次发布：2025-01-14。
【数据内容】
源站原始描述：Large Language Models (LLMs) have shown impressive potential in clinical question answering (QA), with Retrieval Augmented Generation (RAG) emerging as a leading approach for ensuring the factual accuracy of model responses. However, current automated RAG metrics perform poorly in clinical and conversational use cases. Using clinical human evaluations of responses is expensive, unscalable, and not conducive to the continuous iterative development of RAG systems. To address these challenges, we i…
【主题与分类】
主题标签：cs.AI、cs.CL。
【适用方向】
文本类数据可用于医学自然语言处理，如命名实体识别、信息抽取与文本分类。
【获取与许可】
源站页面：https://arxiv.org/abs/2501.08208v1
下载信息
注册下载
需要注册 Kaggle 账号并登录后下载，适合需要跟踪下载记录和使用 API 的用户。
暂未开放公开下载
无需注册即可直接获取公开样本或文档，适合快速预览和评估数据集质量。
免登录有偿下载
公开数据集受托下载与技术交付服务。
提供高速下载与技术交付服务（收技术服务费，非数据销售）
暂未开放当前数据集主要来源为 Kaggle 公开托管，完整影像包建议通过原始链接或 Kaggle API 获取。
使用方式
数据集说明
ASTRID临床问答评估数据集 对应论文数据集（arXiv 预印本）。
数据获取指引
- 打开论文页面获取作者与项目信息：https://arxiv.org/abs/2501.08208v1
- 论文 Data Availability / Code Availability 章节标注了数据实际托管位置；
- 获取到实际数据链接后，按对应平台标准方式下载。
论文摘要：Abstract:Large Language Models (LLMs) have shown impressive potential in clinical question answering (QA), with Retrieval Augmented Generation (RAG) emerging as a leading approach for ensuring the factual accuracy of model responses. However, current automated RAG metrics perform poorly in clinical and conversational use cases. Using clinical human evaluations of responses is expensive, unscalable, and not conducive to the continuous iterative development of RAG systems. To address these challenges, we introduce ASTRID - an Automated and Scalable TRIaD for evaluating clinical QA systems leveraging RAG - consisting of three metrics: Context Relevance (CR), Refusal Accuracy (RA), and Conversational Faithfulness (CF). Our novel evaluation metric, CF, is designed to better capture the faithfulness of a model's response to the knowledge base without penalising conversational elements. To validate our triad, we curate a dataset of over 200 real-world patient questions posed to an LLM-based QA agent during surgical follow-up for cataract surgery - the highest volume operation in the world - augmented with clinician-selected questions for emergency, clinical, and non-clinical out-of-domain scenarios. We demonstrate that CF can predict human ratings of faithfulness better than existing definitions for conversational use cases. Furthermore, we show that evaluation using our triad consisting of CF, RA, and CR exhibits alignment with clinician assessment for inappropriate, harmful, or unhelpful responses. Finally, using nine different LLMs, we demonstrate that the three metrics can
数据缺失？
依托陕西、四川两大基地，我们与超过十家三甲医院建立直接合作关系，覆盖合规授权、采集治理、专业标注、数据交付的全流程，为AI医疗团队提供即拿即用的高质量临床数据。
⚡️ 需要数据支持或标注服务？立即联系我们获取专业方案。
