理解NLP工具的性能与可靠性：四种NLP工具预测放射学报告中卒中表型的比较（数据表1）

数据集获取（Figshare）

• 源站标题：Datasheet1_Understanding the performance and reliability of NLP tools: a comparison of four NLP tools predicting stroke phenotypes in radiology reports.docx
• DOI：10.3389/fdgth.2023.1184919.s001
• 发布日期：2023-09-28
• 文件数：1 个
• 文件总体积：17.90 KB
• 源站页面：https://frontiersin.figshare.com/articles/dataset/Datasheet1_Understanding_the_performance_and_reliability_of_NLP_tools_a_comparison_of_four_NLP_tools_predicting_stroke_phenotypes_in_radiology_reports_docx/24210219

命令行下载

curl -L -o "Datasheet1_Understanding the performance and reliability of NLP tools: a comparison of four NLP tools predicting stroke phenotypes in radiology report.docx" "https://ndownloader.figshare.com/files/42490386"

文件清单

• Datasheet1_Understanding the performance and reliability of NLP tools: a comparison of four NLP tools predicting stroke phenotypes in radiology report.docx（17.90 KB）

文件总体积 17.90 KB（源站 API 实测）。

数据加载示例（图像类）

from PIL import Image
import glob, os

files = (glob.glob(os.path.join(path, "**", "*.png"), recursive=True)
       + glob.glob(os.path.join(path, "**", "*.jpg"), recursive=True)
       + glob.glob(os.path.join(path, "**", "*.tif"), recursive=True))
print("图像文件数:", len(files))
img = Image.open(files[0]); print("尺寸/模式:", img.size, img.mode)

# torchvision Dataset 方式:
# from torchvision import datasets
# ds = datasets.ImageFolder(path)  # 要求 子目录=类别

目录组织与标注格式以源站说明和下载后实际文件为准。

数据说明

BackgroundNatural language processing (NLP) has the potential to automate the reading of radiology reports, but there is a need to demonstrate that NLP methods are adaptable and reliable for use in real-world clinical applications.
MethodsWe tested the F1 score, precision, and recall to compare NLP tools on a cohort from a study on delirium using images and radiology reports from NHS Fife and a population-based cohort (Generation Scotland) that spans multiple National Health Service health boards. We compared four off-the-shelf rule-based and neural NLP tools (namely, EdIE-R, ALARM+, ESPRESSO, and Sem-EHR) and reported on their performance for three cerebrovascular phenotypes, namely, ischaemic stroke, small vessel disease (SVD), and atrophy. Clinical experts from the EdIE-R team defined phenotypes using labelling techniques developed in the development of EdIE-R, in conjunction with an expert researcher who read underlying images.
ResultsEdIE-R obtained the highest F1 score in both cohorts for ischaemic stroke, ≥93%, followed by ALARM+, ≥87%. The F1 score of ESPRESSO was ≥74%, whilst that of Sem-EHR is ≥66%, although ESPRESSO had the highest precision in both cohorts, 90% and 98%. For F1 scores for SVD, EdIE-R scored ≥98% and ALARM+ ≥90%. ESPRESSO scored lowest with ≥77% and Sem-EHR ≥81%. In NHS Fife, F1 scores for atrophy by EdIE-R and ALARM+ were 99%, dropping in Generation Scotland to 96% for EdIE-R and 91% for ALARM+. Sem-EHR performed lowest for atrophy at 89% in NHS Fife and 73% in Generation Scotland. When comparing NLP tool output with brain image reads using F1 scores, ALARM+ scored 80%, outperforming EdIE-R at 66% in ischaemic stroke. For SVD, EdIE-R performed best, scoring 84%, with Sem-EHR 82%. For atrophy, EdIE-R and both ALARM+ versions were comparable at 80%.
ConclusionsThe four NLP tools show varying F1 (and precision/recall) scores across all three phenotypes, although more apparent for ischaemic stroke. If NLP tools are to be used in clinical settings, this cannot be performed “out of the box.” It is essential to understand the context of their development to assess whether they are suitable for the task at hand or whether further training, re

许可

CC BY 4.0

引用

学术使用请引用 DOI 10.3389/fdgth.2023.1184919.s001（source: https://frontiersin.figshare.com/articles/dataset/Datasheet1_Understanding_the_performance_and_reliability_of_NLP_tools_a_comparison_of_four_NLP_tools_predicting_stroke_phenotypes_in_radiology_reports_docx/24210219）。