BlueScrubs-v1-fixed（医疗文本语料库）
BlueScrubs-v1-fixed是修复后的医疗文本语料库，含1108万条训练文本，来自多个公开来源，附带医疗概率和肿瘤学标签，用于训练临床大语言模型。
基本信息
资源简介
TheBlueScrubs-v1-fixed是一个大规模精选医疗文本语料库，专为临床大语言模型设计。数据集包含约1108万条训练文本，来源于SlimPajama/RedPajama（涵盖Common Crawl、C4、GitHub、图书、arXiv、维基百科、StackExchange），经过逻辑回归筛选和Llama-3.1-70B评估，提供文本级医疗概率分数（0.8-1.0）以及三项LLM评分（相关性、精准性、安全性）。全语料中约110亿Token附带肿瘤学分类标签。此版本修复了上游训练子集中meta列的序列化错误，移除meta列，仅保留text字段，确保在datasets中流式加载顺畅。
下载信息
注册下载
Tips: 该数据集需要在对应的数据源网站注册通过后，才能进行数据下载，注册有对应要求，或者需要收费。
暂未开放公开下载
Tips: 该数据集属于公开下载，应该可以免费公开下载。
免登录有偿下载
Tips: 该数据集 Qianfanghub 可以协助提供有偿下载服务，注意，服务不针对数据相关产权，只是技术服务费。
提供高速下载与技术交付服务（收技术服务费，非数据销售）
暂未开放千方医数集，医疗数据集部分，是为社区服务的公开医疗数据集搜索引擎，并不存储或者下载原始的任何数据。 如果您有其他医疗数据需求，可以和客服联系，或者下工单。我们有强大的三甲医疗机构帮助您提供个性化的医疗数据定制、采集、标注服务。
使用方式
数据集获取（ModelScope）
方式一：MsDataset（Python）
# 前置依赖: pip install modelscope
from modelscope.msdatasets import MsDataset
ds = MsDataset.load("openmed-community/TheBlueScrubs-v1-fixed", subset_name="default", split="train")
print(ds)
方式二：CLI（命令行）
pip install modelscope
modelscope download dataset openmed-community/TheBlueScrubs-v1-fixed
数据集卡片摘录（源站）
openmed-community/TheBlueScrubs-v1-fixed
What is this?
TheBlueScrubs-v1-fixed is a maintenance fork of the upstream TheBlueScrubs/TheBlueScrubs-v1 train split that resolves a schema bug in the meta column.
In the original train files, some rows serialized meta incorrectly (appearing as the literal string "dict"). This fork re-exports the entire train split without meta column, preserving text field and values.
- Document count: 11,080,331 texts (train)
- Tokens (upstream estimate across all splits): ~20B tokens
- Sources: Curated from SlimPajama/RedPajama (Common Crawl, C4, GitHub, Books, arXiv, Wikipedia, StackExchange)
- Quality signals: per-text medical probability (0.8–1.0) + three 1–5 LLM-based scores (relevance, precision/factual detail, safety/ethics); oncology label covering ~11B tokens across the full corpus.
Upstream details: The Blue Scrubs is a large, curated medical corpus designed for clinical LLMs, filtered via a logistic-regression screen and then Llama-3.1-70B evaluation; clinician and external checks reported high concordance. An oncology classifier adds cancer labels at scale.
Why this fork?
-
Fix: Removes the
metacolumn, unblocking usage withdatasetsstreaming and dataframe backends. -
Scope: Content is otherwise unchanged relative to upstream train split (same rows, fields, and values).
-
Goal: Provide a drop-in train split that loads cleanly in
datasetswithout ad-hoc parsing workarounds.
Data fields (train)
| Field | Type | Description |
|---|---|---|
text |
string | Raw medical text extracted from SlimPajama/RedPajama sources. |
Splits
This repository publishes the train split only (11,080,331 documents). For methods, scope, and aggregate corpus statistics (including validation/test in the upstream project), see the original dataset card and paper.
How to load
from datasets import load_dataset
# streaming
ds = load_dataset("openmed-community/TheBlueScrubs-v1-fixed", split="train", streaming=True)
row = next(iter(ds))
row["text"]
# non-streaming (if you have local storage/network bandwidth)
ds = load_dataset("openmed-community/TheBlueScrubs-v1-fixed", split="train")
ds.features
## 许可
apache-2.0
## 数据加载示例（表格/文本类）
```python
import pandas as pd, glob, os
files = (glob.glob(os.path.join(path, "**", "*.csv"), recursive=True)
+ glob.glob(os.path.join(path, "**", "*.tsv"), recursive=True)
+ glob.glob(os.path.join(path, "**", "*.xlsx"), recursive=True))
print("数据文件:", files)
df = pd.read_csv(files[0])
print(df.shape); print(df.columns.tolist()); print(df.head(3))
源站：modelscope.cn/datasets/openmed-community/TheBlueScrubs-v1-fixed
精度瓶颈？数据缺失？
当前公开数据无法满足您的算法精度？千方提供针对 癌症（总论） 的高质量、多模态真实临床数据定制解决方案。
