BlueScrubs-v1-fixed（医疗文本语料库）
BlueScrubs-v1-fixed是修复后的医疗文本语料库，含1108万条训练文本，来自多个公开来源，附带医疗概率和肿瘤学标签，用于训练临床大语言模型。
基本信息
资源简介
TheBlueScrubs-v1-fixed是一个大规模精选医疗文本语料库，专为临床大语言模型设计。数据集包含约1108万条训练文本，来源于SlimPajama/RedPajama（涵盖Common Crawl、C4、GitHub、图书、arXiv、维基百科、StackExchange），经过逻辑回归筛选和Llama-3.1-70B评估，提供文本级医疗概率分数（0.8-1.0）以及三项LLM评分（相关性、精准性、安全性）。全语料中约110亿Token附带肿瘤学分类标签。此版本修复了上游训练子集中meta列的序列化错误，移除meta列，仅保留text字段，确保在datasets中流式加载顺畅。
下载信息
注册下载
需要注册 Kaggle 账号并登录后下载，适合需要跟踪下载记录和使用 API 的用户。
暂未开放公开下载
无需注册即可直接获取公开样本或文档，适合快速预览和评估数据集质量。
免登录有偿下载
公开数据集受托下载与技术交付服务。
提供高速下载与技术交付服务（收技术服务费，非数据销售）
暂未开放当前数据集主要来源为 Kaggle 公开托管，完整影像包建议通过原始链接或 Kaggle API 获取。
使用方式
数据集获取（ModelScope）
方式一：MsDataset（Python）
# 前置依赖: pip install modelscope
from modelscope.msdatasets import MsDataset
ds = MsDataset.load("openmed-community/TheBlueScrubs-v1-fixed", subset_name="default", split="train")
print(ds)
方式二：CLI（命令行）
pip install modelscope
modelscope download dataset openmed-community/TheBlueScrubs-v1-fixed
数据集卡片摘录（源站）
openmed-community/TheBlueScrubs-v1-fixed
What is this?
TheBlueScrubs-v1-fixed is a maintenance fork of the upstream TheBlueScrubs/TheBlueScrubs-v1 train split that resolves a schema bug in the meta column.
In the original train files, some rows serialized meta incorrectly (appearing as the literal string "dict"). This fork re-exports the entire train split without meta column, preserving text field and values.
- Document count: 11,080,331 texts (train)
- Tokens (upstream estimate across all splits): ~20B tokens
- Sources: Curated from SlimPajama/RedPajama (Common Crawl, C4, GitHub, Books, arXiv, Wikipedia, StackExchange)
- Quality signals: per-text medical probability (0.8–1.0) + three 1–5 LLM-based scores (relevance, precision/factual detail, safety/ethics); oncology label covering ~11B tokens across the full corpus.
Upstream details: The Blue Scrubs is a large, curated medical corpus designed for clinical LLMs, filtered via a logistic-regression screen and then Llama-3.1-70B evaluation; clinician and external checks reported high concordance. An oncology classifier adds cancer labels at scale.
Why this fork?
-
Fix: Removes the
metacolumn, unblocking usage withdatasetsstreaming and dataframe backends. -
Scope: Content is otherwise unchanged relative to upstream train split (same rows, fields, and values).
-
Goal: Provide a drop-in train split that loads cleanly in
datasetswithout ad-hoc parsing workarounds.
Data fields (train)
| Field | Type | Description |
|---|---|---|
text |
string | Raw medical text extracted from SlimPajama/RedPajama sources. |
Splits
This repository publishes the train split only (11,080,331 documents). For methods, scope, and aggregate corpus statistics (including validation/test in the upstream project), see the original dataset card and paper.
How to load
from datasets import load_dataset
# streaming
ds = load_dataset("openmed-community/TheBlueScrubs-v1-fixed", split="train", streaming=True)
row = next(iter(ds))
row["text"]
# non-streaming (if you have local storage/network bandwidth)
ds = load_dataset("openmed-community/TheBlueScrubs-v1-fixed", split="train")
ds.features
## 许可
apache-2.0
## 数据加载示例（表格/文本类）
```python
import pandas as pd, glob, os
files = (glob.glob(os.path.join(path, "**", "*.csv"), recursive=True)
+ glob.glob(os.path.join(path, "**", "*.tsv"), recursive=True)
+ glob.glob(os.path.join(path, "**", "*.xlsx"), recursive=True))
print("数据文件:", files)
df = pd.read_csv(files[0])
print(df.shape); print(df.columns.tolist()); print(df.head(3))
源站：modelscope.cn/datasets/openmed-community/TheBlueScrubs-v1-fixed
数据缺失？
依托陕西、四川两大基地，我们与超过十家三甲医院建立直接合作关系，覆盖合规授权、采集治理、专业标注、数据交付的全流程，为AI医疗团队提供即拿即用的高质量临床数据。
⚡️ 需要数据支持或标注服务？立即联系我们获取专业方案。
