数据集获取（ModelScope）

方式一：MsDataset（Python）

# 前置依赖: pip install modelscope
from modelscope.msdatasets import MsDataset

ds = MsDataset.load("NeuML/pubmed-hmpv", subset_name="default", split="train")
print(ds)

方式二：CLI（命令行）

pip install modelscope
modelscope download dataset NeuML/pubmed-hmpv

数据集卡片摘录（源站）

PubMed HMPV Articles

Current as of January 7, 2025

This dataset is metadata (id, publication date, title, link) from PubMed articles related to HMPV. It was created using paperetl and the PubMed Baseline.

The 37 million articles were filtered to match either of the following criteria.

• MeSH code = D029121
• Keyword of HMPV in either the title or abstract

Retrieve article abstracts

The full article abstracts can be retrieved via the PubMed API. This method accepts batches of PubMed IDs.

Alternatively, the dataset can be recreated using the following steps and loading the abstracts into the dataset (see step 5).

Download and build

The following steps recreate this dataset.

1. Create the following directories and files

   mkdir -p pubmed/config pubmed/data
   
   echo "D029121" > pubmed/config/codes
   echo "HMPV" > pubmed/config/keywords
   

2. Install paperetl and download PubMed Baseline + Updates into pubmed/data.

   pip install paperetl datasets
   

3. Parse the PubMed dataset into article metadata

   python -m paperetl.file pubmed/data pubmed/articles pubmed/config
   

4. Export to dataset

   from datasets import Dataset
   
   ds = Dataset.from_sql(
       ("SELECT id id, published published, title title, reference reference FROM articles "
       "ORDER BY published DESC"),
       f"sqlite:///pubmed/articles/articles.sqlite"
   )
   ds.to_csv(f"pubmed-hmpv/articles.csv")
   

5. Optional Export to dataset with all fields

   paperetl parses all metadata and article abstracts. If you’'d like to create a local dataset with the abstracts, run the following instead of step 4.

   import sqlite3
   import uuid
   
   from datasets import Dataset
   
   class Export:
       def __init__(self, dbfile):
           # Load database
           self.conevent-blocked= sqlite3.connect(dbfile)
           self.connection.row_factory = sqlite3.Row
   
       def __call__(self):
           # Create cursors
           cursor1 = self.connection.cursor()
           cursor2 = self.connection.cursor()
   
           # Get article metadata
          
   
   
   

许可

Apache License 2.0

数据加载示例（表格/文本类）

import pandas as pd, glob, os

files = (glob.glob(os.path.join(path, "**", "*.csv"), recursive=True)
       + glob.glob(os.path.join(path, "**", "*.tsv"), recursive=True)
       + glob.glob(os.path.join(path, "**", "*.xlsx"), recursive=True))
print("数据文件:", files)
df = pd.read_csv(files[0])
print(df.shape); print(df.columns.tolist()); print(df.head(3))

源站：modelscope.cn/datasets/NeuML/pubmed-hmpv