数据集获取

git clone https://github.com/JerryWeiAI/COVID-Q.git

curl -L -o repo.zip https://github.com/JerryWeiAI/COVID-Q/archive/refs/heads/master.zip
unzip repo.zip

源站 README 摘录（使用方式）

COVID-Q: 1,690 Questions about COVID-19

Full dataset for the paper “What are People Asking About COVID-19? A Question Classification Dataset”
The dataset CSV file can be found at final_master_dataset.csv.
This dataset consists of COVID-19 questions which have been annotated into a broad category (e.g. Transmission, Prevention) and a more specific class such that questions in the same class are all asking the same thing.
Note: Formal definitions of our categories can be found at this link.

Folders included in this respository:

1. code - all code needed to split dataset into training/testing datasets and to run basic BERT baselines.
2. data - contains original data (TSVs, CSVs, PDFs) to document all question sources
3. dataset_categories - contains usable training and testing data for question-category classification.
4. dataset_classes - contains usable training and testing data for question-class classification.

Datasets

Question-Category Classification

The question-category classification task assigns each question to one of 15 broad categories (e.g. Transmission, Prevention). The goal is to be able to match a given question to the category that best describes what type of information the question is asking for.
In the dataset_categories folder, the following files are included:

1. question_embeddings_pooled.pickle - dictionary of BERT embeddings for every question in the dataset. Please note that augmented questions’’ embeddings are not included in this pickle and require recreating the pickle file.
2. testA.csv - contains testing questions from real sources. Column 1 is the question and Column 2 is the category.
3. testB.csv - contains testing questions that were generated by the author. Column 1 is the question and Column 2 is the category.
4. train20_augmented.csv - contains training questions with 20 questions given per category for a total of 300 real questions. Data augmentation is used to generate 16 more examples for each question. Column 1 is the question and Column 2 is the category.
5. train_20.csv - contains training questions with 20 questions given per category for a total of 300

数据加载示例（表格/文本类）

import pandas as pd, glob, os

files = (glob.glob(os.path.join(path, "**", "*.csv"), recursive=True)
       + glob.glob(os.path.join(path, "**", "*.tsv"), recursive=True)
       + glob.glob(os.path.join(path, "**", "*.xlsx"), recursive=True))
print("数据文件:", files)
df = pd.read_csv(files[0])
print(df.shape); print(df.columns.tolist()); print(df.head(3))

完整仓库：github.com/JerryWeiAI/COVID-Q