自动标注的Twitter COVID-19数据集

数据集获取

git clone https://github.com/thepanacealab/annotated_twitter_covid19_dataset.git

curl -L -o repo.zip https://github.com/thepanacealab/annotated_twitter_covid19_dataset/archive/refs/heads/main.zip
unzip repo.zip

源站 README 摘录（使用方式）

A biomedically oriented automatically annotated Twitter COVID-19 Dataset

Authors: Luis Alberto Robles Hernandez, Tiffany Callahan and Juan M. Banda

The use of social media data, like Twitter, for biomedical research has been gradually increasing over the years. With the COVID-19 pandemic, researchers have turned to more non-traditional sources of clinical data to characterize the disease in near-real time, study the societal implications of interventions, as well as the sequelae that recovered COVID-19 cases present (Long-). However, manually curated social media datasets are difficult to come by due to the expensive costs of manual annotation and the efforts needed to identify the correct texts. When datasets are available, they are usually very small and their annotations don’t generalize well over time or to larger sets of documents. As part of the 2021 Biomedical Linked Annotation Hackathon, we release our dataset of over 120 million automatically annotated tweets for biomedical research purposes. Incorporating best-practices, we identify tweets with potentially high clinical relevance. We evaluated our work by comparing several SpaCy-based annotation frameworks against a manually annotated gold-standard dataset. Selecting the best method to use for automatic annotation, we then annotated 120 million tweets and released them publicly for future downstream usage within the biomedical domain.
Some of the components of this project have been worked on during the COVID-19 Biohackathon and published at the EMNLP NLP COVID-19 Workshop - Part 2.
Dataset details:
The file columns are: Tweet_id, annotation_concept_id, annotationStart, annotationEnd
In order to use this resource, the annotation_concept_id column needs to be joined with the The Observational Health Data Sciences and Informatics Vocabulary.
We used a subset of all available vocabularies which included (version next to the name):

• ICD9CM - ICD9CM v32 master descriptions
• ICD10PCS - ICD10PCS 2021
• CPT4 - 2020 Release
• NDFRT - RXNORM 2018-08-12
• HCPCS - 2020 Alpha Numeric HCPCS File
• MeSH - 2020 Release
• ICD10 - 2020 Release
• ICD9Proc - ICD9CM v32 master des

数据加载示例（表格/文本类）

import pandas as pd, glob, os

files = (glob.glob(os.path.join(path, "**", "*.csv"), recursive=True)
       + glob.glob(os.path.join(path, "**", "*.tsv"), recursive=True)
       + glob.glob(os.path.join(path, "**", "*.xlsx"), recursive=True))
print("数据文件:", files)
df = pd.read_csv(files[0])
print(df.shape); print(df.columns.tolist()); print(df.head(3))

完整仓库：github.com/thepanacealab/annotated_twitter_covid19_dataset