数据集获取

git clone https://github.com/SarahAlqurashi/COVID-19-Arabic-Tweets-Dataset.git

curl -L -o repo.zip https://github.com/SarahAlqurashi/COVID-19-Arabic-Tweets-Dataset/archive/refs/heads/master.zip
unzip repo.zip

源站 README 摘录（使用方式）

COVID-19-Arabic-Tweets-Dataset

The repository contains a collection of Arabic tweets IDs related to novel coronavirus COVID-19. The dataset contains Tweets ids starting from January ,2020 . The Twitter search API was used to gather real-time tweets that contained specific keywords in the Arabic language. To comply with Twitter’s Terms of Service, only the ids of the tweets are released. This dataset is for non-commercial research use only.

Data Organization

• As of Jan 26, 2021 we have tweets from January,2020 unitl May 30, 2020 tweets. We plan to add more months in upcoming days
  and continuosly update this page.
• Tweet-ID files are stored in folders that indicate the year and month of the collection
• The Tweet-ID files contain the tweets ids, all files name have the same structure, with a prefix “COVID19-tweetID-year-month-day"

Dataset collection

• Only tweets in Arabic language were collected from January 1,2020 to May 30, 2020.
• The keywords.txt file contains the updated keywords along with the date we began tracing them. The Hashtags.txt files contain the hashtags that we followed in our Twitter data-set the number of tweets collected for each hashtag along with the date we began tracing them.
• Since Twitter’s search API have a restriction on the amount of the retrieved data there are missing hours of data.
• We provided preliminary statistics of the data-set in the associated paper to this repository. The preliminary statistics will be automatically updated with every update of the dataset.
• For retrieving, the full object of the tweet consider the following tools Hydrator and twarc .

Dataset Statistics

The following statistics is from Tweets colected until May,30,2020.<br/>
The Number of Tweets: 6,086,085 <br/>
The Number of Tweets with geolocation :3925 <br>
The Average of Tweets Collected Daily : 40573 <br/>

Guideline to Hydrate

Using TWARC Notebook

To hydrate the tweets-ID from our COVID-19-Arabic-Tweets-Dataset GitHub repository you can use our [Hydrate_TweetIDs_Arabic_COVID19](https://github.com/SarahAlqurashi/COVID-19-Arabic-Tweets-Dataset/blob/master/Guideline to Hydrate/Hydrate_TweetIDs_Arabic_COVID19.

数据加载示例（表格/文本类）

import pandas as pd, glob, os

files = (glob.glob(os.path.join(path, "**", "*.csv"), recursive=True)
       + glob.glob(os.path.join(path, "**", "*.tsv"), recursive=True)
       + glob.glob(os.path.join(path, "**", "*.xlsx"), recursive=True))
print("数据文件:", files)
df = pd.read_csv(files[0])
print(df.shape); print(df.columns.tolist()); print(df.head(3))

完整仓库：github.com/SarahAlqurashi/COVID-19-Arabic-Tweets-Dataset