数据集获取

git clone https://github.com/nafis-neehal/CTBench_LLM.git

curl -L -o repo.zip https://github.com/nafis-neehal/CTBench_LLM/archive/refs/heads/main.zip
unzip repo.zip

源站 README 摘录（使用方式）

CTBench_LLM

This repository contains the CT-Repo and CT-Pub datasets and demo codes to run the benchmark.

Steps to run

1. Create a separate virtual environment in Python3. Let’'s name it ‘‘my_venv’’.

python3 -m venv my_venv

Then activate the virtual environment -

source my_venv/bin/activate

2. Install the requirements from requirements.txt

pip3 install -r requirements.txt 

3. Running the Scripts
   1. Run the Demo Scripts Single and Demo Scripts Batch notebook to explore example Generation (zero/three shot) and Evaluation with GPT-4o codes for a single example and for the whole batch of the data respectively. The batch script will generate two files (CT-Pub-With-Examples-Corrected-allgen.csv, CT-Pub-With-Examples-Corrected-allgpteval.csv). The first one contains generation responses from our models (GPT4o and Llama3.1) in both 0/3 shot setting. The second one contains matched responses for all 4 model/ICL settings evaluated by GPT4o.
   2. For getting scores (precision, recall, f1) based on GPT4o evaluation, explore the GPT 4 Scoring notebook. Use the CT-Pub-With-Examples-Corrected-allgpteval.csv file as input which was generated after 3.1.
   3. Also run the BERT Score Calculate notebook to explore the BERT Score evaluations at 0.7 threshold for CT_PUB data (use CT-Pub-With-Examples-Corrected-allgen.csv generated in step 3.1).
4. This can be modified for CT_REPO data as well in same process. Use the same notebooks, just alter the datafile (Use CT-Repo Dataset), and use ‘‘API_BaselineMeasures_Corrected’’ column instead of ‘‘Paper_BaselineMeasures_Corrected’’ column as reference list.

Data (Version 1.0)

1. CT-Pub Dataset (including 3 examples for three-shot setting) - 103 Trials
2. CT-Repo Dataset (including 3 examples for three-shot setting) after pre-processing - 1693 Trials

Metadata (Croissant Format) of CT-Pub Dataset

{
  "@type": "sc:Dataset",
  "name": "CT-Pub Dataset",
  "description": "This dataset includes clinical trial metadata with additional manually annotated baseline features from related p


## 数据加载示例（图像类）

```python
from PIL import Image
import glob, os

files = (glob.glob(os.path.join(path, "**", "*.png"), recursive=True)
       + glob.glob(os.path.join(path, "**", "*.jpg"), recursive=True)
       + glob.glob(os.path.join(path, "**", "*.tif"), recursive=True))
print("图像文件数:", len(files))
img = Image.open(files[0]); print("尺寸/模式:", img.size, img.mode)

# torchvision Dataset 方式:
# from torchvision import datasets
# ds = datasets.ImageFolder(path)  # 要求 子目录=类别

目录组织与标注格式以源站说明和下载后实际文件为准。

完整仓库：github.com/nafis-neehal/CTBench_LLM