数据集获取

git clone https://github.com/Arjun-08/Sequence-to-sequence-networks-for-multi-text-document-summarization.git

curl -L -o repo.zip https://github.com/Arjun-08/Sequence-to-sequence-networks-for-multi-text-document-summarization/archive/refs/heads/main.zip
unzip repo.zip

源站 README 摘录（使用方式）

Sequence-to-sequence networks for multi-text document summarization on medical dataset

Overview

This project explores the application of advanced sequence-to-sequence networks and transformer models for multi-text document summarization in the medical domain. Given the exponential growth of medical literature, efficiently summarizing this vast amount of information is crucial for healthcare professionals, researchers, and policymakers. Our research focuses on leveraging state-of-the-art natural language processing (NLP) techniques to generate concise, informative, and coherent summaries of medical documents.

Project Objectives

• Develop and fine-tune transformer models: Utilize models such as BART, BERT, PEGASUS, and T5 for the task of summarizing biomedical research articles.
• Evaluate model performance: Assess the summarization quality using metrics like ROUGE to ensure the generated summaries are accurate and meaningful.
• Address domain-specific challenges: Tackle issues related to specialized medical terminology, document diversity, and the complexity of medical texts.

Methodology

Datasets

1. CORD-19 Dataset: A large collection of over 1,000,000 scholarly articles on COVID-19 and related coronaviruses, freely accessible to support NLP and AI research.
2. Biomedical Abstracts Dataset: A dataset from Hugging Face with 210,000 training abstracts and 45,000 each for validation and test sets, used for text summarization and NLP tasks.

Models Implemented

• BART (Bidirectional and Auto-Regressive Transformers)
• BERT (Bidirectional Encoder Representations from Transformers)
• PEGASUS
• T5 (Text-To-Text Transfer Transformer)

Evaluation Metrics

• ROUGE (Recall-Oriented Understudy for Gisting Evaluation): Measures the quality of machine-generated summaries by comparing them to reference summaries, evaluating precision, recall, and F1-score.

Implementation

1. Data Preprocessing: Tokenization and preparation of datasets for training.
2. Model Training: Fine-tuning pre-trained transformer models using the biomedical dataset.
3. Evaluation: Assessing model performance using ROUGE scores to ensure high-quality summarization.
4. Summarization: Generating summaries for biomedical research articles and evaluating them qualitatively.

Results and Discussions

• Performance: The fi

数据加载示例（图像类）

from PIL import Image
import glob, os

files = (glob.glob(os.path.join(path, "**", "*.png"), recursive=True)
       + glob.glob(os.path.join(path, "**", "*.jpg"), recursive=True)
       + glob.glob(os.path.join(path, "**", "*.tif"), recursive=True))
print("图像文件数:", len(files))
img = Image.open(files[0]); print("尺寸/模式:", img.size, img.mode)

# torchvision Dataset 方式:
# from torchvision import datasets
# ds = datasets.ImageFolder(path)  # 要求 子目录=类别

目录组织与标注格式以源站说明和下载后实际文件为准。

完整仓库：github.com/Arjun-08/Sequence-to-sequence-networks-for-multi-text-document-summarization