Update README.md

README.md

@@ -1,33 +1,34 @@
----
-language:
-- en
-license: apache-2.0
-library_name: transformers
-tags:
-- chest X-ray report generation
-- radiology report generation
-- image captioning
-- chest X-ray
-- X-ray
-- radiology
-- cxrmate
-- cxrmate-ed
-- report
-- radiology report
-- multimodal
-- patient data
-- patient records
-- mimic-cxr
-- mimic-iv-ed
----
-
-# CXRMate-ED: The Impact of Auxiliary Patient Data on Automated Chest X-Ray Report Generation and How to Incorporate It
+---
+language:
+- en
+license: apache-2.0
+library_name: transformers
+tags:
+- chest X-ray report generation
+- radiology report generation
+- image captioning
+- chest X-ray
+- X-ray
+- radiology
+- cxrmate
+- cxrmate-ed
+- report
+- radiology report
+- multimodal
+- patient data
+- patient records
+- mimic-cxr
+- mimic-iv-ed
+pipeline_tag: image-text-to-text
+---
+
+# **CXRMate-ED**: The Impact of Auxiliary Patient Data on Automated Chest X-Ray Report Generation and How to Incorporate It
 
 This is the model and data pipeline for the CXRMate-ED model from: https://arxiv.org/pdf/2406.13181.
 
 The abstract from the paper:
 
-"This study investigates the integration of diverse patient data sources into multimodal language models for automated chest X-ray (CXR) report generation. Traditionally, CXR report generation relies solely on CXR images and limited radiology data, overlooking valuable information from patient health records, particularly from emergency departments. Utilising the MIMIC-CXR and MIMIC-IV-ED datasets, we incorporate detailed patient information such as aperiodic vital signs, medications, and clinical history to enhance diagnostic accuracy. We introduce a novel approach to transform these heterogeneous data sources into embeddings that prompt a multimodal language model, significantly enhancing the diagnostic accuracy of generated radiology reports. Our comprehensive evaluation demonstrates the benefits of using a broader set of patient data, underscoring the potential for enhanced diagnostic capabilities and better patient outcomes through the integration of multimodal data in CXR report generation."
+"This study investigates the integration of diverse patient data sources into multimodal language models for automated chest X-ray (CXR) report generation. Traditionally, CXR report generation relies solely on CXR images and limited radiology data, overlooking valuable information from patient health records, particularly from emergency departments. Utilising the MIMIC-CXR and MIMIC-IV-ED datasets, we incorporate detailed patient information such as vital signs, medicines, and clinical history to enhance diagnostic accuracy. We introduce a novel approach to transform these heterogeneous data sources into embeddings that prompt a multimodal language model; this significantly enhances the diagnostic accuracy of generated radiology reports. Our comprehensive evaluation demonstrates the benefits of using a broader set of patient data, underscoring the potential for enhanced diagnostic capabilities and better patient outcomes through the integration of multimodal data in CXR report generation."
 
 ## MIMIC-CXR & MIMIC-IV-ED Dataset:
 
@@ -39,6 +40,7 @@ mimic-cxr  mimic-cxr-jpg  mimic-iv-ed
 ```
 
 ### Download MIMIC-CXR-JPG:
+
 Download the MIMIC-CXR-JPG dataset from https://physionet.org/content/mimic-cxr-jpg, e.g.,
 ```shell
 wget -r -N -c -np --user <username> --ask-password https://physionet.org/files/mimic-cxr-jpg/2.1.0/
@@ -46,6 +48,7 @@ wget -r -N -c -np --user <username> --ask-password https://physionet.org/files/m
 Note that you must be a credentialised user to access this dataset.
 
 ### Download the reports from MIMIC-CXR:
+
 MIMIC-CXR-JPG does not include the radiology reports and are instead included with MIMIC-CXR (the DICOM version of the dataset). To download this dataset and avoid downloading the DICOM files (which are very large), use `--reject dcm` with the wget command from https://physionet.org/content/mimic-cxr, e.g, 
 ```shell
 wget -r -N -c -np --reject dcm --user <username> --ask-password https://physionet.org/files/mimic-cxr/2.0.0/
@@ -53,6 +56,7 @@ wget -r -N -c -np --reject dcm --user <username> --ask-password https://physione
 Note that you must be a credentialised user to access this dataset.
 
 ### Download MIMIC-IV-ED:
+
 Download the MIMIC-IV-ED dataset from https://physionet.org/content/mimic-iv-ed, e.g.,
 ```shell
 wget -r -N -c -np --user <username> --ask-password https://physionet.org/files/mimic-iv-ed/2.2/
@@ -60,92 +64,45 @@ wget -r -N -c -np --user <username> --ask-password https://physionet.org/files/m
 Note that you must be a credentialised user to access this dataset.
 
 ### Prepare the dataset:
+
 ```python
 import transformers
 
 # Paths:
 physionet_dir = '/.../physionet.org/files'  # Where MIMIC-CXR, MIMIC-CXR-JPG, and MIMIC-IV-ED are stored.
-database_dir = '/.../database/cxrmate_ed'  # The LMDB database for the JPGs and the DuckDB database for the tables will be saved here.
+database_dir = '/.../database/cxrmate_ed'  # The Hugging Face dataset will be saved here.
 
-# Prepare the MIMIC-CXR & MIMIC-IV-ED dataset:
+# Prepare the Hugging Face MIMIC-CXR & MIMIC-IV-ED dataset:
 model = transformers.AutoModel.from_pretrained('aehrc/cxrmate-ed', trust_remote_code=True)
-model.prepare_data(
-    physionet_dir=physionet_dir,
-    database_dir=database_dir,
-)
+model.prepare_data(physionet_dir=physionet_dir, database_dir=database_dir)
 ```
-Note: dataset preperation should take roughly 2-3 hours.
-
 
 ## Generate a report
 
 ```python
 import torch
 import transformers
-from timm.data.constants import IMAGENET_DEFAULT_MEAN, IMAGENET_DEFAULT_STD
-from torch.utils.data import DataLoader
-from torchvision.transforms import v2
-import os
-import pprint
-import matplotlib.pyplot as plt
-from torchvision.utils import make_grid
 
 # Device and paths:
 device = 'cuda'
-physionet_dir = '/.../physionet.org/files'  # Where MIMIC-CXR, MIMIC-CXR-JPG, and MIMIC-IV-ED are stored.
-database_dir = '/.../database/cxrmate_ed'  # The LMDB database for the JPGs and the DuckDB database for the tables will be saved here.
 
 # Download model checkpoint:
-model = transformers.AutoModel.from_pretrained('aehrc/cxrmate-ed', trust_remote_code=True).to(device=device)
-model.eval()
-
-# Download tokenizer:
+model = transformers.AutoModelForCausalLM.from_pretrained('aehrc/cxrmate-ed', trust_remote_code=True).to(device=device)
 tokenizer = transformers.PreTrainedTokenizerFast.from_pretrained('aehrc/cxrmate-ed')
-os.environ['TOKENIZERS_PARALLELISM'] = 'false'
-
-# Image transforms:
-image_size = 384
-test_transforms = v2.Compose(
-    [
-        v2.Grayscale(num_output_channels=3),
-        v2.Resize(
-            size=image_size, 
-            antialias=True,
-            interpolation=v2.InterpolationMode.BICUBIC,
-        ),
-        v2.CenterCrop(size=[image_size, image_size]),
-        v2.ToDtype(torch.float32, scale=True),
-        v2.Normalize(mean=IMAGENET_DEFAULT_MEAN, std=IMAGENET_DEFAULT_STD),
-    ]
-)
-
-# Prepare the MIMIC-CXR & MIMIC-IV-ED dataset:
-model.prepare_data(
-    physionet_dir=physionet_dir,
-    database_dir=database_dir,
-)
-
-# Get the test set dataset & dataloader:
-test_set = model.get_dataset(split='test', transforms=test_transforms, database_dir=database_dir)
-test_dataloader = DataLoader(
-    test_set,
-    batch_size=1, 
-    num_workers=5,
-    shuffle=True,
-    collate_fn=model.collate_fn,
-    pin_memory=True,
-)
-
-# Get an example:
-batch = next(iter(test_dataloader))
-
-# Move tensors in the batch to the device:
-for key, value in batch.items():
-    if isinstance(value, torch.Tensor):
-        batch[key] = value.to(device)
+
+# Get the Hugging Face MIMIC-CXR & MIMIC-IV-ED test set:
+test_set = model.get_dataset(database_dir=database_dir, test_set_only=True)
+
+# Get an example, add mini-batch dimension and move to device:
+example = test_set[0]
+for k, v in example.items():
+    if isinstance(v, torch.Tensor):
+        example[k] = v.to(device).unsqueeze(0)
+    else:
+        example[k] = [v]
 
 # Convert the patient data in the batch into embeddings:
-inputs_embeds, attention_mask, token_type_ids, position_ids, bos_token_ids = model.prepare_inputs(tokenizer=tokenizer, **batch)
+inputs_embeds, attention_mask, token_type_ids, position_ids, bos_token_ids = model.prepare_inputs(tokenizer=tokenizer, **example)
     
 # Generate reports:
 output_ids = model.generate(
@@ -155,7 +112,6 @@ output_ids = model.generate(
     prompt_attention_mask=attention_mask,
     prompt_position_ids=position_ids,
     special_token_ids=[tokenizer.sep_token_id],
-    token_type_id_sections=model.decoder.config.section_ids,
     max_length=256,
     num_beams=4,
     return_dict_in_generate=True,
@@ -165,13 +121,13 @@ output_ids = model.generate(
 findings, impression = model.split_and_decode_sections(output_ids, [tokenizer.sep_token_id, tokenizer.eos_token_id], tokenizer)
 for i,j in zip(findings, impression):
     print(f'Findings:\t{i}\nImpression:\t{j}\n\n')
-
 ```
 
 # Environment requirements
 
-Environment requirements can be found here: https://github.com/aehrc/anon/blob/main/requirements.txt.
+Environment requirements can be found here: https://github.com/aehrc/cxrmate-ed/blob/main/requirements.txt.
 
 
 # Code repository
-The code repository, which includes the training pipeline for CXRMate-ED, is available at: https://github.com/aehrc/anon.
+
+The code repository, which includes the training pipeline for CXRMate-ED, is available at: https://github.com/aehrc/cxrmate-ed.